Spark running in Standalone mode on Windows not working



.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;








0















On a single windows machine, I am running both Spark Master and Worker.
I am testing my application code. So, I am running only 1 worker.



Note: I am using cygwin to start master and worker.



Master start command:



java -cp "C:Sparkspark-2.2.2-bin-hadoop2.7/conf;C:Sparkspark-2.2.2-bin-hadoop2.7jars*"
 -Xmx1g org.apache.spark.deploy.master.Master --host 10.111.30.22 --port 7077 
 --webui-port 8080


Worker start command:



java -cp "C:Sparkspark-2.2.2-bin-hadoop2.7/conf;C:Sparkspark-2.2.2-bin-hadoop2.7jars*" 
 -Xmx1g org.apache.spark.deploy.worker.Worker 
 --webui-port 8081 spark://10.111.30.22:7077 --cores 1 --memory 1g


After starting, I am able to see spark master and worker.



I opened one more another prompt, and ran the following, to test spark sample example code:



./bin/spark-submit --class org.apache.spark.examples.SparkPi 
 --master spark://10.111.30.22:7077 --executor-memory 2G 
 --total-executor-cores 2 
 C:Sparkspark-2.2.2-bin-hadoop2.7examplesjarsspark-examples_2.11-2.2.2.jar 2


I am getting the error in the worker, attached image. And the job never finishes, it keeps running forever.



The following is the error dumped on executor console:




$ java -cp "C:Sparkspark-2.2.2-bin-hadoop2.7/conf;C:Sparkspark-2.2.2-bin-hadoop2.7jars*" -Xmx1g org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://10.111.30.22:7077 --cores 1 --memory 1g
18/11/19 13:40:32 ERROR ExecutorRunner: Error running executor
java.lang.IllegalStateException: Cannot find any build directories.
 at org.apache.spark.launcher.CommandBuilderUtils.checkState(CommandBuilderUtils.java:248)
 at org.apache.spark.launcher.AbstractCommandBuilder.getScalaVersion(AbstractCommandBuilder.java:240)
 at org.apache.spark.launcher.AbstractCommandBuilder.buildClassPath(AbstractCommandBuilder.java:194)
 at org.apache.spark.launcher.AbstractCommandBuilder.buildJavaCommand(AbstractCommandBuilder.java:117)
 at org.apache.spark.launcher.WorkerCommandBuilder.buildCommand(WorkerCommandBuilder.scala:39)
 at org.apache.spark.launcher.WorkerCommandBuilder.buildCommand(WorkerCommandBuilder.scala:45)
 at org.apache.spark.deploy.worker.CommandUtils$.buildCommandSeq(CommandUtils.scala:63)
 at org.apache.spark.deploy.worker.CommandUtils$.buildProcessBuilder(CommandUtils.scala:51)
 at org.apache.spark.deploy.worker.ExecutorRunner.org$apache$spark$deploy$worker$ExecutorRunner$$fetchAndRunExecutor(ExecutorRunner.scala:145)
 at org.apache.spark.deploy.worker.ExecutorRunner$$anon$1.run(ExecutorRunner.scala:73)
18/11/19 13:40:32 ERROR ExecutorRunner: Error running executor
java.lang.IllegalStateException: Cannot find any build directories.
 at org.apache.spark.launcher.CommandBuilderUtils.checkState(CommandBuilderUtils.java:248)
 at org.apache.spark.launcher.AbstractCommandBuilder.getScalaVersion(AbstractCommandBuilder.java:240)
 at org.apache.spark.launcher.AbstractCommandBuilder.buildClassPath(AbstractCommandBuilder.java:194)
 at org.apache.spark.launcher.AbstractCommandBuilder.buildJavaCommand(AbstractCommandBuilder.java:117)
 at org.apache.spark.launcher.WorkerCommandBuilder.buildCommand(WorkerCommandBuilder.scala:39)
 at org.apache.spark.launcher.WorkerCommandBuilder.buildCommand(WorkerCommandBuilder.scala:45)
 at org.apache.spark.deploy.worker.CommandUtils$.buildCommandSeq(CommandUtils.scala:63)
 at org.apache.spark.deploy.worker.CommandUtils$.buildProcessBuilder(CommandUtils.scala:51)
 at org.apache.spark.deploy.worker.ExecutorRunner.org$apache$spark$deploy$worker$ExecutorRunner$$fetchAndRunExecutor(ExecutorRunner.scala:145)
 at org.apache.spark.deploy.worker.ExecutorRunner$$anon$1.run(ExecutorRunner.scala:73)
18/11/19 13:40:32 ERROR ExecutorRunner: Error running executor
java.lang.IllegalStateException: Cannot find any build directories.
 at org.apache.spark.launcher.CommandBuilderUtils.checkState(CommandBuilderUtils.java:248)
 at org.apache.spark.launcher.AbstractCommandBuilder.getScalaVersion(AbstractCommandBuilder.java:240)
 at org.apache.spark.launcher.AbstractCommandBuilder.buildClassPath(AbstractCommandBuilder.java:194)
 at org.apache.spark.launcher.AbstractCommandBuilder.buildJavaCommand(AbstractCommandBuilder.java:117)
 at org.apache.spark.launcher.WorkerCommandBuilder.buildCommand(WorkerCommandBuilder.scala:39)
 at org.apache.spark.launcher.WorkerCommandBuilder.buildCommand(WorkerCommandBuilder.scala:45)
 at org.apache.spark.deploy.worker.CommandUtils$.buildCommandSeq(CommandUtils.scala:63)
 at org.apache.spark.deploy.worker.CommandUtils$.buildProcessBuilder(CommandUtils.scala:51)
 at org.apache.spark.deploy.worker.ExecutorRunner.org$apache$spark$deploy$worker$ExecutorRunner$$fetchAndRunExecutor(ExecutorRunner.scala:145)
 at org.apache.spark.deploy.worker.ExecutorRunner$$anon$1.run(ExecutorRunner.scala:73)
18/11/19 13:40:33 ERROR ExecutorRunner: Error running executor,
.

=======================================================================



Environment:
OS: Windows. Using Cygwin to run Spark Master and Worker.
Only 1 Worker is used for testing.



I do not see any error in Spark Master console. I see a number of executor created and status failed. No executor is getting completed.



Anybody faced this issue? What is the solution? Much appreciate your response.



I am using following commands to manually start Master and Worker:
Master:
java -cp "C:Sparkspark-2.2.2-bin-hadoop2.7/conf;C:Sparkspark-2.2.2-bin-hadoop2.7jars*" -Xmx1g org.apache.spark.deploy.master.Master --host 10.111.30.22 --port 7077 --webui-port 8080



Worker:
java -cp "C:Sparkspark-2.2.2-bin-hadoop2.7/conf;C:Sparkspark-2.2.2-bin-hadoop2.7jars*" -Xmx1g org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://10.111.30.22:7077 --cores 1 --memory 1g



Spark Pi Job:
spark-submit --class org.apache.spark.examples.SparkPi --master spark://10.111.30.22:7077 C:Sparkspark-2.2.2-bin-hadoop2.7examplesjarsspark-examples_2.11-2.2.2.jar 2






windows apache-spark







share|improve this question
























  • Paste error message as text

    – barbsan
    Nov 16 '18 at 12:52

















0















On a single windows machine, I am running both Spark Master and Worker.
I am testing my application code. So, I am running only 1 worker.



Note: I am using cygwin to start master and worker.



Master start command:



java -cp "C:Sparkspark-2.2.2-bin-hadoop2.7/conf;C:Sparkspark-2.2.2-bin-hadoop2.7jars*"
 -Xmx1g org.apache.spark.deploy.master.Master --host 10.111.30.22 --port 7077 
 --webui-port 8080


Worker start command:



java -cp "C:Sparkspark-2.2.2-bin-hadoop2.7/conf;C:Sparkspark-2.2.2-bin-hadoop2.7jars*" 
 -Xmx1g org.apache.spark.deploy.worker.Worker 
 --webui-port 8081 spark://10.111.30.22:7077 --cores 1 --memory 1g


After starting, I am able to see spark master and worker.



I opened one more another prompt, and ran the following, to test spark sample example code:



./bin/spark-submit --class org.apache.spark.examples.SparkPi 
 --master spark://10.111.30.22:7077 --executor-memory 2G 
 --total-executor-cores 2 
 C:Sparkspark-2.2.2-bin-hadoop2.7examplesjarsspark-examples_2.11-2.2.2.jar 2


I am getting the error in the worker, attached image. And the job never finishes, it keeps running forever.



The following is the error dumped on executor console:




$ java -cp "C:Sparkspark-2.2.2-bin-hadoop2.7/conf;C:Sparkspark-2.2.2-bin-hadoop2.7jars*" -Xmx1g org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://10.111.30.22:7077 --cores 1 --memory 1g
18/11/19 13:40:32 ERROR ExecutorRunner: Error running executor
java.lang.IllegalStateException: Cannot find any build directories.
 at org.apache.spark.launcher.CommandBuilderUtils.checkState(CommandBuilderUtils.java:248)
 at org.apache.spark.launcher.AbstractCommandBuilder.getScalaVersion(AbstractCommandBuilder.java:240)
 at org.apache.spark.launcher.AbstractCommandBuilder.buildClassPath(AbstractCommandBuilder.java:194)
 at org.apache.spark.launcher.AbstractCommandBuilder.buildJavaCommand(AbstractCommandBuilder.java:117)
 at org.apache.spark.launcher.WorkerCommandBuilder.buildCommand(WorkerCommandBuilder.scala:39)
 at org.apache.spark.launcher.WorkerCommandBuilder.buildCommand(WorkerCommandBuilder.scala:45)
 at org.apache.spark.deploy.worker.CommandUtils$.buildCommandSeq(CommandUtils.scala:63)
 at org.apache.spark.deploy.worker.CommandUtils$.buildProcessBuilder(CommandUtils.scala:51)
 at org.apache.spark.deploy.worker.ExecutorRunner.org$apache$spark$deploy$worker$ExecutorRunner$$fetchAndRunExecutor(ExecutorRunner.scala:145)
 at org.apache.spark.deploy.worker.ExecutorRunner$$anon$1.run(ExecutorRunner.scala:73)
18/11/19 13:40:32 ERROR ExecutorRunner: Error running executor
java.lang.IllegalStateException: Cannot find any build directories.
 at org.apache.spark.launcher.CommandBuilderUtils.checkState(CommandBuilderUtils.java:248)
 at org.apache.spark.launcher.AbstractCommandBuilder.getScalaVersion(AbstractCommandBuilder.java:240)
 at org.apache.spark.launcher.AbstractCommandBuilder.buildClassPath(AbstractCommandBuilder.java:194)
 at org.apache.spark.launcher.AbstractCommandBuilder.buildJavaCommand(AbstractCommandBuilder.java:117)
 at org.apache.spark.launcher.WorkerCommandBuilder.buildCommand(WorkerCommandBuilder.scala:39)
 at org.apache.spark.launcher.WorkerCommandBuilder.buildCommand(WorkerCommandBuilder.scala:45)
 at org.apache.spark.deploy.worker.CommandUtils$.buildCommandSeq(CommandUtils.scala:63)
 at org.apache.spark.deploy.worker.CommandUtils$.buildProcessBuilder(CommandUtils.scala:51)
 at org.apache.spark.deploy.worker.ExecutorRunner.org$apache$spark$deploy$worker$ExecutorRunner$$fetchAndRunExecutor(ExecutorRunner.scala:145)
 at org.apache.spark.deploy.worker.ExecutorRunner$$anon$1.run(ExecutorRunner.scala:73)
18/11/19 13:40:32 ERROR ExecutorRunner: Error running executor
java.lang.IllegalStateException: Cannot find any build directories.
 at org.apache.spark.launcher.CommandBuilderUtils.checkState(CommandBuilderUtils.java:248)
 at org.apache.spark.launcher.AbstractCommandBuilder.getScalaVersion(AbstractCommandBuilder.java:240)
 at org.apache.spark.launcher.AbstractCommandBuilder.buildClassPath(AbstractCommandBuilder.java:194)
 at org.apache.spark.launcher.AbstractCommandBuilder.buildJavaCommand(AbstractCommandBuilder.java:117)
 at org.apache.spark.launcher.WorkerCommandBuilder.buildCommand(WorkerCommandBuilder.scala:39)
 at org.apache.spark.launcher.WorkerCommandBuilder.buildCommand(WorkerCommandBuilder.scala:45)
 at org.apache.spark.deploy.worker.CommandUtils$.buildCommandSeq(CommandUtils.scala:63)
 at org.apache.spark.deploy.worker.CommandUtils$.buildProcessBuilder(CommandUtils.scala:51)
 at org.apache.spark.deploy.worker.ExecutorRunner.org$apache$spark$deploy$worker$ExecutorRunner$$fetchAndRunExecutor(ExecutorRunner.scala:145)
 at org.apache.spark.deploy.worker.ExecutorRunner$$anon$1.run(ExecutorRunner.scala:73)
18/11/19 13:40:33 ERROR ExecutorRunner: Error running executor,
.

=======================================================================



Environment:
OS: Windows. Using Cygwin to run Spark Master and Worker.
Only 1 Worker is used for testing.



I do not see any error in Spark Master console. I see a number of executor created and status failed. No executor is getting completed.



Anybody faced this issue? What is the solution? Much appreciate your response.



I am using following commands to manually start Master and Worker:
Master:
java -cp "C:Sparkspark-2.2.2-bin-hadoop2.7/conf;C:Sparkspark-2.2.2-bin-hadoop2.7jars*" -Xmx1g org.apache.spark.deploy.master.Master --host 10.111.30.22 --port 7077 --webui-port 8080



Worker:
java -cp "C:Sparkspark-2.2.2-bin-hadoop2.7/conf;C:Sparkspark-2.2.2-bin-hadoop2.7jars*" -Xmx1g org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://10.111.30.22:7077 --cores 1 --memory 1g



Spark Pi Job:
spark-submit --class org.apache.spark.examples.SparkPi --master spark://10.111.30.22:7077 C:Sparkspark-2.2.2-bin-hadoop2.7examplesjarsspark-examples_2.11-2.2.2.jar 2






windows apache-spark







share|improve this question
























  • Paste error message as text

    – barbsan
    Nov 16 '18 at 12:52













0












0








0








On a single windows machine, I am running both Spark Master and Worker.
I am testing my application code. So, I am running only 1 worker.



Note: I am using cygwin to start master and worker.



Master start command:



java -cp "C:Sparkspark-2.2.2-bin-hadoop2.7/conf;C:Sparkspark-2.2.2-bin-hadoop2.7jars*"
 -Xmx1g org.apache.spark.deploy.master.Master --host 10.111.30.22 --port 7077 
 --webui-port 8080


Worker start command:



java -cp "C:Sparkspark-2.2.2-bin-hadoop2.7/conf;C:Sparkspark-2.2.2-bin-hadoop2.7jars*" 
 -Xmx1g org.apache.spark.deploy.worker.Worker 
 --webui-port 8081 spark://10.111.30.22:7077 --cores 1 --memory 1g


After starting, I am able to see spark master and worker.



I opened one more another prompt, and ran the following, to test spark sample example code:



./bin/spark-submit --class org.apache.spark.examples.SparkPi 
 --master spark://10.111.30.22:7077 --executor-memory 2G 
 --total-executor-cores 2 
 C:Sparkspark-2.2.2-bin-hadoop2.7examplesjarsspark-examples_2.11-2.2.2.jar 2


I am getting the error in the worker, attached image. And the job never finishes, it keeps running forever.



The following is the error dumped on executor console:




$ java -cp "C:Sparkspark-2.2.2-bin-hadoop2.7/conf;C:Sparkspark-2.2.2-bin-hadoop2.7jars*" -Xmx1g org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://10.111.30.22:7077 --cores 1 --memory 1g
18/11/19 13:40:32 ERROR ExecutorRunner: Error running executor
java.lang.IllegalStateException: Cannot find any build directories.
 at org.apache.spark.launcher.CommandBuilderUtils.checkState(CommandBuilderUtils.java:248)
 at org.apache.spark.launcher.AbstractCommandBuilder.getScalaVersion(AbstractCommandBuilder.java:240)
 at org.apache.spark.launcher.AbstractCommandBuilder.buildClassPath(AbstractCommandBuilder.java:194)
 at org.apache.spark.launcher.AbstractCommandBuilder.buildJavaCommand(AbstractCommandBuilder.java:117)
 at org.apache.spark.launcher.WorkerCommandBuilder.buildCommand(WorkerCommandBuilder.scala:39)
 at org.apache.spark.launcher.WorkerCommandBuilder.buildCommand(WorkerCommandBuilder.scala:45)
 at org.apache.spark.deploy.worker.CommandUtils$.buildCommandSeq(CommandUtils.scala:63)
 at org.apache.spark.deploy.worker.CommandUtils$.buildProcessBuilder(CommandUtils.scala:51)
 at org.apache.spark.deploy.worker.ExecutorRunner.org$apache$spark$deploy$worker$ExecutorRunner$$fetchAndRunExecutor(ExecutorRunner.scala:145)
 at org.apache.spark.deploy.worker.ExecutorRunner$$anon$1.run(ExecutorRunner.scala:73)
18/11/19 13:40:32 ERROR ExecutorRunner: Error running executor
java.lang.IllegalStateException: Cannot find any build directories.
 at org.apache.spark.launcher.CommandBuilderUtils.checkState(CommandBuilderUtils.java:248)
 at org.apache.spark.launcher.AbstractCommandBuilder.getScalaVersion(AbstractCommandBuilder.java:240)
 at org.apache.spark.launcher.AbstractCommandBuilder.buildClassPath(AbstractCommandBuilder.java:194)
 at org.apache.spark.launcher.AbstractCommandBuilder.buildJavaCommand(AbstractCommandBuilder.java:117)
 at org.apache.spark.launcher.WorkerCommandBuilder.buildCommand(WorkerCommandBuilder.scala:39)
 at org.apache.spark.launcher.WorkerCommandBuilder.buildCommand(WorkerCommandBuilder.scala:45)
 at org.apache.spark.deploy.worker.CommandUtils$.buildCommandSeq(CommandUtils.scala:63)
 at org.apache.spark.deploy.worker.CommandUtils$.buildProcessBuilder(CommandUtils.scala:51)
 at org.apache.spark.deploy.worker.ExecutorRunner.org$apache$spark$deploy$worker$ExecutorRunner$$fetchAndRunExecutor(ExecutorRunner.scala:145)
 at org.apache.spark.deploy.worker.ExecutorRunner$$anon$1.run(ExecutorRunner.scala:73)
18/11/19 13:40:32 ERROR ExecutorRunner: Error running executor
java.lang.IllegalStateException: Cannot find any build directories.
 at org.apache.spark.launcher.CommandBuilderUtils.checkState(CommandBuilderUtils.java:248)
 at org.apache.spark.launcher.AbstractCommandBuilder.getScalaVersion(AbstractCommandBuilder.java:240)
 at org.apache.spark.launcher.AbstractCommandBuilder.buildClassPath(AbstractCommandBuilder.java:194)
 at org.apache.spark.launcher.AbstractCommandBuilder.buildJavaCommand(AbstractCommandBuilder.java:117)
 at org.apache.spark.launcher.WorkerCommandBuilder.buildCommand(WorkerCommandBuilder.scala:39)
 at org.apache.spark.launcher.WorkerCommandBuilder.buildCommand(WorkerCommandBuilder.scala:45)
 at org.apache.spark.deploy.worker.CommandUtils$.buildCommandSeq(CommandUtils.scala:63)
 at org.apache.spark.deploy.worker.CommandUtils$.buildProcessBuilder(CommandUtils.scala:51)
 at org.apache.spark.deploy.worker.ExecutorRunner.org$apache$spark$deploy$worker$ExecutorRunner$$fetchAndRunExecutor(ExecutorRunner.scala:145)
 at org.apache.spark.deploy.worker.ExecutorRunner$$anon$1.run(ExecutorRunner.scala:73)
18/11/19 13:40:33 ERROR ExecutorRunner: Error running executor,
.

=======================================================================



Environment:
OS: Windows. Using Cygwin to run Spark Master and Worker.
Only 1 Worker is used for testing.



I do not see any error in Spark Master console. I see a number of executor created and status failed. No executor is getting completed.



Anybody faced this issue? What is the solution? Much appreciate your response.



I am using following commands to manually start Master and Worker:
Master:
java -cp "C:Sparkspark-2.2.2-bin-hadoop2.7/conf;C:Sparkspark-2.2.2-bin-hadoop2.7jars*" -Xmx1g org.apache.spark.deploy.master.Master --host 10.111.30.22 --port 7077 --webui-port 8080



Worker:
java -cp "C:Sparkspark-2.2.2-bin-hadoop2.7/conf;C:Sparkspark-2.2.2-bin-hadoop2.7jars*" -Xmx1g org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://10.111.30.22:7077 --cores 1 --memory 1g



Spark Pi Job:
spark-submit --class org.apache.spark.examples.SparkPi --master spark://10.111.30.22:7077 C:Sparkspark-2.2.2-bin-hadoop2.7examplesjarsspark-examples_2.11-2.2.2.jar 2






windows apache-spark







share|improve this question
















On a single windows machine, I am running both Spark Master and Worker.
I am testing my application code. So, I am running only 1 worker.



Note: I am using cygwin to start master and worker.



Master start command:



java -cp "C:Sparkspark-2.2.2-bin-hadoop2.7/conf;C:Sparkspark-2.2.2-bin-hadoop2.7jars*"
 -Xmx1g org.apache.spark.deploy.master.Master --host 10.111.30.22 --port 7077 
 --webui-port 8080


Worker start command:



java -cp "C:Sparkspark-2.2.2-bin-hadoop2.7/conf;C:Sparkspark-2.2.2-bin-hadoop2.7jars*" 
 -Xmx1g org.apache.spark.deploy.worker.Worker 
 --webui-port 8081 spark://10.111.30.22:7077 --cores 1 --memory 1g


After starting, I am able to see spark master and worker.



I opened one more another prompt, and ran the following, to test spark sample example code:



./bin/spark-submit --class org.apache.spark.examples.SparkPi 
 --master spark://10.111.30.22:7077 --executor-memory 2G 
 --total-executor-cores 2 
 C:Sparkspark-2.2.2-bin-hadoop2.7examplesjarsspark-examples_2.11-2.2.2.jar 2


I am getting the error in the worker, attached image. And the job never finishes, it keeps running forever.



The following is the error dumped on executor console:




$ java -cp "C:Sparkspark-2.2.2-bin-hadoop2.7/conf;C:Sparkspark-2.2.2-bin-hadoop2.7jars*" -Xmx1g org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://10.111.30.22:7077 --cores 1 --memory 1g
18/11/19 13:40:32 ERROR ExecutorRunner: Error running executor
java.lang.IllegalStateException: Cannot find any build directories.
 at org.apache.spark.launcher.CommandBuilderUtils.checkState(CommandBuilderUtils.java:248)
 at org.apache.spark.launcher.AbstractCommandBuilder.getScalaVersion(AbstractCommandBuilder.java:240)
 at org.apache.spark.launcher.AbstractCommandBuilder.buildClassPath(AbstractCommandBuilder.java:194)
 at org.apache.spark.launcher.AbstractCommandBuilder.buildJavaCommand(AbstractCommandBuilder.java:117)
 at org.apache.spark.launcher.WorkerCommandBuilder.buildCommand(WorkerCommandBuilder.scala:39)
 at org.apache.spark.launcher.WorkerCommandBuilder.buildCommand(WorkerCommandBuilder.scala:45)
 at org.apache.spark.deploy.worker.CommandUtils$.buildCommandSeq(CommandUtils.scala:63)
 at org.apache.spark.deploy.worker.CommandUtils$.buildProcessBuilder(CommandUtils.scala:51)
 at org.apache.spark.deploy.worker.ExecutorRunner.org$apache$spark$deploy$worker$ExecutorRunner$$fetchAndRunExecutor(ExecutorRunner.scala:145)
 at org.apache.spark.deploy.worker.ExecutorRunner$$anon$1.run(ExecutorRunner.scala:73)
18/11/19 13:40:32 ERROR ExecutorRunner: Error running executor
java.lang.IllegalStateException: Cannot find any build directories.
 at org.apache.spark.launcher.CommandBuilderUtils.checkState(CommandBuilderUtils.java:248)
 at org.apache.spark.launcher.AbstractCommandBuilder.getScalaVersion(AbstractCommandBuilder.java:240)
 at org.apache.spark.launcher.AbstractCommandBuilder.buildClassPath(AbstractCommandBuilder.java:194)
 at org.apache.spark.launcher.AbstractCommandBuilder.buildJavaCommand(AbstractCommandBuilder.java:117)
 at org.apache.spark.launcher.WorkerCommandBuilder.buildCommand(WorkerCommandBuilder.scala:39)
 at org.apache.spark.launcher.WorkerCommandBuilder.buildCommand(WorkerCommandBuilder.scala:45)
 at org.apache.spark.deploy.worker.CommandUtils$.buildCommandSeq(CommandUtils.scala:63)
 at org.apache.spark.deploy.worker.CommandUtils$.buildProcessBuilder(CommandUtils.scala:51)
 at org.apache.spark.deploy.worker.ExecutorRunner.org$apache$spark$deploy$worker$ExecutorRunner$$fetchAndRunExecutor(ExecutorRunner.scala:145)
 at org.apache.spark.deploy.worker.ExecutorRunner$$anon$1.run(ExecutorRunner.scala:73)
18/11/19 13:40:32 ERROR ExecutorRunner: Error running executor
java.lang.IllegalStateException: Cannot find any build directories.
 at org.apache.spark.launcher.CommandBuilderUtils.checkState(CommandBuilderUtils.java:248)
 at org.apache.spark.launcher.AbstractCommandBuilder.getScalaVersion(AbstractCommandBuilder.java:240)
 at org.apache.spark.launcher.AbstractCommandBuilder.buildClassPath(AbstractCommandBuilder.java:194)
 at org.apache.spark.launcher.AbstractCommandBuilder.buildJavaCommand(AbstractCommandBuilder.java:117)
 at org.apache.spark.launcher.WorkerCommandBuilder.buildCommand(WorkerCommandBuilder.scala:39)
 at org.apache.spark.launcher.WorkerCommandBuilder.buildCommand(WorkerCommandBuilder.scala:45)
 at org.apache.spark.deploy.worker.CommandUtils$.buildCommandSeq(CommandUtils.scala:63)
 at org.apache.spark.deploy.worker.CommandUtils$.buildProcessBuilder(CommandUtils.scala:51)
 at org.apache.spark.deploy.worker.ExecutorRunner.org$apache$spark$deploy$worker$ExecutorRunner$$fetchAndRunExecutor(ExecutorRunner.scala:145)
 at org.apache.spark.deploy.worker.ExecutorRunner$$anon$1.run(ExecutorRunner.scala:73)
18/11/19 13:40:33 ERROR ExecutorRunner: Error running executor,
.

=======================================================================



Environment:
OS: Windows. Using Cygwin to run Spark Master and Worker.
Only 1 Worker is used for testing.



I do not see any error in Spark Master console. I see a number of executor created and status failed. No executor is getting completed.



Anybody faced this issue? What is the solution? Much appreciate your response.



I am using following commands to manually start Master and Worker:
Master:
java -cp "C:Sparkspark-2.2.2-bin-hadoop2.7/conf;C:Sparkspark-2.2.2-bin-hadoop2.7jars*" -Xmx1g org.apache.spark.deploy.master.Master --host 10.111.30.22 --port 7077 --webui-port 8080



Worker:
java -cp "C:Sparkspark-2.2.2-bin-hadoop2.7/conf;C:Sparkspark-2.2.2-bin-hadoop2.7jars*" -Xmx1g org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://10.111.30.22:7077 --cores 1 --memory 1g



Spark Pi Job:
spark-submit --class org.apache.spark.examples.SparkPi --master spark://10.111.30.22:7077 C:Sparkspark-2.2.2-bin-hadoop2.7examplesjarsspark-examples_2.11-2.2.2.jar 2






windows apache-spark




windows apache-spark






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Nov 19 '18 at 12:41









Ashish Kamble

729620




729620










asked Nov 16 '18 at 11:13









ChariChari

11




11












  • Paste error message as text

    – barbsan
    Nov 16 '18 at 12:52

















  • Paste error message as text

    – barbsan
    Nov 16 '18 at 12:52
















Paste error message as text

– barbsan
Nov 16 '18 at 12:52





Paste error message as text

– barbsan
Nov 16 '18 at 12:52












0






active

oldest

votes












Your Answer






StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);













draft saved

draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53336739%2fspark-running-in-standalone-mode-on-windows-not-working%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown

























0






active

oldest

votes








0






active

oldest

votes









active

oldest

votes






active

oldest

votes















draft saved

draft discarded
















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid …


  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53336739%2fspark-running-in-standalone-mode-on-windows-not-working%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







-

Popular posts from this blog

27

Image
www.msyz777.com_www.msyz888.com_www.msyz999.com window._bd_share_config="common":"bdSnsKey":,"bdText":"","bdMini":"2","bdMiniList":false,"bdPic":"","bdStyle":"1","bdSize":"16","share":,"image":"viewList":["qzone","tsina","tqq","renren","t163"],"viewText":"��������","viewSize":"16","selectShare":"bdContainerClass":null,"bdSelectMiniList":["qzone","tsina","tqq","renren","t163"];with(document)0[(getElementsByTagName('head')[0]||body).appendChild(createElement('script')).src='http://bdimg.share.baidu.com/static/...
Read more

Top Tejano songwriter Luis Silva dead of heart attack at 64

Image
Ramiro Burr's New Blog - to go back: www.ramiroburr.com From Latin rock to reggaeton, boleros to blues,Tex-Mex to Tejano, conjunto to corridos and beyond, Ramiro Burr has it covered. If you have a new CD release, a trivia question or are looking for tour info, post a message here or e-mail Ramiro directly at: musicreporter@gmail.com Top Tejano songwriter Luis Silva dead of heart attack at 64 By Ramiro Burr on October 23, 2008 8:40 AM | Permalink | Comments (12) | TrackBacks (0) UPDATE: Luis Silva Funeral Service details released Visitation 4-9 p.m. Saturday, Rosary service 6 p.m. Saturday at Porter Loring, 1101 McCullough Ave Funeral Service 10:30 a.m. Monday St. Anthony De Padua Catholic Church, Burial Service at Chapel Hills, 7735 Gibbs Sprawl Road. Porter Loring (210) 227-8221 Related New Flash: Irma Laura Lopez: long time record promoter killed in accident NewsFlash: 9:02 a.m. (New comments below) Luis Silva , one of the most well-known ...
Read more

Category:Rhetoric

Image
Help Category:Rhetoric From Wikipedia, the free encyclopedia Jump to navigation Jump to search Library cataloging and classification main topic rhetoric Library of Congress P301 Universal Decimal 808 Wikimedia Commons has media related to Rhetoric . The main article for this category is Rhetoric . Rhetorical terms, i.e. terms used to describe or analyze rhetorical arguments, including those in persuasive speeches and opinion editorials. Contents Top 0–9 A B C D E F G H I J K L M N O P Q R S T U V W X Y Z Subcategories This category has the following 17 subcategories, out of 17 total. A ► Analogy‎ (2 C, 23 P) ► Apologetics‎ (5 C, 12 P) D ► Dialectic‎ (2 C, 21 P) ► Dialogues‎ (2 C, 38 P) F ► Fiction-writing mode‎ (4 C, 11 P) H ► Homiletics‎ (21 P) ► Hypocrisy‎ (11 P) P ► Populism‎ (13 C, 49 P) R ► Rhetoric theorists‎ (3 C,...
Read more