Question: Use from Java app

[beradrian]

I'm using your docker-compose.yml in a Docker container.

How can I setup a SparkSession?

	SparkSession spark = SparkSession
			.builder()
			.appName("Java Spark SQL basic example")
			.master("spark://master:7077")
				.getOrCreate();

I actually set in etc/hosts master as the docker machine IP.

The one above does not seem to work.

[yeikel]

Hi , I also need help to complete the configuration

I was able to connect setting the master URL to spark://localhost:7077 but for some reason, the command line execution is including my host machine in the "CoarseGrainedScheduler" parameter.

I tried : conf.set("spark.dynamicAllocation.enabled","false"); but that did not work

Spark Executor Command: "/usr/jdk1.8.0_131/bin/java" "-cp" "/conf:/usr/spark-2.3.0/jars/*:/usr/hadoop-2.8.3/etc/hadoop/:/usr/hadoop-2.8.3/etc/hadoop/*:/usr/hadoop-2.8.3/share/hadoop/common/lib/*:/usr/hadoop-2.8.3/share/hadoop/common/*:/usr/hadoop-2.8.3/share/hadoop/hdfs/*:/usr/hadoop-2.8.3/share/hadoop/hdfs/lib/*:/usr/hadoop-2.8.3/share/hadoop/yarn/lib/*:/usr/hadoop-2.8.3/share/hadoop/yarn/*:/usr/hadoop-2.8.3/share/hadoop/mapreduce/lib/*:/usr/hadoop-2.8.3/share/hadoop/mapreduce/*:/usr/hadoop-2.8.3/share/hadoop/tools/lib/*" "-Xmx1024M" "-Dspark.driver.port=56310" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@yeikel-pc:56310" "--executor-id" "22" "--hostname" "172.19.0.3" "--cores" "2" "--app-id" "app-20180401002500-0001" "--worker-url" "spark://Worker@172.19.0.3:8881"

Which produces the following error :

Caused by: java.io.IOException: Failed to connect to yeikel-pc:56310
	at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:245)
	at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:187)
	at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:198)
	at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:194)
	at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:190)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.UnknownHostException: yeikel-pc