Add Spark Shuffle Service Support

[jafreck]

Enable shuffle service by default by setting spark.shuffle.service.enabled true in spark-defaults.conf.

Also, open port 7337 in container.

https://spark.apache.org/docs/latest/configuration.html#shuffle-behavior

[jiata]

why are we setting non-default configurations for spark? Shouldn't this be something we allow them to enable if they wish.

[paselem]

This is similar to the decision we made regarding the chunk size for data. It's a 'better' default for distributed spark clusters since it makes it more reliable to nodes leaving the cluster. In this case, we can rely on Spark's shuffle system to move data around and not force a re-compute if a node goes away. With our focus on low-pri nodes, I feel this is probably a good default for AZTK.

[jafreck]

Additionally, enabling the shuffle service is required for dynamic allocation. And dynamic allocation allows re-scaling of the cluster during Spark application execution.

So without the shuffle service and dynamic allocation enabled by default, resizing your cluster will only affect subsequent jobs.

This is a very large deal for job submission mode, where the spark submits are executed immediately after the master is elected, and potentially before all workers in the cluster are up.

[jiata]

makes sense. thanks guys