Skip to content
This repository has been archived by the owner on Feb 3, 2021. It is now read-only.

Bug: Spark shuffle service worker registration fail #492

Merged
merged 5 commits into from
Apr 9, 2018
Merged

Bug: Spark shuffle service worker registration fail #492

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
40d1d98
stop calling start-shuffle-service.sh script
jafreck Apr 2, 2018
45a535d
whitespace
jafreck Apr 6, 2018
4264897
Merge branch 'master' into bug/spark-shuffle-service-worker-registrat…
jafreck Apr 6, 2018
d3b8e49
remove unused method
jafreck Apr 9, 2018
59cc3d1
Merge branch 'bug/spark-shuffle-service-worker-registration-fail' of …
jafreck Apr 9, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 0 additions & 9 deletions aztk/node_scripts/install/spark.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,15 +96,6 @@ def start_spark_worker():
print("Connecting to master with '{0}'".format(" ".join(cmd)))
call(cmd)

# enable the shuffle service on all slaves
start_shuffle_service()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so are you just not starting it now?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I talked about this in #491, but apparently for standalone mode, it is not necessary to launch the shuffle service using this script. If you just set spark.shuffle.service.enabled to true, the shuffle service is started automatically.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When you submit a spark application with that setting (and not using the start script), the logs indicate that the external shuffle service is running.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so do you even need to keep the method below or you just keep for later usage?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was keeping it in case, but I don't think it is necessary at all. I'll remove it.



def start_shuffle_service():
exe = os.path.join(spark_home, "sbin", "start-shuffle-service.sh")
print("Starting the shuffle service with {}".format(exe))
call([exe, " &"])


def copyfile(src, dest):
try:
Expand Down