Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set benchmark-specific thread limits for the Spark local executor #284

Merged
merged 1 commit into from
Jul 4, 2021

Conversation

lbulej
Copy link
Member

@lbulej lbulej commented Jul 2, 2021

The limits reflect the recommendations from #260, with the limits for gauss-mix, chi-square and dec-tree bumped up by 2 threads (providing a bit of slack that does not help but should not hurt), resulting in the following defaults for the spark_thread_limit parameter:

  • 4 in chi-square and gauss-mix
  • 6 in dec-tree
  • 8 in movie-lens
  • 12 in als, als-ml, log-regression and page-rank
  • unlimited in naive-bayes

The limits reflect the recommendations in #260 (comment)
@lbulej lbulej requested a review from ceresek July 2, 2021 11:11
@vhotspur
Copy link
Member

vhotspur commented Jul 2, 2021

Related to #283: perhaps there should be some hard minimum too?

@lbulej
Copy link
Member Author

lbulej commented Jul 2, 2021

Spark should be fine even with 1 CPU. On my machine (32G RAM, 4*2 CPUs), Spark benchmarks run (albeit slowly) under taskset -c 0, both on JDK8 and JDK11.

@lbulej
Copy link
Member Author

lbulej commented Jul 2, 2021

If the thread limits are fine, I'd suggest merging this, because the issue with #283 appears to be related to thread stack size. If we end up imposing a minimal number of threads to work around the thread stack size, it would be better to do it in another PR anyway.

@lbulej lbulej changed the title Set thread limits for the Spark local executor Set benchmark-specific thread limits for the Spark local executor Jul 2, 2021
@lbulej lbulej merged commit bcecf23 into master Jul 4, 2021
@lbulej lbulej deleted the topic/spark-thread-limits branch July 4, 2021 07:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants