Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Regressions in spark-shell usage of benchmark utilities #1400

Closed
andygrove opened this issue Dec 15, 2020 · 0 comments · Fixed by #1444
Closed

[BUG] Regressions in spark-shell usage of benchmark utilities #1400

andygrove opened this issue Dec 15, 2020 · 0 comments · Fixed by #1444
Assignees
@andygrove
Labels
benchmark Benchmarking, benchmarking tools bug Something isn't working
Milestone
Jan 4 - Jan 15

Comments

@andygrove
Copy link
Contributor

andygrove commented Dec 15, 2020 •
edited
Loading

Describe the bug
During recent work to support running benchmarks and comparisons from spark-submit, the BenchUtils code was refactored to support CLI usage. This has led to regressions in the UX when using spark-shell and also means that the documentation for spark-shell usage is now outdated but this is more than just a documentation bug because we need to make it easy to run everything from spark-shell again.

We should also add unit tests to mimic the documented spark-shell usage so that we do not have future regressions like this.

Steps/Code to reproduce bug
Follow the benchmarks documentation and try and reproduce all examples of spark-shell usage.

Expected behavior
It should be possible to run all benchmarks and comparisons from spark-shell following the documentation.
@andygrove andygrove added bug Something isn't working ? - Needs Triage Need team to review and classify benchmark Benchmarking, benchmarking tools labels Dec 15, 2020
@andygrove andygrove added this to the Jan 4 - Jan 15 milestone Dec 15, 2020
@andygrove andygrove self-assigned this Dec 15, 2020
@sameerz sameerz removed the ? - Needs Triage Need team to review and classify label Dec 18, 2020
@andygrove andygrove mentioned this issue Jan 5, 2021
Merged
tgravescs pushed a commit to tgravescs/spark-rapids that referenced this issue Nov 30, 2023
@nvauto
Update submodule cudf to 0190c2921d0278f80328240b76a22e6628cb24f7 (NV…
5991f80 
…IDIA#1400)

Signed-off-by: spark-rapids automation <70000568+nvauto@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@andygrove andygrove

Labels
benchmark Benchmarking, benchmarking tools bug Something isn't working
Projects
None yet
Milestone
Jan 4 - Jan 15
Development

Successfully merging a pull request may close this issue.

Improve UX when running benchmarks from Spark shell andygrove/spark-rapids
2 participants
@andygrove @sameerz