Support spilling to disk directly via cuFile/GDS #1313
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Rong Ou <rong.ou@gmail.com>
rongou
added
performance
revans2
reviewed
Dec 8, 2020
abellina
reviewed
Dec 8, 2020
sql-plugin/src/main/scala/com/nvidia/spark/rapids/shuffle/BufferSendState.scala
Show resolved
Hide resolved
abellina
requested changes
Dec 8, 2020
sql-plugin/src/main/scala/com/nvidia/spark/rapids/RapidsGdsStore.scala
Outdated
Show resolved
Hide resolved
abellina
reviewed
Dec 8, 2020
sql-plugin/src/main/scala/com/nvidia/spark/rapids/RapidsGdsStore.scala
Outdated
Show resolved
Hide resolved
|
@revans2 added wording to the config to specify it's an experimental feature, and link to gds docs. |
revans2
previously approved these changes
Dec 9, 2020
jlowe
requested changes
Dec 9, 2020
sql-plugin/src/main/scala/com/nvidia/spark/rapids/RapidsGdsStore.scala
Outdated
Show resolved
Hide resolved
Signed-off-by: Rong Ou <rong.ou@gmail.com>
jlowe
approved these changes
Dec 10, 2020
|
build |
|
build |
|
build |
1 similar comment
|
build |
abellina
approved these changes
Dec 17, 2020
nartal1
pushed a commit
to nartal1/spark-rapids
that referenced
this pull request
Jun 9, 2021
* support spilling to GDS Signed-off-by: Rong Ou <rong.ou@gmail.com> * address review comments * review feedback Signed-off-by: Rong Ou <rong.ou@gmail.com>
nartal1
pushed a commit
to nartal1/spark-rapids
that referenced
this pull request
Jun 9, 2021
* support spilling to GDS Signed-off-by: Rong Ou <rong.ou@gmail.com> * address review comments * review feedback Signed-off-by: Rong Ou <rong.ou@gmail.com>
tgravescs
pushed a commit
to tgravescs/spark-rapids
that referenced
this pull request
Nov 30, 2023
…p ci] [bot] (NVIDIA#1313) * Update submodule cudf to 9c559c94fcaa1525d7c95faf94e5486fcd992ef1 Signed-off-by: spark-rapids automation <70000568+nvauto@users.noreply.github.com> * Update submodule cudf to b7994bc16b1b1743b0743860b4f02ac4da8245d5 Signed-off-by: spark-rapids automation <70000568+nvauto@users.noreply.github.com> * Update submodule cudf to 15cc5011902f1026e04662e725b880f48d38ba8d Signed-off-by: spark-rapids automation <70000568+nvauto@users.noreply.github.com> --------- Signed-off-by: spark-rapids automation <70000568+nvauto@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds an option to spill to disk directly via GDS. Default is off and shouldn't have any effect on existing behavior.
Initial test on my local desktop shows that it performs similarly to spilling to host memory, with a single Samsung 970 EVO Plus 2TB NVMe M.2 drive. Need to do more benchmarking in real deployment environments.
Depends on rapidsai/cudf#6940.