Disable GPU accelerated row-column transpose for Pascal GPUs: #5122
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Works around the failures described in NVIDIA#4980. This commit bypasses GPU accelerated row-column conversion for Pascal GPUs. Other architectures should remain unaffected by this shunt. Further investigation might be required to find the actual problem in CUDF's `fixed_width_convert_to_rows()` implementation. Signed-off-by: MithunR <mythrocks@gmail.com>
jlowe
added
the
cudf_dependency
jlowe
reviewed
Apr 1, 2022
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuColumnarToRowExec.scala
Outdated
Show resolved
Hide resolved
1. Converted `isAcceleratedTransposeSupported` to a `lazy val`. 2. Changed to use `Cuda.getComputeCapabilityMajor`.
|
I've tested that this bypass works for Pascal, and that Turing still uses the fast path. |
bdice
reviewed
Apr 1, 2022
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuColumnarToRowExec.scala
Outdated
Show resolved
Hide resolved
ajschmidt8
pushed a commit
to rapidsai/cudf
that referenced
this pull request
Apr 1, 2022
This commit introduces JNI bindings to retrieve the major and minor CUDA compute capability versions for the current CUDA device. This feature enables introspection from `spark-rapids` to detect the GPU architecture, for model-specific behaviour. This is required from NVIDIA/spark-rapids/pull/5122, to work around the erroneous behaviour of JNI `fixed_width_convert_to_rows()` on Pascal GPUs (#10569), (which in turn produces failures like NVIDIA/spark-rapids/issues/4980). Authors: - MithunR (https://github.com/mythrocks) Approvers: - https://github.com/nvdbaranec - Jason Lowe (https://github.com/jlowe) - Nghia Truong (https://github.com/ttnghia)
Rephrase description of the workaround for column->row transposition bug.
jlowe
approved these changes
Apr 4, 2022
|
Build |
mythrocks
added a commit
to mythrocks/spark-rapids
that referenced
this pull request
Apr 8, 2022
Fixes NVIDIA#5181. The change in NVIDIA#5122 modifies the row->column conversion code to run differently on Pascal GPUs. It does so by checking the GPU architecture to detect whether the current device is Pascal. In a distributed setup, this check should run on the Spark executor. As it currently stands, the check is erroneously run on the driver. This fix postpones the GPU architecture check till the executor task attempts to fetch result rows. Signed-off-by: MithunR <mythrocks@gmail.com>
pxLi
pushed a commit
that referenced
this pull request
Apr 12, 2022
Fixes #5181. The change in #5122 modifies the row->column conversion code to run differently on Pascal GPUs. It does so by checking the GPU architecture to detect whether the current device is Pascal. In a distributed setup, this check should run on the Spark executor. As it currently stands, the check is erroneously run on the driver. This fix postpones the GPU architecture check till the executor task attempts to fetch result rows. Signed-off-by: MithunR <mythrocks@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Depends on rapidsai/cudf/pull/10568.
Works around the failures described in #4980.
This commit bypasses GPU accelerated row-column conversion for Pascal GPUs.
Other architectures should remain unaffected by this shunt.
Further investigation might be required to find the actual problem in
CUDF's
fixed_width_convert_to_rows()implementation.Signed-off-by: MithunR mythrocks@gmail.com