Support for non-string key-types for GetMapValue and element_at()
#4944
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This can't be merged until rapidsai/cudf#10380 has been merged. The code still needs to be rebased to latest. Note, also, that the new tests in |
mythrocks
force-pushed
the
map-non-string-scalars
branch
from
af4306e
to
655cd2a
Compare
This commit adds support for non-string keys and values
in map columns. Specifically:
1. `GetMapValue` supports keys of integral, floating point,
chrono, and string scalars. The returned column can be
of any CUDF type.
2. Similarly, `ElementAt` on Map inputs now supports keys
of integral, floating point, chrono, and string scalars.
The returned column can be of any supported CUDF type.
Signed-off-by: MithunR <mythrocks@gmail.com>
mythrocks
force-pushed
the
map-non-string-scalars
branch
from
655cd2a
to
66f3e2d
Compare
|
Moved out of |
|
Build |
revans2
reviewed
Mar 16, 2022
sql-plugin/src/main/311until320-nondb/scala/com/nvidia/spark/rapids/shims/Spark31XShims.scala
Show resolved
Hide resolved
sql-plugin/src/main/311until320-nondb/scala/com/nvidia/spark/rapids/shims/Spark31XShims.scala
Show resolved
Hide resolved
1. Allow Map lookup for values of type Decimal, Array, Struct, Maps, etc. 2. Reduce map row size in tests. 3. Added tests for STRUCT, DECIMAL values.
|
Build |
|
Build |
revans2
reviewed
Mar 17, 2022
sql-plugin/src/main/311until320-nondb/scala/com/nvidia/spark/rapids/shims/Spark31XShims.scala
Show resolved
Hide resolved
revans2
approved these changes
Mar 18, 2022
|
Build |
mythrocks
added a commit
to mythrocks/spark-rapids
that referenced
this pull request
Apr 8, 2022
Fixes NVIDIA#5180. Map lookup is currently supported only in cases where the keys are scalar values. In case the keys are specified as a vector (e.g. expressions), the plugin should fall back to CPU. NVIDIA#4944 introduced a bug in how literal signatures are specified for multiple data types. This breaks CPU fallback. This commit fixes the specification of literals-only `TypeSig`.
This was referenced Apr 8, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #3245.
Depends on rapidsai/cudf#10380, which adds support for non-string key types in map columns.
This commit adds non-string key-type support in the
spark-rapidsplugin for map lookups withGetMapValueandelement_at().Note that nested key types are still not supported. The keys must be the common primitive types (integral, floating point, strings, chrono types).