Auto-register UDF extention when main plugin is set #1102
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Allen Xu <allxu@nvidia.com>
wjxiz1992
changed the title
wjxiz1992
changed the title
|
build |
abellina
approved these changes
Nov 12, 2020
|
We should also update documentation:
Could happen in a different PR IMHO. |
abellina
requested changes
Nov 12, 2020
Removing current extension requires adding main sql plugin config. But that will fail the OpcodeSuite. |
|
I would like to see the documentation updated as a part of this PR, just so it is not forgotten. |
Signed-off-by: Allen Xu <allxu@nvidia.com>
revans2
previously approved these changes
Nov 13, 2020
docs/compatibility.md
Outdated
| @@ -290,10 +290,10 @@ Casting from string to timestamp currently has the following limitations. | |||
| Only timezone 'Z' (UTC) is supported. Casting unsupported formats will result in null values. | |||
|
|
|||
| ## UDF to Catalyst Expressions | |||
| To speedup the process of UDF, spark-rapids introduces a udf-compiler extension to translate UDFs to Catalyst expressions. | |||
| To speedup the process of UDF, spark-rapids introduces a udf-compiler extension to translate UDFs to Catalyst expressions. This compiler will be injected automatically to spark extensions by setting `spark.plugins=com.nvidia.spark.SQLPlugin` and is disabled by default. | |||
There was a problem hiding this comment.
nit: I don't think we need this line. I don't think the user cares about the internal details of how we implemented it, just how to enable it, which is covered by the change below.
There was a problem hiding this comment.
Right, reasonable to me. Updated.
Signed-off-by: Allen Xu <allxu@nvidia.com>
revans2
approved these changes
Nov 13, 2020
|
build |
Sounds good, sorry I missed that. |
abellina
approved these changes
Nov 13, 2020
sperlingxx
pushed a commit
to sperlingxx/spark-rapids
that referenced
this pull request
Nov 20, 2020
* Auto-register UDF extention when main plugin is set Signed-off-by: Allen Xu <allxu@nvidia.com> * Update udf-compiler descriptions in related docs Signed-off-by: Allen Xu <allxu@nvidia.com> * Remove unecessary lines to make more user-friendly Signed-off-by: Allen Xu <allxu@nvidia.com> Co-authored-by: Allen Xu <allxu@nvidia.com>
nartal1
pushed a commit
to nartal1/spark-rapids
that referenced
this pull request
Jun 9, 2021
* Auto-register UDF extention when main plugin is set Signed-off-by: Allen Xu <allxu@nvidia.com> * Update udf-compiler descriptions in related docs Signed-off-by: Allen Xu <allxu@nvidia.com> * Remove unecessary lines to make more user-friendly Signed-off-by: Allen Xu <allxu@nvidia.com> Co-authored-by: Allen Xu <allxu@nvidia.com>
nartal1
pushed a commit
to nartal1/spark-rapids
that referenced
this pull request
Jun 9, 2021
* Auto-register UDF extention when main plugin is set Signed-off-by: Allen Xu <allxu@nvidia.com> * Update udf-compiler descriptions in related docs Signed-off-by: Allen Xu <allxu@nvidia.com> * Remove unecessary lines to make more user-friendly Signed-off-by: Allen Xu <allxu@nvidia.com> Co-authored-by: Allen Xu <allxu@nvidia.com>
tgravescs
pushed a commit
to tgravescs/spark-rapids
that referenced
this pull request
Nov 30, 2023
…IDIA#1102) Signed-off-by: spark-rapids automation <70000568+nvauto@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Allen Xu allxu@nvidia.com
To deal with #688.
With this change, the udf-compiler extension will be registered automatically which has the same behavior as
SQLExecPlugin.