-
Notifications
You must be signed in to change notification settings - Fork 230
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement getShuffleRDD and fixup mismatched output types on shuffle reuse [databricks] #4257
Conversation
…reuse Signed-off-by: Jason Lowe <jlowe@nvidia.com>
build |
Databricks 9.1 build failed due to access restrictions on |
build |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
import org.apache.spark.sql.execution.adaptive.{QueryStageExec, ShuffleQueryStageExec} | ||
|
||
/** Utility methods for manipulating Catalyst classes involved in Adaptive Query Execution */ | ||
object AQEUtils { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
same as the db version? hopefully this will be combined after the commonizing PR #4235
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think #4235 is going to address this. This change is only needed by spark312db, and there's no source directory for "everything except spark312db." I didn't bother to create one for this since the code is so small.
…reuse [databricks] (NVIDIA#4257) * Implement getShuffleRDD and fixup mismatched output types on shuffle reuse Signed-off-by: Jason Lowe <jlowe@nvidia.com> * Fix Databricks 9.1 build
…reuse [databricks] (NVIDIA#4257) * Implement getShuffleRDD and fixup mismatched output types on shuffle reuse Signed-off-by: Jason Lowe <jlowe@nvidia.com> * Fix Databricks 9.1 build
Fixes #4216.
This implements the getShuffleRDD interface for GpuShuffleExchangeExec and fixes an issue with ReusedExchangeExec caching CPU aggregation buffer types before they're updated with GPU aggregation buffer types. The plan is searched for instances where the ReusedShuffleExec output types do not match the underlying GPU shuffle output types, fixing the output types of ReusedShuffleExec to match the GPU shuffle.