Fix canonicalization of GpuFileSourceScanExec, GpuShuffleCoalesceExec #1310
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
At least part of the issue in #1308 is caused by parts of the query being executed multiple times redundantly because Spark is unable to determine that subsections of the query are identical. This is caused by broken canonicalization where a GPU Exec node is not comparing correctly with a semantically-equivalent copy of itself.
In this case it's the use of a
RapidsConf
instance as a parameter of the case class.RapidsConf
doesn't have anequals
method, so two instances will be considered different. However I don't think adding anequals
method is the real fix here, as the configs could be slightly different in ways unrelated to the semantics of the node's execution and therefore should still be considered equal with respect to that node's canonicalization.In the case of
GpuShuffleCoalesceExec
it was easy, there was only one config value being used, so I replaced the conf parameter with the value from the conf. ForGpuFileSourceScanExec
it is trickier since there are many config parameters being used, and it already has quite a few arguments. In this case I opted to place the conf instance in a separate parameter list which is excluded from the automatically generated equals and hashcode comparison which is used by Spark plan canonicalization. In our case, there are no RAPIDS config settings that change the semantics for what data is read and how it is presented, so we can safely ignore any RAPIDS configs for purposes of canonicalization.