Support split broadcast join condition into ast and non-ast [databricks] #9760

winningsix · 2023-11-16T22:52:35Z

This fixes #9759. Following similar approach as nested loop join case.

Signed-off-by: Ferdinand Xu <ferdinandx@nvidia.com>

winningsix · 2023-11-16T22:52:59Z

build

winningsix · 2023-11-17T03:46:11Z

Will take care of DBX failure.

winningsix · 2023-11-20T11:46:02Z

build

winningsix · 2023-11-20T23:45:57Z

build

winningsix · 2023-11-21T03:15:34Z

build

integration_tests/src/main/python/join_test.py

...ugin/src/main/scala/org/apache/spark/sql/rapids/execution/GpuBroadcastHashJoinExecBase.scala

...src/main/spark311/scala/org/apache/spark/sql/rapids/execution/GpuBroadcastHashJoinExec.scala

sql-plugin/src/main/scala/org/apache/spark/sql/rapids/execution/GpuHashJoin.scala

winningsix · 2023-11-22T23:56:18Z

Trying to avoid wrapping a placeholder project node to workaround output attribute.

Update all versions

winningsix · 2023-11-24T16:57:18Z

build

winningsix · 2023-11-25T01:05:10Z

build

winningsix · 2023-11-25T08:04:01Z

build

winningsix · 2023-11-26T01:10:27Z

Comments addressed. Please help take further look. @jlowe Thanks!

integration_tests/src/main/python/join_test.py

...ugin/src/main/scala/org/apache/spark/sql/rapids/execution/GpuBroadcastHashJoinExecBase.scala

…n/GpuBroadcastHashJoinExecBase.scala Co-authored-by: Jason Lowe <jlowe@nvidia.com>

winningsix · 2023-12-07T11:14:39Z

DB CI failed due to #9977

winningsix · 2023-12-07T22:18:57Z

build

jlowe

Looks good overall, just minor cleanups

sql-plugin/src/main/scala/com/nvidia/spark/rapids/AstUtil.scala

sql-plugin/src/main/scala/org/apache/spark/sql/rapids/execution/GpuHashJoin.scala

Co-authored-by: Jason Lowe <jlowe@nvidia.com>

…n/GpuHashJoin.scala Co-authored-by: Jason Lowe <jlowe@nvidia.com>

Co-authored-by: Jason Lowe <jlowe@nvidia.com>

winningsix · 2023-12-07T23:11:20Z

build

winningsix · 2023-12-08T05:07:30Z

@NVnavkumar can you help take a look? All comments are addressed with checks passed. Thanks!

…ks] (NVIDIA#9760) Signed-off-by: Ferdinand Xu <ferdinandx@nvidia.com> Co-authored-by: Jason Lowe <jlowe@nvidia.com> Co-authored-by: Navin Kumar <97137715+NVnavkumar@users.noreply.github.com>

* Merged NVDA/branch-23.12 * Support split broadcast join condition into ast and non-ast [databricks] (NVIDIA#9760) --------- Signed-off-by: Ferdinand Xu <ferdinandx@nvidia.com> Co-authored-by: Ferdinand Xu <ferdinandx@nvidia.com> Co-authored-by: Jason Lowe <jlowe@nvidia.com> Co-authored-by: Navin Kumar <97137715+NVnavkumar@users.noreply.github.com>

…databricks] (NVIDIA#9760)" This reverts commit 7c307d4.

…databricks] (NVIDIA#9760)" This reverts commit 7c307d4. Signed-off-by: Ferdinand Xu <ferdinandx@nvidia.com>

…databricks] (#9760)" This reverts commit 7c307d4. Signed-off-by: Ferdinand Xu <ferdinandx@nvidia.com>

…ks] (NVIDIA#9760) * Support split broadcast join condition into ast and non-ast Signed-off-by: Ferdinand Xu <ferdinandx@nvidia.com> * Fix * Fix compile * Fix * Fix * Address comments * Fix Update all versions * Fix * Fix * Fix Spark 311 * Update sql-plugin/src/main/scala/org/apache/spark/sql/rapids/execution/GpuBroadcastHashJoinExecBase.scala Co-authored-by: Jason Lowe <jlowe@nvidia.com> * Address comments * Update sql-plugin/src/main/scala/com/nvidia/spark/rapids/AstUtil.scala Co-authored-by: Jason Lowe <jlowe@nvidia.com> * Update sql-plugin/src/main/scala/com/nvidia/spark/rapids/AstUtil.scala Co-authored-by: Jason Lowe <jlowe@nvidia.com> * Update sql-plugin/src/main/scala/com/nvidia/spark/rapids/AstUtil.scala Co-authored-by: Jason Lowe <jlowe@nvidia.com> * Update sql-plugin/src/main/scala/com/nvidia/spark/rapids/AstUtil.scala Co-authored-by: Jason Lowe <jlowe@nvidia.com> * Fix scala 2.13, code style, refactor * Minor fix * minor * Fix scala 2.13 * Fix DBX * Update sql-plugin/src/main/scala/com/nvidia/spark/rapids/AstUtil.scala Co-authored-by: Navin Kumar <97137715+NVnavkumar@users.noreply.github.com> * Update sql-plugin/src/main/scala/com/nvidia/spark/rapids/AstUtil.scala Co-authored-by: Jason Lowe <jlowe@nvidia.com> * Fix scala 2.13, refactor * Revert unnecessary changes * Fix failed UT * Update sql-plugin/src/main/scala/com/nvidia/spark/rapids/AstUtil.scala Co-authored-by: Jason Lowe <jlowe@nvidia.com> * Update sql-plugin/src/main/scala/com/nvidia/spark/rapids/AstUtil.scala Co-authored-by: Jason Lowe <jlowe@nvidia.com> * Update sql-plugin/src/main/scala/com/nvidia/spark/rapids/AstUtil.scala Co-authored-by: Jason Lowe <jlowe@nvidia.com> * Update sql-plugin/src/main/scala/org/apache/spark/sql/rapids/execution/GpuHashJoin.scala Co-authored-by: Jason Lowe <jlowe@nvidia.com> * Update sql-plugin/src/main/scala/org/apache/spark/sql/rapids/execution/GpuHashJoin.scala Co-authored-by: Jason Lowe <jlowe@nvidia.com> * Update sql-plugin/src/main/scala/org/apache/spark/sql/rapids/execution/GpuHashJoin.scala Co-authored-by: Jason Lowe <jlowe@nvidia.com> * Update sql-plugin/src/main/scala/org/apache/spark/sql/rapids/execution/GpuHashJoin.scala Co-authored-by: Jason Lowe <jlowe@nvidia.com> * Update sql-plugin/src/main/scala/com/nvidia/spark/rapids/AstUtil.scala Co-authored-by: Jason Lowe <jlowe@nvidia.com> * Update sql-plugin/src/main/scala/com/nvidia/spark/rapids/AstUtil.scala Co-authored-by: Jason Lowe <jlowe@nvidia.com> --------- Signed-off-by: Ferdinand Xu <ferdinandx@nvidia.com> Co-authored-by: Jason Lowe <jlowe@nvidia.com> Co-authored-by: Navin Kumar <97137715+NVnavkumar@users.noreply.github.com>

Support split broadcast join condition into ast and non-ast

a9c32a4

Signed-off-by: Ferdinand Xu <ferdinandx@nvidia.com>

winningsix marked this pull request as ready for review November 16, 2023 22:52

winningsix added 3 commits November 20, 2023 18:28

Fix

4f1ce38

Fix compile

3fe880b

Fix

5823e65

Fix

b6d4c3e

winningsix mentioned this pull request Nov 21, 2023

[BUG] Code dump in rapids-4-spark-api-validation_2.12 test for RegularExpressionTranspilerSuite test #9808

Closed

winningsix requested review from revans2 and jlowe November 21, 2023 14:07

jlowe reviewed Nov 21, 2023

View reviewed changes

winningsix marked this pull request as draft November 22, 2023 23:55

winningsix added 2 commits November 23, 2023 18:19

Address comments

9538f36

Fix

73bb81f

Update all versions

winningsix force-pushed the asthbj branch from 42d5399 to 73bb81f Compare November 24, 2023 16:29

Fix

16876f1

Fix

94e27a5

winningsix marked this pull request as ready for review November 25, 2023 01:05

Fix Spark 311

8a31238

jlowe reviewed Nov 27, 2023

View reviewed changes

Update sql-plugin/src/main/scala/org/apache/spark/sql/rapids/executio…

944f64d

…n/GpuBroadcastHashJoinExecBase.scala Co-authored-by: Jason Lowe <jlowe@nvidia.com>

Fix failed UT

2d99043

jlowe reviewed Dec 7, 2023

View reviewed changes

winningsix and others added 9 commits December 8, 2023 06:57

Update sql-plugin/src/main/scala/com/nvidia/spark/rapids/AstUtil.scala

514282d

Co-authored-by: Jason Lowe <jlowe@nvidia.com>

Update sql-plugin/src/main/scala/com/nvidia/spark/rapids/AstUtil.scala

441dd75

Co-authored-by: Jason Lowe <jlowe@nvidia.com>

Update sql-plugin/src/main/scala/com/nvidia/spark/rapids/AstUtil.scala

660c357

Co-authored-by: Jason Lowe <jlowe@nvidia.com>

Update sql-plugin/src/main/scala/org/apache/spark/sql/rapids/executio…

ca5e80a

…n/GpuHashJoin.scala Co-authored-by: Jason Lowe <jlowe@nvidia.com>

Update sql-plugin/src/main/scala/org/apache/spark/sql/rapids/executio…

447c24a

…n/GpuHashJoin.scala Co-authored-by: Jason Lowe <jlowe@nvidia.com>

Update sql-plugin/src/main/scala/org/apache/spark/sql/rapids/executio…

1ae0711

…n/GpuHashJoin.scala Co-authored-by: Jason Lowe <jlowe@nvidia.com>

Update sql-plugin/src/main/scala/org/apache/spark/sql/rapids/executio…

ea5738f

…n/GpuHashJoin.scala Co-authored-by: Jason Lowe <jlowe@nvidia.com>

Update sql-plugin/src/main/scala/com/nvidia/spark/rapids/AstUtil.scala

e7ec35a

Co-authored-by: Jason Lowe <jlowe@nvidia.com>

Update sql-plugin/src/main/scala/com/nvidia/spark/rapids/AstUtil.scala

fbde5c9

Co-authored-by: Jason Lowe <jlowe@nvidia.com>

jlowe approved these changes Dec 7, 2023

View reviewed changes

NVnavkumar approved these changes Dec 11, 2023

View reviewed changes

winningsix merged commit 7c307d4 into NVIDIA:branch-24.02 Dec 12, 2023
38 checks passed

winningsix deleted the asthbj branch December 12, 2023 00:00

This was referenced Dec 12, 2023

Update 23.12 to the top of NVDA/branch-23.12 #10022

Closed

Update 23.12 to the top of NVDA/branch-23.12 firestarman/spark-rapids#12

Merged

jbrennan333 mentioned this pull request Dec 13, 2023

[BUG] CudfException during conditional hash join while running nds query64 #10047

Closed

winningsix added a commit to winningsix/spark-rapids that referenced this pull request Dec 18, 2023

Revert "Support split broadcast join condition into ast and non-ast […

67f071a

…databricks] (NVIDIA#9760)" This reverts commit 7c307d4.

winningsix mentioned this pull request Dec 18, 2023

Revert "Support split broadcast join condition into ast and non-ast [… #10069

Merged

winningsix added a commit to winningsix/spark-rapids that referenced this pull request Dec 18, 2023

Revert "Support split broadcast join condition into ast and non-ast […

8342d09

…databricks] (NVIDIA#9760)" This reverts commit 7c307d4. Signed-off-by: Ferdinand Xu <ferdinandx@nvidia.com>

jbrennan333 pushed a commit that referenced this pull request Dec 19, 2023

Revert "Support split broadcast join condition into ast and non-ast […

a613427

…databricks] (#9760)" This reverts commit 7c307d4. Signed-off-by: Ferdinand Xu <ferdinandx@nvidia.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support split broadcast join condition into ast and non-ast [databricks] #9760

Support split broadcast join condition into ast and non-ast [databricks] #9760

winningsix commented Nov 16, 2023

winningsix commented Nov 16, 2023

winningsix commented Nov 17, 2023

winningsix commented Nov 20, 2023

winningsix commented Nov 20, 2023

winningsix commented Nov 21, 2023

winningsix commented Nov 22, 2023

winningsix commented Nov 24, 2023

winningsix commented Nov 25, 2023

winningsix commented Nov 25, 2023

winningsix commented Nov 26, 2023

winningsix commented Dec 7, 2023 •

edited

Loading

winningsix commented Dec 7, 2023

jlowe left a comment

winningsix commented Dec 7, 2023

winningsix commented Dec 8, 2023

Support split broadcast join condition into ast and non-ast [databricks] #9760

Support split broadcast join condition into ast and non-ast [databricks] #9760

Conversation

winningsix commented Nov 16, 2023

winningsix commented Nov 16, 2023

winningsix commented Nov 17, 2023

winningsix commented Nov 20, 2023

winningsix commented Nov 20, 2023

winningsix commented Nov 21, 2023

winningsix commented Nov 22, 2023

winningsix commented Nov 24, 2023

winningsix commented Nov 25, 2023

winningsix commented Nov 25, 2023

winningsix commented Nov 26, 2023

winningsix commented Dec 7, 2023 • edited Loading

winningsix commented Dec 7, 2023

jlowe left a comment

Choose a reason for hiding this comment

winningsix commented Dec 7, 2023

winningsix commented Dec 8, 2023

winningsix commented Dec 7, 2023 •

edited

Loading