Added Spark-3.4.2 Shims #9967

razajafri · 2023-12-05T21:39:24Z

This PR adds shims for Spark-3.4.2.

Changes Made:
Pretty much the result of running

mvn generate-sources -Dshimplify=true -Dshimplify.move=true -Dshimplify.overwrite=true -Dshimplify.add.shim=342 -Dshimplify.add.base=341

The only files that were manually changed were pom.xml, SparkShimsSuite.scala and ShimServiceProvider.scala to add the 3.4.2 version to the VERSIONNAMES.
Removed some empty lines as a result of the above Shimplify command.
Moved DecimalMultiply128.scala and GetSequenceSize.scala to the appropriate shim directory to maintain convention

Tests:
All IT were run locally.

fixes #9260

Signed-off-by: Raza Jafri <rjafri@nvidia.com>

tgravescs · 2023-12-07T16:11:28Z

note this should have same issue with decimal multiply - #9859

razajafri · 2023-12-20T02:47:22Z

This uses changes from #9962 so until that is merged this will stay a draft PR

The following tests are failing but should be fixed when #9962 is merged

====================================================================================== short test summary info 
FAILED ../../src/main/python/csv_test.py::test_csv_prefer_date_with_infer_schema[DATAGEN_SEED=1703008777, ALLOW_NON_GPU(FileSourceScanExec,CollectLimitExec,DeserializeToObjectExec)] - ValueError: year 0 is out of range
FAILED ../../src/main/python/arithmetic_ops_test.py::test_multiplication[Decimal(38,10)][DATAGEN_SEED=1703008777, INJECT_OOM] - AssertionError: GPU and CPU decimal values are different at [907, '(CAST(-12 AS DECIMAL(38,10)) * b)']
FAILED ../../src/main/python/arithmetic_ops_test.py::test_multiplication_mixed[Integer-Decimal(30,10)][DATAGEN_SEED=1703008777, INJECT_OOM] - AssertionError: GPU and CPU decimal values are different at [32, '(a * b)']
FAILED ../../src/main/python/collection_ops_test.py::test_sequence_too_long_sequence[Integer(not_null)][DATAGEN_SEED=1703008777, INJECT_OOM] - AssertionError: Expected error 'Too long sequence' did not appear in 'py4j.protocol.Py4JJavaError: An error occurred while calling o45025.collectToPython.
FAILED ../../src/main/python/collection_ops_test.py::test_sequence_too_long_sequence[Long(not_null)][DATAGEN_SEED=1703008777] - AssertionError: Expected error 'Too long sequence' did not appear in 'py4j.protocol.Py4JJavaError: An error occurred while calling o45109.collectToPython.
FAILED ../../src/main/python/arithmetic_ops_test.py::test_multiplication_mixed[Decimal(10,-2)-Decimal(30,10)][DATAGEN_SEED=1703008777, INJECT_OOM] - AssertionError: GPU and CPU decimal values are different at [50, '(a * b)']
FAILED ../../src/main/python/arithmetic_ops_test.py::test_multiplication_mixed[Decimal(15,3)-Decimal(30,10)][DATAGEN_SEED=1703008777, INJECT_OOM] - AssertionError: GPU and CPU decimal values are different at [15, '(a * b)']
FAILED ../../src/main/python/arithmetic_ops_test.py::test_multiplication_mixed[Decimal(30,12)-Integer][DATAGEN_SEED=1703008777] - AssertionError: GPU and CPU decimal values are different at [61, '(a * b)']
FAILED ../../src/main/python/arithmetic_ops_test.py::test_multiplication_mixed[Decimal(30,12)-Decimal(16,7)][DATAGEN_SEED=1703008777, INJECT_OOM] - AssertionError: GPU and CPU decimal values are different at [37, '(a * b)']
FAILED ../../src/main/python/arithmetic_ops_test.py::test_multiplication_mixed[Decimal(27,7)-Decimal(16,7)][DATAGEN_SEED=1703008777, INJECT_OOM] - AssertionError: GPU and CPU decimal values are different at [29, '(a * b)']
=============================================== 10 failed, 20648 passed, 1044 skipped, 380 xfailed, 294 xpassed, 8544 warnings in 2588.35s (0:43:08) ==========

razajafri · 2023-12-26T21:11:33Z

This uses changes from #9962 so until that is merged this will stay a draft PR

The following tests are failing but should be fixed when #9962 is merged

====================================================================================== short test summary info 
FAILED ../../src/main/python/csv_test.py::test_csv_prefer_date_with_infer_schema[DATAGEN_SEED=1703008777, ALLOW_NON_GPU(FileSourceScanExec,CollectLimitExec,DeserializeToObjectExec)] - ValueError: year 0 is out of range
FAILED ../../src/main/python/arithmetic_ops_test.py::test_multiplication[Decimal(38,10)][DATAGEN_SEED=1703008777, INJECT_OOM] - AssertionError: GPU and CPU decimal values are different at [907, '(CAST(-12 AS DECIMAL(38,10)) * b)']
FAILED ../../src/main/python/arithmetic_ops_test.py::test_multiplication_mixed[Integer-Decimal(30,10)][DATAGEN_SEED=1703008777, INJECT_OOM] - AssertionError: GPU and CPU decimal values are different at [32, '(a * b)']
FAILED ../../src/main/python/collection_ops_test.py::test_sequence_too_long_sequence[Integer(not_null)][DATAGEN_SEED=1703008777, INJECT_OOM] - AssertionError: Expected error 'Too long sequence' did not appear in 'py4j.protocol.Py4JJavaError: An error occurred while calling o45025.collectToPython.
FAILED ../../src/main/python/collection_ops_test.py::test_sequence_too_long_sequence[Long(not_null)][DATAGEN_SEED=1703008777] - AssertionError: Expected error 'Too long sequence' did not appear in 'py4j.protocol.Py4JJavaError: An error occurred while calling o45109.collectToPython.
FAILED ../../src/main/python/arithmetic_ops_test.py::test_multiplication_mixed[Decimal(10,-2)-Decimal(30,10)][DATAGEN_SEED=1703008777, INJECT_OOM] - AssertionError: GPU and CPU decimal values are different at [50, '(a * b)']
FAILED ../../src/main/python/arithmetic_ops_test.py::test_multiplication_mixed[Decimal(15,3)-Decimal(30,10)][DATAGEN_SEED=1703008777, INJECT_OOM] - AssertionError: GPU and CPU decimal values are different at [15, '(a * b)']
FAILED ../../src/main/python/arithmetic_ops_test.py::test_multiplication_mixed[Decimal(30,12)-Integer][DATAGEN_SEED=1703008777] - AssertionError: GPU and CPU decimal values are different at [61, '(a * b)']
FAILED ../../src/main/python/arithmetic_ops_test.py::test_multiplication_mixed[Decimal(30,12)-Decimal(16,7)][DATAGEN_SEED=1703008777, INJECT_OOM] - AssertionError: GPU and CPU decimal values are different at [37, '(a * b)']
FAILED ../../src/main/python/arithmetic_ops_test.py::test_multiplication_mixed[Decimal(27,7)-Decimal(16,7)][DATAGEN_SEED=1703008777, INJECT_OOM] - AssertionError: GPU and CPU decimal values are different at [29, '(a * b)']
=============================================== 10 failed, 20648 passed, 1044 skipped, 380 xfailed, 294 xpassed, 8544 warnings in 2588.35s (0:43:08) ==========

All of the above failing tests are now passing PTAL

razajafri · 2023-12-26T21:51:25Z

build

jlowe

MockTaskContextBase needs a 342 shim directive added to it.

razajafri · 2023-12-27T21:36:04Z

MockTaskContextBase needs a 342 shim directive added to it.

Done, PTAL

razajafri · 2023-12-27T21:48:13Z

build

gerashegalov · 2023-12-27T22:04:39Z

MockTaskContextBase needs a 342 shim directive added to it.

@jlowe are you OK with getting rid of MockTaskContextBase per #10096 (comment)

jlowe · 2023-12-27T22:05:19Z

are you OK with getting rid of MockTaskContextBase

Yes, that's fine with me.

gerashegalov · 2023-12-27T22:23:46Z

@razajafri I would drop MockTaskContextBase per #10096 (comment)

razajafri · 2023-12-28T00:34:59Z

@razajafri I would drop MockTaskContextBase per #10096 (comment)

Keeping in line with keeping the PRs focused on what they are supposed to do, I think that should be a separate PR. What do you think about that?

gerashegalov · 2023-12-28T01:09:49Z

Keeping in line with keeping the PRs focused on what they are supposed to do, I think that should be a separate PR. What do you think about that?

It is a valid point. Just thought of it because it would have preempted this comment #9967 (review) in this PR.

razajafri added 2 commits December 5, 2023 13:25

added 342 shims

fe9ed16

Signing off

e31d260

Signed-off-by: Raza Jafri <rjafri@nvidia.com>

sameerz added the feature request New feature or request label Dec 5, 2023

razajafri added 5 commits December 18, 2023 13:56

upmerged

82ee22b

missed from upmerge

a245e32

Fixed Sequence size limit check

3217520

Fixed the overflow check for addition and subtraction

fdac493

Updated test conditions

660a173

razajafri added 2 commits December 26, 2023 12:26

upmerged

b7b3209

removed unused file

31e2dc1

razajafri marked this pull request as ready for review December 26, 2023 21:11

razajafri requested review from jlowe, revans2, tgravescs, GaryShen2008 and NvTimLiu as code owners December 26, 2023 21:11

jlowe reviewed Dec 27, 2023

View reviewed changes

razajafri added 2 commits December 27, 2023 13:17

Merge remote-tracking branch 'origin/branch-24.02' into SP-9260-342-shim

52de05d

Added MockContextBase to 342

82fdbb8

jlowe approved these changes Dec 27, 2023

View reviewed changes

razajafri mentioned this pull request Dec 28, 2023

Add isFailed() to MockTaskContext and Remove MockTaskContextBase.scala [databricks] #10106

Merged

razajafri merged commit ff62883 into NVIDIA:branch-24.02 Dec 28, 2023
40 checks passed

razajafri deleted the SP-9260-342-shim branch December 28, 2023 02:38

nvliyuan mentioned this pull request May 20, 2024

[DOC] v2402 already support spark342 #10840

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added Spark-3.4.2 Shims #9967

Added Spark-3.4.2 Shims #9967

razajafri commented Dec 5, 2023 •

edited

Loading

tgravescs commented Dec 7, 2023

razajafri commented Dec 20, 2023

razajafri commented Dec 26, 2023

razajafri commented Dec 26, 2023

jlowe left a comment

razajafri commented Dec 27, 2023

razajafri commented Dec 27, 2023

gerashegalov commented Dec 27, 2023

jlowe commented Dec 27, 2023

gerashegalov commented Dec 27, 2023

razajafri commented Dec 28, 2023

gerashegalov commented Dec 28, 2023

Added Spark-3.4.2 Shims #9967

Added Spark-3.4.2 Shims #9967

Conversation

razajafri commented Dec 5, 2023 • edited Loading

tgravescs commented Dec 7, 2023

razajafri commented Dec 20, 2023

razajafri commented Dec 26, 2023

razajafri commented Dec 26, 2023

jlowe left a comment

Choose a reason for hiding this comment

razajafri commented Dec 27, 2023

razajafri commented Dec 27, 2023

gerashegalov commented Dec 27, 2023

jlowe commented Dec 27, 2023

gerashegalov commented Dec 27, 2023

razajafri commented Dec 28, 2023

gerashegalov commented Dec 28, 2023

razajafri commented Dec 5, 2023 •

edited

Loading