Add missed shims for scala2.13 #10465

NvTimLiu · 2024-02-22T08:08:30Z

Add spark334 and spark342 shims for the private scala2.13 jars

To fix: #10464

Signed-off-by: Tim Liu <timl@nvidia.com>

NvTimLiu · 2024-02-22T08:17:59Z

build

gerashegalov · 2024-02-22T21:44:11Z

According to the PR checks spark-rapids-private sclala2.13 shims for 334 and 342 are not published either

NVnavkumar · 2024-02-23T05:30:24Z

pom.xml

            340,
            341,
+            342,
            350
        </noSnapshotScala213.buildvers>
        <snapshotScala213.buildvers>


Need to add 351 to snapshotScala213.buildvers as well

Need to add 351 to snapshotScala213.buildvers as well

We do not support 3.5.1 in 24.02, right?

This is for 24.02 release build, snapshot shim doesn't matter.

Need to add 351 to snapshotScala213.buildvers as well

Good catch! Let me update it.

Need to add 351 to snapshotScala213.buildvers as well

We do not support 3.5.1 in 24.02, right?

Yes, we only build/test against spark-3.5.1-SNAPSHOT

Signed-off-by: Tim Liu <timl@nvidia.com>

NvTimLiu · 2024-02-26T10:01:20Z

Oops,

spark released 3.5.1 days ago : https://archive.apache.org/dist/spark/spark-3.5.1/

But apache spark does not announce it in its download page: https://spark.apache.org/docs/latest/

For branch-24.02, we'd better remove 351 from list instead?
https://github.com/NVIDIA/spark-rapids/blob/branch-24.02/pom.xml#L837

For branch-24.04, we should move 351 into
https://github.com/NVIDIA/spark-rapids/blob/branch-24.04/pom.xml#L835?

@sameerz , @GaryShen2008

NvTimLiu · 2024-02-26T23:12:21Z

According to the PR checks spark-rapids-private sclala2.13 shims for 334 and 342 are not published either

Yes, as private v24.02.0 dependency jars have been published to maven central without 334 and 342 shims,

we'll re-publish v24.02.1 private jars to maven central including the 334 and 342 shims,

then change the spark-rapids-private dependency to v24.02.1 here again to PASS the pre-merge.

sameerz · 2024-02-26T23:50:35Z

For branch-24.02, we'd better remove 351 from list instead?
https://github.com/NVIDIA/spark-rapids/blob/branch-24.02/pom.xml#L837

For branch-24.04, we should move 351 into
https://github.com/NVIDIA/spark-rapids/blob/branch-24.04/pom.xml#L835?

Yes we can remove 351 from 24.02 and add to 24.04.

Signed-off-by: Tim Liu <timl@nvidia.com>

NVnavkumar · 2024-02-27T00:36:15Z

scala2.13/pom.xml

            350
        </noSnapshotScala213.buildvers>
        <snapshotScala213.buildvers>
+            351


Actually, we should remove this too since it is removed from snapshot.buildvers

Oh I forget it, let me update

Signed-off-by: Tim Liu <timl@nvidia.com>

sameerz · 2024-02-27T06:59:24Z

build

NvTimLiu · 2024-02-27T13:43:58Z

I'll try to remove 351 shim source code to PASS the mvn build. @razajafri can you help review it? Thanks!

Ran `mvn generate-sources -Dshimplify=true -Dshimplify.move=true -Dshimplify.remove.shim=351` to remove 351 shim's jason string, and fix some unnecessary empty lines that were introduced Signed-off-by: Tim Liu <timl@nvidia.com>

NvTimLiu · 2024-02-27T14:15:09Z

build

The pre-merge CI always failed due to no spark-rapids-private spark334/spark342 v24.02.0 shims until we release the new v24.02.1 private jars (which will definitely including 334/342shims)

Auto copyright by below scripts ``` export SPARK_RAPIDS_AUTO_COPYRIGHTER=ON ./scripts/auto-copyrighter.sh $(git diff --name-only origin/branch-24.04..HEAD) ``` Signed-off-by: Tim Liu <timl@nvidia.com>

Move 351 shims into noSnapshot buildvers as spark has release it. Follow up of NVIDIA#10465 (comment) Signed-off-by: Tim Liu <timl@nvidia.com>

* Move 351 shims into noSnapshot buildvers Move 351 shims into noSnapshot buildvers as spark has release it. Follow up of #10465 (comment) Signed-off-by: Tim Liu <timl@nvidia.com> * 351 shim for scala 2.13 Signed-off-by: Tim Liu <timl@nvidia.com> --------- Signed-off-by: Tim Liu <timl@nvidia.com>

jlowe · 2024-02-29T23:17:25Z

...gen/src/main/spark320/scala/org/apache/spark/sql/tests/datagen/datagen/DataGenExprBase.scala

@@ -36,7 +36,6 @@
 {"spark": "341db"}
 {"spark": "342"}
 {"spark": "350"}
-{"spark": "351"}


Why does this PR remove 351 when it says it's adding shims?

it's actually adding the 334 and 342 shim versions to the pom file for scala 2.13.

351 context is here: #10465 (comment)

We should not be removing 351 in the source code this late. Do we want to not build 351 support in the 24.02 jar? Fine. But this much source churn this late is not good. Modify the pom to not ask for a 351 build rather than fully ripping out the source code.

And this will cause a merge nightmare for 24.04 if done this way.

And this will cause a merge nightmare for 24.04 if done this way.

exactly

NvTimLiu · 2024-02-29T23:30:42Z

build

This reverts commit 8482847.

This reverts commit 78d1f00.

razajafri · 2024-03-01T01:44:31Z

@jlowe PTAL
@gerashegalov if you have time PTAL

sameerz · 2024-03-01T01:56:08Z

build

Run the script `bash build/make-scala-version-build-files.sh 2.13` Signed-off-by: Tim Liu <timl@nvidia.com>

NvTimLiu · 2024-03-01T02:37:05Z

build

Signed-off-by: Gera Shegalov <gera@apache.org>

…0465-1

Signed-off-by: Tim Liu <timl@nvidia.com>

gerashegalov

LGTM

NvTimLiu · 2024-03-01T03:38:08Z

build

GaryShen2008

Good to be clean now.

razajafri

LGTM

NvTimLiu · 2024-03-01T06:08:00Z

premerge failed: rebuild

 ERROR SparkUI: Failed to bind SparkUI
java.net.BindException: Failed to bind to /0.0.0.0:4056: Service 'SparkUI' failed after 16 retries (starting from 4040)! Consider explicitly setting the appropriate port for the service 'SparkUI' (for example spark.ui.port for SparkUI) to an available port or increasing spark.port.maxRetries.

NvTimLiu · 2024-03-01T06:08:05Z

build

* Update rapids jni and private dependency version to 24.02.1 (#10511) Signed-off-by: Tim Liu <timl@nvidia.com> * Add missed shims for scala2.13 (#10465) * Add missed shims for scala2.13 Signed-off-by: Tim Liu <timl@nvidia.com> * Add 351 snapshot shim for the scala2.13 version of plugin jar Signed-off-by: Tim Liu <timl@nvidia.com> * Remove 351 snapshot shim as spark 3.5.1 has been released Signed-off-by: Tim Liu <timl@nvidia.com> * Remove scala2.13 351 snapshot shim Signed-off-by: Tim Liu <timl@nvidia.com> * Remove 351 shim's jason string Ran `mvn generate-sources -Dshimplify=true -Dshimplify.move=true -Dshimplify.remove.shim=351` to remove 351 shim's jason string, and fix some unnecessary empty lines that were introduced Signed-off-by: Tim Liu <timl@nvidia.com> * Update Copyright 2024 Auto copyright by below scripts ``` export SPARK_RAPIDS_AUTO_COPYRIGHTER=ON ./scripts/auto-copyrighter.sh $(git diff --name-only origin/branch-24.04..HEAD) ``` Signed-off-by: Tim Liu <timl@nvidia.com> * Revert "Update Copyright 2024" This reverts commit 8482847. * Revert "Remove 351 shim's jason string" This reverts commit 78d1f00. * skip 351 from strict checking * Alien scala2.13/pom.xml to scala2.12 one Run the script `bash build/make-scala-version-build-files.sh 2.13` Signed-off-by: Tim Liu <timl@nvidia.com> * pretend 351 is a snapshot in 24.02 Signed-off-by: Gera Shegalov <gera@apache.org> * pretend 351 is a SNAPSHOT version * Revert change of build/shimplify.py Signed-off-by: Tim Liu <timl@nvidia.com> --------- Signed-off-by: Tim Liu <timl@nvidia.com> Signed-off-by: Gera Shegalov <gera@apache.org> Co-authored-by: Raza Jafri <rjafri@nvidia.com> Co-authored-by: Gera Shegalov <gera@apache.org> * Update changelog for v24.02.0 release (#10525) Signed-off-by: Tim Liu <timl@nvidia.com> --------- Signed-off-by: Tim Liu <timl@nvidia.com> Signed-off-by: Gera Shegalov <gera@apache.org> Co-authored-by: Raza Jafri <rjafri@nvidia.com> Co-authored-by: Gera Shegalov <gera@apache.org>

Add missed shims for scala2.13

22641d5

Signed-off-by: Tim Liu <timl@nvidia.com>

NvTimLiu added the build Related to CI / CD or cleanly building label Feb 22, 2024

NvTimLiu requested a review from sameerz February 22, 2024 08:08

NvTimLiu self-assigned this Feb 22, 2024

NvTimLiu requested review from jlowe, revans2, tgravescs, GaryShen2008 and gerashegalov as code owners February 22, 2024 08:08

jlowe previously approved these changes Feb 22, 2024

View reviewed changes

NVnavkumar requested changes Feb 23, 2024

View reviewed changes

NVnavkumar previously approved these changes Feb 26, 2024

View reviewed changes

Add 351 snapshot shim for the scala2.13 version of plugin jar

8a6539b

Signed-off-by: Tim Liu <timl@nvidia.com>

NvTimLiu dismissed stale reviews from NVnavkumar and jlowe via 8a6539b February 26, 2024 08:28

Remove 351 snapshot shim as spark 3.5.1 has been released

6898bdf

Signed-off-by: Tim Liu <timl@nvidia.com>

NVnavkumar reviewed Feb 27, 2024

View reviewed changes

Remove scala2.13 351 snapshot shim

913d9a7

Signed-off-by: Tim Liu <timl@nvidia.com>

NVnavkumar previously approved these changes Feb 27, 2024

View reviewed changes

Remove 351 shim's jason string

78d1f00

Ran `mvn generate-sources -Dshimplify=true -Dshimplify.move=true -Dshimplify.remove.shim=351` to remove 351 shim's jason string, and fix some unnecessary empty lines that were introduced Signed-off-by: Tim Liu <timl@nvidia.com>

NvTimLiu dismissed NVnavkumar’s stale review via 78d1f00 February 27, 2024 14:06

NvTimLiu requested a review from razajafri February 27, 2024 14:11

Update Copyright 2024

8482847

Auto copyright by below scripts ``` export SPARK_RAPIDS_AUTO_COPYRIGHTER=ON ./scripts/auto-copyrighter.sh $(git diff --name-only origin/branch-24.04..HEAD) ``` Signed-off-by: Tim Liu <timl@nvidia.com>

NvTimLiu added a commit to NvTimLiu/spark-rapids that referenced this pull request Feb 28, 2024

Move 351 shims into noSnapshot buildvers

d52b53f

Move 351 shims into noSnapshot buildvers as spark has release it. Follow up of NVIDIA#10465 (comment) Signed-off-by: Tim Liu <timl@nvidia.com>

NvTimLiu mentioned this pull request Feb 28, 2024

Move 351 shims into noSnapshot buildvers #10508

Merged

jlowe reviewed Feb 29, 2024

View reviewed changes

razajafri added 3 commits February 29, 2024 17:19

Revert "Remove 351 shim's jason string"

62eda3f

This reverts commit 78d1f00.

skip 351 from strict checking

a3656da

Alien scala2.13/pom.xml to scala2.12 one

53ecc05

Run the script `bash build/make-scala-version-build-files.sh 2.13` Signed-off-by: Tim Liu <timl@nvidia.com>

gerashegalov and others added 4 commits February 29, 2024 19:12

pretend 351 is a snapshot in 24.02

7961b7f

Signed-off-by: Gera Shegalov <gera@apache.org>

Merge remote-tracking branch 'origin/branch-24.02' into pr/NvTimLiu/1…

328fe34

…0465-1

pretend 351 is a SNAPSHOT version

9830124

Revert change of build/shimplify.py

40f843b

Signed-off-by: Tim Liu <timl@nvidia.com>

gerashegalov approved these changes Mar 1, 2024

View reviewed changes

GaryShen2008 approved these changes Mar 1, 2024

View reviewed changes

razajafri approved these changes Mar 1, 2024

View reviewed changes

NvTimLiu merged commit c4abf14 into NVIDIA:branch-24.02 Mar 1, 2024
40 of 41 checks passed

gerashegalov mentioned this pull request Jun 13, 2024

[BUG] Shims inadvertently missed for various artifacts #11061

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add missed shims for scala2.13 #10465

Add missed shims for scala2.13 #10465

NvTimLiu commented Feb 22, 2024 •

edited by gerashegalov

Loading

NvTimLiu commented Feb 22, 2024

gerashegalov commented Feb 22, 2024

NVnavkumar Feb 23, 2024

sameerz Feb 23, 2024

GaryShen2008 Feb 23, 2024

NvTimLiu Feb 26, 2024 •

edited

Loading

NvTimLiu Feb 26, 2024

NvTimLiu commented Feb 26, 2024 •

edited

Loading

NvTimLiu commented Feb 26, 2024

sameerz commented Feb 26, 2024

NVnavkumar Feb 27, 2024

NvTimLiu Feb 27, 2024

sameerz commented Feb 27, 2024

NvTimLiu commented Feb 27, 2024 •

edited

Loading

NvTimLiu commented Feb 27, 2024

jlowe Feb 29, 2024

NVnavkumar Feb 29, 2024

jlowe Feb 29, 2024

jlowe Feb 29, 2024

NvTimLiu Mar 1, 2024

NvTimLiu commented Feb 29, 2024

razajafri commented Mar 1, 2024

sameerz commented Mar 1, 2024

NvTimLiu commented Mar 1, 2024

gerashegalov left a comment

NvTimLiu commented Mar 1, 2024

GaryShen2008 left a comment

razajafri left a comment

NvTimLiu commented Mar 1, 2024 •

edited

Loading

NvTimLiu commented Mar 1, 2024

Add missed shims for scala2.13 #10465

Add missed shims for scala2.13 #10465

Conversation

NvTimLiu commented Feb 22, 2024 • edited by gerashegalov Loading

NvTimLiu commented Feb 22, 2024

gerashegalov commented Feb 22, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NvTimLiu Feb 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NvTimLiu commented Feb 26, 2024 • edited Loading

NvTimLiu commented Feb 26, 2024

sameerz commented Feb 26, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sameerz commented Feb 27, 2024

NvTimLiu commented Feb 27, 2024 • edited Loading

NvTimLiu commented Feb 27, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NvTimLiu commented Feb 29, 2024

razajafri commented Mar 1, 2024

sameerz commented Mar 1, 2024

NvTimLiu commented Mar 1, 2024

gerashegalov left a comment

Choose a reason for hiding this comment

NvTimLiu commented Mar 1, 2024

GaryShen2008 left a comment

Choose a reason for hiding this comment

razajafri left a comment

Choose a reason for hiding this comment

NvTimLiu commented Mar 1, 2024 • edited Loading

NvTimLiu commented Mar 1, 2024

NvTimLiu commented Feb 22, 2024 •

edited by gerashegalov

Loading

NvTimLiu Feb 26, 2024 •

edited

Loading

NvTimLiu commented Feb 26, 2024 •

edited

Loading

NvTimLiu commented Feb 27, 2024 •

edited

Loading

NvTimLiu commented Mar 1, 2024 •

edited

Loading