Get cudf/spark dependency from the correct .m2 dir #1062

NvTimLiu · 2020-11-04T10:37:00Z

'WORKSPACE' & 'M2DIR' vars are needed for shims to gen the correct cudf/spark dependency info in shims.

Below error in 'spark*-info.properties' is due to unset of 'WORKSPACE' & 'M2DIR':
build/dependency-info.sh: line 30: /jenkins/printJarVersion.sh: No such file or directory
build/dependency-info.sh: line 33: /jenkins/printJarVersion.sh: No such file or directory

To fix the error, we set the default values for them in 'build/dependency-info.sh':
'M2DIR=$HOME/.m2/repository'
'WORKSPACE=../..'

We also need to explicitly set the correct 'M2DIR' path, in case we change it by '-Dmaven.repo.local=$M2DIR'.
Already updated Jenkins scripts to set the correct 'M2DIR'.

Signed-off-by: Tim Liu timl@nvidia.com

'WORKSPACE' & 'M2DIR' vars are needed for shims to gen the correct cudf/spark dependency info in shims. Below error in 'spark*-info.properties' is due to unset of 'WORKSPACE' & 'M2DIR': build/dependency-info.sh: line 30: /jenkins/printJarVersion.sh: No such file or directory build/dependency-info.sh: line 33: /jenkins/printJarVersion.sh: No such file or directory To fix the error, we set the default values for them in 'build/dependency-info.sh': 'M2DIR=$HOME/.m2/repository' 'WORKSPACE=../..' We also need to explicitly set the correct 'M2DIR' path, in case we change it by '-Dmaven.repo.local=$M2DIR'. Already updated Jenkins scripts to set the correct 'M2DIR'. Signed-off-by: Tim Liu <timl@nvidia.com>

NvTimLiu · 2020-11-04T10:37:21Z

build

jlowe · 2020-11-04T14:10:34Z

The changes look good, but it doesn't completely fix the issue. If the script fails to run successfully for some reason then it will still not fail the build. I would like to see changes related to error handling in addition to fixing the paths when run outside of Jenkins. The improved error handling doesn't have to be part of this PR, but we shouldn't close #1031 until that is addressed.

NvTimLiu · 2020-11-05T09:07:55Z

@jlowe @revans2, I agree if the script fails again later, then we'll stop the build.

I tried to add some scripts to check how to let mvn package fails in case the script 'build/dependency-info.sh' runs failure.

As the dependency-info.sh is script is running in the child process forked by mvn package, but the mvn process does not check the forked process's status, so even the child process fails, mvn process still run builds.

I did not find a good way to let mvn process automatically stop when 'dependency-info.sh' fails.

Have you any suggestions?

mvn process: 21245 18897 java -Dmaven.home=/home/timl/anaconda3/opt/maven
dependency-info.sh: 21304 21245 bash build/dependency-info.sh 0.17-SNAPSHOT cuda10-1 3.0.0

jlowe · 2020-11-05T14:37:49Z

I did not find a good way to let mvn process automatically stop when 'dependency-info.sh' fails.

The subprocess is launched by the Maven antrun plugin which in turn is using Ant to perform the tasks. In this case it's the Ant exec task which is documented here. There is a failonerror attribute that defaults to false. Specifying that attribute as true will cause the build to fail if the program exits with an error code. This should be done for every place in the build using the Ant exec task to ensure the build fails if a program we're trying to launch as part of the build fails.

That's the first step. The next step is to ensure the script will fail if anything launched by the script fails. That can be done by placing a set -e or set -o errexit command as the first command in the script. Alternatively you can add a -e argument to every bash command line in an antrun exec directive.

NvTimLiu · 2020-11-05T16:00:34Z

@jlowe I tried the way you suggested, the failonerror attribute works for me.
Thanks very much for your help. I'll update it on the PR.

…uns failure

jlowe · 2020-11-05T16:46:01Z

build/dependency-info.sh

@@ -23,6 +23,7 @@
 #   SPARK_VER - The version of spark

 # Parse cudf and spark dependency versions
+set -e


The same needs to happen in build/build-info and the corresponding failonerror pom change is needed in the top-level pom's use of the antrun plugin.

The same needs to happen in build/build-info and the corresponding failonerror pom change is needed in the top-level pom's use of the antrun plugin.

@jlowe Thanks very much for reminding me, updated.

Needs 2020 copyrights on build/build-info

Signed-off-by: Tim Liu <timl@nvidia.com>

NvTimLiu · 2020-11-06T00:52:00Z

build

NvTimLiu · 2020-11-06T15:58:34Z

build

jlowe · 2020-11-06T17:14:36Z

build/build-info

@@ -1,7 +1,7 @@
 #!/usr/bin/env bash

 #
-# Copyright (c) 2019, NVIDIA CORPORATION. All rights reserved.
+# Copyright (c) 2020, NVIDIA CORPORATION. All rights reserved.


This needs to be 2019-2020. We don't give up copyright for older years.

This needs to be 2019-2020. We don't give up copyright for older years.

Ok

jlowe · 2020-11-06T17:17:11Z

build

Signed-off-by: Tim Liu <timl@nvidia.com>

NvTimLiu · 2020-11-06T18:40:24Z

build

jlowe · 2020-11-06T18:57:23Z

build

This reverts commit 0afbca6.

This reverts commit 0afbca6. Signed-off-by: Jason Lowe <jlowe@nvidia.com>

…1083) This reverts commit 0afbca6. Signed-off-by: Jason Lowe <jlowe@nvidia.com>

* Get cudf/spark dependency from the correct .m2 dir 'WORKSPACE' & 'M2DIR' vars are needed for shims to gen the correct cudf/spark dependency info in shims. Below error in 'spark*-info.properties' is due to unset of 'WORKSPACE' & 'M2DIR': build/dependency-info.sh: line 30: /jenkins/printJarVersion.sh: No such file or directory build/dependency-info.sh: line 33: /jenkins/printJarVersion.sh: No such file or directory To fix the error, we set the default values for them in 'build/dependency-info.sh': 'M2DIR=$HOME/.m2/repository' 'WORKSPACE=../..' We also need to explicitly set the correct 'M2DIR' path, in case we change it by '-Dmaven.repo.local=$M2DIR'. Already updated Jenkins scripts to set the correct 'M2DIR'. Signed-off-by: Tim Liu <timl@nvidia.com> * let mvn package fails in case the script 'build/dependency-info.sh' runs failure * Stop mvn build if `build/build-info` fails Signed-off-by: Tim Liu <timl@nvidia.com> * Copyright 2020 Signed-off-by: Tim Liu <timl@nvidia.com>

* Get cudf/spark dependency from the correct .m2 dir (#1062) * Get cudf/spark dependency from the correct .m2 dir 'WORKSPACE' & 'M2DIR' vars are needed for shims to gen the correct cudf/spark dependency info in shims. Below error in 'spark*-info.properties' is due to unset of 'WORKSPACE' & 'M2DIR': build/dependency-info.sh: line 30: /jenkins/printJarVersion.sh: No such file or directory build/dependency-info.sh: line 33: /jenkins/printJarVersion.sh: No such file or directory To fix the error, we set the default values for them in 'build/dependency-info.sh': 'M2DIR=$HOME/.m2/repository' 'WORKSPACE=../..' We also need to explicitly set the correct 'M2DIR' path, in case we change it by '-Dmaven.repo.local=$M2DIR'. Already updated Jenkins scripts to set the correct 'M2DIR'. Signed-off-by: Tim Liu <timl@nvidia.com> * let mvn package fails in case the script 'build/dependency-info.sh' runs failure * Stop mvn build if `build/build-info` fails Signed-off-by: Tim Liu <timl@nvidia.com> * Copyright 2020 Signed-off-by: Tim Liu <timl@nvidia.com> * List the latest SNAPSHOT jar file in local maven repo Signed-off-by: Tim Liu <timl@nvidia.com> * Get the path of 'dependency-info.sh', then set 'WORKSPACE' relative to it Signed-off-by: Tim Liu <timl@nvidia.com> * Only collect dependency info on Jenkins build * Only collect timestamped dependency in Jenkins build Collect snapshot dependency info only in Jenkins build, In dev build, print 'SNAPSHOT' tag without time stamp, e.g.: cudf-0.17-SNAPSHOT.jar * simplifying the dependencylogic

* Get cudf/spark dependency from the correct .m2 dir 'WORKSPACE' & 'M2DIR' vars are needed for shims to gen the correct cudf/spark dependency info in shims. Below error in 'spark*-info.properties' is due to unset of 'WORKSPACE' & 'M2DIR': build/dependency-info.sh: line 30: /jenkins/printJarVersion.sh: No such file or directory build/dependency-info.sh: line 33: /jenkins/printJarVersion.sh: No such file or directory To fix the error, we set the default values for them in 'build/dependency-info.sh': 'M2DIR=$HOME/.m2/repository' 'WORKSPACE=../..' We also need to explicitly set the correct 'M2DIR' path, in case we change it by '-Dmaven.repo.local=$M2DIR'. Already updated Jenkins scripts to set the correct 'M2DIR'. Signed-off-by: Tim Liu <timl@nvidia.com> * let mvn package fails in case the script 'build/dependency-info.sh' runs failure * Stop mvn build if `build/build-info` fails Signed-off-by: Tim Liu <timl@nvidia.com> * Copyright 2020 Signed-off-by: Tim Liu <timl@nvidia.com>

…)" (NVIDIA#1083) This reverts commit 0afbca6. Signed-off-by: Jason Lowe <jlowe@nvidia.com>

* Get cudf/spark dependency from the correct .m2 dir (NVIDIA#1062) * Get cudf/spark dependency from the correct .m2 dir 'WORKSPACE' & 'M2DIR' vars are needed for shims to gen the correct cudf/spark dependency info in shims. Below error in 'spark*-info.properties' is due to unset of 'WORKSPACE' & 'M2DIR': build/dependency-info.sh: line 30: /jenkins/printJarVersion.sh: No such file or directory build/dependency-info.sh: line 33: /jenkins/printJarVersion.sh: No such file or directory To fix the error, we set the default values for them in 'build/dependency-info.sh': 'M2DIR=$HOME/.m2/repository' 'WORKSPACE=../..' We also need to explicitly set the correct 'M2DIR' path, in case we change it by '-Dmaven.repo.local=$M2DIR'. Already updated Jenkins scripts to set the correct 'M2DIR'. Signed-off-by: Tim Liu <timl@nvidia.com> * let mvn package fails in case the script 'build/dependency-info.sh' runs failure * Stop mvn build if `build/build-info` fails Signed-off-by: Tim Liu <timl@nvidia.com> * Copyright 2020 Signed-off-by: Tim Liu <timl@nvidia.com> * List the latest SNAPSHOT jar file in local maven repo Signed-off-by: Tim Liu <timl@nvidia.com> * Get the path of 'dependency-info.sh', then set 'WORKSPACE' relative to it Signed-off-by: Tim Liu <timl@nvidia.com> * Only collect dependency info on Jenkins build * Only collect timestamped dependency in Jenkins build Collect snapshot dependency info only in Jenkins build, In dev build, print 'SNAPSHOT' tag without time stamp, e.g.: cudf-0.17-SNAPSHOT.jar * simplifying the dependencylogic

* Get cudf/spark dependency from the correct .m2 dir 'WORKSPACE' & 'M2DIR' vars are needed for shims to gen the correct cudf/spark dependency info in shims. Below error in 'spark*-info.properties' is due to unset of 'WORKSPACE' & 'M2DIR': build/dependency-info.sh: line 30: /jenkins/printJarVersion.sh: No such file or directory build/dependency-info.sh: line 33: /jenkins/printJarVersion.sh: No such file or directory To fix the error, we set the default values for them in 'build/dependency-info.sh': 'M2DIR=$HOME/.m2/repository' 'WORKSPACE=../..' We also need to explicitly set the correct 'M2DIR' path, in case we change it by '-Dmaven.repo.local=$M2DIR'. Already updated Jenkins scripts to set the correct 'M2DIR'. Signed-off-by: Tim Liu <timl@nvidia.com> * let mvn package fails in case the script 'build/dependency-info.sh' runs failure * Stop mvn build if `build/build-info` fails Signed-off-by: Tim Liu <timl@nvidia.com> * Copyright 2020 Signed-off-by: Tim Liu <timl@nvidia.com>

…)" (NVIDIA#1083) This reverts commit 0afbca6. Signed-off-by: Jason Lowe <jlowe@nvidia.com>

* Get cudf/spark dependency from the correct .m2 dir (NVIDIA#1062) * Get cudf/spark dependency from the correct .m2 dir 'WORKSPACE' & 'M2DIR' vars are needed for shims to gen the correct cudf/spark dependency info in shims. Below error in 'spark*-info.properties' is due to unset of 'WORKSPACE' & 'M2DIR': build/dependency-info.sh: line 30: /jenkins/printJarVersion.sh: No such file or directory build/dependency-info.sh: line 33: /jenkins/printJarVersion.sh: No such file or directory To fix the error, we set the default values for them in 'build/dependency-info.sh': 'M2DIR=$HOME/.m2/repository' 'WORKSPACE=../..' We also need to explicitly set the correct 'M2DIR' path, in case we change it by '-Dmaven.repo.local=$M2DIR'. Already updated Jenkins scripts to set the correct 'M2DIR'. Signed-off-by: Tim Liu <timl@nvidia.com> * let mvn package fails in case the script 'build/dependency-info.sh' runs failure * Stop mvn build if `build/build-info` fails Signed-off-by: Tim Liu <timl@nvidia.com> * Copyright 2020 Signed-off-by: Tim Liu <timl@nvidia.com> * List the latest SNAPSHOT jar file in local maven repo Signed-off-by: Tim Liu <timl@nvidia.com> * Get the path of 'dependency-info.sh', then set 'WORKSPACE' relative to it Signed-off-by: Tim Liu <timl@nvidia.com> * Only collect dependency info on Jenkins build * Only collect timestamped dependency in Jenkins build Collect snapshot dependency info only in Jenkins build, In dev build, print 'SNAPSHOT' tag without time stamp, e.g.: cudf-0.17-SNAPSHOT.jar * simplifying the dependencylogic

* Get cudf/spark dependency from the correct .m2 dir 'WORKSPACE' & 'M2DIR' vars are needed for shims to gen the correct cudf/spark dependency info in shims. Below error in 'spark*-info.properties' is due to unset of 'WORKSPACE' & 'M2DIR': build/dependency-info.sh: line 30: /jenkins/printJarVersion.sh: No such file or directory build/dependency-info.sh: line 33: /jenkins/printJarVersion.sh: No such file or directory To fix the error, we set the default values for them in 'build/dependency-info.sh': 'M2DIR=$HOME/.m2/repository' 'WORKSPACE=../..' We also need to explicitly set the correct 'M2DIR' path, in case we change it by '-Dmaven.repo.local=$M2DIR'. Already updated Jenkins scripts to set the correct 'M2DIR'. Signed-off-by: Tim Liu <timl@nvidia.com> * let mvn package fails in case the script 'build/dependency-info.sh' runs failure * Stop mvn build if `build/build-info` fails Signed-off-by: Tim Liu <timl@nvidia.com> * Copyright 2020 Signed-off-by: Tim Liu <timl@nvidia.com>

…)" (NVIDIA#1083) This reverts commit 0afbca6. Signed-off-by: Jason Lowe <jlowe@nvidia.com>

* Get cudf/spark dependency from the correct .m2 dir (NVIDIA#1062) * Get cudf/spark dependency from the correct .m2 dir 'WORKSPACE' & 'M2DIR' vars are needed for shims to gen the correct cudf/spark dependency info in shims. Below error in 'spark*-info.properties' is due to unset of 'WORKSPACE' & 'M2DIR': build/dependency-info.sh: line 30: /jenkins/printJarVersion.sh: No such file or directory build/dependency-info.sh: line 33: /jenkins/printJarVersion.sh: No such file or directory To fix the error, we set the default values for them in 'build/dependency-info.sh': 'M2DIR=$HOME/.m2/repository' 'WORKSPACE=../..' We also need to explicitly set the correct 'M2DIR' path, in case we change it by '-Dmaven.repo.local=$M2DIR'. Already updated Jenkins scripts to set the correct 'M2DIR'. Signed-off-by: Tim Liu <timl@nvidia.com> * let mvn package fails in case the script 'build/dependency-info.sh' runs failure * Stop mvn build if `build/build-info` fails Signed-off-by: Tim Liu <timl@nvidia.com> * Copyright 2020 Signed-off-by: Tim Liu <timl@nvidia.com> * List the latest SNAPSHOT jar file in local maven repo Signed-off-by: Tim Liu <timl@nvidia.com> * Get the path of 'dependency-info.sh', then set 'WORKSPACE' relative to it Signed-off-by: Tim Liu <timl@nvidia.com> * Only collect dependency info on Jenkins build * Only collect timestamped dependency in Jenkins build Collect snapshot dependency info only in Jenkins build, In dev build, print 'SNAPSHOT' tag without time stamp, e.g.: cudf-0.17-SNAPSHOT.jar * simplifying the dependencylogic

NvTimLiu requested review from GaryShen2008, jlowe, revans2 and tgravescs as code owners November 4, 2020 10:37

revans2 previously approved these changes Nov 4, 2020

View reviewed changes

let mvn package fails in case the script 'build/dependency-info.sh' r…

84e7e94

…uns failure

NvTimLiu dismissed revans2’s stale review via 84e7e94 November 5, 2020 16:26

jlowe reviewed Nov 5, 2020

View reviewed changes

jlowe added the build Related to CI / CD or cleanly building label Nov 5, 2020

Stop mvn build if build/build-info fails

dfa7f0b

Signed-off-by: Tim Liu <timl@nvidia.com>

revans2 previously approved these changes Nov 6, 2020

View reviewed changes

jlowe mentioned this pull request Nov 6, 2020

[FEA] Support Databricks 7.3 LTS Runtime #1076

Merged

jlowe reviewed Nov 6, 2020

View reviewed changes

jlowe dismissed revans2’s stale review via 757724a November 6, 2020 17:16

NvTimLiu force-pushed the shims-dependency-info branch from 757724a to e143254 Compare November 6, 2020 18:40

jlowe approved these changes Nov 6, 2020

View reviewed changes

jlowe merged commit 0afbca6 into NVIDIA:branch-0.3 Nov 6, 2020

jlowe added a commit that referenced this pull request Nov 6, 2020

Revert "Get cudf/spark dependency from the correct .m2 dir (#1062)"

2de8700

This reverts commit 0afbca6.

jlowe mentioned this pull request Nov 6, 2020

Revert "Get cudf/spark dependency from the correct .m2 dir" #1083

Merged

jlowe added a commit that referenced this pull request Nov 6, 2020

Revert "Get cudf/spark dependency from the correct .m2 dir (#1062)"

0d75075

This reverts commit 0afbca6. Signed-off-by: Jason Lowe <jlowe@nvidia.com>

revans2 pushed a commit that referenced this pull request Nov 6, 2020

Revert "Get cudf/spark dependency from the correct .m2 dir (#1062)" (#…

f6fdfe4

…1083) This reverts commit 0afbca6. Signed-off-by: Jason Lowe <jlowe@nvidia.com>

sperlingxx pushed a commit to sperlingxx/spark-rapids that referenced this pull request Nov 20, 2020

Revert "Get cudf/spark dependency from the correct .m2 dir (NVIDIA#1062…

3d29b75

…)" (NVIDIA#1083) This reverts commit 0afbca6. Signed-off-by: Jason Lowe <jlowe@nvidia.com>

nartal1 pushed a commit to nartal1/spark-rapids that referenced this pull request Jun 9, 2021

Revert "Get cudf/spark dependency from the correct .m2 dir (NVIDIA#1062…

dc1a053

…)" (NVIDIA#1083) This reverts commit 0afbca6. Signed-off-by: Jason Lowe <jlowe@nvidia.com>

nartal1 pushed a commit to nartal1/spark-rapids that referenced this pull request Jun 9, 2021

Revert "Get cudf/spark dependency from the correct .m2 dir (NVIDIA#1062…

ed2d680

…)" (NVIDIA#1083) This reverts commit 0afbca6. Signed-off-by: Jason Lowe <jlowe@nvidia.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Get cudf/spark dependency from the correct .m2 dir #1062

Get cudf/spark dependency from the correct .m2 dir #1062

NvTimLiu commented Nov 4, 2020

NvTimLiu commented Nov 4, 2020

jlowe commented Nov 4, 2020

NvTimLiu commented Nov 5, 2020

jlowe commented Nov 5, 2020

NvTimLiu commented Nov 5, 2020

jlowe Nov 5, 2020

NvTimLiu Nov 6, 2020

jlowe Nov 6, 2020 •

edited

Loading

NvTimLiu commented Nov 6, 2020

NvTimLiu commented Nov 6, 2020

jlowe Nov 6, 2020

NvTimLiu Nov 6, 2020

jlowe commented Nov 6, 2020

NvTimLiu commented Nov 6, 2020

jlowe commented Nov 6, 2020

Get cudf/spark dependency from the correct .m2 dir #1062

Get cudf/spark dependency from the correct .m2 dir #1062

Conversation

NvTimLiu commented Nov 4, 2020

NvTimLiu commented Nov 4, 2020

jlowe commented Nov 4, 2020

NvTimLiu commented Nov 5, 2020

jlowe commented Nov 5, 2020

NvTimLiu commented Nov 5, 2020

jlowe Nov 5, 2020

Choose a reason for hiding this comment

NvTimLiu Nov 6, 2020

Choose a reason for hiding this comment

jlowe Nov 6, 2020 • edited Loading

Choose a reason for hiding this comment

NvTimLiu commented Nov 6, 2020

NvTimLiu commented Nov 6, 2020

jlowe Nov 6, 2020

Choose a reason for hiding this comment

NvTimLiu Nov 6, 2020

Choose a reason for hiding this comment

jlowe commented Nov 6, 2020

NvTimLiu commented Nov 6, 2020

jlowe commented Nov 6, 2020

jlowe Nov 6, 2020 •

edited

Loading