Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Dependencies missing of spark-rapids v21.12.0 release jars #4253

Closed
NvTimLiu opened this issue Dec 1, 2021 · 3 comments · Fixed by #4265
Closed

[BUG] Dependencies missing of spark-rapids v21.12.0 release jars #4253

NvTimLiu opened this issue Dec 1, 2021 · 3 comments · Fixed by #4265
Assignees
@gerashegalov
Labels
bug Something isn't working
Milestone
Nov 30 - Dec 10

Comments

@NvTimLiu
Copy link
Collaborator

NvTimLiu commented Dec 1, 2021 •
edited
Loading

Describe the bug
We use “mvn dependency:get” command to download the rapids-4-spark jar from the OSS staging repo for the release test spark-tests.sh#L33-L34.
What we want is to download the rapids-4-spark jar only, maven tries to get its dependency jars as blow, which are not available, because we will NOT release these dependencies.

We used denpendency-reduce before in dist/pom.xml to void some these denpendencies, but not on branch-21.12, we remove the denpendency-reduce from dist/pom.xml

Related issue on branch-21.10: #3929 #3927

mvn dependency:get -B -Ddest=snap -DremoteRepositories=https://oss.sonatype.org/content/repositories/staging -DgroupId=com.nvidia -DartifactId=rapids-4-spark_2.12 -Dversion=21.12.0 -s spark-rapids/jenkins/settings.xml -Dpackaging=jar

22:57:31  [INFO] ------------------------------------------------------------------------
22:57:31  [INFO] Reactor Summary for RAPIDS Accelerator for Apache Spark Root Project 21.12.0-SNAPSHOT:
22:57:31  [INFO] 
22:57:31  [INFO] RAPIDS Accelerator for Apache Spark Root Project ... FAILURE [ 14.257 s]
22:57:31  [INFO] RAPIDS Accelerator for Apache Spark SQL Plugin ..... SKIPPED
22:57:31  [INFO] RAPIDS Accelerator for Apache Spark Shuffle Plugin . SKIPPED
22:57:31  [INFO] RAPIDS Accelerator for Apache Spark SQL Plugin Shims SKIPPED
22:57:31  [INFO] RAPIDS Accelerator for Apache Spark SQL Plugin Spark 3.0.1 Shim SKIPPED
22:57:31  [INFO] RAPIDS Accelerator for Apache Spark Scala UDF Plugin SKIPPED
22:57:31  [INFO] RAPIDS Accelerator for Apache Spark Aggregator ..... SKIPPED
22:57:31  [INFO] RAPIDS Accelerator for Apache Spark Distribution ... SKIPPED
22:57:31  [INFO] RAPIDS Accelerator for Apache Spark UDF Examples ... SKIPPED
22:57:31  [INFO] RAPIDS Accelerator for Apache Spark Tests .......... SKIPPED
22:57:31  [INFO] rapids-4-spark-integration-tests_2.12 .............. SKIPPED
22:57:31  [INFO] rapids-4-spark-api-validation ...................... SKIPPED
22:57:31  [INFO] ------------------------------------------------------------------------
22:57:31  [INFO] BUILD FAILURE
22:57:31  [INFO] ------------------------------------------------------------------------
22:57:31  [INFO] Total time:  14.574 s
22:57:31  [INFO] Finished at: 2021-12-01T14:57:31Z
22:57:31  [INFO] ------------------------------------------------------------------------
22:57:31  [ERROR] Failed to execute goal org.apache.maven.plugins:maven-dependency-plugin:2.8:get (default-cli) on project rapids-4-spark-parent: Couldn't download artifact: Missing:
22:57:31  [ERROR] ----------
22:57:31  [ERROR] 1) com.nvidia:rapids-4-spark-shuffle_2.12:jar:spark301:21.12.0
22:57:31  [ERROR] 
22:57:31  [ERROR]   Try downloading the file manually from the project website.
22:57:31  [ERROR] 
22:57:31  [ERROR]   Then, install it using the command: 
22:57:31  [ERROR]       mvn install:install-file -DgroupId=com.nvidia -DartifactId=rapids-4-spark-shuffle_2.12 -Dversion=21.12.0 -Dclassifier=spark301 -Dpackaging=jar -Dfile=/path/to/file
22:57:31  [ERROR] 
22:57:31  [ERROR]   Alternatively, if you host your own repository you can deploy the file there: 
22:57:31  [ERROR]       mvn deploy:deploy-file -DgroupId=com.nvidia -DartifactId=rapids-4-spark-shuffle_2.12 -Dversion=21.12.0 -Dclassifier=spark301 -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
22:57:31  [ERROR] 
22:57:31  [ERROR]   Path to dependency: 
22:57:31  [ERROR]   	1) org.apache.maven.plugins:maven-downloader-plugin:jar:1.0
22:57:31  [ERROR]   	2) com.nvidia:rapids-4-spark_2.12:jar:21.12.0
22:57:31  [ERROR]   	3) com.nvidia:rapids-4-spark-aggregator_2.12:jar:spark301:21.12.0
22:57:31  [ERROR]   	4) com.nvidia:rapids-4-spark-shuffle_2.12:jar:spark301:21.12.0
22:57:31  [ERROR] 
22:57:31  [ERROR] 2) com.nvidia:rapids-4-spark-sql_2.12:jar:spark301:21.12.0
22:57:31  [ERROR] 
22:57:31  [ERROR]   Try downloading the file manually from the project website.
22:57:31  [ERROR] 
22:57:31  [ERROR]   Then, install it using the command: 
22:57:31  [ERROR]       mvn install:install-file -DgroupId=com.nvidia -DartifactId=rapids-4-spark-sql_2.12 -Dversion=21.12.0 -Dclassifier=spark301 -Dpackaging=jar -Dfile=/path/to/file
22:57:31  [ERROR] 
22:57:31  [ERROR]   Alternatively, if you host your own repository you can deploy the file there: 
22:57:31  [ERROR]       mvn deploy:deploy-file -DgroupId=com.nvidia -DartifactId=rapids-4-spark-sql_2.12 -Dversion=21.12.0 -Dclassifier=spark301 -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
22:57:31  [ERROR] 
22:57:31  [ERROR]   Path to dependency: 
22:57:31  [ERROR]   	1) org.apache.maven.plugins:maven-downloader-plugin:jar:1.0
22:57:31  [ERROR]   	2) com.nvidia:rapids-4-spark_2.12:jar:21.12.0
22:57:31  [ERROR]   	3) com.nvidia:rapids-4-spark-aggregator_2.12:jar:spark301:21.12.0
22:57:31  [ERROR]   	4) com.nvidia:rapids-4-spark-sql_2.12:jar:spark301:21.12.0
22:57:31  [ERROR] 
22:57:31  [ERROR] 3) com.nvidia:rapids-4-spark-udf_2.12:jar:spark301:21.12.0
22:57:31  [ERROR] 
22:57:31  [ERROR]   Try downloading the file manually from the project website.
22:57:31  [ERROR] 
22:57:31  [ERROR]   Then, install it using the command: 
22:57:31  [ERROR]       mvn install:install-file -DgroupId=com.nvidia -DartifactId=rapids-4-spark-udf_2.12 -Dversion=21.12.0 -Dclassifier=spark301 -Dpackaging=jar -Dfile=/path/to/file
22:57:31  [ERROR] 
22:57:31  [ERROR]   Alternatively, if you host your own repository you can deploy the file there: 
22:57:31  [ERROR]       mvn deploy:deploy-file -DgroupId=com.nvidia -DartifactId=rapids-4-spark-udf_2.12 -Dversion=21.12.0 -Dclassifier=spark301 -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
22:57:31  [ERROR] 
22:57:31  [ERROR]   Path to dependency: 
22:57:31  [ERROR]   	1) org.apache.maven.plugins:maven-downloader-plugin:jar:1.0
22:57:31  [ERROR]   	2) com.nvidia:rapids-4-spark_2.12:jar:21.12.0
22:57:31  [ERROR]   	3) com.nvidia:rapids-4-spark-aggregator_2.12:jar:spark301:21.12.0
22:57:31  [ERROR]   	4) com.nvidia:rapids-4-spark-udf_2.12:jar:spark301:21.12.0
22:57:31  [ERROR] 
22:57:31  [ERROR] 4) com.nvidia:rapids-4-spark-aggregator_2.12:jar:spark301:21.12.0
22:57:31  [ERROR] 
22:57:31  [ERROR]   Try downloading the file manually from the project website.
22:57:31  [ERROR] 
22:57:31  [ERROR]   Then, install it using the command: 
22:57:31  [ERROR]       mvn install:install-file -DgroupId=com.nvidia -DartifactId=rapids-4-spark-aggregator_2.12 -Dversion=21.12.0 -Dclassifier=spark301 -Dpackaging=jar -Dfile=/path/to/file
22:57:31  [ERROR] 
22:57:31  [ERROR]   Alternatively, if you host your own repository you can deploy the file there: 
22:57:31  [ERROR]       mvn deploy:deploy-file -DgroupId=com.nvidia -DartifactId=rapids-4-spark-aggregator_2.12 -Dversion=21.12.0 -Dclassifier=spark301 -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
22:57:31  [ERROR] 
22:57:31  [ERROR]   Path to dependency: 
22:57:31  [ERROR]   	1) org.apache.maven.plugins:maven-downloader-plugin:jar:1.0
22:57:31  [ERROR]   	2) com.nvidia:rapids-4-spark_2.12:jar:21.12.0
22:57:31  [ERROR]   	3) com.nvidia:rapids-4-spark-aggregator_2.12:jar:spark301:21.12.0
22:57:31  [ERROR] 
22:57:31  [ERROR] 5) com.nvidia:rapids-4-spark-shims-spark301_2.12:jar:21.12.0
22:57:31  [ERROR] 
22:57:31  [ERROR]   Try downloading the file manually from the project website.
22:57:31  [ERROR] 
22:57:31  [ERROR]   Then, install it using the command: 
22:57:31  [ERROR]       mvn install:install-file -DgroupId=com.nvidia -DartifactId=rapids-4-spark-shims-spark301_2.12 -Dversion=21.12.0 -Dpackaging=jar -Dfile=/path/to/file
22:57:31  [ERROR] 
22:57:31  [ERROR]   Alternatively, if you host your own repository you can deploy the file there: 
22:57:31  [ERROR]       mvn deploy:deploy-file -DgroupId=com.nvidia -DartifactId=rapids-4-spark-shims-spark301_2.12 -Dversion=21.12.0 -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
22:57:31  [ERROR] 
22:57:31  [ERROR]   Path to dependency: 
22:57:31  [ERROR]   	1) org.apache.maven.plugins:maven-downloader-plugin:jar:1.0
22:57:31  [ERROR]   	2) com.nvidia:rapids-4-spark_2.12:jar:21.12.0
22:57:31  [ERROR]   	3) com.nvidia:rapids-4-spark-aggregator_2.12:jar:spark301:21.12.0
22:57:31  [ERROR]   	4) com.nvidia:rapids-4-spark-shims-spark301_2.12:jar:21.12.0
22:57:31  [ERROR] 
22:57:31  [ERROR] ----------
22:57:31  [ERROR] 5 required artifacts are missing.
22:57:31  [ERROR] 
22:57:31  [ERROR] for artifact: 
22:57:31  [ERROR]   org.apache.maven.plugins:maven-downloader-plugin:jar:1.0
22:57:31  [ERROR] 
22:57:31  [ERROR] from the specified remote repositories:
22:57:31  [ERROR]   apache-snapshots-repo (https://urm.nvidia.com/artifactory/sw-spark-maven, releases=true, snapshots=true),
22:57:31  [ERROR]   apache.snapshots.https (https://urm.nvidia.com/artifactory/sw-spark-maven, releases=true, snapshots=true),
22:57:31  [ERROR]   apache.snapshots (https://urm.nvidia.com/artifactory/sw-spark-maven, releases=true, snapshots=true),
22:57:31  [ERROR]   central (https://urm.nvidia.com/artifactory/sw-spark-maven, releases=true, snapshots=false),
22:57:31  [ERROR]   snapshots (https://urm.nvidia.com/artifactory/sw-spark-maven, releases=true, snapshots=true),
22:57:31  [ERROR]   snapshots-repo (https://oss.sonatype.org/content/repositories/snapshots, releases=false, snapshots=true),
22:57:31  [ERROR]   temp (https://oss.sonatype.org/content/repositories/staging, releases=true, snapshots=true)
22:57:31  [ERROR] -> [Help 1]
22:57:31  [ERROR] 
22:57:31  [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
22:57:31  [ERROR] Re-run Maven using the -X switch to enable full debug logging.
22:57:31  [ERROR] 
22:57:31  [ERROR] For more information about the errors and possible solutions, please read the following articles:
22:57:31  [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
@NvTimLiu NvTimLiu added bug Something isn't working ? - Needs Triage Need team to review and classify labels Dec 1, 2021
@NvTimLiu NvTimLiu changed the title [BUG] Dependency missing of spark-rapids v21.12.0 release jars [BUG] Dependencies missing of spark-rapids v21.12.0 release jars Dec 1, 2021
@NvTimLiu
Copy link
Collaborator Author

NvTimLiu commented Dec 1, 2021

@gerashegalov Could you please help to check? thanks!

gerashegalov added a commit to gerashegalov/spark-rapids that referenced this issue Dec 2, 2021
@gerashegalov
Use intransitive dependency:get for artifacts
5d03755 
Fixes NVIDIA#4253

Signed-off-by: Gera Shegalov <gera@apache.org>
@gerashegalov gerashegalov mentioned this issue Dec 2, 2021
Merged
@gerashegalov
Copy link
Collaborator

@NvTimLiu I added -Dtransitive=false in #4265 . I don't think we should change anything about dependency graph just because the get mojo in the dependency plugin tries to pull even provided dependencies.

@gerashegalov gerashegalov removed the ? - Needs Triage Need team to review and classify label Dec 2, 2021
@gerashegalov gerashegalov added this to the Nov 30 - Dec 10 milestone Dec 2, 2021
@jlowe
Copy link
Member

jlowe commented Dec 2, 2021

I don't think we should change anything about dependency graph just because the get mojo in the dependency plugin tries to pull even provided dependencies.

IMO it is still weird to have the public, published pom list artifacts that are completely missing, even if they're marked provided. It's not even correct in this case, as the pom says it depends on the aggregator jar when it in fact contains it instead.

NvTimLiu pushed a commit to NvTimLiu/spark-rapids that referenced this issue Dec 3, 2021
@gerashegalov @NvTimLiu
Use intransitive dependency:get for artifacts
fa09265 
Fixes NVIDIA#4253

Signed-off-by: Gera Shegalov <gera@apache.org>
@nvauto nvauto closed this as completed in 3e27acc Dec 6, 2021
@res-life res-life mentioned this issue Jan 26, 2022
Merged
@res-life res-life mentioned this issue Feb 15, 2022
Merged
@gerashegalov gerashegalov mentioned this issue Jan 8, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gerashegalov gerashegalov

Labels
bug Something isn't working
Projects
None yet
Milestone
Nov 30 - Dec 10
Development

Successfully merging a pull request may close this issue.

Remove aggregator dependency before deploying dist artifact gerashegalov/spark-rapids
3 participants
@jlowe @gerashegalov @NvTimLiu