Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] nightly spark311cdh build failed #3844

Closed
pxLi opened this issue Oct 18, 2021 · 1 comment · Fixed by #3846
Closed

[BUG] nightly spark311cdh build failed #3844

pxLi opened this issue Oct 18, 2021 · 1 comment · Fixed by #3846
Assignees
Labels
bug Something isn't working build Related to CI / CD or cleanly building

Comments

@pxLi
Copy link
Collaborator

pxLi commented Oct 18, 2021

Describe the bug
could be related to #3523, build pass w/o that commit

nightly spark311cdh build failed Failure to find org.apache.spark:spark-hive_2.12:jar:3.1.1.3.1.7270.0-253 in https://urm.nvidia.com/artifactory/sw-spark-maven was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced

[2021-10-17T09:46:49.954Z] [INFO] Reactor Summary for RAPIDS Accelerator for Apache Spark Root Project 21.12.0-SNAPSHOT:
[2021-10-17T09:46:49.954Z] [INFO] 
[2021-10-17T09:46:49.954Z] [INFO] RAPIDS Accelerator for Apache Spark Root Project ... SUCCESS [ 36.269 s]
[2021-10-17T09:46:49.954Z] [INFO] RAPIDS Accelerator for Apache Spark SQL Plugin ..... SUCCESS [03:28 min]
[2021-10-17T09:46:49.954Z] [INFO] RAPIDS Accelerator for Apache Spark Shuffle Plugin . SUCCESS [ 16.594 s]
[2021-10-17T09:46:49.954Z] [INFO] RAPIDS Accelerator for Apache Spark SQL Plugin Shims SUCCESS [  1.531 s]
[2021-10-17T09:46:49.954Z] [INFO] RAPIDS Accelerator for Apache Spark SQL Plugin Spark 3.1.1 Cloudera Shim SUCCESS [ 11.889 s]
[2021-10-17T09:46:49.954Z] [INFO] RAPIDS Accelerator for Apache Spark Scala UDF Plugin SUCCESS [ 56.070 s]
[2021-10-17T09:46:49.954Z] [INFO] RAPIDS Accelerator for Apache Spark Aggregator ..... SUCCESS [ 13.407 s]
[2021-10-17T09:46:49.954Z] [INFO] RAPIDS Accelerator for Apache Spark Distribution ... FAILURE [ 16.410 s]
[2021-10-17T09:46:49.954Z] [INFO] RAPIDS Accelerator for Apache Spark UDF Examples ... SKIPPED
[2021-10-17T09:46:49.954Z] [INFO] RAPIDS Accelerator for Apache Spark Tests .......... SKIPPED
[2021-10-17T09:46:49.954Z] [INFO] rapids-4-spark-integration-tests_2.12 .............. SKIPPED
[2021-10-17T09:46:49.954Z] [INFO] RAPIDS Accelerator for Apache Spark Tests For 3.1.X+ SKIPPED
[2021-10-17T09:46:49.954Z] [INFO] ------------------------------------------------------------------------
[2021-10-17T09:46:49.954Z] [INFO] BUILD FAILURE
[2021-10-17T09:46:49.954Z] [INFO] ------------------------------------------------------------------------
[2021-10-17T09:46:49.954Z] [INFO] Total time:  06:00 min
[2021-10-17T09:46:49.955Z] [INFO] Finished at: 2021-10-17T09:46:49Z
[2021-10-17T09:46:49.955Z] [INFO] ------------------------------------------------------------------------
[2021-10-17T09:46:49.955Z] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.8:run (generate-build-info) on project rapids-4-spark_2.12: Execution generate-build-info of goal org.apache.maven.plugins:maven-antrun-plugin:1.8:run failed: Plugin org.apache.maven.plugins:maven-antrun-plugin:1.8 or one of its dependencies could not be resolved: The following artifacts could not be resolved: org.apache.spark:spark-hive_2.12:jar:3.1.1.3.1.7270.0-253, org.apache.spark:spark-core_2.12:jar:3.1.1.3.1.7270.0-253, org.apache.hadoop:hadoop-client:jar:3.1.1.7.2.7.0-184, org.apache.hadoop:hadoop-common:jar:3.1.1.7.2.7.0-184, org.apache.curator:curator-client:jar:4.3.0.7.2.7.0-184, org.apache.hadoop:hadoop-hdfs-client:jar:3.1.1.7.2.7.0-184, org.apache.hadoop:hadoop-yarn-client:jar:3.1.1.7.2.7.0-184, org.apache.hadoop:hadoop-mapreduce-client-core:jar:3.1.1.7.2.7.0-184, org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:3.1.1.7.2.7.0-184, org.apache.hadoop:hadoop-mapreduce-client-common:jar:3.1.1.7.2.7.0-184, org.apache.hadoop:hadoop-annotations:jar:3.1.1.7.2.7.0-184, org.apache.spark:spark-launcher_2.12:jar:3.1.1.3.1.7270.0-253, org.apache.spark:spark-kvstore_2.12:jar:3.1.1.3.1.7270.0-253, org.apache.spark:spark-network-common_2.12:jar:3.1.1.3.1.7270.0-253, org.apache.spark:spark-network-shuffle_2.12:jar:3.1.1.3.1.7270.0-253, org.apache.spark:spark-unsafe_2.12:jar:3.1.1.3.1.7270.0-253, org.apache.curator:curator-recipes:jar:4.3.0.7.2.7.0-SNAPSHOT, org.apache.zookeeper:zookeeper:jar:3.5.5.7.2.7.0-184, org.apache.zookeeper:zookeeper-jute:jar:3.5.5.7.2.7.0-184, log4j:log4j:jar:1.2.17-cloudera1, org.apache.spark:spark-tags_2.12:jar:3.1.1.3.1.7270.0-253, org.apache.spark:spark-sql_2.12:jar:3.1.1.3.1.7270.0-253, org.apache.spark:spark-sketch_2.12:jar:3.1.1.3.1.7270.0-253, org.apache.spark:spark-catalyst_2.12:jar:3.1.1.3.1.7270.0-253, org.apache.parquet:parquet-column:jar:1.10.99.7.2.7.0-184, org.apache.parquet:parquet-common:jar:1.10.99.7.2.7.0-184, org.apache.parquet:parquet-encoding:jar:1.10.99.7.2.7.0-184, org.apache.parquet:parquet-hadoop:jar:1.10.99.7.2.7.0-184, org.apache.parquet:parquet-format-structures:jar:1.10.99.7.2.7.0-184, org.apache.parquet:parquet-jackson:jar:1.10.99.7.2.7.0-184, org.apache.hive:hive-common:jar:3.1.3000.7.2.7.0-184, org.apache.hive:hive-classification:jar:3.1.3000.7.2.7.0-184, org.apache.hive:hive-standalone-metastore:jar:3.1.3000.7.2.7.0-184, org.apache.curator:curator-framework:jar:4.3.0.7.2.7.0-184, org.apache.hive:hive-exec:jar:3.1.3000.7.2.7.0-184, org.apache.hive:hive-vector-code-gen:jar:3.1.3000.7.2.7.0-184, org.cloudera.logredactor:logredactor:jar:2.0.8, org.apache.hadoop:hadoop-yarn-registry:jar:3.1.1.7.2.7.0-184, org.apache.hadoop:hadoop-auth:jar:3.1.1.7.2.7.0-184, org.apache.hadoop:hadoop-yarn-common:jar:3.1.1.7.2.7.0-184, org.apache.hadoop:hadoop-yarn-api:jar:3.1.1.7.2.7.0-184, org.apache.kafka:kafka-clients:jar:2.5.0.7.2.7.0-184, org.apache.hive:hive-metastore:jar:3.1.3000.7.2.7.0-184, org.apache.hive:hive-serde:jar:3.1.3000.7.2.7.0-184, org.apache.arrow:arrow-vector:jar:0.8.0.7.2.7.0-184, org.apache.arrow:arrow-format:jar:0.8.0.7.2.7.0-184, org.apache.arrow:arrow-memory:jar:0.8.0.7.2.7.0-184, org.apache.hive:hive-shims:jar:3.1.3000.7.2.7.0-184, org.apache.hive.shims:hive-shims-common:jar:3.1.3000.7.2.7.0-184, org.apache.hive.shims:hive-shims-0.20:jar:3.1.3000.7.2.7.0-184, org.apache.hive.shims:hive-shims-0.23:jar:3.1.3000.7.2.7.0-184, org.apache.hive.shims:hive-shims-scheduler:jar:3.1.3000.7.2.7.0-184, org.apache.hive:hive-llap-common:jar:3.1.3000.7.2.7.0-184, org.apache.hive:hive-llap-client:jar:3.1.3000.7.2.7.0-184, org.apache.avro:avro:jar:1.8.2.7.2.7.0-184, org.apache.avro:avro-mapred:jar:hadoop2:1.8.2.7.2.7.0-184, org.apache.avro:avro-ipc:jar:1.8.2.7.2.7.0-184, org.apache.calcite:calcite-core:jar:1.21.0.7.2.7.0-184, org.apache.calcite.avatica:avatica-core:jar:1.16.0.7.2.7.0-184, org.apache.calcite.avatica:avatica-metrics:jar:1.16.0.7.2.7.0-184, org.apache.calcite:calcite-linq4j:jar:1.21.0.7.2.7.0-184, org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13-cloudera.1: Failure to find org.apache.spark:spark-hive_2.12:jar:3.1.1.3.1.7270.0-253 in https://urm.nvidia.com/artifactory/sw-spark-maven was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced -> [Help 1]

To reproduce locally,
mvn -U -B -Dbuildver=311cdh clean install -Dcuda.version=cuda11

@pxLi pxLi added bug Something isn't working build Related to CI / CD or cleanly building labels Oct 18, 2021
@gerashegalov
Copy link
Collaborator

gerashegalov commented Oct 18, 2021

git bisect identifies #3523 as @pxLi suggested. However, I have no clue in which way. The offending PR only changes the build output destination.

@tgravescs points out offline that #3523 switches the plugin for running RapidsConf.

This was done because scala-maven-plugin does not honor a non-default project.build.directory. Previously this was working by reusing the transitive build path of the aggregator. For ant-run we had to add spark-hive as a dependency.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working build Related to CI / CD or cleanly building
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants