Change Databricks image from 8.2 to 9.1 [skip ci] #4049
Conversation
NvTimLiu
requested review from
jlowe,
gerashegalov,
revans2,
tgravescs,
pxLi and
GaryShen2008
NvTimLiu
changed the title
|
build |
There was a problem hiding this comment.
since this change includes the jenkinsfile change, so it would possibly failed the test and we may need to skip ci here, can you test against https://github.com/pxLi/spark-rapids/ repo?
| @@ -390,7 +390,7 @@ pipeline { | |||
| } | |||
| } // end of DB runtime 7.3 | |||
|
|
|||
| stage('DB runtime 8.2') { | |||
| stage('DB runtime 9.1') { | |||
There was a problem hiding this comment.
please also update stage expression,
db_build && major_ver >= 21 && minor_ver >= 6
==>
db_build && major_ver >= 21 && minor_ver >= 12
as we added spark312db shim later for 21.12 or later
NvTimLiu
changed the title
pxLi
changed the title
NvTimLiu
changed the title
NvTimLiu
changed the title
|
build |
|
@tgravescs @jlowe @revans2 Could you please help to review? Thanks! Please keep this PR |
Converting this to a draft PR so it cannot be accidentally merged. |
|
@NvTimLiu has this been tested? |
| ${base}/envs/mamba/bin/mamba install -y -c rapidsai -c rapidsai-nightly -c nvidia -c conda-forge -c defaults cudf=$CUDF_VER cudatoolkit=11.0 | ||
| conda env remove -n mamba | ||
| # Create and activate 'cudf-udf' conda env for cudf-udf tests | ||
| conda create -y -n cudf-udf && source activate && conda activate cudf-udf |
There was a problem hiding this comment.
We change to create a common 'cudf-udf' conda environment for cudf udf tests, because
1, Not like 7.3 & 8.2, Databricks 9.1 does not have the default databricks-ml-gpu conda ENV
2, In 7.3 & 8.2, the script init_cudf_udf.sh create the empty MAMBA conda environment, but install everything (using MAMBA)into the default databricks-ml-gpu ENV, this looks odd.
3, We create a common and clean cudf-udf ENV specified for the cudf udf test.
Yes, this has been tested against both 7.3 & 9.1 build/IT runtime. On 7.3, got everything PASS, On 9.1, FAILED on the issue: rapidsai/cudf#9622 Will verify after the issue get fixed. |
| [[ -z $SPARK_SHIM_VER ]] && export SPARK_SHIM_VER=spark${BASE_SPARK_VER//.}db | ||
|
|
||
| # Try to use "cudf-udf" conda environment for the python cudf-udf tests. | ||
| if [ -d "/databricks/conda/envs/cudf-udf" ]; then |
There was a problem hiding this comment.
Use "cudf-udf" conda environment (created by jenkins/databricks/init_cudf_udf.sh) for the python cudf-udf tests.
Databricks 8.2 has been deprecated, change to Databricks 9.1 LTS. Signed-off-by: Tim Liu <timl@nvidia.com>
Signed-off-by: Tim Liu <timl@nvidia.com>
Signed-off-by: Tim Liu <timl@nvidia.com>
|
DB 9.1 tests failed on #4069 |
|
build |
|
Going to create a dummy verification PR |
Databricks 8.2 has been deprecated, change to Databricks 9.1 LTS.
Signed-off-by: Tim Liu timl@nvidia.com