Fix gptl include and remove Kokkos_ROOT settings #2866

jgfouca · 2024-06-13T18:50:38Z

Confirmed that this fixes all build errors for eamxx CIME cases on pm-cpu.

ndkeen · 2024-06-13T18:54:00Z

Are we going to have differences in e3sm/scream repo? Or will this be downstreamed to e3sm?

bartgol · 2024-06-13T18:54:11Z

Jim, do you think we should fix this in the upstream? Or maybe just do a downstream merge (which we were planning anyways) when this is merged?

Edit: ha! this time Noel beat me!

jgfouca · 2024-06-13T18:55:26Z

I'm making the same change upstream, so it should be OK.

E3SM-Autotester · 2024-06-13T19:17:47Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5537
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE
PULLREQUESTNUM	2866
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`c65c84e`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`b7f853c`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5806
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE
PULLREQUESTNUM	2866
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`c65c84e`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`b7f853c`
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: jgfouca

E3SM-Autotester · 2024-06-13T20:01:45Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5537
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE
PULLREQUESTNUM	2866
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`c65c84e`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`b7f853c`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5806
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE
PULLREQUESTNUM	2866
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`c65c84e`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`b7f853c`
TEST_REPO_ALIAS	SCREAM

SCREAM_PullRequest_Autotester_Mappy # 5537 FAILED (click to see last 100 lines of console output)


[ 94%] Built target vertical_layer
[ 94%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_main_tests.cpp.o
[ 94%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pblintd_height_tests.cpp.o
[ 94%] Built target cld_fraction_standalone
[ 94%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_pdf_vv_parameters_tests.cpp.o
[ 94%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_pdf_thl_parameters_tests.cpp.o
[ 94%] Built target p3_standalone
[ 94%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_vd_shoc_decomp_and_solve_tests.cpp.o
[ 94%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pblintd_surf_temp_tests.cpp.o
[ 94%] Built target spa_standalone
[ 94%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pblintd_check_pblh_tests.cpp.o
[ 94%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_pdf_qw_parameters_tests.cpp.o
[ 94%] Linking CXX executable aerocom_cld
[ 94%] Linking CXX executable shortwave_cloud_forcing
[ 94%] Built target shoc_standalone
[ 94%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pblintd_tests.cpp.o
[ 94%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_pdf_inplume_corr_tests.cpp.o
[ 95%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_compute_shoc_temperature_tests.cpp.o
[ 95%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_pdf_computetemp_tests.cpp.o
[ 95%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_pdf_compute_qs_tests.cpp.o
[ 95%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_pdf_compute_s_tests.cpp.o
[ 95%] Linking CXX executable precip_surf_mass_flux
[ 96%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_pdf_compute_buoyflux_tests.cpp.o
[ 96%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_pdf_compute_liqflux_tests.cpp.o
[ 96%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_pdf_compute_cloudvar_tests.cpp.o
[ 96%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_pdf_compute_sgsliq_tests.cpp.o
[ 96%] Linking CXX executable ad_ut
[ 96%] Built target virtual_temperature
[ 96%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_diag_second_mom_ubycond_test.cpp.o
[ 96%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_pblintd_init_pot_test.cpp.o
[ 96%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_unit_tests.cpp.o
[ 97%] Linking CXX executable relative_humidity
[ 97%] Linking CXX executable aodvis
[ 97%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_diag_second_moments_lbycond_tests.cpp.o
[ 97%] Linking CXX executable vapor_flux
[ 98%] Linking CXX executable surface_upward_latent_heat_flux
[ 98%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_diag_second_moments_tests.cpp.o
[ 99%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_diag_second_shoc_moments_tests.cpp.o
[ 99%] Linking CXX executable longwave_cloud_forcing
[ 99%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_pblintd_cldcheck_tests.cpp.o
[ 99%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_compute_shoc_vapor_tests.cpp.o
[ 99%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_update_prognostics_implicit_tests.cpp.o
[ 99%] Built target atmosphere_density
[ 99%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_main_tests.cpp.o
[ 99%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_pblintd_height_tests.cpp.o
[ 99%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_vd_shoc_decomp_and_solve_tests.cpp.o
[ 99%] Built target surface_coupling
[ 99%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_pblintd_surf_temp_tests.cpp.o
[ 99%] Built target shoc_p3
[ 99%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_pblintd_check_pblh_tests.cpp.o
[ 99%] Linking CXX executable shoc_tests
[100%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_pblintd_tests.cpp.o
[100%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_compute_shoc_temperature_tests.cpp.o
[100%] Built target shoc_p3_nudging
[100%] Built target sea_level_pressure
[100%] Built target dry_static_energy
[100%] Built target aerocom_cld
[100%] Built target mam4_optics_standalone
[100%] Built target shortwave_cloud_forcing
[100%] Built target shoc_tests
[100%] Linking CXX executable shoc_sk_tests
[100%] Built target precip_surf_mass_flux
[100%] Built target aodvis
[100%] Built target relative_humidity
[100%] Built target surface_upward_latent_heat_flux
[100%] Built target vapor_flux
[100%] Built target shoc_sk_tests
[100%] Built target longwave_cloud_forcing
[100%] Built target ad_ut
Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on mappy with cmd: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5537/scream/components/eamxx && module purge && module load sems-archive-env acme-env acme-cmake/3.26.3 acme-gcc/11.2.0 sems-archive-git/2.10.1 acme-openmpi/4.1.4 acme-netcdf/4.7.4/acme && export GATOR_INITIAL_MB=4000MB && export PATH=/ascldap/users/jgfouca/packages/valgrind-3.22.0/bin:$PATH && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m mappy

RUN: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5537/scream/components/eamxx && module purge && module load sems-archive-env acme-env acme-cmake/3.26.3 acme-gcc/11.2.0 sems-archive-git/2.10.1 acme-openmpi/4.1.4 acme-netcdf/4.7.4/acme && export GATOR_INITIAL_MB=4000MB && export PATH=/ascldap/users/jgfouca/packages/valgrind-3.22.0/bin:$PATH && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m mappy

FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5537/scream/components/eamxx

mappy failed

######################################################

Build step 'Execute shell' marked build as failure

$ ssh-agent -k

unset SSH_AUTH_SOCK;

unset SSH_AGENT_PID;

echo Agent pid 125202 killed;

[ssh-agent] Stopped.

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs
squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel
[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins5633816042309870377.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE

SCREAM_PullRequest_Autotester_Weaver # 5806 PASSED (click to see last 100 lines of console output)


        Start 117: shoc_p3_nudging_glob_novert
117/134 Test #117: shoc_p3_nudging_glob_novert .............................   Passed    2.86 sec
        Start 118: homme_shoc_cld_p3_rrtmgp_np1
118/134 Test #118: homme_shoc_cld_p3_rrtmgp_np1 ............................   Passed   11.43 sec
        Start 119: homme_shoc_cld_p3_rrtmgp_baseline_cmp
119/134 Test #119: homme_shoc_cld_p3_rrtmgp_baseline_cmp ...................   Passed    0.10 sec
        Start 120: homme_shoc_cld_p3_rrtmgp_pg2_np1
120/134 Test #120: homme_shoc_cld_p3_rrtmgp_pg2_np1 ........................   Passed   11.98 sec
        Start 121: homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp
121/134 Test #121: homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp ...............   Passed    0.10 sec
        Start 122: model_baseline
122/134 Test #122: model_baseline ..........................................   Passed   12.55 sec
        Start 123: model_initial
123/134 Test #123: model_initial ...........................................   Passed    5.66 sec
        Start 124: model_restart
124/134 Test #124: model_restart ...........................................   Passed    7.00 sec
        Start 125: restarted_vs_monolithic_check_np1
125/134 Test #125: restarted_vs_monolithic_check_np1 .......................   Passed    0.11 sec
        Start 126: homme_shoc_cld_spa_p3_rrtmgp_np1
126/134 Test #126: homme_shoc_cld_spa_p3_rrtmgp_np1 ........................   Passed   12.03 sec
        Start 127: homme_shoc_cld_spa_p3_rrtmgp_baseline_cmp
127/134 Test #127: homme_shoc_cld_spa_p3_rrtmgp_baseline_cmp ...............   Passed    0.12 sec
        Start 128: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1
128/134 Test #128: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1 ..............   Passed   15.52 sec
        Start 129: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1
129/134 Test #129: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1 ...   Passed    1.40 sec
        Start 130: homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp
130/134 Test #130: homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp .....   Passed    0.65 sec
        Start 131: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np1
131/134 Test #131: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np1 .................   Passed   19.02 sec
        Start 132: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_baseline_cmp
132/134 Test #132: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_baseline_cmp ........   Passed    0.10 sec
        Start 133: homme_shoc_cld_p3_mam_optics_rrtmgp_np1
133/134 Test #133: homme_shoc_cld_p3_mam_optics_rrtmgp_np1 .................   Passed   18.19 sec
        Start 134: homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp
134/134 Test #134: homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp ........   Passed    0.21 sec
100% tests passed, 0 tests failed out of 134
Label Time Summary:

baseline_cmp        =  95.88 secproc (16 tests)

baseline_gen        = 209.34 secproc (18 tests)

bfbhash             =   0.69 secproc (1 test)

check               =   0.70 secproc (1 test)

cld                 =  18.00 secproc (3 tests)

cld_fraction        =   1.00 secproc (1 test)

cxx baseline_cmp    =   4.54 secproc (2 tests)

diagnostics         =  44.33 secproc (22 tests)

driver              =  42.40 secproc (8 tests)

dynamics            =   9.25 secproc (3 tests)

fail                =  27.94 secproc (5 tests)

io                  =  59.55 secproc (13 tests)

mam4_optics         =   9.23 secproc (1 test)

nudging             =  13.83 secproc (2 tests)

p3                  =  66.32 secproc (7 tests)

p3_sk               =  34.67 secproc (2 tests)

physics             = 130.67 secproc (19 tests)

remap               =   6.02 secproc (1 test)

rrtmgp              =  39.38 secproc (9 tests)

shoc                =  21.99 secproc (7 tests)

spa                 =  13.16 secproc (4 tests)

surface_coupling    =   4.67 secproc (1 test)
Total Test time (real) = 602.32 sec
Testing '''c65c84e3fc3e2d16428ee931cf6e6f8293fd6c35''' for test '''full_sp_debug'''
RUN: taskset -c 52-103 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5806/scream/components/eamxx/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5806/scream/components/eamxx/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5806/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5806/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_sp_debug" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5806/scream/components/eamxx/ctest-build/full_sp_debug
Testing '''c65c84e3fc3e2d16428ee931cf6e6f8293fd6c35''' for test '''full_debug'''
RUN: taskset -c 0-51 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5806/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5806/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5806/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5806/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DKokkos_ENABLE_DEBUG_BOUNDS_CHECK=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5806/scream/components/eamxx/ctest-build/full_debug
Testing '''c65c84e3fc3e2d16428ee931cf6e6f8293fd6c35''' for test '''release'''
RUN: taskset -c 104-155 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5806/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5806/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5806/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5806/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5806/scream/components/eamxx/ctest-build/release

OVERALL STATUS: PASS

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5806/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5806/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5806/scream/components/eamxx

Completed analysis on weaver'

[[ 0 != 0 ]]
[[ 1 == 0 ]]
[[ weaver == \m\a\p\p\y ]]
set +x

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins9725240449007020491.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Finished: SUCCESS

jgfouca · 2024-06-13T21:14:19Z

Re-running the tests. Looking at the mappy fail, it's hard to see what went wrong. It claims the build failed but I don't see a build error anywhere. Also, it seems to work OK when I run it myself.

ndkeen · 2024-06-13T21:49:59Z

When I try to build on pm-cpu with ERP_Ln9.ne4pg2_ne4pg2.F2010-SCREAMv1.pm-cpu_gnu

/global/cfs/cdirs/e3sm/ndk/repos/jgfouca_cleanup_kokkos_tril/components/eamxx/src/share/atm_process/atmosphere_process_dag.cpp: In lambda function:
/global/cfs/cdirs/e3sm/ndk/repos/jgfouca_cleanup_kokkos_tril/components/eamxx/src/share/atm_process/atmosphere_process_dag.cpp:149:37: error: 'const Units' {aka 'const class ekat::units::Units'} has no member named 'get_string'\
; did you mean 'get_si_string'?
  149 |         s += " [" + fid.get_units().get_string() + "]";
      |                                     ^~~~~~~~~~
      |                                     get_si_string

If I replace all of those get_string with get_si_string, it gets further

./eamxx/src/share/atm_process/atmosphere_process_dag.cpp:149:        s += " [" + fid.get_units().get_string() + "]";
./eamxx/src/share/field/field_identifier.cpp:51:  m_identifier += " [" + m_units.get_string() + "]";
./eamxx/src/share/io/scorpio_output.cpp:965:    std::string units = fid.get_units().get_string();

but still fails:

/global/cfs/cdirs/e3sm/ndk/repos/jgfouca_cleanup_kokkos_tril/components/eamxx/src/share/field/field_manager.cpp:49:69: error: invalid initialization of reference of type 'const scream::FieldLayout&' from expression of type 'const Un\
its' {aka 'const ekat::units::Units'}
   49 |           "         - input field units:  " + to_string(id.get_units()) + "\n"
      |                                                         ~~~~~~~~~~~~^~

jgfouca · 2024-06-13T21:56:41Z

@bartgol , did something change in Ekat::Units? @ndkeen , are you sure your submodules are updated?

E3SM-Autotester · 2024-06-13T22:03:39Z

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

E3SM-Autotester · 2024-06-13T22:04:28Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5539
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: RETEST;AT: AUTOMERGE;AT: Skip weaver
PULLREQUESTNUM	2866
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`c65c84e`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`b7f853c`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: -1
Status: SKIPPED

Jenkins Parameters

Parameter Name	Value

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: jgfouca

E3SM-Autotester · 2024-06-13T22:48:04Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5539
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: RETEST;AT: AUTOMERGE;AT: Skip weaver
PULLREQUESTNUM	2866
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`c65c84e`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`b7f853c`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: -1
Status: SKIPPED

Jenkins Parameters

Parameter Name	Value

SCREAM_PullRequest_Autotester_Mappy # 5539 FAILED (click to see last 100 lines of console output)


[ 93%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_unit_tests.cpp.o
[ 93%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_diag_second_moments_lbycond_tests.cpp.o
[ 93%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_diag_second_moments_tests.cpp.o
[ 93%] Linking CXX executable ad_ut
[ 93%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pdf_compute_qs_tests.cpp.o
[ 93%] Linking CXX executable aodvis
[ 93%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pdf_compute_s_tests.cpp.o
[ 93%] Linking CXX executable precip_surf_mass_flux
[ 93%] Linking CXX executable vapor_flux
[ 94%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_diag_second_shoc_moments_tests.cpp.o
[ 94%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pdf_compute_buoyflux_tests.cpp.o
[ 95%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pdf_compute_liqflux_tests.cpp.o
[ 96%] Linking CXX executable relative_humidity
[ 96%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_pblintd_cldcheck_tests.cpp.o
[ 96%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_compute_shoc_vapor_tests.cpp.o
[ 97%] Linking CXX executable surface_upward_latent_heat_flux
[ 97%] Linking CXX executable longwave_cloud_forcing
[ 97%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pdf_compute_cloudvar_tests.cpp.o
[ 97%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pdf_compute_sgsliq_tests.cpp.o
[ 97%] Linking CXX executable aerocom_cld
[ 97%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_diag_second_mom_ubycond_test.cpp.o
[ 97%] Built target virtual_temperature
[ 97%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_update_prognostics_implicit_tests.cpp.o
[ 97%] Built target dry_static_energy
[ 97%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_main_tests.cpp.o
[ 97%] Linking CXX executable shortwave_cloud_forcing
[ 97%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_pblintd_height_tests.cpp.o
[ 97%] Built target atmosphere_density
[ 97%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_vd_shoc_decomp_and_solve_tests.cpp.o
[ 97%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pblintd_init_pot_test.cpp.o
[ 97%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_pblintd_surf_temp_tests.cpp.o
[ 97%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_unit_tests.cpp.o
[ 97%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_pblintd_check_pblh_tests.cpp.o
[ 97%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_diag_second_moments_lbycond_tests.cpp.o
[ 97%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_diag_second_moments_tests.cpp.o
[ 97%] Built target cld_fraction_standalone
[ 97%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_diag_second_shoc_moments_tests.cpp.o
[ 97%] Built target sea_level_pressure
[ 98%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pblintd_cldcheck_tests.cpp.o
[ 98%] Built target shoc_standalone
[ 98%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_compute_shoc_vapor_tests.cpp.o
[ 99%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_pblintd_tests.cpp.o
[ 99%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_update_prognostics_implicit_tests.cpp.o
[ 99%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_main_tests.cpp.o
[ 99%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_compute_shoc_temperature_tests.cpp.o
[ 99%] Built target spa_standalone
[ 99%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pblintd_height_tests.cpp.o
[ 99%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_vd_shoc_decomp_and_solve_tests.cpp.o
[ 99%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pblintd_surf_temp_tests.cpp.o
[ 99%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pblintd_check_pblh_tests.cpp.o
[ 99%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pblintd_tests.cpp.o
[100%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_compute_shoc_temperature_tests.cpp.o
[100%] Built target shoc_p3
[100%] Built target vapor_flux
[100%] Built target aodvis
[100%] Built target shoc_p3_nudging
[100%] Built target relative_humidity
[100%] Built target precip_surf_mass_flux
[100%] Built target surface_upward_latent_heat_flux
[100%] Built target longwave_cloud_forcing
[100%] Linking CXX executable shoc_sk_tests
[100%] Built target aerocom_cld
[100%] Built target surface_coupling
[100%] Built target ad_ut
[100%] Built target shortwave_cloud_forcing
[100%] Built target mam4_optics_standalone
[100%] Linking CXX executable shoc_tests
[100%] Built target shoc_sk_tests
[100%] Built target shoc_tests
Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on mappy with cmd: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5539/scream/components/eamxx && module purge && module load sems-archive-env acme-env acme-cmake/3.26.3 acme-gcc/11.2.0 sems-archive-git/2.10.1 acme-openmpi/4.1.4 acme-netcdf/4.7.4/acme && export GATOR_INITIAL_MB=4000MB && export PATH=/ascldap/users/jgfouca/packages/valgrind-3.22.0/bin:$PATH && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m mappy

RUN: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5539/scream/components/eamxx && module purge && module load sems-archive-env acme-env acme-cmake/3.26.3 acme-gcc/11.2.0 sems-archive-git/2.10.1 acme-openmpi/4.1.4 acme-netcdf/4.7.4/acme && export GATOR_INITIAL_MB=4000MB && export PATH=/ascldap/users/jgfouca/packages/valgrind-3.22.0/bin:$PATH && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m mappy

FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5539/scream/components/eamxx

mappy failed

######################################################

Build step 'Execute shell' marked build as failure

$ ssh-agent -k

unset SSH_AUTH_SOCK;

unset SSH_AGENT_PID;

echo Agent pid 22811 killed;

[ssh-agent] Stopped.

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs
squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel
[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins14698651648106922841.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE

SCREAM_PullRequest_Autotester_Weaver # -1 SKIPPED

ndkeen · 2024-06-13T22:52:50Z

Jim: you said this fixed build errors on pm-cpu. What tests did you run? You can see the test I ran to get build error.

jgfouca · 2024-06-13T23:03:05Z

@ndkeen something is up with your repo. ./create_test e3sm_scream_v1_medres --compiler=gnu --no-run all built fine for me.

bartgol · 2024-06-13T23:12:51Z

Jim: you said this fixed build errors on pm-cpu. What tests did you run? You can see the test I ran to get build error.

Noel, i think you tried the e3sm branch though, no? At least that's my understanding from the slack convo. This PR is in the scream repo, where get_string is not used anymore...

ndkeen · 2024-06-13T23:20:18Z

ah, ok. i just made this comment on the wrong PR.
The scream repo does not have trouble building -- though i do see a number of runtime fails.

E3SM-Autotester · 2024-06-17T17:41:27Z

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

E3SM-Autotester · 2024-06-17T17:44:15Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5550
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: RETEST;AT: AUTOMERGE;AT: Skip weaver
PULLREQUESTNUM	2866
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`c65c84e`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`b7f853c`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: -1
Status: SKIPPED

Jenkins Parameters

Parameter Name	Value

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: jgfouca

E3SM-Autotester · 2024-06-17T19:53:11Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5550
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: RETEST;AT: AUTOMERGE;AT: Skip weaver
PULLREQUESTNUM	2866
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`c65c84e`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`b7f853c`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: -1
Status: SKIPPED

Jenkins Parameters

Parameter Name	Value

SCREAM_PullRequest_Autotester_Mappy # 5550 FAILED (click to see last 100 lines of console output)


+++ MODULEPATH=/projects/sems/modulefiles/rhel7-x86_64/sems-archive/devpack:/projects/sems/modulefiles/rhel7-x86_64/sems-archive/compiler:/projects/sems/modulefiles/rhel7-x86_64/sems-archive/tpl:/projects/sems/modulefiles/rhel7-x86_64/sems-archive/utility:/projects/sems/modulefiles/projects:/projects/sems/cee-sierra-modules:/projects/sems/modulefiles/rhel7-x86_64/sems/linux-rhel7-x86_64/compilers:/projects/sems/modulefiles/rhel7-x86_64/sems/linux-rhel7-x86_64/Core:/projects/sems/modulefiles/rhel7-x86_64/sems/linux-rhel7-x86_64/project-modulefiles:/projects/aue/modules/cee/x86_64/rhel7:/usr/share/Modules/modulefiles:/etc/modulefiles
+++ export MODULEPATH
+++ __LMOD_REF_COUNT_PATH='/projects/sems/install/rhel7-x86_64/sems/utility/git/2.10.1/sbin:1;/projects/sems/install/rhel7-x86_64/sems/utility/git/2.10.1/bin:1;/ascldap/users/jgfouca/packages/Python-3.8.5/bin:1;/usr/lib64/qt-3.3/bin:1;/usr/condabin:1;/usr/local/sbin:1;/usr/local/bin:1;/sbin:1;/bin:1;/usr/sbin:1;/usr/bin:1;/root/bin:1;/opt/dell/srvadmin/sbin:1;/ascldap/users/e3sm-jenkins/.local/bin:1;/ascldap/users/e3sm-jenkins/bin:1'
+++ export __LMOD_REF_COUNT_PATH
+++ PATH=/projects/sems/install/rhel7-x86_64/sems/utility/git/2.10.1/sbin:/projects/sems/install/rhel7-x86_64/sems/utility/git/2.10.1/bin:/ascldap/users/jgfouca/packages/Python-3.8.5/bin:/usr/lib64/qt-3.3/bin:/usr/condabin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/opt/dell/srvadmin/sbin:/ascldap/users/e3sm-jenkins/.local/bin:/ascldap/users/e3sm-jenkins/bin
+++ export PATH
+++ SEMS_GIT_LOCAL_COMPILER_VERSION=4.8.5
+++ export SEMS_GIT_LOCAL_COMPILER_VERSION
+++ SEMS_GIT_LOCAL_PYTHON_VERSION=2.7.5
+++ export SEMS_GIT_LOCAL_PYTHON_VERSION
+++ SEMS_GIT_ROOT=/projects/sems/install/rhel7-x86_64/sems/utility/git/2.10.1
+++ export SEMS_GIT_ROOT
+++ SEMS_GIT_VERSION=2.10.1
+++ export SEMS_GIT_VERSION
+++ __LMOD_REF_COUNT__LMFILES_='/projects/sems/modulefiles/projects/sems-archive-env:1;/projects/sems/modulefiles/rhel7-x86_64/sems-archive/utility/sems-archive-git/2.10.1:1'
+++ export __LMOD_REF_COUNT__LMFILES_
+++ _LMFILES_=/projects/sems/modulefiles/projects/sems-archive-env:/projects/sems/modulefiles/rhel7-x86_64/sems-archive/utility/sems-archive-git/2.10.1
+++ export _LMFILES_
+++ _ModuleTable001_=X01vZHVsZVRhYmxlXz17WyJNVHZlcnNpb24iXT0zLFsiY19yZWJ1aWxkVGltZSJdPWZhbHNlLFsiY19zaG9ydFRpbWUiXT1mYWxzZSxkZXB0aFQ9e30sZmFtaWx5PXt9LG1UPXtbInNlbXMtYXJjaGl2ZS1lbnYiXT17WyJmbiJdPSIvcHJvamVjdHMvc2Vtcy9tb2R1bGVmaWxlcy9wcm9qZWN0cy9zZW1zLWFyY2hpdmUtZW52IixbImZ1bGxOYW1lIl09InNlbXMtYXJjaGl2ZS1lbnYiLFsibG9hZE9yZGVyIl09MSxwcm9wVD17fSxbInN0YWNrRGVwdGgiXT0wLFsic3RhdHVzIl09ImFjdGl2ZSIsWyJ1c2VyTmFtZSJdPSJzZW1zLWFyY2hpdmUtZW52Iix9LFsic2Vtcy1hcmNoaXZlLWdpdCJdPXtbImZuIl09Ii9wcm9qZWN0cy9zZW1zL21vZHVsZWZpbGVzL3JoZWw3LXg4Nl82NC9z
+++ export _ModuleTable001_
+++ _ModuleTable002_=ZW1zLWFyY2hpdmUvdXRpbGl0eS9zZW1zLWFyY2hpdmUtZ2l0LzIuMTAuMSIsWyJmdWxsTmFtZSJdPSJzZW1zLWFyY2hpdmUtZ2l0LzIuMTAuMSIsWyJsb2FkT3JkZXIiXT0yLHByb3BUPXt9LFsic3RhY2tEZXB0aCJdPTAsWyJzdGF0dXMiXT0iYWN0aXZlIixbInVzZXJOYW1lIl09InNlbXMtYXJjaGl2ZS1naXQvMi4xMC4xIix9LH0sbXBhdGhBPXsiL3Byb2plY3RzL3NlbXMvbW9kdWxlZmlsZXMvcmhlbDcteDg2XzY0L3NlbXMtYXJjaGl2ZS9kZXZwYWNrIiwiL3Byb2plY3RzL3NlbXMvbW9kdWxlZmlsZXMvcmhlbDcteDg2XzY0L3NlbXMtYXJjaGl2ZS9jb21waWxlciIsIi9wcm9qZWN0cy9zZW1zL21vZHVsZWZpbGVzL3JoZWw3LXg4Nl82NC9zZW1zLWFyY2hpdmUvdHBsIiwi
+++ export _ModuleTable002_
+++ _ModuleTable003_=L3Byb2plY3RzL3NlbXMvbW9kdWxlZmlsZXMvcmhlbDcteDg2XzY0L3NlbXMtYXJjaGl2ZS91dGlsaXR5IiwiL3Byb2plY3RzL3NlbXMvbW9kdWxlZmlsZXMvcHJvamVjdHMiLCIvcHJvamVjdHMvc2Vtcy9jZWUtc2llcnJhLW1vZHVsZXMiLCIvcHJvamVjdHMvc2Vtcy9tb2R1bGVmaWxlcy9yaGVsNy14ODZfNjQvc2Vtcy9saW51eC1yaGVsNy14ODZfNjQvY29tcGlsZXJzIiwiL3Byb2plY3RzL3NlbXMvbW9kdWxlZmlsZXMvcmhlbDcteDg2XzY0L3NlbXMvbGludXgtcmhlbDcteDg2XzY0L0NvcmUiLCIvcHJvamVjdHMvc2Vtcy9tb2R1bGVmaWxlcy9yaGVsNy14ODZfNjQvc2Vtcy9saW51eC1yaGVsNy14ODZfNjQvcHJvamVjdC1tb2R1bGVmaWxlcyIsIi9wcm9qZWN0cy9hdWUv
+++ export _ModuleTable003_
+++ _ModuleTable004_=bW9kdWxlcy9jZWUveDg2XzY0L3JoZWw3IiwiL3Vzci9zaGFyZS9Nb2R1bGVzL21vZHVsZWZpbGVzIiwiL2V0Yy9tb2R1bGVmaWxlcyIsfSxbInN5c3RlbUJhc2VNUEFUSCJdPSIvdXNyL3NoYXJlL01vZHVsZXMvbW9kdWxlZmlsZXM6L2V0Yy9tb2R1bGVmaWxlcyIsfQ==
+++ export _ModuleTable004_
+++ _ModuleTable_Sz_=4
+++ export _ModuleTable_Sz_
++ source /ascldap/users/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5550/scream/components/eamxx/scripts/jenkins/sandia_son_proxy
+++ export http_proxy=http://proxy.sandia.gov:80
+++ http_proxy=http://proxy.sandia.gov:80
+++ export RSYNC_PROXY=proxy.sandia.gov:80
+++ RSYNC_PROXY=proxy.sandia.gov:80
+++ export rsync_proxy=proxy.sandia.gov:80
+++ rsync_proxy=proxy.sandia.gov:80
+++ export HTTPS_PROXY=http://proxy.sandia.gov:80
+++ HTTPS_PROXY=http://proxy.sandia.gov:80
+++ export https_proxy=http://proxy.sandia.gov:80
+++ https_proxy=http://proxy.sandia.gov:80
+++ export HTTP_PROXY=http://proxy.sandia.gov:80
+++ HTTP_PROXY=http://proxy.sandia.gov:80
++ SCREAM_MACHINE=mappy
+ [[ 0 == 1 ]]
+ [[ 1 == 1 ]]
+ [[ mappy == \w\e\a\v\e\r ]]
+ [[ 0 == 1 ]]
++ whoami
+ [[ e3sm-jenkins == \e\3\s\m\-\j\e\n\k\i\n\s ]]
+ git config --local user.email jenkins@ignore.com
+ git config --local user.name 'Jenkins Jenkins'
+ declare -i fails=0
+ BASELINES_DIR=AUTO
+ TAS_ARGS='--baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m $machine'
+ [[ mappy == \p\m\-\g\p\u ]]
+ set +e
+ '[' -n 2866 ']'
+ is_at_run=1
+ SA_FAILURES_DETAILS=
+ '[' 1 -eq 1 ']'
++ ./scripts/gather-all-data './scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m $machine' -l -m mappy
Build timed out (after 120 minutes). Marking the build as failed.
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 91027 killed;
[ssh-agent] Stopped.
Build was aborted
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs
squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel
[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins6745347741604427191.sh

Traceback (most recent call last):

File "./scripts/test-all-scream", line 155, in 

_main_func(doc)

File "./scripts/test-all-scream", line 146, in _main_func

success = tas.test_all_scream()

File "/home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5550/scream/components/eamxx/scripts/test_all_scream.py", line 862, in test_all_scream

success = self.generate_all_baselines()

File "/home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5550/scream/components/eamxx/scripts/test_all_scream.py", line 713, in generate_all_baselines

success &= future.result()

File "/ascldap/users/jgfouca/packages/Python-3.8.5/lib/python3.8/concurrent/futures/_base.py", line 432, in result

return self.__get_result()

File "/ascldap/users/jgfouca/packages/Python-3.8.5/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result

raise self.exception

concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

./scream/components/eamxx/scripts/jenkins/jenkins_common.sh: line 7: 24361 Terminated              $JENKINS_SCRIPT_DIR/jenkins_common_impl.sh 2>&1

24362 Done                    | tee JENKINS$DATE_STAMP

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE

SCREAM_PullRequest_Autotester_Weaver # -1 SKIPPED

* origin/master: (151 commits) Update EKAT submodule EAMxx: fix relative humidity unit test EAMxx: fix usage of code deprecated in Kokkos 4 Update haero submodule Fix uninitialized variable and a few Cuda compiler warnings. EAMxx: fixes to coarsening/vertical remappers, related to masks EAMxx: refactor vertical interpolation EAMxx: add utility method to FieldLayout Turns out that the MurphyKoop function only needs the ice flag to determine whether to take ice into consideration. So the PR reduces to changing that flag to true. allows for computing qv_sat_ice when T < T_freeze for RH diagnostic Adds a cime test for aci Fixed a layout and Clang format Fixes LABLES of the tests to change mam to mam4_aci for time reporting Fixes warnings and changes [=] to [&] GPU fixes: changes array of views to views in Kokkos loop Changes TeamThreadRange to TeamVectorRange; [=] to [&]; removes aitken dia func with deep_copy Adds some TODO comments and minor cleanup Fixes np1_vs_npX standalone tests and adds mam4xx vars to output Changes after rebase Updates CMAKE for the aci tests to mimic P3 CMAKE to enable np1 vs npX tests ...

E3SM-Autotester · 2024-06-17T21:12:28Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5552
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE;AT: Skip weaver
PULLREQUESTNUM	2866
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`5637a2f`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`687dd31`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: -1
Status: SKIPPED

Jenkins Parameters

Parameter Name	Value

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: jgfouca

E3SM-Autotester · 2024-06-17T22:01:23Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5552
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE;AT: Skip weaver
PULLREQUESTNUM	2866
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`5637a2f`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`687dd31`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: -1
Status: SKIPPED

Jenkins Parameters

Parameter Name	Value

Fix gptl include and remove Kokkos_ROOT settings

c65c84e

jgfouca added the AT: AUTOMERGE Inform the autotester (AT) that it can merge this PR if reviewers approved, and tests pass label Jun 13, 2024

jgfouca requested review from bartgol and ndkeen June 13, 2024 18:50

jgfouca added AT: Skip weaver Do not run AT testing on weaver for this PR AT: RETEST Force the autotester (AT) to retest the PR labels Jun 13, 2024

bartgol approved these changes Jun 13, 2024

View reviewed changes

E3SM-Autotester removed the AT: RETEST Force the autotester (AT) to retest the PR label Jun 13, 2024

ndkeen approved these changes Jun 13, 2024

View reviewed changes

bartgol added the AT: RETEST Force the autotester (AT) to retest the PR label Jun 17, 2024

E3SM-Autotester removed the AT: RETEST Force the autotester (AT) to retest the PR label Jun 17, 2024

E3SM-Autotester merged commit aeb08b7 into master Jun 17, 2024
15 checks passed

E3SM-Autotester deleted the jgfouca/pm-cpu-fixes branch June 17, 2024 22:01

E3SM-Autotester removed the AT: AUTOMERGE Inform the autotester (AT) that it can merge this PR if reviewers approved, and tests pass label Jun 17, 2024

Fix gptl include and remove Kokkos_ROOT settings #2866

Fix gptl include and remove Kokkos_ROOT settings #2866

Conversation

jgfouca commented Jun 13, 2024 • edited Loading

ndkeen commented Jun 13, 2024

bartgol commented Jun 13, 2024 • edited Loading

jgfouca commented Jun 13, 2024

E3SM-Autotester commented Jun 13, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Using Repos:

E3SM-Autotester commented Jun 13, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs

Total Test time (real) = 602.32 sec

Testing '''c65c84e3fc3e2d16428ee931cf6e6f8293fd6c35''' for test '''full_sp_debug'''

Testing '''c65c84e3fc3e2d16428ee931cf6e6f8293fd6c35''' for test '''full_debug'''

Testing '''c65c84e3fc3e2d16428ee931cf6e6f8293fd6c35''' for test '''release'''

jgfouca commented Jun 13, 2024

ndkeen commented Jun 13, 2024 • edited Loading

jgfouca commented Jun 13, 2024

E3SM-Autotester commented Jun 13, 2024

E3SM-Autotester commented Jun 13, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Using Repos:

E3SM-Autotester commented Jun 13, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs

ndkeen commented Jun 13, 2024

jgfouca commented Jun 13, 2024

bartgol commented Jun 13, 2024

ndkeen commented Jun 13, 2024 • edited Loading

E3SM-Autotester commented Jun 17, 2024

E3SM-Autotester commented Jun 17, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Using Repos:

E3SM-Autotester commented Jun 17, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs

E3SM-Autotester commented Jun 17, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Using Repos:

E3SM-Autotester commented Jun 17, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

jgfouca commented Jun 13, 2024 •

edited

Loading

bartgol commented Jun 13, 2024 •

edited

Loading

ndkeen commented Jun 13, 2024 •

edited

Loading

ndkeen commented Jun 13, 2024 •

edited

Loading