Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pbl entrainment budget diags #2822

Closed
wants to merge 6 commits into from
Closed

Conversation

mahf708
Copy link
Contributor

@mahf708 mahf708 commented May 13, 2024

adding pbl entrainment budget diags to eamxx

@mahf708 mahf708 added the AT: WIP Inform the autotester (AT) that the PR is a work in progress, and should not be tested label May 13, 2024
components/eamxx/src/diagnostics/extraaci.cpp Outdated Show resolved Hide resolved
components/eamxx/src/diagnostics/extraaci.cpp Outdated Show resolved Hide resolved
components/eamxx/src/diagnostics/extraaci.cpp Outdated Show resolved Hide resolved

auto ws = wsm.get_workspace(team);
ekat::Unmanaged<WSMgr::view_1d<Real>> tm_grad;
ws.take_many_contiguous_unsafe<wsms>({"tm_grad"}, {&tm_grad});
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not wrong, but since you use only one slot, you can prob just use auto tm_grad = ws.take("tm_grad").

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do want to keep the full wsm stuff around because this is not the final place these diags will be in. I will likely add at least a few more temp views here soon

team.team_barrier();

// release stuff from wsm
ws.release_many_contiguous<wsms>({&tm_grad});
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you switch to take, then this can become a simple release()

components/eamxx/src/diagnostics/extraaci.cpp Outdated Show resolved Hide resolved
size = 0;

// pblp
index_map["p+"] = size++;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are + and ^ in the diags names?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+: above the inversion (so at min_tm_grad_lev-1 in the algo)
^: integrated under the inversion (so integrated from num_levs to min_tm_grad_lev)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these "common" names in this context? If not, I would rather use some more wordy names...

Copy link
Contributor Author

@mahf708 mahf708 May 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They kind of are defined like this. I mean, that's how they're defined in the reference eqs I used. But, we don't have to follow what these eqs do and we can do our own more verbose descriptions

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(I may also decide to add technical docs with this PR either now or later...)

components/eamxx/src/diagnostics/tests/extraaci_test.cpp Outdated Show resolved Hide resolved
@mahf708 mahf708 force-pushed the mahf708/diags/extra-swaci-diags branch from 9555a54 to 8e42850 Compare May 14, 2024 02:25
@mahf708 mahf708 changed the title extra swaci diags entrainment budget diags May 14, 2024
@mahf708 mahf708 force-pushed the mahf708/diags/extra-swaci-diags branch from 8e42850 to f445669 Compare May 14, 2024 04:18
@mahf708 mahf708 marked this pull request as ready for review May 14, 2024 04:19
@mahf708 mahf708 added priority:low diagnostic and removed AT: WIP Inform the autotester (AT) that the PR is a work in progress, and should not be tested labels May 14, 2024
@mahf708 mahf708 force-pushed the mahf708/diags/extra-swaci-diags branch from f445669 to be0c10e Compare May 14, 2024 04:25
@mahf708 mahf708 added the AT: RETEST Force the autotester (AT) to retest the PR label May 14, 2024
@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

@mahf708 mahf708 changed the title entrainment budget diags pbl entrainment budget diags May 14, 2024
@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5422
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: RETEST;priority:low;diagnostic
PULLREQUESTNUM 2822
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 93ef4a4
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 8ff32a8
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5702
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: RETEST;priority:low;diagnostic
PULLREQUESTNUM 2822
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 93ef4a4
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 8ff32a8
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)
  • Branch: mahf708/diags/extra-swaci-diags
  • SHA: 93ef4a4
  • Mode: TEST_REPO

Pull Request Author: mahf708

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5422
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: RETEST;priority:low;diagnostic
PULLREQUESTNUM 2822
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 93ef4a4
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 8ff32a8
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5702
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: RETEST;priority:low;diagnostic
PULLREQUESTNUM 2822
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 93ef4a4
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 8ff32a8
TEST_REPO_ALIAS SCREAM
SCREAM_PullRequest_Autotester_Mappy # 5422 FAILED (click to see last 100 lines of console output)

[ 94%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_diag_second_moments_tests.cpp.o
[ 94%] Built target water_path
[ 94%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_diag_second_shoc_moments_tests.cpp.o
[ 94%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_pblintd_cldcheck_tests.cpp.o
[ 94%] Linking CXX executable dry_static_energy
[ 94%] Linking CXX executable shortwave_cloud_forcing
[ 94%] Building CXX object src/physics/spa/tests/CMakeFiles/spa_read_data_test.dir/spa_read_data_from_file_test.cpp.o
[ 94%] Building CXX object src/physics/spa/tests/CMakeFiles/spa_one_to_one_remap_test.dir/spa_one_to_one_remap_test.cpp.o
[ 94%] Linking CXX executable precip_surf_mass_flux
[ 94%] Linking CXX executable p3_standalone
[ 94%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_compute_shoc_vapor_tests.cpp.o
[ 94%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pdf_inplume_corr_tests.cpp.o
[ 94%] Built target wind_speed
[ 94%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pdf_computetemp_tests.cpp.o
[ 94%] Linking CXX executable aodvis
[ 94%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pdf_compute_qs_tests.cpp.o
[ 94%] Linking CXX executable aerocom_cld
[ 94%] Linking CXX executable relative_humidity
[ 94%] Linking CXX executable vapor_flux
[ 94%] Linking CXX executable create_vert_remap_and_weights
[ 94%] Linking CXX executable longwave_cloud_forcing
[ 95%] Linking CXX executable cld_fraction_standalone
[ 95%] Linking CXX executable shoc_standalone
[ 96%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pdf_compute_s_tests.cpp.o
[ 96%] Linking CXX executable spa_standalone
[ 97%] Linking CXX executable surface_upward_latent_heat_flux
[ 97%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_update_prognostics_implicit_tests.cpp.o
[ 97%] Built target atmosphere_density
[ 97%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_main_tests.cpp.o
[ 97%] Built target number_paths
[ 97%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_pblintd_height_tests.cpp.o
[ 97%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_vd_shoc_decomp_and_solve_tests.cpp.o
[ 97%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pdf_compute_buoyflux_tests.cpp.o
[ 97%] Linking CXX executable ad_ut
[ 97%] Built target create_vert_remap_and_weights
[ 97%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pdf_compute_liqflux_tests.cpp.o
[ 97%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pdf_compute_cloudvar_tests.cpp.o
[ 97%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pdf_compute_sgsliq_tests.cpp.o
[ 98%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_pblintd_surf_temp_tests.cpp.o
[ 98%] Built target pbl_entrainment_budget
[ 98%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_pblintd_check_pblh_tests.cpp.o
[ 98%] Linking CXX executable shoc_p3
[ 98%] Built target sea_level_pressure
[ 98%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_pblintd_tests.cpp.o
[ 98%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_sk_tests.dir/shoc_compute_shoc_temperature_tests.cpp.o
[ 98%] Linking CXX executable shoc_p3_nudging
[ 98%] Built target dry_static_energy
[ 98%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_diag_second_mom_ubycond_test.cpp.o
[ 98%] Built target shortwave_cloud_forcing
[ 98%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pblintd_init_pot_test.cpp.o
[ 98%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_unit_tests.cpp.o
[ 98%] Linking CXX executable mam4_optics_standalone
[ 98%] Built target precip_surf_mass_flux
[ 98%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_diag_second_moments_lbycond_tests.cpp.o
[ 99%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_diag_second_moments_tests.cpp.o
[ 99%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_diag_second_shoc_moments_tests.cpp.o
[ 99%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_compute_shoc_vapor_tests.cpp.o
[ 99%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_update_prognostics_implicit_tests.cpp.o
[ 99%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pblintd_cldcheck_tests.cpp.o
[ 99%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_main_tests.cpp.o
[ 99%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pblintd_height_tests.cpp.o
[ 99%] Linking CXX executable surface_coupling
[ 99%] Built target aodvis
[ 99%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_vd_shoc_decomp_and_solve_tests.cpp.o
[ 99%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pblintd_surf_temp_tests.cpp.o
[100%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pblintd_check_pblh_tests.cpp.o
[100%] Built target aerocom_cld
[100%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pblintd_tests.cpp.o
[100%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_compute_shoc_temperature_tests.cpp.o
[100%] Built target vapor_flux
[100%] Built target relative_humidity
[100%] Linking CXX executable shoc_sk_tests
[100%] Built target longwave_cloud_forcing
[100%] Built target surface_upward_latent_heat_flux
[100%] Built target p3_standalone
[100%] Linking CXX executable shoc_tests
[100%] Built target ad_ut
[100%] Built target shoc_sk_tests
[100%] Built target cld_fraction_standalone
[100%] Built target shoc_standalone
[100%] Built target spa_standalone
[100%] Linking CXX executable spa_read_data_test
[100%] Built target shoc_tests
[100%] Linking CXX executable spa_one_to_one_remap_test
[100%] Built target shoc_p3
[100%] Built target mam4_optics_standalone
[100%] Built target surface_coupling
[100%] Built target shoc_p3_nudging
[100%] Built target spa_read_data_test
[100%] Built target spa_one_to_one_remap_test
Starting analysis on mappy with cmd: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5422/scream/components/eamxx && module purge && module load sems-archive-env acme-env acme-cmake/3.26.3 acme-gcc/11.2.0 sems-archive-git/2.10.1 acme-openmpi/4.1.4 acme-netcdf/4.7.4/acme && export GATOR_INITIAL_MB=4000MB && export PATH=/ascldap/users/jgfouca/packages/valgrind-3.22.0/bin:$PATH && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m mappy
RUN: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5422/scream/components/eamxx && module purge && module load sems-archive-env acme-env acme-cmake/3.26.3 acme-gcc/11.2.0 sems-archive-git/2.10.1 acme-openmpi/4.1.4 acme-netcdf/4.7.4/acme && export GATOR_INITIAL_MB=4000MB && export PATH=/ascldap/users/jgfouca/packages/valgrind-3.22.0/bin:$PATH && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m mappy
FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5422/scream/components/eamxx
mappy failed'
./scream/components/eamxx/scripts/jenkins/jenkins_common.sh: line 7: 70347 Terminated              $JENKINS_SCRIPT_DIR/jenkins_common_impl.sh 2>&1
     70348 Done                    | tee JENKINS_$DATE_STAMP
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Sending e-mails to: lbertag@sandia.gov
Finished: FAILURE

SCREAM_PullRequest_Autotester_Weaver # 5702 FAILED (click to see last 100 lines of console output)

===============================================================================
Generating baseline for test full_debug with config '\''cmake -C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5702/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='\''ON'\'' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_ONLY_GENERATE_BASELINES=ON -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug -DSCREAM_TEST_MAX_TOTAL_THREADS=1'\''
===============================================================================
RUN: cmake -C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5702/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='\''ON'\'' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_ONLY_GENERATE_BASELINES=ON -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug -DSCREAM_TEST_MAX_TOTAL_THREADS=1 /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5702/scream/components/eamxx
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5702/scream/components/eamxx/ctest-build/tas_baseline_build/full_debug
RUN: taskset -c 0-51 sh -c '\''make -j52'\''
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5702/scream/components/eamxx/ctest-build/tas_baseline_build/full_debug
RUN: ctest -j1 -L baseline_gen --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5702/scream/components/eamxx/ctest-build/tas_baseline_build/full_debug/ctest_resource_file.json
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5702/scream/components/eamxx/ctest-build/tas_baseline_build/full_debug
WARNING: Failed to create baselines (run phase):
Errors while running CTest
Output from these tests are in: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5702/scream/components/eamxx/ctest-build/tas_baseline_build/full_debug/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
Error(s) occurred during baselines generation phase
OVERALL STATUS: FAIL
Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5702/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver
RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5702/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5702/scream/components/eamxx
weaver failed'
+ [[ 1 != 0 ]]
+ fails=0+1
+ sa_fail=1
+ [[ 1 == 1 ]]
++ grep -m1 -A 100000 'Build type '
++ echo 'Job <73062> is submitted to queue .
Note: no value passed for --make-parallel-level. Using the default for this machine: 160
Note: no value passed for --ctest-parallel-level. Using the default for this machine: 4
Test full_debug can use 52 jobs to compile, and 1 jobs for test
Test full_sp_debug can use 52 jobs to compile, and 1 jobs for test
Test release can use 52 jobs to compile, and 1 jobs for test
RUN: { source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export CTEST_PARALLEL_LEVEL=4;  } > /dev/null && env | sort
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5702/scream/components/eamxx
Last commit on ref '\''f1a0982ecc7426a0c372a2fd3892d7eac8c71a15'\'': f1a0982ecc Merge commit '\''93ef4a4f16cc158e13e0cdff32b136f4ad216a1d'\'' into HEAD
Checking baselines directory: /home/projects/e3sm/scream/pr-autotester/master-baselines/weaver
 -> Test full_debug appears to have baselines
 -> Test full_sp_debug appears to have baselines
 -> Test release appears to have baselines
 -> Test full_debug baselines are expired because origin/master is ahead of the existing baseline commit 8ff32a8c9cd50b8cd9cbb02509c009cd561b0362 by 4
 -> Test full_sp_debug baselines are valid and do not need to be regenerated
 -> Test release baselines are expired because origin/master is ahead of the existing baseline commit 8ff32a8c9cd50b8cd9cbb02509c009cd561b0362 by 4
Merge of '\''origin/master'\'' not necessary. Current HEAD is already ahead.
###############################################################################
Generating baselines from git ref origin/master
###############################################################################
===============================================================================
Generating baseline for test release with config '\''cmake -C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5702/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='\''ON'\'' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_ONLY_GENERATE_BASELINES=ON -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release -DSCREAM_TEST_MAX_TOTAL_THREADS=1'\''
===============================================================================
RUN: cmake -C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5702/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='\''ON'\'' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_ONLY_GENERATE_BASELINES=ON -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release -DSCREAM_TEST_MAX_TOTAL_THREADS=1 /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5702/scream/components/eamxx
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5702/scream/components/eamxx/ctest-build/tas_baseline_build/release
RUN: taskset -c 104-155 sh -c '\''make -j52'\''
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5702/scream/components/eamxx/ctest-build/tas_baseline_build/release
RUN: ctest -j1 -L baseline_gen --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5702/scream/components/eamxx/ctest-build/tas_baseline_build/release/ctest_resource_file.json
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5702/scream/components/eamxx/ctest-build/tas_baseline_build/release
WARNING: Failed to create baselines (run phase):
Errors while running CTest
Output from these tests are in: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5702/scream/components/eamxx/ctest-build/tas_baseline_build/release/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
===============================================================================
Generating baseline for test full_debug with config '\''cmake -C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5702/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='\''ON'\'' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_ONLY_GENERATE_BASELINES=ON -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug -DSCREAM_TEST_MAX_TOTAL_THREADS=1'\''
===============================================================================
RUN: cmake -C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5702/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='\''ON'\'' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_ONLY_GENERATE_BASELINES=ON -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug -DSCREAM_TEST_MAX_TOTAL_THREADS=1 /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5702/scream/components/eamxx
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5702/scream/components/eamxx/ctest-build/tas_baseline_build/full_debug
RUN: taskset -c 0-51 sh -c '\''make -j52'\''
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5702/scream/components/eamxx/ctest-build/tas_baseline_build/full_debug
RUN: ctest -j1 -L baseline_gen --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5702/scream/components/eamxx/ctest-build/tas_baseline_build/full_debug/ctest_resource_file.json
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5702/scream/components/eamxx/ctest-build/tas_baseline_build/full_debug
WARNING: Failed to create baselines (run phase):
Errors while running CTest
Output from these tests are in: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5702/scream/components/eamxx/ctest-build/tas_baseline_build/full_debug/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
Error(s) occurred during baselines generation phase
OVERALL STATUS: FAIL
Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5702/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver
RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5702/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5702/scream/components/eamxx
weaver failed'
+ errors=
+ SA_FAILURES_DETAILS+=
+ [[ 1 == 0 ]]
+ [[ weaver == \m\a\p\p\y ]]
+ set +x
######################################################
FAILS DETECTED:
  SCREAM STANDALONE TESTING FAILED!

######################################################
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/

./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins15640253031249656208.sh
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Sending e-mails to: lbertag@sandia.gov
Finished: FAILURE

@E3SM-Autotester E3SM-Autotester removed the AT: RETEST Force the autotester (AT) to retest the PR label May 15, 2024
@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5424
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS priority:low;diagnostic
PULLREQUESTNUM 2822
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 93ef4a4
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 8ff32a8
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5704
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS priority:low;diagnostic
PULLREQUESTNUM 2822
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 93ef4a4
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 8ff32a8
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)
  • Branch: mahf708/diags/extra-swaci-diags
  • SHA: 93ef4a4
  • Mode: TEST_REPO

Pull Request Author: mahf708

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5424
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS priority:low;diagnostic
PULLREQUESTNUM 2822
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 93ef4a4
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 8ff32a8
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5704
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS priority:low;diagnostic
PULLREQUESTNUM 2822
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 93ef4a4
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 8ff32a8
TEST_REPO_ALIAS SCREAM
SCREAM_PullRequest_Autotester_Mappy # 5424 FAILED (click to see last 100 lines of console output)

383/412 Test #164: p3_sk_tests_fail_omp12 ..................................   Passed    1.92 sec
        Start 180: shoc_tests_omp11
384/412 Test #180: shoc_tests_omp11 ........................................   Passed    1.25 sec
        Start 181: shoc_tests_omp12
385/412 Test #136: p3_tests_fail_omp16 .....................................   Passed    1.94 sec
        Start 152: p3_sk_tests_omp16
386/412 Test #181: shoc_tests_omp12 ........................................   Passed    1.24 sec
        Start 196: shoc_sk_tests_omp11
387/412 Test #196: shoc_sk_tests_omp11 .....................................   Passed    0.52 sec
        Start 197: shoc_sk_tests_omp12
388/412 Test #197: shoc_sk_tests_omp12 .....................................   Passed    0.50 sec
        Start 226: tms_tests_omp8
389/412 Test #226: tms_tests_omp8 ..........................................   Passed    0.26 sec
        Start 227: tms_tests_omp9
390/412 Test #227: tms_tests_omp9 ..........................................   Passed    0.26 sec
        Start 228: tms_tests_omp10
391/412 Test #152: p3_sk_tests_omp16 .......................................   Passed    2.29 sec
        Start 165: p3_sk_tests_fail_omp13
392/412 Test #228: tms_tests_omp10 .........................................   Passed    0.25 sec
        Start 166: p3_sk_tests_fail_omp14
393/412 Test #165: p3_sk_tests_fail_omp13 ..................................   Passed    1.57 sec
        Start 182: shoc_tests_omp13
394/412 Test #166: p3_sk_tests_fail_omp14 ..................................   Passed    1.88 sec
        Start 167: p3_sk_tests_fail_omp15
395/412 Test #182: shoc_tests_omp13 ........................................   Passed    1.26 sec
        Start 198: shoc_sk_tests_omp13
396/412 Test #198: shoc_sk_tests_omp13 .....................................   Passed    0.52 sec
        Start 229: tms_tests_omp11
397/412 Test #167: p3_sk_tests_fail_omp15 ..................................   Passed    1.39 sec
        Start 168: p3_sk_tests_fail_omp16
398/412 Test #229: tms_tests_omp11 .........................................   Passed    0.25 sec
        Start 230: tms_tests_omp12
399/412 Test #230: tms_tests_omp12 .........................................   Passed    0.24 sec
400/412 Test #168: p3_sk_tests_fail_omp16 ..................................   Passed    1.32 sec
        Start 183: shoc_tests_omp14
        Start 199: shoc_sk_tests_omp14
401/412 Test #199: shoc_sk_tests_omp14 .....................................   Passed    0.45 sec
        Start 231: tms_tests_omp13
402/412 Test #231: tms_tests_omp13 .........................................   Passed    0.26 sec
        Start 232: tms_tests_omp14
403/412 Test #232: tms_tests_omp14 .........................................   Passed    0.25 sec
404/412 Test #183: shoc_tests_omp14 ........................................   Passed    1.15 sec
        Start 184: shoc_tests_omp15
405/412 Test #184: shoc_tests_omp15 ........................................   Passed    1.12 sec
        Start 185: shoc_tests_omp16
406/412 Test #185: shoc_tests_omp16 ........................................   Passed    1.11 sec
        Start 200: shoc_sk_tests_omp15
407/412 Test #200: shoc_sk_tests_omp15 .....................................   Passed    0.40 sec
        Start 201: shoc_sk_tests_omp16
408/412 Test #201: shoc_sk_tests_omp16 .....................................   Passed    0.42 sec
        Start 233: tms_tests_omp15
409/412 Test #233: tms_tests_omp15 .........................................   Passed    0.25 sec
        Start 234: tms_tests_omp16
410/412 Test #234: tms_tests_omp16 .........................................   Passed    0.24 sec
411/412 Test #409: homme_shoc_cld_p3_mam_optics_rrtmgp_np4 .................   Passed  133.17 sec
        Start 412: homme_shoc_cld_p3_mam_optics_rrtmgp_np4_vs_np1
412/412 Test #412: homme_shoc_cld_p3_mam_optics_rrtmgp_np4_vs_np1 ..........   Passed    0.08 sec

99% tests passed, 1 tests failed out of 412

Label Time Summary:
PEM = 5.41 secproc (45 tests)
bfbhash = 2.43 sec
proc (1 test)
check = 2.20 secproc (1 test)
cld = 9046.79 sec
proc (60 tests)
cld_fraction = 2.16 secproc (1 test)
cosp = 15.15 sec
proc (4 tests)
diagnostics = 644.42 secproc (26 tests)
dp = 1366.31 sec
proc (7 tests)
driver = 11158.07 secproc (71 tests)
dynamics = 7860.82 sec
proc (61 tests)
fail = 459.49 secproc (33 tests)
io = 626.42 sec
proc (49 tests)
mam4_optics = 2892.70 secproc (21 tests)
nudging = 817.41 sec
proc (8 tests)
p3 = 9782.35 secproc (103 tests)
p3_sk = 558.95 sec
proc (32 tests)
pg2 = 477.61 secproc (10 tests)
physics = 12076.15 sec
proc (187 tests)
remap = 29.24 secproc (4 tests)
rrtmgp = 9995.29 sec
proc (77 tests)
shoc = 9184.12 secproc (74 tests)
spa = 4133.47 sec
proc (43 tests)
surface_coupling = 13.75 secproc (4 tests)
tms = 1843.79 sec
proc (14 tests)

Total Test time (real) = 449.28 sec

The following tests FAILED:
256 - pbl_entrainment_budget (Failed)
Starting analysis on mappy with cmd: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5424/scream/components/eamxx && module purge && module load sems-archive-env acme-env acme-cmake/3.26.3 acme-gcc/11.2.0 sems-archive-git/2.10.1 acme-openmpi/4.1.4 acme-netcdf/4.7.4/acme && export GATOR_INITIAL_MB=4000MB && export PATH=/ascldap/users/jgfouca/packages/valgrind-3.22.0/bin:$PATH && export OMP_PROC_BIND=spread && true && ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m mappy
RUN: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5424/scream/components/eamxx && module purge && module load sems-archive-env acme-env acme-cmake/3.26.3 acme-gcc/11.2.0 sems-archive-git/2.10.1 acme-openmpi/4.1.4 acme-netcdf/4.7.4/acme && export GATOR_INITIAL_MB=4000MB && export PATH=/ascldap/users/jgfouca/packages/valgrind-3.22.0/bin:$PATH && export OMP_PROC_BIND=spread && true && ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m mappy
FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5424/scream/components/eamxx
mappy failed'
./scream/components/eamxx/scripts/jenkins/jenkins_common.sh: line 7: 14792 Terminated $JENKINS_SCRIPT_DIR/jenkins_common_impl.sh 2>&1
14793 Done | tee JENKINS_$DATE_STAMP
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Sending e-mails to: lbertag@sandia.gov
Finished: FAILURE

SCREAM_PullRequest_Autotester_Weaver # 5704 FAILED (click to see last 100 lines of console output)

For compatibility with older versions of CMake, option is clearing the
normal variable 'HAERO_ENABLE_GPU'.
This warning is for project developers. Use -Wno-dev to suppress it.

CMake Warning (dev) at /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5704/scream/externals/haero/CMakeLists.txt:11 (option):
Policy CMP0077 is not set: option() honors normal variables. Run "cmake
--help-policy CMP0077" for policy details. Use the cmake_policy command to
set the policy and suppress this warning.

For compatibility with older versions of CMake, option is clearing the
normal variable 'HAERO_ENABLE_MPI'.
This warning is for project developers. Use -Wno-dev to suppress it.

CMake Warning (dev) at /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5704/scream/externals/haero/CMakeLists.txt:12 (option):
Policy CMP0077 is not set: option() honors normal variables. Run "cmake
--help-policy CMP0077" for policy details. Use the cmake_policy command to
set the policy and suppress this warning.

For compatibility with older versions of CMake, option is clearing the
normal variable 'HAERO_ENABLE_TESTS'.
This warning is for project developers. Use -Wno-dev to suppress it.

-- Configuring with build type: Release
-- Building for CPU
-- Building Haero within another project.
-- Generating project files in build directory: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5704/scream/components/eamxx/ctest-build/release/externals/haero
-- C++ compiler is /projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicxx (GNU 11.3.0)
-- C compiler is /projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicc (GNU 11.3.0)
-- Configuring haero v0.1.0
-- Using double precision floating point numbers
-- Configuring on Linux.
-- Installation prefix is /usr/local
-- MPI is enabled
-- Configuring with build type: Release
-- Generating project files in build directory: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5704/scream/components/eamxx/ctest-build/release/externals/mam4xx
-- C++ compiler is /projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicxx (GNU 11.3.0)
-- C compiler is /projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicc (GNU 11.3.0)
-- Configuring MAM4xx v0.1.0
-- Using double precision floating point numbers
-- Installation prefix is /usr/local
CMake Error at cmake/ScreamUtils.cmake:212 (message):
Could not download cam/scam/iop/DYCOMSrf02_iopfile_4scam.nc, output was:
WARNING:root:wget failed with output: and errput File
/home/projects/e3sm/scream/data/atm/cam/scam/iop/DYCOMSrf02_iopfile_4scam.nc
already there; not retrieving.

Checking for file atm/cam/scam/iop/DYCOMSrf02_iopfile_4scam.nc within
/home/projects/e3sm/scream/data

Input file /home/projects/e3sm/scream/data/atm/cam/scam/iop/DYCOMSrf02_iopfile_4scam.nc needs to be downloaded.
Attempting to download atm/cam/scam/iop/DYCOMSrf02_iopfile_4scam.nc from https://web.lcrc.anl.gov/public/e3sm/inputdata/ with protocol wget
FAILED

Traceback (most recent call last):

File "/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5704/scream/components/eamxx/src/../scripts/check-input", line 45, in <module>
  _main_func(__doc__)
File "/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5704/scream/components/eamxx/src/../scripts/check-input", line 39, in _main_func
  success = check_input(**vars(parse_command_line(sys.argv, description)))
File "/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5704/scream/components/eamxx/scripts/check_input.py", line 55, in check_input
  success = download_file(input_root, the_file)
File "/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5704/scream/components/eamxx/scripts/check_input.py", line 23, in download_file
  from CIME.Servers import SVN

ImportError: cannot import name 'SVN' from 'CIME.Servers'
(/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5704/scream/components/eamxx/scripts/../../../cime/CIME/Servers/init.py)

Call Stack (most recent call first):
src/diagnostics/tests/CMakeLists.txt:76 (GetInputFile)

-- Configuring incomplete, errors occurred!
See also "/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5704/scream/components/eamxx/ctest-build/release/CMakeFiles/CMakeOutput.log".
See also "/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5704/scream/components/eamxx/ctest-build/release/CMakeFiles/CMakeError.log".

Error(s) occurred during test phase
OVERALL STATUS: FAIL
Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5704/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver
RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5704/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5704/scream/components/eamxx
weaver failed
######################################################
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/

./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins1918583283000897028.sh
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Sending e-mails to: lbertag@sandia.gov
Finished: FAILURE

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5428
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS priority:low;diagnostic
PULLREQUESTNUM 2822
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 93ef4a4
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 8ff32a8
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5706
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS priority:low;diagnostic
PULLREQUESTNUM 2822
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 93ef4a4
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 8ff32a8
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)
  • Branch: mahf708/diags/extra-swaci-diags
  • SHA: 93ef4a4
  • Mode: TEST_REPO

Pull Request Author: mahf708

@mahf708 mahf708 added the AT: RETEST Force the autotester (AT) to retest the PR label May 18, 2024
@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5445
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: RETEST;priority:low;diagnostic
PULLREQUESTNUM 2822
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 7dd6802
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 8ff32a8
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5723
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: RETEST;priority:low;diagnostic
PULLREQUESTNUM 2822
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 7dd6802
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 8ff32a8
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)
  • Branch: mahf708/diags/extra-swaci-diags
  • SHA: 7dd6802
  • Mode: TEST_REPO

Pull Request Author: mahf708

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5445
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: RETEST;priority:low;diagnostic
PULLREQUESTNUM 2822
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 7dd6802
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 8ff32a8
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5723
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: RETEST;priority:low;diagnostic
PULLREQUESTNUM 2822
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 7dd6802
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 8ff32a8
TEST_REPO_ALIAS SCREAM

@E3SM-Autotester E3SM-Autotester removed the AT: RETEST Force the autotester (AT) to retest the PR label May 18, 2024
@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
THE LAST COMMIT TO THIS PULL REQUEST HAS NOT BEEN REVIEWED YET!

@E3SM-Autotester
Copy link
Collaborator

All Jobs Finished; status = PASSED, target_sha=5555fa2c22ee08aa2ac6b17c06968c9858d09387, However Inspection must be performed before merge can occur...

@mahf708 mahf708 added the AT: WIP Inform the autotester (AT) that the PR is a work in progress, and should not be tested label May 18, 2024
@tcclevenger
Copy link
Contributor

What is the status of this? It's labeled WIP, is it a part of the output fails you are trying to debug?

@mahf708
Copy link
Contributor Author

mahf708 commented May 20, 2024

What is the status of this? It's labeled WIP, is it a part of the output fails you are trying to debug?

No, it is not part of debugging (the fail happens with/without it). I made it WIP because I didn't want it to keep testing. I decided to use an earlier commit in my runs (so I will cherry-pick it and there is no rush to get this integrated)

Do you think we should merge it?

@tcclevenger
Copy link
Contributor

If you aren't using this version of the diag (and no one else is needing it) then I guess we should leave it unmerged. Re-request reviews if/when you want to merge it.

@mahf708
Copy link
Contributor Author

mahf708 commented May 20, 2024

If you aren't using this version of the diag (and no one else is needing it) then I guess we should leave it unmerged. Re-request reviews if/when you want to merge it.

I do want to use it, so let me rerequest reviews right now and take out the WIP label

@mahf708 mahf708 removed the AT: WIP Inform the autotester (AT) that the PR is a work in progress, and should not be tested label May 20, 2024
@E3SM-Autotester
Copy link
Collaborator

All Jobs Finished; status = PASSED, target_sha=5555fa2c22ee08aa2ac6b17c06968c9858d09387, However Inspection must be performed before merge can occur...

Copy link
Contributor

@bartgol bartgol left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see any mods to scorpio_output.cpp. The user can still request it using PBLEntrainmentBudget as field in the output yaml, but there's no control over the options. If you will ever want to support the other pblinvalg options, you will need to have multiple strings to map to that diag, which means that scorpio_output.cpp will have to handle them, setting the proper values for the diag parameter list.

I'm assuming for now you're just interested in the default temperature-inversion case. Perhaps add a comment in the header file, explaining that the only supported case is temperature-inversion for now?

// Get pseudo_density
add_field<Required>("pseudo_density", scalar2d_layout, Pa, grid_name);
// Get radiation up and down terms
add_field<Required>("SW_flux_dn", scalar2d_layout, W / m * m, grid_name);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Watch out, these units here will resolve to W, since you don't have parentheses!

Note: this will have zero impact on this code working, since I/O does not check the units of the requested fields. But we should fix this, a) to avoid confusion and b) in case I/O starts to check units of the diags required fields.

m_units_map = eadu.units_map;
m_ndiag = eadu.size;

if(eadu.pblinvalg == "temperature-inversion") {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the only possible scenario, since eadu is default-constructed, so it will always have "temperature-inversion" as pblinvalg. In the long run, I'm guessing this code should change into something like

  PBLEntrainmentBudgetDiagUtil eadu;
  m_index_map = eadu.index_map;
  m_units_map = eadu.units_map;
  m_ndiag     = eadu.size;
  eadu.pblinvalg = m_params.get<std::string>("pblinvalg");
  if (eadu.pblinvalg=="temperature-inversion") {
    m_pblinvalg = 0;
  } else if (...) {
  } else {
    EKAT_ERROR_MSG ("Watch out, silly: invalid value for pblinvalg!\n");
  }

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's right...

So, I don't know what to do here. On the one hand, I already implemented three different methods to do this below (all rely on "state inversion" --- an inversion in the gradient of one "state" variable, namely temperature, liquid potential temperature, or water content. There are other more weird ways to determine this, including relationships in derived fluxes and so on, which I could also implement, but my thinking is, we should likely stick to state variables.

The reason I didn't implement this like AeroComCldTop/Bot is because.... I don't know what to name the variables yet 😄 PBLEntrainmentBudgetTemperature? PBLEntrainmentBudget_t? PBLEntrainmentBudget1? Do you have ideas? :D

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have no clue what these variables represent, so I can't advise on this. But it's confusing to have code that is guaranteed to never be executed. I would pick some name for those diagnostics, and just do the mods to support them all. Alternatively, remove all other implementations, and hard-code the version for "temperature-inversion", without giving the idea that this is a configurable option...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it okay with you if I do PBLEntrainmentBudget0, PBLEntrainmentBudget1, etc.?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ugh, if there's no other way, then ok. But I would prefer "Temperature" or "T", as you suggested. The name length is already super long, so no point in being cheap (and cryptic) with the last word...

@E3SM-Autotester
Copy link
Collaborator

The base branch has been updated since the last successful testing.

  • last PASS base branch sha: 5555fa2
  • current base branch sha : f5b824a
    The AutoTester will discard the last PASS, and re-test the PR from scratch

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5449
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS priority:low;diagnostic
PULLREQUESTNUM 2822
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 7dd6802
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 8ff32a8
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5727
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS priority:low;diagnostic
PULLREQUESTNUM 2822
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 7dd6802
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 8ff32a8
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)
  • Branch: mahf708/diags/extra-swaci-diags
  • SHA: 7dd6802
  • Mode: TEST_REPO

Pull Request Author: mahf708

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5449
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS priority:low;diagnostic
PULLREQUESTNUM 2822
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 7dd6802
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 8ff32a8
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5727
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS priority:low;diagnostic
PULLREQUESTNUM 2822
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 7dd6802
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 8ff32a8
TEST_REPO_ALIAS SCREAM
SCREAM_PullRequest_Autotester_Mappy # 5449 FAILED (click to see last 100 lines of console output)

Starting SHAREDLIB_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 with 1 procs
Starting SETUP for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 with 1 procs
Finished SETUP for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 in 2.620029 seconds (PASS)
Starting SHAREDLIB_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 with 1 procs
Finished SETUP for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 2.595107 seconds (PASS)
Starting SHAREDLIB_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 with 1 procs
Finished SETUP for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 in 2.094882 seconds (PASS)
Finished SETUP for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 2.540606 seconds (PASS)
Starting SHAREDLIB_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 with 1 procs
Starting SHAREDLIB_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 with 1 procs
Finished SETUP for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 in 2.547677 seconds (PASS)
Starting SHAREDLIB_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 with 1 procs
Finished SHAREDLIB_BUILD for test SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics in 55.003748 seconds (PASS)
Starting MODEL_BUILD for test SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics with 16 procs
Finished SHAREDLIB_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 in 54.854227 seconds (PASS)
Starting MODEL_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 with 16 procs
Finished SHAREDLIB_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 in 54.994124 seconds (PASS)
Starting MODEL_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 with 16 procs
Finished SHAREDLIB_BUILD for test ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu in 70.565083 seconds (PASS)
Starting MODEL_BUILD for test ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu with 16 procs
Finished SHAREDLIB_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 in 69.843913 seconds (PASS)
Finished SHAREDLIB_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 80.361874 seconds (PASS)
Finished SHAREDLIB_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 166.306256 seconds (PASS)
Finished SHAREDLIB_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 in 194.477449 seconds (PASS)
Starting MODEL_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 with 16 procs
Finished MODEL_BUILD for test SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics in 177.407787 seconds (PASS)
Starting RUN for test SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics in 1.082796 seconds (PEND). [COMPLETED 1 of 8]
Starting MODEL_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 with 16 procs
Finished MODEL_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 in 177.510594 seconds (PASS)
Starting RUN for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 with 1 proc on interactive node and 64 procs on compute nodes
Finished MODEL_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 in 175.463969 seconds (PASS)
Starting RUN for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 in 0.939526 seconds (PEND). [COMPLETED 2 of 8]
Starting MODEL_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 with 16 procs
Finished RUN for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 in 0.926369 seconds (PEND). [COMPLETED 3 of 8]
Starting MODEL_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 with 16 procs
Finished MODEL_BUILD for test ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu in 309.983637 seconds (PASS)
Starting RUN for test ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu with 1 proc on interactive node and 16 procs on compute nodes
Finished RUN for test ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu in 8.046538 seconds (PEND). [COMPLETED 4 of 8]
Finished MODEL_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 in 325.823139 seconds (PASS)
Starting RUN for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 in 1.112494 seconds (PEND). [COMPLETED 5 of 8]
Finished MODEL_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 346.052372 seconds (PASS)
Starting RUN for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 0.934709 seconds (PEND). [COMPLETED 6 of 8]
Finished MODEL_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 445.725169 seconds (PASS)
Starting RUN for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 1.717432 seconds (PEND). [COMPLETED 7 of 8]
Finished MODEL_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 in 532.983427 seconds (PASS)
Starting RUN for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 in 0.954512 seconds (PEND). [COMPLETED 8 of 8]
Waiting for tests to finish
PASS ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240520_140715_66w5qf
PASS ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240520_140715_66w5qf
PASS ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240520_140715_66w5qf
PASS ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240520_140715_66w5qf
PASS ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240520_140715_66w5qf
PASS PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240520_140715_66w5qf
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240520_140715_66w5qf
PASS SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240520_140715_66w5qf
test-scheduler took 808.4955139160156 seconds'
+ [[ 0 != 0 ]]
+ set +x
######################################################
FAILS DETECTED:
  SCREAM STANDALONE TESTING FAILED!

######################################################
Build step 'Execute shell' marked build as failure
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 93932 killed;
[ssh-agent] Stopped.
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/

./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs

squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel

[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins17299583503081189101.sh
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Sending e-mails to: lbertag@sandia.gov
Finished: FAILURE

SCREAM_PullRequest_Autotester_Weaver # 5727 FAILED (click to see last 100 lines of console output)

/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5727/scream/externals/YAKL/src/extensions/YAKL_netcdf.h: In member function '\''void yakl::SimpleNetCDF::NcVar::print()'\'':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5727/scream/externals/YAKL/src/extensions/YAKL_netcdf.h:213:19: warning: comparison of integer expressions of different signedness: '\''int'\'' and '\''std::vector::size_type'\'' {aka '\''long unsigned int'\''} [-Wsign-compare]
  213 |         for (int i=0; i < dims.size(); i++) {
      |                 ~~^~~~~~~~~~~~~~~
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5727/scream/externals/YAKL/src/extensions/YAKL_netcdf.h: In member function '\''yakl::SimpleNetCDF::NcVar yakl::SimpleNetCDF::NcFile::addVar(std::string, int, std::vector&)'\'':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5727/scream/externals/YAKL/src/extensions/YAKL_netcdf.h:302:19: warning: comparison of integer expressions of different signedness: '\''int'\'' and '\''std::vector::size_type'\'' {aka '\''long unsigned int'\''} [-Wsign-compare]
  302 |         for (int i=0; i < dims.size(); i++) { dimids[i] = dims[i].getId(); }
      |                 ~~^~~~~~~~~~~~~
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5727/scream/externals/YAKL/src/extensions/YAKL_netcdf.h: In instantiation of '\''void yakl::SimpleNetCDF::read(yakl::Array&, std::string) [with T = double; int rank = 2; int myMem = 1; int myStyle = 2; std::string = std::__cxx11::basic_string]'\'':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5727/scream/components/eamxx/src/physics/rrtmgp/rrtmgp_test_utils.cpp:205:8:   required from here
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5727/scream/externals/YAKL/src/extensions/YAKL_netcdf.h:531:19: warning: comparison of integer expressions of different signedness: '\''int'\'' and '\''std::vector::size_type'\'' {aka '\''long unsigned int'\''} [-Wsign-compare]
  531 |           for (int i=0; i < varDims.size(); i++) { dimSizes[i] = varDims[varDims.size()-1-i].getSize(); }
      |                 ~~^~~~~~~~~~~~~~~~
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5727/scream/externals/YAKL/src/extensions/YAKL_netcdf.h:535:19: warning: comparison of integer expressions of different signedness: '\''int'\'' and '\''std::vector::size_type'\'' {aka '\''long unsigned int'\''} [-Wsign-compare]
  535 |           for (int i=0; i < dimSizes.size(); i++) {
      |                 ~~^~~~~~~~~~~~~~~~~
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5727/scream/externals/YAKL/src/extensions/YAKL_netcdf.h:536:17: warning: comparison of integer expressions of different signedness: '\''__gnu_cxx::__alloc_traits, int>::value_type'\'' {aka '\''int'\''} and '\''yakl::index_t'\'' {aka '\''long unsigned int'\''} [-Wsign-compare]
  536 |             if (dimSizes[i] != arr.extent(i)) {
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5727/scream/externals/YAKL/src/extensions/YAKL_netcdf.h: In instantiation of '\''void yakl::SimpleNetCDF::write(const yakl::Array&, std::string, std::vector >) [with T = double; int rank = 2; int myMem = 1; int myStyle = 2; std::string = std::__cxx11::basic_string]'\'':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5727/scream/components/eamxx/src/physics/rrtmgp/rrtmgp_test_utils.cpp:248:9:   required from here
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5727/scream/externals/YAKL/src/extensions/YAKL_netcdf.h:406:19: warning: comparison of integer expressions of different signedness: '\''int'\'' and '\''std::vector::size_type'\'' {aka '\''long unsigned int'\''} [-Wsign-compare]
  406 |         for (int i=0; i < varDims.size(); i++) {
      |                 ~~^~~~~~~~~~~~~~~~
nvcc_internal_extended_lambda_implementation: In static member function '\''static void* __nv_hdl_wrapper_t::manager::do_copy(void*) [with Lambda = rrtmgpTest::dummy_clouds(CloudOptics&, real2d&, real2d&, real2d&, real2d&, real2d&, real2d&, real2d&)::; Tag = __nv_dl_tag&, yakl::Array&, yakl::Array&, yakl::Array&, yakl::Array&, yakl::Array&, yakl::Array&), rrtmgpTest::dummy_clouds, 1>; OpFuncR = void; OpFuncArgs = {int, int}; F1 = yakl::Array; F2 = yakl::Array; F3 = yakl::Array; F4 = yakl::Array; F5 = yakl::Array; F6 = yakl::Array; F7 = double; F8 = yakl::Array; F9 = double]'\'':
nvcc_internal_extended_lambda_implementation:955:19: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5727/scream/components/eamxx/src/physics/rrtmgp/tests/generate_baseline.cpp:2:10: fatal error: physics/rrtmgp/mo_garand_atmos_io.h: No such file or directory
    2 | #include "physics/rrtmgp/mo_garand_atmos_io.h"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [src/physics/rrtmgp/tests/CMakeFiles/generate_baseline.dir/build.make:76: src/physics/rrtmgp/tests/CMakeFiles/generate_baseline.dir/generate_baseline.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:3954: src/physics/rrtmgp/tests/CMakeFiles/generate_baseline.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5727/scream/components/eamxx/src/physics/spa/tests/spa_main_test.cpp: In lambda function:
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5727/scream/components/eamxx/src/physics/spa/tests/spa_main_test.cpp:202:72: note: parameter passing for argument of type '\''std::pair'\'' when C++17 is enabled changed to match C++14 in GCC 10.1
  202 |   auto max_min = [&] (const Real v0, const Real v1) -> std::pair {
      |                                                                        ^~~~~~               
nvcc_internal_extended_lambda_implementation: In static member function '\''static void* __nv_hdl_wrapper_t::manager::do_copy(void*) [with Lambda = scream::spa::SPAFunctions >::perform_vertical_interpolation(scream::spa::SPAFunctions >::view_2d >&, scream::spa::SPAFunctions >::view_2d >&, const scream::spa::SPAFunctions >::SPAData&, const scream::spa::SPAFunctions >::SPAData&)::; Tag = __nv_dl_tag**, Kokkos::LayoutRight, Kokkos::Device, Kokkos::MemoryTraits<0> >&, const Kokkos::View**, Kokkos::LayoutRight, Kokkos::Device, Kokkos::MemoryTraits<0> >&, const scream::spa::SPAFunctions >::SPAData&, const scream::spa::SPAFunctions >::SPAData&), scream::spa::SPAFunctions >::perform_vertical_interpolation, 1>; OpFuncR = void; OpFuncArgs = {const Kokkos::Impl::CudaTeamMember&}; F1 = ekat::LinInterp >; F2 = const Kokkos::View**, Kokkos::LayoutRight, Kokkos::Device, Kokkos::MemoryTraits<0> >; F3 = const Kokkos::View**, Kokkos::LayoutRight, Kokkos::Device, Kokkos::MemoryTraits<0> >]'\'':
nvcc_internal_extended_lambda_implementation:337:19: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
nvcc_internal_extended_lambda_implementation: In member function '\''void scream::control::IntensiveObservationPeriod::set_fields_from_iop_data(scream::control::IntensiveObservationPeriod::field_mgr_ptr)'\'':
nvcc_internal_extended_lambda_implementation:1311:339: warning: '\''ps_iop'\'' may be used uninitialized in this function [-Wmaybe-uninitialized]
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5727/scream/components/eamxx/src/control/intensive_observation_period.cpp:773:6: note: '\''ps_iop'\'' was declared here
  773 |   Real ps_iop;
      |      ^~~~~~
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5727/scream/components/eamxx/src/share/field/field_alloc_prop.hpp: In copy constructor '\''scream::FieldAllocProp::FieldAllocProp(const scream::FieldAllocProp&)'\'':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5727/scream/components/eamxx/src/share/field/field_alloc_prop.hpp:89:1: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
   89 |   FieldAllocProp (const FieldAllocProp&) = default;
      | ^ ~~~~~~~~~~~~
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5727/scream/externals/ekat/extern/kokkos/core/src/KokkosExp_MDRangePolicy.hpp: In copy constructor '\''Kokkos::MDRangePolicy, Kokkos::IndexType >::MDRangePolicy(const Kokkos::MDRangePolicy, Kokkos::IndexType >&)'\'':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5727/scream/externals/ekat/extern/kokkos/core/src/KokkosExp_MDRangePolicy.hpp:185:8: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
  185 | struct MDRangePolicy : public Kokkos::Impl::PolicyTraits {
      |        ^~~~~~~~~~~~~
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5727/scream/externals/ekat/extern/kokkos/core/src/KokkosExp_MDRangePolicy.hpp: In copy constructor '\''Kokkos::MDRangePolicy, Kokkos::IndexType >::MDRangePolicy(const Kokkos::MDRangePolicy, Kokkos::IndexType >&)'\'':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5727/scream/externals/ekat/extern/kokkos/core/src/KokkosExp_MDRangePolicy.hpp:185:8: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
  185 | struct MDRangePolicy : public Kokkos::Impl::PolicyTraits {
      |        ^~~~~~~~~~~~~
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5727/scream/components/eamxx/src/share/field/field_alloc_prop.hpp: In copy constructor '\''scream::FieldAllocProp::FieldAllocProp(const scream::FieldAllocProp&)'\'':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5727/scream/components/eamxx/src/share/field/field_alloc_prop.hpp:89:1: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
   89 |   FieldAllocProp (const FieldAllocProp&) = default;
      | ^ ~~~~~~~~~~~~
make: *** [Makefile:166: all] Error 2
===============================================================================
Generating baseline for test full_sp_debug with config '\''cmake -C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5727/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DISABLE_TPL_WARNINGS='\''ON'\'' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_ONLY_GENERATE_BASELINES=ON -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_sp_debug -DSCREAM_TEST_MAX_TOTAL_THREADS=1'\''
===============================================================================
RUN: cmake -C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5727/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DISABLE_TPL_WARNINGS='\''ON'\'' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_ONLY_GENERATE_BASELINES=ON -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_sp_debug -DSCREAM_TEST_MAX_TOTAL_THREADS=1 /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5727/scream/components/eamxx
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5727/scream/components/eamxx/ctest-build/tas_baseline_build/full_sp_debug
RUN: taskset -c 52-103 sh -c '\''make -j52'\''
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5727/scream/components/eamxx/ctest-build/tas_baseline_build/full_sp_debug
RUN: ctest -j1 -L baseline_gen --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5727/scream/components/eamxx/ctest-build/tas_baseline_build/full_sp_debug/ctest_resource_file.json
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5727/scream/components/eamxx/ctest-build/tas_baseline_build/full_sp_debug

Error(s) occurred during baselines generation phase
OVERALL STATUS: FAIL
Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5727/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver
RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5727/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5727/scream/components/eamxx
weaver failed'

  • errors=
  • SA_FAILURES_DETAILS+=
  • [[ 1 == 0 ]]
  • [[ weaver == \m\a\p\p\y ]]
  • set +x
    ######################################################
    FAILS DETECTED:
    SCREAM STANDALONE TESTING FAILED!

######################################################
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/

./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins16069266164443868222.sh
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Sending e-mails to: lbertag@sandia.gov
Finished: FAILURE

@mahf708 mahf708 added the AT: RETEST Force the autotester (AT) to retest the PR label May 22, 2024
@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5456
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: RETEST;priority:low;diagnostic
PULLREQUESTNUM 2822
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 7dd6802
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 8ff32a8
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5734
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: RETEST;priority:low;diagnostic
PULLREQUESTNUM 2822
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 7dd6802
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 8ff32a8
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)
  • Branch: mahf708/diags/extra-swaci-diags
  • SHA: 7dd6802
  • Mode: TEST_REPO

Pull Request Author: mahf708

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5456
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: RETEST;priority:low;diagnostic
PULLREQUESTNUM 2822
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 7dd6802
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 8ff32a8
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5734
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: RETEST;priority:low;diagnostic
PULLREQUESTNUM 2822
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 7dd6802
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 8ff32a8
TEST_REPO_ALIAS SCREAM
SCREAM_PullRequest_Autotester_Mappy # 5456 FAILED (click to see last 100 lines of console output)

    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240522_041157_x69osm
PASS ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240522_041157_x69osm
PASS ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240522_041157_x69osm
PASS ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240522_041157_x69osm
PASS ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240522_041157_x69osm
PASS PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240522_041157_x69osm
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240522_041157_x69osm
FAIL SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240522_041157_x69osm
test-scheduler took 800.0068647861481 seconds'
+ errors='Waiting for tests to finish
PASS ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240522_041157_x69osm
PASS ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240522_041157_x69osm
PASS ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240522_041157_x69osm
PASS ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240522_041157_x69osm
PASS ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240522_041157_x69osm
PASS PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240522_041157_x69osm
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240522_041157_x69osm
FAIL SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240522_041157_x69osm
test-scheduler took 800.0068647861481 seconds'
+ V1_FAILURES_DETAILS+='Waiting for tests to finish
PASS ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240522_041157_x69osm
PASS ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240522_041157_x69osm
PASS ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240522_041157_x69osm
PASS ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240522_041157_x69osm
PASS ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240522_041157_x69osm
PASS PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240522_041157_x69osm
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240522_041157_x69osm
FAIL SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240522_041157_x69osm
test-scheduler took 800.0068647861481 seconds'
+ set +x
######################################################
FAILS DETECTED:
  SCREAM STANDALONE TESTING FAILED!

SCREAM V1 TESTING FAILED!
Waiting for tests to finish
PASS ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240522_041157_x69osm
PASS ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 RUN
Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240522_041157_x69osm
PASS ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 RUN
Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240522_041157_x69osm
PASS ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 RUN
Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240522_041157_x69osm
PASS ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu RUN
Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240522_041157_x69osm
PASS PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240522_041157_x69osm
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics RUN
Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240522_041157_x69osm
FAIL SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 (phase RUN)
Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240522_041157_x69osm
test-scheduler took 800.0068647861481 seconds
######################################################
Build step 'Execute shell' marked build as failure
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 107676 killed;
[ssh-agent] Stopped.
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/

./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs

squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel

[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins2336358104318312431.sh
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Sending e-mails to: lbertag@sandia.gov
Finished: FAILURE

SCREAM_PullRequest_Autotester_Weaver # 5734 PASSED (click to see last 100 lines of console output)

116/133 Test #116: shoc_p3_nudging_glob_novert .............................   Passed    2.48 sec
        Start 117: homme_shoc_cld_p3_rrtmgp_np1
117/133 Test #117: homme_shoc_cld_p3_rrtmgp_np1 ............................   Passed    7.05 sec
        Start 118: homme_shoc_cld_p3_rrtmgp_baseline_cmp
118/133 Test #118: homme_shoc_cld_p3_rrtmgp_baseline_cmp ...................   Passed    0.10 sec
        Start 119: homme_shoc_cld_p3_rrtmgp_pg2_np1
119/133 Test #119: homme_shoc_cld_p3_rrtmgp_pg2_np1 ........................   Passed    6.91 sec
        Start 120: homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp
120/133 Test #120: homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp ...............   Passed    0.10 sec
        Start 121: model_baseline
121/133 Test #121: model_baseline ..........................................   Passed    7.74 sec
        Start 122: model_initial
122/133 Test #122: model_initial ...........................................   Passed    4.47 sec
        Start 123: model_restart
123/133 Test #123: model_restart ...........................................   Passed    5.42 sec
        Start 124: restarted_vs_monolithic_check_np1
124/133 Test #124: restarted_vs_monolithic_check_np1 .......................   Passed    0.09 sec
        Start 125: homme_shoc_cld_spa_p3_rrtmgp_np1
125/133 Test #125: homme_shoc_cld_spa_p3_rrtmgp_np1 ........................   Passed    7.80 sec
        Start 126: homme_shoc_cld_spa_p3_rrtmgp_baseline_cmp
126/133 Test #126: homme_shoc_cld_spa_p3_rrtmgp_baseline_cmp ...............   Passed    0.12 sec
        Start 127: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1
127/133 Test #127: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1 ..............   Passed    9.80 sec
        Start 128: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1
128/133 Test #128: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1 ...   Passed    1.50 sec
        Start 129: homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp
129/133 Test #129: homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp .....   Passed    0.56 sec
        Start 130: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np1
130/133 Test #130: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np1 .................   Passed   10.23 sec
        Start 131: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_baseline_cmp
131/133 Test #131: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_baseline_cmp ........   Passed    0.09 sec
        Start 132: homme_shoc_cld_p3_mam_optics_rrtmgp_np1
132/133 Test #132: homme_shoc_cld_p3_mam_optics_rrtmgp_np1 .................   Passed    6.91 sec
        Start 133: homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp
133/133 Test #133: homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp ........   Passed    0.14 sec

100% tests passed, 0 tests failed out of 133

Label Time Summary:
baseline_cmp = 86.97 secproc (16 tests)
baseline_gen = 167.29 sec
proc (18 tests)
bfbhash = 0.69 secproc (1 test)
check = 0.76 sec
proc (1 test)
cld = 12.76 secproc (3 tests)
cld_fraction = 2.74 sec
proc (1 test)
cxx baseline_cmp = 6.56 secproc (2 tests)
diagnostics = 38.73 sec
proc (23 tests)
driver = 30.75 secproc (8 tests)
dynamics = 5.13 sec
proc (3 tests)
fail = 34.62 secproc (4 tests)
io = 44.09 sec
proc (13 tests)
mam4_optics = 6.40 secproc (1 test)
nudging = 9.59 sec
proc (2 tests)
p3 = 137.79 secproc (7 tests)
p3_sk = 121.48 sec
proc (2 tests)
physics = 282.97 secproc (19 tests)
remap = 4.62 sec
proc (1 test)
rrtmgp = 27.47 secproc (9 tests)
shoc = 16.72 sec
proc (7 tests)
spa = 9.08 secproc (4 tests)
surface_coupling = 3.23 sec
proc (1 test)

Total Test time (real) = 675.10 sec

Testing '''c7eefe345af2ff65fd01c7dfa272c4046c5a7f23''' for test '''full_sp_debug'''

RUN: taskset -c 52-103 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5734/scream/components/eamxx/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5734/scream/components/eamxx/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5734/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5734/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_sp_debug" '''
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5734/scream/components/eamxx/ctest-build/full_sp_debug

Testing '''c7eefe345af2ff65fd01c7dfa272c4046c5a7f23''' for test '''release'''

RUN: taskset -c 104-155 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5734/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5734/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5734/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5734/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release" '''
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5734/scream/components/eamxx/ctest-build/release

Testing '''c7eefe345af2ff65fd01c7dfa272c4046c5a7f23''' for test '''full_debug'''

RUN: taskset -c 0-51 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5734/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5734/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5734/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5734/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug" '''
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5734/scream/components/eamxx/ctest-build/full_debug
OVERALL STATUS: PASS
Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5734/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver
RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5734/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5734/scream/components/eamxx
Completed analysis on weaver'

  • [[ 0 != 0 ]]
  • [[ 1 == 0 ]]
  • [[ weaver == \m\a\p\p\y ]]
  • set +x
    Performing Post build task...
    Match found for : : True
    Logical operation result is TRUE
    Running script : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/

./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins8734200918454060680.sh
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Sending e-mails to: lbertag@sandia.gov
Finished: SUCCESS

@E3SM-Autotester E3SM-Autotester removed the AT: RETEST Force the autotester (AT) to retest the PR label May 22, 2024
@mahf708 mahf708 added the AT: WIP Inform the autotester (AT) that the PR is a work in progress, and should not be tested label May 29, 2024
@mahf708
Copy link
Contributor Author

mahf708 commented Jun 3, 2024

WIP status: I will try to merge this after upstream merge to bring kokkos 4.2. The PR doesn't need kokkos 4.2, but I can edit some parts in it to take advantage of kokkos 4.2

@mahf708 mahf708 marked this pull request as draft June 3, 2024 20:55
@mahf708
Copy link
Contributor Author

mahf708 commented Jul 8, 2024

Update: Lost motivation/drive to finish this PR. Closing. People interested in these diagnostics can try to incorporate them. I will revisit later when I have more clarity and bandwidth.

@mahf708 mahf708 closed this Jul 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
AT: WIP Inform the autotester (AT) that the PR is a work in progress, and should not be tested diagnostic priority:low
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants