Enable pyeamxx testing in SP builds #2885

bartgol · 2024-06-28T18:46:44Z

We were not unit testing pyeamxx, so enable it in SP builds (which are faster, since there's no dynamics or rrtmgp).

~~Also, add a documentation page for pyeamxx to our docs~~ Removed docs commit, since we'll do it in another PR.

E3SM-Autotester · 2024-06-28T23:04:43Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5601
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE;documentation;AT: Skip v1 Testing;pyeamxx
PULLREQUESTNUM	2885
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`3d14934`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`a1646c8`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5850
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE;documentation;AT: Skip v1 Testing;pyeamxx
PULLREQUESTNUM	2885
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`3d14934`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`a1646c8`
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: bartgol

E3SM-Autotester · 2024-06-28T23:17:03Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5601
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE;documentation;AT: Skip v1 Testing;pyeamxx
PULLREQUESTNUM	2885
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`3d14934`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`a1646c8`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5850
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE;documentation;AT: Skip v1 Testing;pyeamxx
PULLREQUESTNUM	2885
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`3d14934`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`a1646c8`
TEST_REPO_ALIAS	SCREAM

SCREAM_PullRequest_Autotester_Mappy # 5601 FAILED (click to see last 100 lines of console output)


+++ export _ModuleTable001_
+++ _ModuleTable002_=ZW1zLWFyY2hpdmUvdXRpbGl0eS9zZW1zLWFyY2hpdmUtZ2l0LzIuMTAuMSIsWyJmdWxsTmFtZSJdPSJzZW1zLWFyY2hpdmUtZ2l0LzIuMTAuMSIsWyJsb2FkT3JkZXIiXT0yLHByb3BUPXt9LFsic3RhY2tEZXB0aCJdPTAsWyJzdGF0dXMiXT0iYWN0aXZlIixbInVzZXJOYW1lIl09InNlbXMtYXJjaGl2ZS1naXQvMi4xMC4xIix9LH0sbXBhdGhBPXsiL3Byb2plY3RzL3NlbXMvbW9kdWxlZmlsZXMvcmhlbDcteDg2XzY0L3NlbXMtYXJjaGl2ZS9kZXZwYWNrIiwiL3Byb2plY3RzL3NlbXMvbW9kdWxlZmlsZXMvcmhlbDcteDg2XzY0L3NlbXMtYXJjaGl2ZS9jb21waWxlciIsIi9wcm9qZWN0cy9zZW1zL21vZHVsZWZpbGVzL3JoZWw3LXg4Nl82NC9zZW1zLWFyY2hpdmUvdHBsIiwi
+++ export _ModuleTable002_
+++ _ModuleTable003_=L3Byb2plY3RzL3NlbXMvbW9kdWxlZmlsZXMvcmhlbDcteDg2XzY0L3NlbXMtYXJjaGl2ZS91dGlsaXR5IiwiL3Byb2plY3RzL3NlbXMvbW9kdWxlZmlsZXMvcHJvamVjdHMiLCIvcHJvamVjdHMvc2Vtcy9jZWUtc2llcnJhLW1vZHVsZXMiLCIvcHJvamVjdHMvc2Vtcy9tb2R1bGVmaWxlcy9yaGVsNy14ODZfNjQvc2Vtcy9saW51eC1yaGVsNy14ODZfNjQvY29tcGlsZXJzIiwiL3Byb2plY3RzL3NlbXMvbW9kdWxlZmlsZXMvcmhlbDcteDg2XzY0L3NlbXMvbGludXgtcmhlbDcteDg2XzY0L0NvcmUiLCIvcHJvamVjdHMvc2Vtcy9tb2R1bGVmaWxlcy9yaGVsNy14ODZfNjQvc2Vtcy9saW51eC1yaGVsNy14ODZfNjQvcHJvamVjdC1tb2R1bGVmaWxlcyIsIi9wcm9qZWN0cy9hdWUv
+++ export _ModuleTable003_
+++ _ModuleTable004_=bW9kdWxlcy9jZWUveDg2XzY0L3JoZWw3IiwiL3Vzci9zaGFyZS9Nb2R1bGVzL21vZHVsZWZpbGVzIiwiL2V0Yy9tb2R1bGVmaWxlcyIsfSxbInN5c3RlbUJhc2VNUEFUSCJdPSIvdXNyL3NoYXJlL01vZHVsZXMvbW9kdWxlZmlsZXM6L2V0Yy9tb2R1bGVmaWxlcyIsfQ==
+++ export _ModuleTable004_
+++ _ModuleTable_Sz_=4
+++ export _ModuleTable_Sz_
++ source /ascldap/users/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5601/scream/components/eamxx/scripts/jenkins/sandia_son_proxy
+++ export http_proxy=http://proxy.sandia.gov:80
+++ http_proxy=http://proxy.sandia.gov:80
+++ export RSYNC_PROXY=proxy.sandia.gov:80
+++ RSYNC_PROXY=proxy.sandia.gov:80
+++ export rsync_proxy=proxy.sandia.gov:80
+++ rsync_proxy=proxy.sandia.gov:80
+++ export HTTPS_PROXY=http://proxy.sandia.gov:80
+++ HTTPS_PROXY=http://proxy.sandia.gov:80
+++ export https_proxy=http://proxy.sandia.gov:80
+++ https_proxy=http://proxy.sandia.gov:80
+++ export HTTP_PROXY=http://proxy.sandia.gov:80
+++ HTTP_PROXY=http://proxy.sandia.gov:80
++ SCREAM_MACHINE=mappy
+ [[ 0 == 1 ]]
+ [[ 0 == 1 ]]
+ [[ 0 == 1 ]]
++ whoami
+ [[ e3sm-jenkins == \e\3\s\m\-\j\e\n\k\i\n\s ]]
+ git config --local user.email jenkins@ignore.com
+ git config --local user.name 'Jenkins Jenkins'
+ declare -i fails=0
+ BASELINES_DIR=AUTO
+ TAS_ARGS='--baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m $machine'
+ [[ mappy == \p\m\-\g\p\u ]]
+ set +e
+ '[' -n 2885 ']'
+ is_at_run=1
+ SA_FAILURES_DETAILS=
+ '[' 1 -eq 1 ']'
++ ./scripts/gather-all-data './scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m $machine' -l -m mappy
Traceback (most recent call last):
  File "./scripts/test-all-scream", line 40, in 
    from test_factory import get_test_name_dict
  File "/home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5601/scream/components/eamxx/scripts/test_factory.py", line 105
    ("EAMXX_ENABLE_PYBIND", "True"],
                                  ^
SyntaxError: closing parenthesis ']' does not match opening parenthesis '('
+ this_output='Starting analysis on mappy with cmd: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5601/scream/components/eamxx && module purge && module load sems-archive-env acme-env acme-cmake/3.26.3 acme-gcc/11.2.0 sems-archive-git/2.10.1 acme-openmpi/4.1.4 acme-netcdf/4.7.4/acme && export GATOR_INITIAL_MB=4000MB && export PATH=/ascldap/users/jgfouca/packages/valgrind-3.22.0/bin:$PATH && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m mappy
RUN: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5601/scream/components/eamxx && module purge && module load sems-archive-env acme-env acme-cmake/3.26.3 acme-gcc/11.2.0 sems-archive-git/2.10.1 acme-openmpi/4.1.4 acme-netcdf/4.7.4/acme && export GATOR_INITIAL_MB=4000MB && export PATH=/ascldap/users/jgfouca/packages/valgrind-3.22.0/bin:$PATH && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m mappy
FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5601/scream/components/eamxx
mappy failed'
+ [[ 1 != 0 ]]
+ fails=0+1
+ sa_fail=1
+ [[ 1 == 1 ]]
++ echo 'Starting analysis on mappy with cmd: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5601/scream/components/eamxx && module purge && module load sems-archive-env acme-env acme-cmake/3.26.3 acme-gcc/11.2.0 sems-archive-git/2.10.1 acme-openmpi/4.1.4 acme-netcdf/4.7.4/acme && export GATOR_INITIAL_MB=4000MB && export PATH=/ascldap/users/jgfouca/packages/valgrind-3.22.0/bin:$PATH && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m mappy
RUN: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5601/scream/components/eamxx && module purge && module load sems-archive-env acme-env acme-cmake/3.26.3 acme-gcc/11.2.0 sems-archive-git/2.10.1 acme-openmpi/4.1.4 acme-netcdf/4.7.4/acme && export GATOR_INITIAL_MB=4000MB && export PATH=/ascldap/users/jgfouca/packages/valgrind-3.22.0/bin:$PATH && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m mappy
FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5601/scream/components/eamxx
mappy failed'
++ grep -m1 -A 100000 'Build type '
+ errors=
+ SA_FAILURES_DETAILS+=
+ [[ 1 == 0 ]]
+ [[ mappy == \m\a\p\p\y ]]
+ [[ 0 == 1 ]]
+ [[ -z '' ]]
+ [[ 1 == 1 ]]
+ [[ 0 == 1 ]]
+ [[ 0 == 1 ]]
+ echo 'SCREAM v1 tests were skipped, since the Github label '\''AT: Skip v1 Testing'\'' was found.\n'
SCREAM v1 tests were skipped, since the Github label 'AT: Skip v1 Testing' was found.\n
+ set +x
######################################################
FAILS DETECTED:
  SCREAM STANDALONE TESTING FAILED!
######################################################

Build step 'Execute shell' marked build as failure

$ ssh-agent -k

unset SSH_AUTH_SOCK;

unset SSH_AGENT_PID;

echo Agent pid 114116 killed;

[ssh-agent] Stopped.

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs
squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel
[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins16045377177382129316.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE

SCREAM_PullRequest_Autotester_Weaver # 5850 FAILED (click to see last 100 lines of console output)


+++ __LMOD_REF_COUNT_PATH='/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe/bin:1;/opt/lsf/10.1/linux3.10-glibc2.17-ppc64le/etc:1;/opt/lsf/10.1/linux3.10-glibc2.17-ppc64le/bin:1;/usr/local/bin:1;/usr/bin:1;/usr/local/sbin:1;/usr/sbin:1;/home/e3sm-jenkins/.local/bin:1;/home/e3sm-jenkins/bin:1'
+++ export __LMOD_REF_COUNT_PATH
+++ PATH=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe/bin:/opt/lsf/10.1/linux3.10-glibc2.17-ppc64le/etc:/opt/lsf/10.1/linux3.10-glibc2.17-ppc64le/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/e3sm-jenkins/.local/bin:/home/e3sm-jenkins/bin
+++ export PATH
+++ __LMOD_REF_COUNT_PKG_CONFIG_PATH=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe/lib/pkgconfig:1
+++ export __LMOD_REF_COUNT_PKG_CONFIG_PATH
+++ PKG_CONFIG_PATH=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe/lib/pkgconfig
+++ export PKG_CONFIG_PATH
+++ PYTHON_BIN=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe/bin
+++ export PYTHON_BIN
+++ PYTHON_INC=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe/include
+++ export PYTHON_INC
+++ PYTHON_LIB=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe/lib
+++ export PYTHON_LIB
+++ PYTHON_ROOT=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe
+++ export PYTHON_ROOT
+++ PYTHON_VERSION=3.10.8
+++ export PYTHON_VERSION
+++ _LMFILES_=/projects/ppc64le-pwr9-rhel8/modulefiles/lmod/utilities/linux-rhel8-ppc64le/Core/python/3.10.8.lua
+++ export _LMFILES_
+++ _ModuleTable001_=X01vZHVsZVRhYmxlXyA9IHsKTVR2ZXJzaW9uID0gMywKY19yZWJ1aWxkVGltZSA9IGZhbHNlLApjX3Nob3J0VGltZSA9IGZhbHNlLApkZXB0aFQgPSB7fSwKZmFtaWx5ID0ge30sCm1UID0gewpweXRob24gPSB7CmZuID0gIi9wcm9qZWN0cy9wcGM2NGxlLXB3cjktcmhlbDgvbW9kdWxlZmlsZXMvbG1vZC91dGlsaXRpZXMvbGludXgtcmhlbDgtcHBjNjRsZS9Db3JlL3B5dGhvbi8zLjEwLjgubHVhIiwKZnVsbE5hbWUgPSAicHl0aG9uLzMuMTAuOCIsCmxvYWRPcmRlciA9IDEsCnByb3BUID0ge30sCnN0YWNrRGVwdGggPSAwLApzdGF0dXMgPSAiYWN0aXZlIiwKdXNlck5hbWUgPSAicHl0aG9uLzMuMTAuOCIsCndWID0gIjAwMDAwMDAwMy4wMDAwMDAwMTAuMDAwMDAwMDA4Lip6
+++ export _ModuleTable001_
+++ _ModuleTable002_=ZmluYWwiLAp9LAp9LAptcGF0aEEgPSB7CiIvcHJvamVjdHMvcHBjNjRsZS1wd3I5LXJoZWw4L21vZHVsZWZpbGVzL2xtb2QvY29tcGlsZXJzIiwgIi9wcm9qZWN0cy9wcGM2NGxlLXB3cjktcmhlbDgvbW9kdWxlZmlsZXMvbG1vZC91dGlsaXRpZXMvbGludXgtcmhlbDgtcHBjNjRsZS9Db3JlIiwKfSwKc3lzdGVtQmFzZU1QQVRIID0gIi9wcm9qZWN0cy9wcGM2NGxlLXB3cjktcmhlbDgvbW9kdWxlZmlsZXMvbG1vZC9jb21waWxlcnM6L3Byb2plY3RzL3BwYzY0bGUtcHdyOS1yaGVsOC9tb2R1bGVmaWxlcy9sbW9kL3V0aWxpdGllcy9saW51eC1yaGVsOC1wcGM2NGxlL0NvcmUiLAp9Cg==
+++ export _ModuleTable002_
+++ _ModuleTable_Sz_=2
+++ export _ModuleTable_Sz_
++ SCREAM_MACHINE=weaver
+ [[ 0 == 1 ]]
+ [[ 0 == 1 ]]
+ [[ 0 == 1 ]]
++ whoami
+ [[ e3sm-jenkins == \e\3\s\m\-\j\e\n\k\i\n\s ]]
+ git config --local user.email jenkins@ignore.com
+ git config --local user.name 'Jenkins Jenkins'
+ declare -i fails=0
+ BASELINES_DIR=AUTO
+ TAS_ARGS='--baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m $machine'
+ [[ weaver == \p\m\-\g\p\u ]]
+ set +e
+ '[' -n 2885 ']'
+ is_at_run=1
+ SA_FAILURES_DETAILS=
+ '[' 1 -eq 1 ']'
++ ./scripts/gather-all-data './scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m $machine' -l -m weaver
***Forced exclusive execution
<>
<>
+ this_output='Job <75242> is submitted to queue .
Traceback (most recent call last):
  File "/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5850/scream/components/eamxx/./scripts/test-all-scream", line 40, in 
    from test_factory import get_test_name_dict
  File "/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5850/scream/components/eamxx/scripts/test_factory.py", line 105
    ("EAMXX_ENABLE_PYBIND", "True"],
                                  ^
SyntaxError: closing parenthesis '\'']'\'' does not match opening parenthesis '\''('\''
Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5850/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver
RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5850/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5850/scream/components/eamxx
weaver failed'
+ [[ 1 != 0 ]]
+ fails=0+1
+ sa_fail=1
+ [[ 1 == 1 ]]
++ echo 'Job <75242> is submitted to queue .
Traceback (most recent call last):
  File "/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5850/scream/components/eamxx/./scripts/test-all-scream", line 40, in 
    from test_factory import get_test_name_dict
  File "/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5850/scream/components/eamxx/scripts/test_factory.py", line 105
    ("EAMXX_ENABLE_PYBIND", "True"],
                                  ^
SyntaxError: closing parenthesis '\'']'\'' does not match opening parenthesis '\''('\''
Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5850/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver
RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5850/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5850/scream/components/eamxx
weaver failed'
++ grep -m1 -A 100000 'Build type '
+ errors=
+ SA_FAILURES_DETAILS+=
+ [[ 1 == 0 ]]
+ [[ weaver == \m\a\p\p\y ]]
+ set +x
######################################################
FAILS DETECTED:
  SCREAM STANDALONE TESTING FAILED!
######################################################

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins8422088230774113154.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE

github-actions · 2024-06-29T00:25:04Z

PR Preview Action v1.4.7
🚀 Deployed preview to https://E3SM-Project.github.io/scream/pr-preview/pr-2885/
on branch `gh-pages` at 2024-07-01 15:38 UTC

E3SM-Autotester · 2024-06-29T06:22:48Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5605
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE;documentation;AT: Skip v1 Testing;pyeamxx
PULLREQUESTNUM	2885
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`fdd97c2`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`a1646c8`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5852
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE;documentation;AT: Skip v1 Testing;pyeamxx
PULLREQUESTNUM	2885
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`fdd97c2`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`a1646c8`
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: bartgol

E3SM-Autotester · 2024-06-29T07:03:46Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5605
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE;documentation;AT: Skip v1 Testing;pyeamxx
PULLREQUESTNUM	2885
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`fdd97c2`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`a1646c8`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5852
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE;documentation;AT: Skip v1 Testing;pyeamxx
PULLREQUESTNUM	2885
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`fdd97c2`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`a1646c8`
TEST_REPO_ALIAS	SCREAM

SCREAM_PullRequest_Autotester_Mappy # 5605 FAILED (click to see last 100 lines of console output)


-- Found OpenMP: TRUE (found version "4.5") found components: Fortran 
SCREAM_DYNAMICS_DYCORE set to 'NONE'. Scream won't enable any test/code that needs dynamics.
-- WARNING: RRTMGP and COSP only supported for double precision builds; skipping

CMake Warning (dev) at /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5605/scream/externals/haero/CMakeLists.txt:10 (option):

Policy CMP0077 is not set: option() honors normal variables.  Run "cmake

--help-policy CMP0077" for policy details.  Use the cmake_policy command to

set the policy and suppress this warning.
For compatibility with older versions of CMake, option is clearing the

normal variable 'HAERO_ENABLE_GPU'.

This warning is for project developers.  Use -Wno-dev to suppress it.
CMake Warning (dev) at /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5605/scream/externals/haero/CMakeLists.txt:11 (option):

Policy CMP0077 is not set: option() honors normal variables.  Run "cmake

--help-policy CMP0077" for policy details.  Use the cmake_policy command to

set the policy and suppress this warning.
For compatibility with older versions of CMake, option is clearing the

normal variable 'HAERO_ENABLE_MPI'.

This warning is for project developers.  Use -Wno-dev to suppress it.
CMake Warning (dev) at /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5605/scream/externals/haero/CMakeLists.txt:12 (option):

Policy CMP0077 is not set: option() honors normal variables.  Run "cmake

--help-policy CMP0077" for policy details.  Use the cmake_policy command to

set the policy and suppress this warning.
For compatibility with older versions of CMake, option is clearing the

normal variable 'HAERO_ENABLE_TESTS'.

This warning is for project developers.  Use -Wno-dev to suppress it.
-- Configuring with build type: Debug

-- Building for CPU

-- Building Haero within another project.

-- Generating project files in build directory: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5605/scream/components/eamxx/ctest-build/full_sp_debug/externals/haero

-- C++ compiler is /projects/sems/install/rhel7-x86_64/acme/compiler/gcc/11.2.0/openmpi/4.1.4/bin/mpicxx (GNU 11.2.0)

-- C compiler is /projects/sems/install/rhel7-x86_64/acme/compiler/gcc/11.2.0/openmpi/4.1.4/bin/mpicc (GNU 11.2.0)

-- Configuring haero v0.1.0

-- Using single precision floating point numbers

-- Configuring on Linux.

-- Installation prefix is /usr/local

-- MPI is enabled

-- Found MPI: TRUE (found version "3.1") found components: C

-- Configuring with build type: Debug

-- Generating project files in build directory: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5605/scream/components/eamxx/ctest-build/full_sp_debug/externals/mam4xx

-- C++ compiler is /projects/sems/install/rhel7-x86_64/acme/compiler/gcc/11.2.0/openmpi/4.1.4/bin/mpicxx (GNU 11.2.0)

-- C compiler is /projects/sems/install/rhel7-x86_64/acme/compiler/gcc/11.2.0/openmpi/4.1.4/bin/mpicc (GNU 11.2.0)

-- Configuring MAM4xx v0.1.0

-- Using single precision floating point numbers

-- Installation prefix is /usr/local

-- WARNING: TMS only supported for double precision builds; skipping

CMake Error at src/python/CMakeLists.txt:1 (find_package):

By not providing "Findpybind11.cmake" in CMAKE_MODULE_PATH this project has

asked CMake to find a package configuration file provided by "pybind11",

but CMake did not find one.
Could not find a package configuration file provided by "pybind11" with any

of the following names:
pybind11Config.cmake
pybind11-config.cmake

Add the installation prefix of "pybind11" to CMAKE_PREFIX_PATH or set

"pybind11_DIR" to a directory containing one of the above files.  If

"pybind11" provides a separate development package or SDK, be sure it has

been installed.
-- Configuring incomplete, errors occurred!
Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on mappy with cmd: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5605/scream/components/eamxx && module purge && module load sems-archive-env acme-env acme-cmake/3.26.3 acme-gcc/11.2.0 sems-archive-git/2.10.1 acme-openmpi/4.1.4 acme-netcdf/4.7.4/acme && export GATOR_INITIAL_MB=4000MB && export PATH=/ascldap/users/jgfouca/packages/valgrind-3.22.0/bin:$PATH && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m mappy

RUN: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5605/scream/components/eamxx && module purge && module load sems-archive-env acme-env acme-cmake/3.26.3 acme-gcc/11.2.0 sems-archive-git/2.10.1 acme-openmpi/4.1.4 acme-netcdf/4.7.4/acme && export GATOR_INITIAL_MB=4000MB && export PATH=/ascldap/users/jgfouca/packages/valgrind-3.22.0/bin:$PATH && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m mappy

FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5605/scream/components/eamxx

mappy failed

######################################################

Build step 'Execute shell' marked build as failure

$ ssh-agent -k

unset SSH_AUTH_SOCK;

unset SSH_AGENT_PID;

echo Agent pid 121821 killed;

[ssh-agent] Stopped.

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs
squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel
[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins15154023867712141745.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE

SCREAM_PullRequest_Autotester_Weaver # 5852 FAILED (click to see last 100 lines of console output)


-- ===== Configuring SCORPIO Documentation... =====
-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE) 
-- Disabling all warnings for target pioc
-- Disabling all warnings for target piof
-- Disabling all warnings for target gptl
-- Found NetCDF: -L/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw/lib -lnetcdff -lnetcdf -lnetcdf -ldl -lm  
-- Found NetCDF: -L/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth/lib -lnetcdf  
-- Disabling all warnings for target cprnc
SCREAM_DYNAMICS_DYCORE set to 'NONE'. Scream won't enable any test/code that needs dynamics.
-- WARNING: RRTMGP and COSP only supported for double precision builds; skipping

CMake Warning (dev) at /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5852/scream/externals/haero/CMakeLists.txt:10 (option):

Policy CMP0077 is not set: option() honors normal variables.  Run "cmake

--help-policy CMP0077" for policy details.  Use the cmake_policy command to

set the policy and suppress this warning.
For compatibility with older versions of CMake, option is clearing the

normal variable 'HAERO_ENABLE_GPU'.

This warning is for project developers.  Use -Wno-dev to suppress it.
-- Configuring with build type: Debug

CMake Warning (dev) at /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5852/scream/externals/haero/CMakeLists.txt:11 (option):

Policy CMP0077 is not set: option() honors normal variables.  Run "cmake

--help-policy CMP0077" for policy details.  Use the cmake_policy command to

set the policy and suppress this warning.
For compatibility with older versions of CMake, option is clearing the

normal variable 'HAERO_ENABLE_MPI'.

This warning is for project developers.  Use -Wno-dev to suppress it.
CMake Warning (dev) at /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5852/scream/externals/haero/CMakeLists.txt:12 (option):

Policy CMP0077 is not set: option() honors normal variables.  Run "cmake

--help-policy CMP0077" for policy details.  Use the cmake_policy command to

set the policy and suppress this warning.
For compatibility with older versions of CMake, option is clearing the

normal variable 'HAERO_ENABLE_TESTS'.

This warning is for project d-- Building for CPU

evelopers.  Use -Wno-dev to suppress it.
-- Building Haero within another project.

-- Generating project files in build directory: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5852/scream/components/eamxx/ctest-build/full_sp_debug/externals/haero

-- C++ compiler is /projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicxx (GNU 11.3.0)

-- C compiler is /projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicc (GNU 11.3.0)

-- Configuring haero v0.1.0

-- Using single precision floating point numbers

-- Configuring on Linux.

-- Installation prefix is /usr/local

-- MPI is enabled

-- Found MPI: TRUE (found version "3.1") found components: C

-- Configuring with build type: Debug

-- Generating project files in build directory: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5852/scream/components/eamxx/ctest-build/full_sp_debug/externals/mam4xx

-- C++ compiler is /projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicxx (GNU 11.3.0)

-- C compiler is /projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicc (GNU 11.3.0)

-- Configuring MAM4xx v0.1.0

-- Using single precision floating point numbers

-- Installation prefix is /usr/local

-- WARNING: TMS only supported for double precision builds; skipping

CMake Error at src/python/CMakeLists.txt:1 (find_package):

By not providing "Findpybind11.cmake" in CMAKE_MODULE_PATH this project has

asked CMake to find a package configuration file provided by "pybind11",

but CMake did not find one.
Could not find a package configuration file provided by "pybind11" with any

of the following names:
pybind11Config.cmake
pybind11-config.cmake

Add the installation prefix of "pybind11" to CMAKE_PREFIX_PATH or set

"pybind11_DIR" to a directory containing one of the above files.  If

"pybind11" provides a separate development package or SDK, be sure it has

been installed.
-- Configuring incomplete, errors occurred!

See also "/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5852/scream/components/eamxx/ctest-build/full_sp_debug/CMakeFiles/CMakeOutput.log".

See also "/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5852/scream/components/eamxx/ctest-build/full_sp_debug/CMakeFiles/CMakeError.log".
Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5852/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5852/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5852/scream/components/eamxx

weaver failed

######################################################

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins1541837855750714231.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE

mahf708

Let's drop the docs part out of this PR for now? I intend to document the python package/interface differently later on, and these docs will go out of date pretty quickly...

E3SM-Autotester · 2024-07-01T17:26:01Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5610
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE;documentation;AT: Skip v1 Testing;pyeamxx
PULLREQUESTNUM	2885
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`24db9af`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`f034619`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5857
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE;documentation;AT: Skip v1 Testing;pyeamxx
PULLREQUESTNUM	2885
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`24db9af`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`f034619`
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: bartgol

E3SM-Autotester · 2024-07-01T18:08:20Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5610
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE;documentation;AT: Skip v1 Testing;pyeamxx
PULLREQUESTNUM	2885
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`24db9af`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`f034619`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5857
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE;documentation;AT: Skip v1 Testing;pyeamxx
PULLREQUESTNUM	2885
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`24db9af`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`f034619`
TEST_REPO_ALIAS	SCREAM

SCREAM_PullRequest_Autotester_Mappy # 5610 FAILED (click to see last 100 lines of console output)


-- Disabling all warnings for target cprnc
-- Found OpenMP: TRUE (found version "4.5") found components: Fortran 
SCREAM_DYNAMICS_DYCORE set to 'NONE'. Scream won't enable any test/code that needs dynamics.
-- WARNING: RRTMGP and COSP only supported for double precision builds; skipping

CMake Warning (dev) at /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5610/scream/externals/haero/CMakeLists.txt:10 (option):

Policy CMP0077 is not set: option() honors normal variables.  Run "cmake

--help-policy CMP0077" for policy details.  Use the cmake_policy command to

set the policy and suppress this warning.
For compatibility with older versions of CMake, option is clearing the

normal variable 'HAERO_ENABLE_GPU'.

This warning is for project developers.  Use -Wno-dev to suppress it.
CMake Warning (dev) at /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5610/scream/externals/haero/CMakeLists.txt:11 (option):

Policy CMP0077 is not set: option() honors normal variables.  Run "cmake

--help-policy CMP0077" for policy details.  Use the cmake_policy command to

set the policy and suppress this warning.
For compatibility with older versions of CMake, option is clearing the

normal variable 'HAERO_ENABLE_MPI'.

This warning is for project developers.  Use -Wno-dev to suppress it.
CMake Warning (dev) at /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5610/scream/externals/haero/CMakeLists.txt:12 (option):

Policy CMP0077 is not set: option() honors normal variables.  Run "cmake

--help-policy CMP0077" for policy details.  Use the cmake_policy command to

set the policy and suppress this warning.
For compatibility with older versions of CMake, option is clearing the

normal variable 'HAERO_ENABLE_TESTS'.

This warning is for pro-- Configuring with build type: Debug

ject developers.  Use -Wno-dev to suppress it.
-- Building for CPU

-- Building Haero within another project.

-- Generating project files in build directory: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5610/scream/components/eamxx/ctest-build/full_sp_debug/externals/haero

-- C++ compiler is /projects/sems/install/rhel7-x86_64/acme/compiler/gcc/11.2.0/openmpi/4.1.4/bin/mpicxx (GNU 11.2.0)

-- C compiler is /projects/sems/install/rhel7-x86_64/acme/compiler/gcc/11.2.0/openmpi/4.1.4/bin/mpicc (GNU 11.2.0)

-- Configuring haero v0.1.0

-- Using single precision floating point numbers

-- Configuring on Linux.

-- Installation prefix is /usr/local

-- MPI is enabled

-- Found MPI: TRUE (found version "3.1") found components: C

-- Configuring with build type: Debug

-- Generating project files in build directory: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5610/scream/components/eamxx/ctest-build/full_sp_debug/externals/mam4xx

-- C++ compiler is /projects/sems/install/rhel7-x86_64/acme/compiler/gcc/11.2.0/openmpi/4.1.4/bin/mpicxx (GNU 11.2.0)

-- C compiler is /projects/sems/install/rhel7-x86_64/acme/compiler/gcc/11.2.0/openmpi/4.1.4/bin/mpicc (GNU 11.2.0)

-- Configuring MAM4xx v0.1.0

-- Using single precision floating point numbers

-- Installation prefix is /usr/local

-- WARNING: TMS only supported for double precision builds; skipping

-- Found PythonInterp: /ascldap/users/jgfouca/packages/Python-3.8.5/bin/python3 (found suitable version "3.8.5", minimum required is "3.7")

-- Found PythonLibs: /ascldap/users/jgfouca/packages/Python-3.8.5/lib/libpython3.8.a

-- Performing Test HAS_FLTO

-- Performing Test HAS_FLTO - Success

-- Found pybind11: /home/e3sm-jenkins/.local/lib/python3.8/site-packages/pybind11/include (found version "2.13.1")

Traceback (most recent call last):

File "", line 1, in 

ModuleNotFoundError: No module named 'mpi4py'

CMake Error at /net/watson/storage/fast/projects/sems/install/rhel7-x86_64/acme/utility/cmake/3.26.3/share/cmake-3.26/Modules/FindPackageHandleStandardArgs.cmake:230 (message):

Could NOT find mpi4py (missing: MPI4PY_INCLUDE_DIR)

Call Stack (most recent call first):

/net/watson/storage/fast/projects/sems/install/rhel7-x86_64/acme/utility/cmake/3.26.3/share/cmake-3.26/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)

cmake/Findmpi4py.cmake:36 (find_package_handle_standard_args)

src/python/CMakeLists.txt:2 (find_package)
-- Configuring incomplete, errors occurred!
Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on mappy with cmd: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5610/scream/components/eamxx && module purge && module load sems-archive-env acme-env acme-cmake/3.26.3 acme-gcc/11.2.0 sems-archive-git/2.10.1 acme-openmpi/4.1.4 acme-netcdf/4.7.4/acme && export GATOR_INITIAL_MB=4000MB && export PATH=/ascldap/users/jgfouca/packages/valgrind-3.22.0/bin:$PATH && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m mappy

RUN: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5610/scream/components/eamxx && module purge && module load sems-archive-env acme-env acme-cmake/3.26.3 acme-gcc/11.2.0 sems-archive-git/2.10.1 acme-openmpi/4.1.4 acme-netcdf/4.7.4/acme && export GATOR_INITIAL_MB=4000MB && export PATH=/ascldap/users/jgfouca/packages/valgrind-3.22.0/bin:$PATH && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m mappy

FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5610/scream/components/eamxx

mappy failed

######################################################

Build step 'Execute shell' marked build as failure

$ ssh-agent -k

unset SSH_AUTH_SOCK;

unset SSH_AGENT_PID;

echo Agent pid 85325 killed;

[ssh-agent] Stopped.

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs
squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel
[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins3216098754150757417.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE

SCREAM_PullRequest_Autotester_Weaver # 5857 FAILED (click to see last 100 lines of console output)


/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5857/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5857/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5857/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5857/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5857/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5857/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5857/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5857/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5857/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5857/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5857/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5857/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5857/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5857/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5857/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5857/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5857/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5857/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5857/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5857/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5857/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5857/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5857/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5857/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5857/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5857/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5857/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5857/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5857/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5857/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5857/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5857/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5857/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5857/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5857/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma
[ 98%] Linking CXX executable aerocom_cld

[ 98%] Built target aerocom_cld

[ 98%] Linking CXX executable shoc_sk_tests

[ 98%] Linking CXX executable shoc_tests

[ 98%] Built target shoc_sk_tests

[ 98%] Built target shoc_tests

gmake: *** [Makefile:166: all] Error 2
Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5857/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5857/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5857/scream/components/eamxx

weaver failed

######################################################

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins13545495289139308571.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE

E3SM-Autotester · 2024-07-01T21:15:59Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5611
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE;documentation;AT: Skip v1 Testing;pyeamxx
PULLREQUESTNUM	2885
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`22ef574`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`f034619`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5858
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE;documentation;AT: Skip v1 Testing;pyeamxx
PULLREQUESTNUM	2885
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`22ef574`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`f034619`
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: bartgol

E3SM-Autotester · 2024-07-01T22:00:20Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5611
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE;documentation;AT: Skip v1 Testing;pyeamxx
PULLREQUESTNUM	2885
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`22ef574`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`f034619`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5858
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE;documentation;AT: Skip v1 Testing;pyeamxx
PULLREQUESTNUM	2885
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`22ef574`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`f034619`
TEST_REPO_ALIAS	SCREAM

SCREAM_PullRequest_Autotester_Mappy # 5611 PASSED (click to see last 100 lines of console output)


tms                 = 2732.79 sec*proc (14 tests)
Total Test time (real) = 1314.36 sec

467/469 Test #465: homme_shoc_cld_p3_mam_optics_rrtmgp_np4 ..................   Passed  514.52 sec

Start 468: homme_shoc_cld_p3_mam_optics_rrtmgp_np4_vs_np1

Start 469: homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp

468/469 Test #468: homme_shoc_cld_p3_mam_optics_rrtmgp_np4_vs_np1 ...........   Passed    0.08 sec

469/469 Test #469: homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp .........   Passed    0.17 sec
100% tests passed, 0 tests failed out of 469
Label Time Summary:

PEM                 =   7.57 secproc (60 tests)

baseline_cmp        =  13.95 secproc (17 tests)

baseline_gen        = 8288.31 secproc (19 tests)

bfbhash             =   0.48 secproc (1 test)

check               =   0.49 secproc (1 test)

cld                 = 15441.15 secproc (75 tests)

cld_fraction        =   0.83 secproc (1 test)

cosp                =  99.08 secproc (3 tests)

cxx baseline_cmp    =  53.98 secproc (2 tests)

diagnostics         = 2823.38 secproc (25 tests)

dp                  = 1119.95 secproc (6 tests)

driver              = 20521.72 secproc (78 tests)

dynamics            = 7413.09 secproc (54 tests)

fail                = 529.56 secproc (35 tests)

io                  = 441.47 secproc (49 tests)

mam4_aci            = 6682.99 secproc (34 tests)

mam4_optics         = 5922.22 secproc (19 tests)

nudging             = 1272.99 secproc (8 tests)

p3                  = 16229.73 secproc (118 tests)

p3_sk               = 664.42 secproc (32 tests)

pg2                 = 312.04 secproc (9 tests)

physics             = 21841.12 secproc (212 tests)

remap               =  20.98 secproc (4 tests)

rrtmgp              = 18036.38 secproc (86 tests)

shoc                = 16057.98 secproc (95 tests)

spa                 = 4096.65 secproc (40 tests)

surface_coupling    =  12.70 secproc (4 tests)

tms                 = 1431.82 secproc (12 tests)
Total Test time (real) = 1300.74 sec
Testing '''e02f4e00a00211512ba429641be585c57a3364fd''' for test '''full_sp_debug'''
RUN: taskset -c 32-63 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=32 CTEST_PARALLEL_LEVEL=32 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5611/scream/components/eamxx/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5611/scream/components/eamxx/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5611/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=mappy -DCMAKE_COMMAND="-C /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5611/scream/components/eamxx/cmake/machine-files/mappy.cmake -DNetCDF_Fortran_PATH=/projects/sems/install/rhel7-x86_64/acme/tpl/netcdf/4.7.4/gcc/11.2.0/openmpi/4.1.4/acme -DNetCDF_C_PATH=/projects/sems/install/rhel7-x86_64/acme/tpl/netcdf/4.7.4/gcc/11.2.0/openmpi/4.1.4/acme -DPnetCDF_C_PATH=/projects/sems/install/rhel7-x86_64/acme/tpl/netcdf/4.7.4/gcc/11.2.0/openmpi/4.1.4/acme -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DEAMXX_ENABLE_PYBIND=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=32 -DSCREAM_BASELINES_DIR=/sems-data-store/ACME/baselines/scream/master-baselines/full_sp_debug" '''

FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5611/scream/components/eamxx/ctest-build/full_sp_debug
Testing '''e02f4e00a00211512ba429641be585c57a3364fd''' for test '''release'''
RUN: taskset -c 96-127 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=32 CTEST_PARALLEL_LEVEL=32 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5611/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5611/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5611/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=mappy -DCMAKE_COMMAND="-C /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5611/scream/components/eamxx/cmake/machine-files/mappy.cmake -DNetCDF_Fortran_PATH=/projects/sems/install/rhel7-x86_64/acme/tpl/netcdf/4.7.4/gcc/11.2.0/openmpi/4.1.4/acme -DNetCDF_C_PATH=/projects/sems/install/rhel7-x86_64/acme/tpl/netcdf/4.7.4/gcc/11.2.0/openmpi/4.1.4/acme -DPnetCDF_C_PATH=/projects/sems/install/rhel7-x86_64/acme/tpl/netcdf/4.7.4/gcc/11.2.0/openmpi/4.1.4/acme -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=32 -DSCREAM_BASELINES_DIR=/sems-data-store/ACME/baselines/scream/master-baselines/release" '''

FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5611/scream/components/eamxx/ctest-build/release
Testing '''e02f4e00a00211512ba429641be585c57a3364fd''' for test '''full_debug'''
RUN: taskset -c 0-31 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=32 CTEST_PARALLEL_LEVEL=32 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5611/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5611/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5611/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=mappy -DCMAKE_COMMAND="-C /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5611/scream/components/eamxx/cmake/machine-files/mappy.cmake -DNetCDF_Fortran_PATH=/projects/sems/install/rhel7-x86_64/acme/tpl/netcdf/4.7.4/gcc/11.2.0/openmpi/4.1.4/acme -DNetCDF_C_PATH=/projects/sems/install/rhel7-x86_64/acme/tpl/netcdf/4.7.4/gcc/11.2.0/openmpi/4.1.4/acme -DPnetCDF_C_PATH=/projects/sems/install/rhel7-x86_64/acme/tpl/netcdf/4.7.4/gcc/11.2.0/openmpi/4.1.4/acme -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DKokkos_ENABLE_DEBUG_BOUNDS_CHECK=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=32 -DSCREAM_BASELINES_DIR=/sems-data-store/ACME/baselines/scream/master-baselines/full_debug" '''

FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5611/scream/components/eamxx/ctest-build/full_debug
Testing '''e02f4e00a00211512ba429641be585c57a3364fd''' for test '''debug_nopack_fpe'''
RUN: taskset -c 64-95 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=32 CTEST_PARALLEL_LEVEL=32 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5611/scream/components/eamxx/ctest-build/debug_nopack_fpe/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5611/scream/components/eamxx/ctest-build/debug_nopack_fpe -DBUILD_NAME_MOD=debug_nopack_fpe -S /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5611/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=mappy -DCMAKE_COMMAND="-C /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5611/scream/components/eamxx/cmake/machine-files/mappy.cmake -DNetCDF_Fortran_PATH=/projects/sems/install/rhel7-x86_64/acme/tpl/netcdf/4.7.4/gcc/11.2.0/openmpi/4.1.4/acme -DNetCDF_C_PATH=/projects/sems/install/rhel7-x86_64/acme/tpl/netcdf/4.7.4/gcc/11.2.0/openmpi/4.1.4/acme -DPnetCDF_C_PATH=/projects/sems/install/rhel7-x86_64/acme/tpl/netcdf/4.7.4/gcc/11.2.0/openmpi/4.1.4/acme -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_PACK_SIZE=1 -DSCREAM_FPE=True -DSCREAM_ENABLE_BASELINE_TESTS=False -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=32 -DSCREAM_ENABLE_BASELINE_TESTS=Off" '''

FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5611/scream/components/eamxx/ctest-build/debug_nopack_fpe

OVERALL STATUS: PASS

Starting analysis on mappy with cmd: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5611/scream/components/eamxx && module purge && module load sems-archive-env acme-env acme-cmake/3.26.3 acme-gcc/11.2.0 sems-archive-git/2.10.1 acme-openmpi/4.1.4 acme-netcdf/4.7.4/acme && export GATOR_INITIAL_MB=4000MB && export PATH=/ascldap/users/jgfouca/packages/valgrind-3.22.0/bin:$PATH && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m mappy

RUN: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5611/scream/components/eamxx && module purge && module load sems-archive-env acme-env acme-cmake/3.26.3 acme-gcc/11.2.0 sems-archive-git/2.10.1 acme-openmpi/4.1.4 acme-netcdf/4.7.4/acme && export GATOR_INITIAL_MB=4000MB && export PATH=/ascldap/users/jgfouca/packages/valgrind-3.22.0/bin:$PATH && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m mappy

FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5611/scream/components/eamxx

Completed analysis on mappy'

[[ 0 != 0 ]]
[[ 1 == 0 ]]
[[ mappy == \m\a\p\p\y ]]
[[ 0 == 1 ]]
[[ -z '' ]]
[[ 1 == 1 ]]
[[ 0 == 1 ]]
[[ 0 == 1 ]]
echo 'SCREAM v1 tests were skipped, since the Github label '''AT: Skip v1 Testing''' was found.\n'

SCREAM v1 tests were skipped, since the Github label 'AT: Skip v1 Testing' was found.\n
set +x

$ ssh-agent -k

unset SSH_AUTH_SOCK;

unset SSH_AGENT_PID;

echo Agent pid 90076 killed;

[ssh-agent] Stopped.

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs
squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel
[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins11292039226240117520.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: SUCCESS

SCREAM_PullRequest_Autotester_Weaver # 5858 FAILED (click to see last 100 lines of console output)


cld_fraction        =   4.44 sec*proc (1 test)
cxx baseline_cmp    =   6.29 sec*proc (2 tests)
diagnostics         =  46.23 sec*proc (22 tests)
driver              =  65.32 sec*proc (12 tests)
dynamics            =   9.51 sec*proc (3 tests)
fail                =  28.65 sec*proc (5 tests)
io                  =  55.85 sec*proc (13 tests)
mam4_aci            =  23.68 sec*proc (4 tests)
mam4_optics         =   7.97 sec*proc (1 test)
nudging             =  12.92 sec*proc (2 tests)
p3                  =  76.65 sec*proc (10 tests)
p3_sk               =  51.43 sec*proc (2 tests)
physics             = 166.85 sec*proc (23 tests)
remap               =   5.92 sec*proc (1 test)
rrtmgp              =  47.23 sec*proc (11 tests)
shoc                =  43.92 sec*proc (11 tests)
spa                 =  11.06 sec*proc (4 tests)
surface_coupling    =   4.41 sec*proc (1 test)
Total Test time (real) = 640.94 sec
Testing '''81273e9968b4f2db301eaa7296470702b92b0251''' for test '''full_sp_debug'''
RUN: taskset -c 52-103 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5858/scream/components/eamxx/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5858/scream/components/eamxx/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5858/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5858/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DEAMXX_ENABLE_PYBIND=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_sp_debug" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5858/scream/components/eamxx/ctest-build/full_sp_debug
Testing '''81273e9968b4f2db301eaa7296470702b92b0251''' for test '''release'''
RUN: taskset -c 104-155 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5858/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5858/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5858/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5858/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5858/scream/components/eamxx/ctest-build/release
Testing '''81273e9968b4f2db301eaa7296470702b92b0251''' for test '''full_debug'''
RUN: taskset -c 0-51 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5858/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5858/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5858/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5858/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DKokkos_ENABLE_DEBUG_BOUNDS_CHECK=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5858/scream/components/eamxx/ctest-build/full_debug

Build type full_sp_debug failed at testing time. Here'''s a list of failed tests:

2:io_utils

102:p3_standalone_py_np1

103:pyp3_cxx_vs_py_np1
Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5858/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5858/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5858/scream/components/eamxx

weaver failed'

errors='Build type full_sp_debug failed at testing time. Here'''s a list of failed tests:

2:io_utils

102:p3_standalone_py_np1

103:pyp3_cxx_vs_py_np1

Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5858/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5858/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5858/scream/components/eamxx

weaver failed'

SA_FAILURES_DETAILS+='Build type full_sp_debug failed at testing time. Here'''s a list of failed tests:

2:io_utils

102:p3_standalone_py_np1

103:pyp3_cxx_vs_py_np1

Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5858/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5858/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5858/scream/components/eamxx

weaver failed'

[[ 1 == 0 ]]
[[ weaver == \m\a\p\p\y ]]
set +x

######################################################

FAILS DETECTED:

SCREAM STANDALONE TESTING FAILED!

Build type full_sp_debug failed at testing time. Here's a list of failed tests:

2:io_utils

102:p3_standalone_py_np1

103:pyp3_cxx_vs_py_np1

Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5858/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5858/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5858/scream/components/eamxx

weaver failed

######################################################

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins8612440663901180369.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE

mahf708

Looks good.

Just curios: Is it struggling to find pybind11 in mappy/weaver? It often "just" finds it...

bartgol · 2024-07-01T23:01:26Z

Looks good.

Just curios: Is it struggling to find pybind11 in mappy/weaver? It often "just" finds it...

On mappy I had to pip install pybind11 and mpi4py, while on weaver it failed b/c of a legit code error.

E3SM-Autotester · 2024-07-01T23:08:54Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5613
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE;AT: Skip v1 Testing;pyeamxx
PULLREQUESTNUM	2885
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`9627fcc`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`3314452`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5860
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE;AT: Skip v1 Testing;pyeamxx
PULLREQUESTNUM	2885
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`9627fcc`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`3314452`
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: bartgol

E3SM-Autotester · 2024-07-01T23:53:26Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5613
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE;AT: Skip v1 Testing;pyeamxx
PULLREQUESTNUM	2885
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`9627fcc`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`3314452`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5860
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE;AT: Skip v1 Testing;pyeamxx
PULLREQUESTNUM	2885
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`9627fcc`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`3314452`
TEST_REPO_ALIAS	SCREAM

SCREAM_PullRequest_Autotester_Mappy # 5613 PASSED (click to see last 100 lines of console output)


shoc                = 14102.86 sec*proc (95 tests)
spa                 = 3601.40 sec*proc (40 tests)
surface_coupling    =  14.92 sec*proc (4 tests)
tms                 = 1256.26 sec*proc (12 tests)
Total Test time (real) = 1156.21 sec

447/450 Test #447: homme_shoc_cld_p3_mam_optics_rrtmgp_np4 ..................   Passed  498.56 sec

Start 450: homme_shoc_cld_p3_mam_optics_rrtmgp_np4_vs_np1

448/450 Test #450: homme_shoc_cld_p3_mam_optics_rrtmgp_np4_vs_np1 ...........   Passed    0.08 sec

449/450 Test #446: homme_shoc_cld_p3_mam_optics_rrtmgp_np3 ..................   Passed  623.32 sec

Start 449: homme_shoc_cld_p3_mam_optics_rrtmgp_np3_vs_np1

450/450 Test #449: homme_shoc_cld_p3_mam_optics_rrtmgp_np3_vs_np1 ...........   Passed    0.10 sec
100% tests passed, 0 tests failed out of 450
Label Time Summary:

PEM                 =   8.02 secproc (60 tests)

bfbhash             =   3.81 secproc (1 test)

check               =   4.11 secproc (1 test)

cld                 = 21227.97 secproc (81 tests)

cld_fraction        =   2.03 secproc (1 test)

cosp                = 226.95 secproc (4 tests)

diagnostics         = 1629.54 secproc (25 tests)

dp                  = 2163.63 secproc (7 tests)

driver              = 27803.78 secproc (91 tests)

dynamics            = 13613.64 secproc (61 tests)

fail                = 439.30 secproc (33 tests)

io                  = 595.60 secproc (53 tests)

mam4_aci            = 6221.40 secproc (35 tests)

mam4_optics         = 8314.70 secproc (21 tests)

nudging             = 1353.82 secproc (8 tests)

p3                  = 21962.95 secproc (124 tests)

p3_sk               = 561.08 secproc (32 tests)

pg2                 = 979.39 secproc (10 tests)

physics             = 28727.62 secproc (222 tests)

remap               =  34.53 secproc (4 tests)

rrtmgp              = 23227.06 secproc (91 tests)

shoc                = 21733.52 secproc (102 tests)

spa                 = 6518.41 secproc (43 tests)

surface_coupling    =  15.25 secproc (4 tests)

tms                 = 3142.89 sec*proc (14 tests)
Total Test time (real) = 1288.46 sec
Testing '''a538a38703d2f79989945ae81a5d5d3a1e51df18''' for test '''full_sp_debug'''
RUN: taskset -c 32-63 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=32 CTEST_PARALLEL_LEVEL=32 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5613/scream/components/eamxx/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5613/scream/components/eamxx/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5613/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=mappy -DCMAKE_COMMAND="-C /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5613/scream/components/eamxx/cmake/machine-files/mappy.cmake -DNetCDF_Fortran_PATH=/projects/sems/install/rhel7-x86_64/acme/tpl/netcdf/4.7.4/gcc/11.2.0/openmpi/4.1.4/acme -DNetCDF_C_PATH=/projects/sems/install/rhel7-x86_64/acme/tpl/netcdf/4.7.4/gcc/11.2.0/openmpi/4.1.4/acme -DPnetCDF_C_PATH=/projects/sems/install/rhel7-x86_64/acme/tpl/netcdf/4.7.4/gcc/11.2.0/openmpi/4.1.4/acme -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DEAMXX_ENABLE_PYBIND=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=32 -DSCREAM_BASELINES_DIR=/sems-data-store/ACME/baselines/scream/master-baselines/full_sp_debug" '''

FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5613/scream/components/eamxx/ctest-build/full_sp_debug
Testing '''a538a38703d2f79989945ae81a5d5d3a1e51df18''' for test '''debug_nopack_fpe'''
RUN: taskset -c 64-95 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=32 CTEST_PARALLEL_LEVEL=32 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5613/scream/components/eamxx/ctest-build/debug_nopack_fpe/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5613/scream/components/eamxx/ctest-build/debug_nopack_fpe -DBUILD_NAME_MOD=debug_nopack_fpe -S /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5613/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=mappy -DCMAKE_COMMAND="-C /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5613/scream/components/eamxx/cmake/machine-files/mappy.cmake -DNetCDF_Fortran_PATH=/projects/sems/install/rhel7-x86_64/acme/tpl/netcdf/4.7.4/gcc/11.2.0/openmpi/4.1.4/acme -DNetCDF_C_PATH=/projects/sems/install/rhel7-x86_64/acme/tpl/netcdf/4.7.4/gcc/11.2.0/openmpi/4.1.4/acme -DPnetCDF_C_PATH=/projects/sems/install/rhel7-x86_64/acme/tpl/netcdf/4.7.4/gcc/11.2.0/openmpi/4.1.4/acme -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_PACK_SIZE=1 -DSCREAM_FPE=True -DSCREAM_ENABLE_BASELINE_TESTS=False -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=32 -DSCREAM_ENABLE_BASELINE_TESTS=Off" '''

FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5613/scream/components/eamxx/ctest-build/debug_nopack_fpe
Testing '''a538a38703d2f79989945ae81a5d5d3a1e51df18''' for test '''release'''
RUN: taskset -c 96-127 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=32 CTEST_PARALLEL_LEVEL=32 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5613/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5613/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5613/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=mappy -DCMAKE_COMMAND="-C /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5613/scream/components/eamxx/cmake/machine-files/mappy.cmake -DNetCDF_Fortran_PATH=/projects/sems/install/rhel7-x86_64/acme/tpl/netcdf/4.7.4/gcc/11.2.0/openmpi/4.1.4/acme -DNetCDF_C_PATH=/projects/sems/install/rhel7-x86_64/acme/tpl/netcdf/4.7.4/gcc/11.2.0/openmpi/4.1.4/acme -DPnetCDF_C_PATH=/projects/sems/install/rhel7-x86_64/acme/tpl/netcdf/4.7.4/gcc/11.2.0/openmpi/4.1.4/acme -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=32 -DSCREAM_BASELINES_DIR=/sems-data-store/ACME/baselines/scream/master-baselines/release" '''

FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5613/scream/components/eamxx/ctest-build/release
Testing '''a538a38703d2f79989945ae81a5d5d3a1e51df18''' for test '''full_debug'''
RUN: taskset -c 0-31 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=32 CTEST_PARALLEL_LEVEL=32 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5613/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5613/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5613/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=mappy -DCMAKE_COMMAND="-C /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5613/scream/components/eamxx/cmake/machine-files/mappy.cmake -DNetCDF_Fortran_PATH=/projects/sems/install/rhel7-x86_64/acme/tpl/netcdf/4.7.4/gcc/11.2.0/openmpi/4.1.4/acme -DNetCDF_C_PATH=/projects/sems/install/rhel7-x86_64/acme/tpl/netcdf/4.7.4/gcc/11.2.0/openmpi/4.1.4/acme -DPnetCDF_C_PATH=/projects/sems/install/rhel7-x86_64/acme/tpl/netcdf/4.7.4/gcc/11.2.0/openmpi/4.1.4/acme -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DKokkos_ENABLE_DEBUG_BOUNDS_CHECK=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=32 -DSCREAM_BASELINES_DIR=/sems-data-store/ACME/baselines/scream/master-baselines/full_debug" '''

FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5613/scream/components/eamxx/ctest-build/full_debug

OVERALL STATUS: PASS

Starting analysis on mappy with cmd: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5613/scream/components/eamxx && module purge && module load sems-archive-env acme-env acme-cmake/3.26.3 acme-gcc/11.2.0 sems-archive-git/2.10.1 acme-openmpi/4.1.4 acme-netcdf/4.7.4/acme && export GATOR_INITIAL_MB=4000MB && export PATH=/ascldap/users/jgfouca/packages/valgrind-3.22.0/bin:$PATH && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m mappy

RUN: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5613/scream/components/eamxx && module purge && module load sems-archive-env acme-env acme-cmake/3.26.3 acme-gcc/11.2.0 sems-archive-git/2.10.1 acme-openmpi/4.1.4 acme-netcdf/4.7.4/acme && export GATOR_INITIAL_MB=4000MB && export PATH=/ascldap/users/jgfouca/packages/valgrind-3.22.0/bin:$PATH && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m mappy

FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5613/scream/components/eamxx

Completed analysis on mappy'

[[ 0 != 0 ]]
[[ 1 == 0 ]]
[[ mappy == \m\a\p\p\y ]]
[[ 0 == 1 ]]
[[ -z '' ]]
[[ 1 == 1 ]]
[[ 0 == 1 ]]
[[ 0 == 1 ]]
echo 'SCREAM v1 tests were skipped, since the Github label '''AT: Skip v1 Testing''' was found.\n'

SCREAM v1 tests were skipped, since the Github label 'AT: Skip v1 Testing' was found.\n
set +x

$ ssh-agent -k

unset SSH_AUTH_SOCK;

unset SSH_AGENT_PID;

echo Agent pid 46407 killed;

[ssh-agent] Stopped.

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs
squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel
[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins4560787432264166252.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Finished: SUCCESS

SCREAM_PullRequest_Autotester_Weaver # 5860 FAILED (click to see last 100 lines of console output)


cld_fraction        =   0.98 sec*proc (1 test)
cxx baseline_cmp    =   8.74 sec*proc (2 tests)
diagnostics         =  45.04 sec*proc (22 tests)
driver              =  70.86 sec*proc (12 tests)
dynamics            =   6.51 sec*proc (3 tests)
fail                =  29.29 sec*proc (5 tests)
io                  =  60.89 sec*proc (14 tests)
mam4_aci            =  33.89 sec*proc (4 tests)
mam4_optics         =   8.08 sec*proc (1 test)
nudging             =  12.12 sec*proc (2 tests)
p3                  =  99.62 sec*proc (10 tests)
p3_sk               =  36.55 sec*proc (2 tests)
physics             = 170.44 sec*proc (23 tests)
remap               =   5.54 sec*proc (1 test)
rrtmgp              =  54.00 sec*proc (11 tests)
shoc                =  53.82 sec*proc (11 tests)
spa                 =  10.71 sec*proc (4 tests)
surface_coupling    =   4.13 sec*proc (1 test)
Total Test time (real) = 653.02 sec
Testing '''41702a6ce1bbaeda35582d4e3fb2d570ddc5b415''' for test '''full_sp_debug'''
RUN: taskset -c 52-103 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5860/scream/components/eamxx/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5860/scream/components/eamxx/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5860/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5860/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DEAMXX_ENABLE_PYBIND=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_sp_debug" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5860/scream/components/eamxx/ctest-build/full_sp_debug
Testing '''41702a6ce1bbaeda35582d4e3fb2d570ddc5b415''' for test '''release'''
RUN: taskset -c 104-155 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5860/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5860/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5860/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5860/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5860/scream/components/eamxx/ctest-build/release
Testing '''41702a6ce1bbaeda35582d4e3fb2d570ddc5b415''' for test '''full_debug'''
RUN: taskset -c 0-51 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5860/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5860/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5860/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5860/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DKokkos_ENABLE_DEBUG_BOUNDS_CHECK=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5860/scream/components/eamxx/ctest-build/full_debug

Build type full_sp_debug failed at testing time. Here'''s a list of failed tests:

2:io_utils

103:p3_standalone_py_np1

104:pyp3_cxx_vs_py_np1
Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5860/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5860/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5860/scream/components/eamxx

weaver failed'

errors='Build type full_sp_debug failed at testing time. Here'''s a list of failed tests:

2:io_utils

103:p3_standalone_py_np1

104:pyp3_cxx_vs_py_np1

Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5860/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5860/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5860/scream/components/eamxx

weaver failed'

SA_FAILURES_DETAILS+='Build type full_sp_debug failed at testing time. Here'''s a list of failed tests:

2:io_utils

103:p3_standalone_py_np1

104:pyp3_cxx_vs_py_np1

Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5860/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5860/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5860/scream/components/eamxx

weaver failed'

[[ 1 == 0 ]]
[[ weaver == \m\a\p\p\y ]]
set +x

######################################################

FAILS DETECTED:

SCREAM STANDALONE TESTING FAILED!

Build type full_sp_debug failed at testing time. Here's a list of failed tests:

2:io_utils

103:p3_standalone_py_np1

104:pyp3_cxx_vs_py_np1

Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5860/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5860/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5860/scream/components/eamxx

weaver failed

######################################################

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins3418282492211141792.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE

E3SM-Autotester · 2024-07-15T17:05:14Z

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

E3SM-Autotester · 2024-07-15T17:06:55Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5658
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: RETEST;AT: AUTOMERGE;AT: Skip v1 Testing;pyeamxx
PULLREQUESTNUM	2885
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`9627fcc`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`3314452`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5900
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: RETEST;AT: AUTOMERGE;AT: Skip v1 Testing;pyeamxx
PULLREQUESTNUM	2885
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`9627fcc`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`3314452`
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: bartgol

E3SM-Autotester · 2024-07-15T17:52:17Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5658
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: RETEST;AT: AUTOMERGE;AT: Skip v1 Testing;pyeamxx
PULLREQUESTNUM	2885
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`9627fcc`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`3314452`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5900
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: RETEST;AT: AUTOMERGE;AT: Skip v1 Testing;pyeamxx
PULLREQUESTNUM	2885
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`9627fcc`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`3314452`
TEST_REPO_ALIAS	SCREAM

SCREAM_PullRequest_Autotester_Mappy # 5658 PASSED (click to see last 100 lines of console output)


tms                 = 3054.34 sec*proc (14 tests)
Total Test time (real) = 1310.40 sec

472/474 Test #470: homme_shoc_cld_p3_mam_optics_rrtmgp_np4 ..................   Passed  514.03 sec

Start 473: homme_shoc_cld_p3_mam_optics_rrtmgp_np4_vs_np1

Start 474: homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp

473/474 Test #473: homme_shoc_cld_p3_mam_optics_rrtmgp_np4_vs_np1 ...........   Passed    0.09 sec

474/474 Test #474: homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp .........   Passed    0.14 sec
100% tests passed, 0 tests failed out of 474
Label Time Summary:

PEM                 =   7.40 secproc (60 tests)

baseline_cmp        =  12.95 secproc (17 tests)

baseline_gen        = 7670.04 secproc (19 tests)

bfbhash             =   0.54 secproc (1 test)

check               =   0.51 secproc (1 test)

cld                 = 15767.58 secproc (75 tests)

cld_fraction        =   0.97 secproc (1 test)

cosp                =  99.39 secproc (3 tests)

cxx baseline_cmp    =  52.25 secproc (2 tests)

diagnostics         = 2815.70 secproc (26 tests)

dp                  = 1196.25 secproc (6 tests)

driver              = 22329.43 secproc (78 tests)

dynamics            = 7279.86 secproc (54 tests)

fail                = 462.98 secproc (35 tests)

io                  = 448.79 secproc (53 tests)

mam4_aci            = 6750.17 secproc (34 tests)

mam4_optics         = 6620.17 secproc (19 tests)

nudging             = 1413.46 secproc (8 tests)

p3                  = 16529.76 secproc (118 tests)

p3_sk               = 626.68 secproc (32 tests)

pg2                 = 488.04 secproc (9 tests)

physics             = 23578.09 secproc (212 tests)

remap               =  22.76 secproc (4 tests)

rrtmgp              = 18566.77 secproc (86 tests)

shoc                = 16395.44 secproc (95 tests)

spa                 = 4355.26 secproc (40 tests)

surface_coupling    =  11.73 secproc (4 tests)

tms                 = 1684.12 secproc (12 tests)
Total Test time (real) = 1339.64 sec
Testing '''1963353d1e93f7a45be1f80a84776b9367d0f989''' for test '''full_sp_debug'''
RUN: taskset -c 32-63 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=32 CTEST_PARALLEL_LEVEL=32 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5658/scream/components/eamxx/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5658/scream/components/eamxx/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5658/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=mappy -DCMAKE_COMMAND="-C /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5658/scream/components/eamxx/cmake/machine-files/mappy.cmake -DNetCDF_Fortran_PATH=/projects/sems/install/rhel7-x86_64/acme/tpl/netcdf/4.7.4/gcc/11.2.0/openmpi/4.1.4/acme -DNetCDF_C_PATH=/projects/sems/install/rhel7-x86_64/acme/tpl/netcdf/4.7.4/gcc/11.2.0/openmpi/4.1.4/acme -DPnetCDF_C_PATH=/projects/sems/install/rhel7-x86_64/acme/tpl/netcdf/4.7.4/gcc/11.2.0/openmpi/4.1.4/acme -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DEAMXX_ENABLE_PYBIND=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=32 -DSCREAM_BASELINES_DIR=/sems-data-store/ACME/baselines/scream/master-baselines/full_sp_debug" '''

FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5658/scream/components/eamxx/ctest-build/full_sp_debug
Testing '''1963353d1e93f7a45be1f80a84776b9367d0f989''' for test '''release'''
RUN: taskset -c 96-127 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=32 CTEST_PARALLEL_LEVEL=32 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5658/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5658/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5658/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=mappy -DCMAKE_COMMAND="-C /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5658/scream/components/eamxx/cmake/machine-files/mappy.cmake -DNetCDF_Fortran_PATH=/projects/sems/install/rhel7-x86_64/acme/tpl/netcdf/4.7.4/gcc/11.2.0/openmpi/4.1.4/acme -DNetCDF_C_PATH=/projects/sems/install/rhel7-x86_64/acme/tpl/netcdf/4.7.4/gcc/11.2.0/openmpi/4.1.4/acme -DPnetCDF_C_PATH=/projects/sems/install/rhel7-x86_64/acme/tpl/netcdf/4.7.4/gcc/11.2.0/openmpi/4.1.4/acme -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=32 -DSCREAM_BASELINES_DIR=/sems-data-store/ACME/baselines/scream/master-baselines/release" '''

FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5658/scream/components/eamxx/ctest-build/release
Testing '''1963353d1e93f7a45be1f80a84776b9367d0f989''' for test '''full_debug'''
RUN: taskset -c 0-31 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=32 CTEST_PARALLEL_LEVEL=32 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5658/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5658/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5658/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=mappy -DCMAKE_COMMAND="-C /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5658/scream/components/eamxx/cmake/machine-files/mappy.cmake -DNetCDF_Fortran_PATH=/projects/sems/install/rhel7-x86_64/acme/tpl/netcdf/4.7.4/gcc/11.2.0/openmpi/4.1.4/acme -DNetCDF_C_PATH=/projects/sems/install/rhel7-x86_64/acme/tpl/netcdf/4.7.4/gcc/11.2.0/openmpi/4.1.4/acme -DPnetCDF_C_PATH=/projects/sems/install/rhel7-x86_64/acme/tpl/netcdf/4.7.4/gcc/11.2.0/openmpi/4.1.4/acme -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DKokkos_ENABLE_DEBUG_BOUNDS_CHECK=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=32 -DSCREAM_BASELINES_DIR=/sems-data-store/ACME/baselines/scream/master-baselines/full_debug" '''

FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5658/scream/components/eamxx/ctest-build/full_debug
Testing '''1963353d1e93f7a45be1f80a84776b9367d0f989''' for test '''debug_nopack_fpe'''
RUN: taskset -c 64-95 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=32 CTEST_PARALLEL_LEVEL=32 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5658/scream/components/eamxx/ctest-build/debug_nopack_fpe/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5658/scream/components/eamxx/ctest-build/debug_nopack_fpe -DBUILD_NAME_MOD=debug_nopack_fpe -S /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5658/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=mappy -DCMAKE_COMMAND="-C /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5658/scream/components/eamxx/cmake/machine-files/mappy.cmake -DNetCDF_Fortran_PATH=/projects/sems/install/rhel7-x86_64/acme/tpl/netcdf/4.7.4/gcc/11.2.0/openmpi/4.1.4/acme -DNetCDF_C_PATH=/projects/sems/install/rhel7-x86_64/acme/tpl/netcdf/4.7.4/gcc/11.2.0/openmpi/4.1.4/acme -DPnetCDF_C_PATH=/projects/sems/install/rhel7-x86_64/acme/tpl/netcdf/4.7.4/gcc/11.2.0/openmpi/4.1.4/acme -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_PACK_SIZE=1 -DSCREAM_FPE=True -DSCREAM_ENABLE_BASELINE_TESTS=False -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=32 -DSCREAM_ENABLE_BASELINE_TESTS=Off" '''

FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5658/scream/components/eamxx/ctest-build/debug_nopack_fpe

OVERALL STATUS: PASS

Starting analysis on mappy with cmd: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5658/scream/components/eamxx && module purge && module load sems-archive-env acme-env acme-cmake/3.26.3 acme-gcc/11.2.0 sems-archive-git/2.10.1 acme-openmpi/4.1.4 acme-netcdf/4.7.4/acme && export GATOR_INITIAL_MB=4000MB && export PATH=/ascldap/users/jgfouca/packages/valgrind-3.22.0/bin:$PATH && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m mappy

RUN: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5658/scream/components/eamxx && module purge && module load sems-archive-env acme-env acme-cmake/3.26.3 acme-gcc/11.2.0 sems-archive-git/2.10.1 acme-openmpi/4.1.4 acme-netcdf/4.7.4/acme && export GATOR_INITIAL_MB=4000MB && export PATH=/ascldap/users/jgfouca/packages/valgrind-3.22.0/bin:$PATH && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m mappy

FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5658/scream/components/eamxx

Completed analysis on mappy'

[[ 0 != 0 ]]
[[ 1 == 0 ]]
[[ mappy == \m\a\p\p\y ]]
[[ 0 == 1 ]]
[[ -z '' ]]
[[ 1 == 1 ]]
[[ 0 == 1 ]]
[[ 0 == 1 ]]
echo 'SCREAM v1 tests were skipped, since the Github label '''AT: Skip v1 Testing''' was found.\n'

SCREAM v1 tests were skipped, since the Github label 'AT: Skip v1 Testing' was found.\n
set +x

$ ssh-agent -k

unset SSH_AUTH_SOCK;

unset SSH_AGENT_PID;

echo Agent pid 27126 killed;

[ssh-agent] Stopped.

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs
squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel
[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins13703333830253417551.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: SUCCESS

SCREAM_PullRequest_Autotester_Weaver # 5900 FAILED (click to see last 100 lines of console output)


cld_fraction        =   4.67 sec*proc (1 test)
cxx baseline_cmp    =   7.84 sec*proc (2 tests)
diagnostics         =  56.13 sec*proc (23 tests)
driver              =  69.04 sec*proc (12 tests)
dynamics            =   9.52 sec*proc (3 tests)
fail                =  29.04 sec*proc (5 tests)
io                  =  61.74 sec*proc (14 tests)
mam4_aci            =  28.51 sec*proc (4 tests)
mam4_optics         =   8.37 sec*proc (1 test)
nudging             =  13.11 sec*proc (2 tests)
p3                  =  84.64 sec*proc (10 tests)
p3_sk               =  31.95 sec*proc (2 tests)
physics             = 153.90 sec*proc (23 tests)
remap               =   6.26 sec*proc (1 test)
rrtmgp              =  50.56 sec*proc (11 tests)
shoc                =  49.34 sec*proc (11 tests)
spa                 =  12.90 sec*proc (4 tests)
surface_coupling    =   4.77 sec*proc (1 test)
Total Test time (real) = 674.42 sec
Testing '''addbfda265f367771e60072a2587b24df7c52353''' for test '''full_sp_debug'''
RUN: taskset -c 52-103 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5900/scream/components/eamxx/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5900/scream/components/eamxx/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5900/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5900/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.1/gcc/11.3.0/openmpi/4.1.6/5tv5psl -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.2/gcc/11.3.0/openmpi/4.1.6/pyuuqd3 -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.6/2s52shy -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DEAMXX_ENABLE_PYBIND=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_sp_debug" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5900/scream/components/eamxx/ctest-build/full_sp_debug
Testing '''addbfda265f367771e60072a2587b24df7c52353''' for test '''release'''
RUN: taskset -c 104-155 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5900/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5900/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5900/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5900/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.1/gcc/11.3.0/openmpi/4.1.6/5tv5psl -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.2/gcc/11.3.0/openmpi/4.1.6/pyuuqd3 -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.6/2s52shy -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5900/scream/components/eamxx/ctest-build/release
Testing '''addbfda265f367771e60072a2587b24df7c52353''' for test '''full_debug'''
RUN: taskset -c 0-51 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5900/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5900/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5900/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5900/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.1/gcc/11.3.0/openmpi/4.1.6/5tv5psl -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.2/gcc/11.3.0/openmpi/4.1.6/pyuuqd3 -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.6/2s52shy -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DKokkos_ENABLE_DEBUG_BOUNDS_CHECK=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5900/scream/components/eamxx/ctest-build/full_debug

Build type full_sp_debug failed at testing time. Here'''s a list of failed tests:

2:io_utils

104:p3_standalone_py_np1

105:pyp3_cxx_vs_py_np1
Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5900/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5900/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5900/scream/components/eamxx

weaver failed'

errors='Build type full_sp_debug failed at testing time. Here'''s a list of failed tests:

2:io_utils

104:p3_standalone_py_np1

105:pyp3_cxx_vs_py_np1

Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5900/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5900/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5900/scream/components/eamxx

weaver failed'

SA_FAILURES_DETAILS+='Build type full_sp_debug failed at testing time. Here'''s a list of failed tests:

2:io_utils

104:p3_standalone_py_np1

105:pyp3_cxx_vs_py_np1

Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5900/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5900/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5900/scream/components/eamxx

weaver failed'

[[ 1 == 0 ]]
[[ weaver == \m\a\p\p\y ]]
set +x

######################################################

FAILS DETECTED:

SCREAM STANDALONE TESTING FAILED!

Build type full_sp_debug failed at testing time. Here's a list of failed tests:

2:io_utils

104:p3_standalone_py_np1

105:pyp3_cxx_vs_py_np1

Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5900/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5900/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5900/scream/components/eamxx

weaver failed

######################################################

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins14591912597887977550.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE

bartgol added AT: AUTOMERGE Inform the autotester (AT) that it can merge this PR if reviewers approved, and tests pass documentation AT: Integrate Without Testing pyeamxx labels Jun 28, 2024

bartgol requested a review from mahf708 June 28, 2024 18:46

bartgol self-assigned this Jun 28, 2024

bartgol added the AT: WIP Inform the autotester (AT) that the PR is a work in progress, and should not be tested label Jun 28, 2024

PyEAMxx: allow passing a python dict to setup PyAtmProc output

f4284a4

bartgol changed the title ~~Add a page on pyeamxx to our documentation~~ Add pyeamxx docs and enable pyeamxx testing in SP builds Jun 28, 2024

bartgol added AT: Skip v1 Testing Do not run v1 CIME-based tests for AT runs and removed AT: Integrate Without Testing AT: WIP Inform the autotester (AT) that the PR is a work in progress, and should not be tested labels Jun 28, 2024

bartgol force-pushed the bartgol/docs/pyeamxx branch from 38cb87a to 3d14934 Compare June 28, 2024 19:03

bartgol force-pushed the bartgol/docs/pyeamxx branch from a4c0c08 to 24db9af Compare July 1, 2024 15:37

mahf708 requested changes Jul 1, 2024

View reviewed changes

bartgol added 4 commits July 1, 2024 16:52

EAMxx: enable pyeamxx in SP build in test-all-scream

836c01e

EAMxx: fix syntax error in test_factory.py

98330e2

EAMxx: add pybind11 root folder to mappy/weaver mach files

98e00e7

PyEAMxx: fix overload issue in pyatmproc.hpp

9627fcc

bartgol force-pushed the bartgol/docs/pyeamxx branch from 22ef574 to 9627fcc Compare July 1, 2024 22:52

bartgol removed the documentation label Jul 1, 2024

bartgol changed the title ~~Add pyeamxx docs and enable pyeamxx testing in SP builds~~ Enable pyeamxx testing in SP builds Jul 1, 2024

bartgol requested a review from mahf708 July 1, 2024 22:53

mahf708 approved these changes Jul 1, 2024

View reviewed changes

mahf708 added the AT: RETEST Force the autotester (AT) to retest the PR label Jul 15, 2024

E3SM-Autotester removed the AT: RETEST Force the autotester (AT) to retest the PR label Jul 15, 2024

mahf708 added the AT: WIP Inform the autotester (AT) that the PR is a work in progress, and should not be tested label Jul 15, 2024

bartgol removed the AT: WIP Inform the autotester (AT) that the PR is a work in progress, and should not be tested label Jul 25, 2024

bartgol closed this Aug 8, 2024

bartgol deleted the bartgol/docs/pyeamxx branch August 29, 2024 19:29

Enable pyeamxx testing in SP builds #2885

Enable pyeamxx testing in SP builds #2885

Conversation

bartgol commented Jun 28, 2024 • edited Loading

E3SM-Autotester commented Jun 28, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Using Repos:

E3SM-Autotester commented Jun 28, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs

github-actions bot commented Jun 29, 2024 • edited Loading

E3SM-Autotester commented Jun 29, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Using Repos:

E3SM-Autotester commented Jun 29, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs

mahf708 left a comment

Choose a reason for hiding this comment

E3SM-Autotester commented Jul 1, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Using Repos:

E3SM-Autotester commented Jul 1, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs

E3SM-Autotester commented Jul 1, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Using Repos:

E3SM-Autotester commented Jul 1, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Total Test time (real) = 1300.74 sec

Testing '''e02f4e00a00211512ba429641be585c57a3364fd''' for test '''full_sp_debug'''

Testing '''e02f4e00a00211512ba429641be585c57a3364fd''' for test '''release'''

Testing '''e02f4e00a00211512ba429641be585c57a3364fd''' for test '''full_debug'''

Testing '''e02f4e00a00211512ba429641be585c57a3364fd''' for test '''debug_nopack_fpe'''

We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs

Total Test time (real) = 640.94 sec

Testing '''81273e9968b4f2db301eaa7296470702b92b0251''' for test '''full_sp_debug'''

Testing '''81273e9968b4f2db301eaa7296470702b92b0251''' for test '''release'''

Testing '''81273e9968b4f2db301eaa7296470702b92b0251''' for test '''full_debug'''

bartgol commented Jun 28, 2024 •

edited

Loading

github-actions bot commented Jun 29, 2024 •

edited

Loading