Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DP-SCREAM v0 diagnostic coordinate fix #2658

Merged
merged 1 commit into from
Jan 29, 2024
Merged

DP-SCREAM v0 diagnostic coordinate fix #2658

merged 1 commit into from
Jan 29, 2024

Conversation

bogensch
Copy link
Contributor

@bogensch bogensch commented Jan 9, 2024

Fixes DP SCREAM output coordinate issue where x-coordinates should be associated with spherep lon values and y-coordinates should be associated with lat values. This is a diagnostic output fix and thus B4B.

@tcclevenger if this section of code has already been converted to v1 could you let me know and I will make the fix.

fixes DP SCREAM output coordinate issue where x-coordinates should be…
4802bc8 
… assocated with spherep lon values and y-coordinates should be associated with lat values
@bogensch bogensch added BFB Bit for bit AT: AUTOMERGE Inform the autotester (AT) that it can merge this PR if reviewers approved, and tests pass bugfix SCREAMv0 DP-SCREAM labels Jan 9, 2024
tcclevenger
tcclevenger approved these changes Jan 9, 2024
View reviewed changes
Copy link
Contributor

@tcclevenger tcclevenger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm actually not sure how this will relate to the EAMxx version since we do not have the same outfld method. I'm assuming this is defining a field to be output? What exactly is being output here?

@bogensch
Copy link
Contributor Author

bogensch commented Jan 9, 2024

The planar grid coordinates are being output in units of meters; which are essential coordinates for diagnostic purposes.

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection is Not Necessary for this Pull Request.

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 4739
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;AT: AUTOMERGE;bugfix;SCREAMv0;DP-SCREAM
PULLREQUESTNUM 2658
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 4802bc8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA f2f2178
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5213
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;AT: AUTOMERGE;bugfix;SCREAMv0;DP-SCREAM
PULLREQUESTNUM 2658
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 4802bc8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA f2f2178
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)
  • Branch: bogensch/DPv0_coordinatefix
  • SHA: 4802bc8
  • Mode: TEST_REPO

Pull Request Author: bogensch

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 4739
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;AT: AUTOMERGE;bugfix;SCREAMv0;DP-SCREAM
PULLREQUESTNUM 2658
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 4802bc8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA f2f2178
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5213
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;AT: AUTOMERGE;bugfix;SCREAMv0;DP-SCREAM
PULLREQUESTNUM 2658
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 4802bc8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA f2f2178
TEST_REPO_ALIAS SCREAM
Console Output (last 100 lines) : SCREAM_PullRequest_Autotester_Mappy # 4739 (click to expand) 

Finished CREATE_NEWCASE for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 1.094512 seconds (PASS)
Starting XML for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 1 procs
Finished XML for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 0.966433 seconds (PASS)
Starting SETUP for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Finished XML for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 1.139920 seconds (PASS)
Finished XML for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 1.200413 seconds (PASS)
Finished XML for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 1.202994 seconds (PASS)
Finished XML for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 1.204975 seconds (PASS)
Starting SETUP for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 with 1 procs
Starting SETUP for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 procs
Starting SETUP for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Starting SETUP for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Finished XML for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 1.029521 seconds (PASS)
Starting SETUP for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 1 procs
Finished SETUP for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 2.024548 seconds (PASS)
Starting SHAREDLIB_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 procs
Finished SETUP for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 2.418462 seconds (PASS)
Starting SHAREDLIB_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Finished SETUP for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 2.365777 seconds (PASS)
Finished SETUP for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 2.408678 seconds (PASS)
Starting SHAREDLIB_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 with 1 procs
Starting SHAREDLIB_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Finished SETUP for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 2.413481 seconds (PASS)
Finished SETUP for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 2.347354 seconds (PASS)
Starting SHAREDLIB_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 1 procs
Starting SHAREDLIB_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Finished SHAREDLIB_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 47.716688 seconds (PASS)
Finished SHAREDLIB_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 47.518284 seconds (PASS)
Starting MODEL_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 16 procs
Starting MODEL_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 with 16 procs
Finished SHAREDLIB_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 61.807777 seconds (PASS)
Starting MODEL_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 16 procs
Finished SHAREDLIB_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 69.243218 seconds (PASS)
Starting MODEL_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 16 procs
Finished SHAREDLIB_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 128.576633 seconds (PASS)
Finished SHAREDLIB_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 156.865369 seconds (PASS)
Starting MODEL_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 with 16 procs
Finished MODEL_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 139.913392 seconds (PASS)
Starting RUN for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 1 proc on interactive node and 64 procs on compute nodes
Finished MODEL_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 140.527511 seconds (PASS)
Starting RUN for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 0.913594 seconds (PEND). [COMPLETED 1 of 6]
Starting MODEL_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 16 procs
Finished RUN for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 0.822818 seconds (PEND). [COMPLETED 2 of 6]
Finished MODEL_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 205.499399 seconds (PASS)
Starting RUN for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 6.507397 seconds (PEND). [COMPLETED 3 of 6]
Finished MODEL_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 231.493499 seconds (PASS)
Starting RUN for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 5.345584 seconds (PEND). [COMPLETED 4 of 6]
Finished MODEL_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 405.402808 seconds (PASS)
Starting RUN for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 1.078320 seconds (PEND). [COMPLETED 5 of 6]
Finished MODEL_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 466.144727 seconds (PASS)
Starting RUN for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 1.676705 seconds (PEND). [COMPLETED 6 of 6]
Waiting for tests to finish
PASS ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9.C.20240109_163309_8p4wks
PASS ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.C.20240109_163309_8p4wks
PASS ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2.C.20240109_163309_8p4wks
PASS ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9.C.20240109_163309_8p4wks
PASS PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.C.20240109_163309_8p4wks
PASS SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9.C.20240109_163309_8p4wks
test-scheduler took 746.4175748825073 seconds'
+ [[ 0 != 0 ]]
+ set +x
######################################################
FAILS DETECTED:
  SCREAM V0 TESTING FAILED!
######################################################
Build step 'Execute shell' marked build as failure
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 118616 killed;
[ssh-agent] Stopped.
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/


./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh


We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs


squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs scancel


[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins11125049698096843593.sh

scancel: error: No job identification provided

POST BUILD TASK : FAILURE

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE
Console Output (last 100 lines) : SCREAM_PullRequest_Autotester_Weaver # 5213 (click to expand) 

 99/112 Test  #99: create_vert_remap_and_weights ...........................   Passed    0.73 sec
        Start 100: shoc_p3_source
100/112 Test #100: shoc_p3_source ..........................................   Passed    2.27 sec
        Start 101: shoc_p3_nudged
101/112 Test #101: shoc_p3_nudged ..........................................   Passed    1.68 sec
        Start 102: shoc_p3_nudged_remapped
102/112 Test #102: shoc_p3_nudged_remapped .................................   Passed    1.74 sec
        Start 103: homme_shoc_cld_p3_rrtmgp_np1
103/112 Test #103: homme_shoc_cld_p3_rrtmgp_np1 ............................   Passed    4.15 sec
        Start 104: model_baseline
104/112 Test #104: model_baseline ..........................................   Passed    5.24 sec
        Start 105: model_initial
105/112 Test #105: model_initial ...........................................   Passed    5.76 sec
        Start 106: model_restart
106/112 Test #106: model_restart ...........................................   Passed    6.18 sec
        Start 107: restarted_vs_monolithic_check_np1
107/112 Test #107: restarted_vs_monolithic_check_np1 .......................   Passed    0.18 sec
        Start 108: homme_shoc_cld_spa_p3_rrtmgp_np1
108/112 Test #108: homme_shoc_cld_spa_p3_rrtmgp_np1 ........................   Passed   15.38 sec
        Start 109: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1
109/112 Test #109: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1 ..............   Passed   15.90 sec
        Start 110: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1
110/112 Test #110: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1 ...   Passed    1.32 sec
        Start 111: homme_shoc_cld_spa_p3_rrtmgp_pg2_np1
111/112 Test #111: homme_shoc_cld_spa_p3_rrtmgp_pg2_np1 ....................   Passed   10.24 sec
        Start 112: homme_shoc_cld_spa_p3_rrtmgp_dp_np1
112/112 Test #112: homme_shoc_cld_spa_p3_rrtmgp_dp_np1 .....................   Passed   22.31 sec

100% tests passed, 0 tests failed out of 107


Label Time Summary:

bfbhash             =   0.89 secproc (1 test)

check               =   0.90 secproc (1 test)

cld                 =  87.29 secproc (8 tests)

cld_fraction        =   1.17 secproc (1 test)

cosp                =   1.39 secproc (1 test)

diagnostics         =  26.62 secproc (17 tests)

dp                  =  22.31 secproc (1 test)

driver              = 126.46 secproc (18 tests)

dynamics            =  75.87 secproc (9 tests)

fail                =  60.13 secproc (4 tests)

horiz_remap         =   3.13 secproc (1 test)

io                  =  37.59 secproc (11 tests)

nudging             =  13.92 secproc (1 test)

p3                  = 227.44 secproc (15 tests)

p3_sk               = 175.44 secproc (2 tests)

pg2                 =  10.24 secproc (1 test)

physics             = 500.73 secproc (32 tests)

physics_nudging     =   2.27 secproc (1 test)

remap               =   3.08 secproc (1 test)

rrtmgp              =  93.60 secproc (14 tests)

shoc                = 154.43 secproc (15 tests)

spa                 =  57.99 secproc (6 tests)

surface_coupling    =   2.31 secproc (1 test)

tms                 =  10.24 secproc (1 test)


Total Test time (real) = 691.57 sec


The following tests did not run:

31 - physics_saturation_baseline_fake (Disabled)

39 - p3_baseline_f90_fake (Disabled)

40 - p3_baseline_cxx_fake (Disabled)

47 - shoc_baseline_f90_fake (Disabled)

48 - shoc_baseline_cxx_fake (Disabled)


Testing '''4802bc8aed0ecb2f32dbeaa5405e9192e8aad4d6''' for test '''release'''


RUN: taskset -c 104-155 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5213/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5213/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5213/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5213/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicxx -DCMAKE_C_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release/data" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5213/scream/components/eamxx/ctest-build/release


Testing '''4802bc8aed0ecb2f32dbeaa5405e9192e8aad4d6''' for test '''full_sp_debug'''


RUN: taskset -c 52-103 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5213/scream/components/eamxx/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5213/scream/components/eamxx/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5213/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5213/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicxx -DCMAKE_C_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_sp_debug/data" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5213/scream/components/eamxx/ctest-build/full_sp_debug


Testing '''4802bc8aed0ecb2f32dbeaa5405e9192e8aad4d6''' for test '''full_debug'''


RUN: taskset -c 0-51 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5213/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5213/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5213/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5213/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicxx -DCMAKE_C_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug/data" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5213/scream/components/eamxx/ctest-build/full_debug

OVERALL STATUS: PASS

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5213/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -p -i -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5213/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -p -i -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5213/scream/components/eamxx

Completed analysis on weaver'


    

  • [[ 0 != 0 ]]
    • 

  • [[ 1 == 0 ]]
    • 

  • [[ weaver == \m\a\p\p\y ]]
    • 

  • set +x
    
Performing Post build task...
    
Match found for : : True
    
Logical operation result is TRUE
    
Running script  : #!/bin/bash -le
    • 



cd $WORKSPACE/${BUILD_ID}/


./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins2250415876603460665.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Finished: SUCCESS

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Failure: Timed out waiting for job SCREAM_PullRequest_Autotester_Mappy to start: Total Wait = 1803

  • Other jobs have been previously started - We must stop them...

@bogensch bogensch added the AT: RETEST Force the autotester (AT) to retest the PR label Jan 11, 2024
@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 4751
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;AT: RETEST;AT: AUTOMERGE;bugfix;SCREAMv0;DP-SCREAM
PULLREQUESTNUM 2658
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 4802bc8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA f2f2178
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5222
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;AT: RETEST;AT: AUTOMERGE;bugfix;SCREAMv0;DP-SCREAM
PULLREQUESTNUM 2658
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 4802bc8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA f2f2178
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)
  • Branch: bogensch/DPv0_coordinatefix
  • SHA: 4802bc8
  • Mode: TEST_REPO

Pull Request Author: bogensch

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 4751
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;AT: RETEST;AT: AUTOMERGE;bugfix;SCREAMv0;DP-SCREAM
PULLREQUESTNUM 2658
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 4802bc8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA f2f2178
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5222
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;AT: RETEST;AT: AUTOMERGE;bugfix;SCREAMv0;DP-SCREAM
PULLREQUESTNUM 2658
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 4802bc8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA f2f2178
TEST_REPO_ALIAS SCREAM
Console Output (last 100 lines) : SCREAM_PullRequest_Autotester_Mappy # 4751 (click to expand) 

Finished CREATE_NEWCASE for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 1.076049 seconds (PASS)
Starting XML for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 1 procs
Finished XML for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 1.037071 seconds (PASS)
Finished XML for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 1.055701 seconds (PASS)
Finished XML for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 1.122426 seconds (PASS)
Finished XML for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 1.126098 seconds (PASS)
Finished XML for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 1.184688 seconds (PASS)
Starting SETUP for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 with 1 procs
Starting SETUP for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Starting SETUP for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 procs
Starting SETUP for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Starting SETUP for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Finished XML for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 0.963642 seconds (PASS)
Starting SETUP for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 1 procs
Finished SETUP for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 2.307795 seconds (PASS)
Starting SHAREDLIB_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 procs
Finished SETUP for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 2.760705 seconds (PASS)
Finished SETUP for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 2.754649 seconds (PASS)
Finished SETUP for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 2.761865 seconds (PASS)
Finished SETUP for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 2.575180 seconds (PASS)
Finished SETUP for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 2.796095 seconds (PASS)
Starting SHAREDLIB_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 1 procs
Starting SHAREDLIB_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 with 1 procs
Starting SHAREDLIB_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Starting SHAREDLIB_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Starting SHAREDLIB_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Finished SHAREDLIB_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 48.126150 seconds (PASS)
Starting MODEL_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 16 procs
Finished SHAREDLIB_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 48.286415 seconds (PASS)
Starting MODEL_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 with 16 procs
Finished SHAREDLIB_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 62.928779 seconds (PASS)
Starting MODEL_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 16 procs
Finished SHAREDLIB_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 71.324000 seconds (PASS)
Starting MODEL_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 16 procs
Finished SHAREDLIB_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 137.099566 seconds (PASS)
Finished SHAREDLIB_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 166.623286 seconds (PASS)
Starting MODEL_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 with 16 procs
Finished MODEL_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 158.753294 seconds (PASS)
Starting RUN for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished MODEL_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 158.968547 seconds (PASS)
Starting RUN for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 1.994373 seconds (PEND). [COMPLETED 1 of 6]
Finished RUN for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 1.983649 seconds (PEND). [COMPLETED 2 of 6]
Starting MODEL_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 16 procs
Finished MODEL_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 220.287374 seconds (PASS)
Starting RUN for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 6.634970 seconds (PEND). [COMPLETED 3 of 6]
Finished MODEL_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 226.094160 seconds (PASS)
Starting RUN for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 0.976320 seconds (PEND). [COMPLETED 4 of 6]
Finished MODEL_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 371.952609 seconds (PASS)
Starting RUN for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 0.977511 seconds (PEND). [COMPLETED 5 of 6]
Finished MODEL_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 470.326674 seconds (PASS)
Starting RUN for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 0.876925 seconds (PEND). [COMPLETED 6 of 6]
Waiting for tests to finish
PASS ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9.C.20240111_104529_o15u1q
PASS ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.C.20240111_104529_o15u1q
PASS ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2.C.20240111_104529_o15u1q
PASS ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9.C.20240111_104529_o15u1q
PASS PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.C.20240111_104529_o15u1q
PASS SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9.C.20240111_104529_o15u1q
test-scheduler took 768.6401371955872 seconds'
+ [[ 0 != 0 ]]
+ set +x
######################################################
FAILS DETECTED:
  SCREAM V0 TESTING FAILED!
######################################################
Build step 'Execute shell' marked build as failure
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 93863 killed;
[ssh-agent] Stopped.
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/


./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh


We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs


squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs scancel


[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins7603390414520952574.sh

scancel: error: No job identification provided

POST BUILD TASK : FAILURE

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE
Console Output (last 100 lines) : SCREAM_PullRequest_Autotester_Weaver # 5222 (click to expand) 

 99/112 Test  #99: create_vert_remap_and_weights ...........................   Passed    0.76 sec
        Start 100: shoc_p3_source
100/112 Test #100: shoc_p3_source ..........................................   Passed    2.04 sec
        Start 101: shoc_p3_nudged
101/112 Test #101: shoc_p3_nudged ..........................................   Passed    1.71 sec
        Start 102: shoc_p3_nudged_remapped
102/112 Test #102: shoc_p3_nudged_remapped .................................   Passed    1.68 sec
        Start 103: homme_shoc_cld_p3_rrtmgp_np1
103/112 Test #103: homme_shoc_cld_p3_rrtmgp_np1 ............................   Passed    3.67 sec
        Start 104: model_baseline
104/112 Test #104: model_baseline ..........................................   Passed    4.67 sec
        Start 105: model_initial
105/112 Test #105: model_initial ...........................................   Passed    4.02 sec
        Start 106: model_restart
106/112 Test #106: model_restart ...........................................   Passed    4.69 sec
        Start 107: restarted_vs_monolithic_check_np1
107/112 Test #107: restarted_vs_monolithic_check_np1 .......................   Passed    0.08 sec
        Start 108: homme_shoc_cld_spa_p3_rrtmgp_np1
108/112 Test #108: homme_shoc_cld_spa_p3_rrtmgp_np1 ........................   Passed   14.64 sec
        Start 109: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1
109/112 Test #109: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1 ..............   Passed   16.04 sec
        Start 110: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1
110/112 Test #110: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1 ...   Passed    1.27 sec
        Start 111: homme_shoc_cld_spa_p3_rrtmgp_pg2_np1
111/112 Test #111: homme_shoc_cld_spa_p3_rrtmgp_pg2_np1 ....................   Passed    9.47 sec
        Start 112: homme_shoc_cld_spa_p3_rrtmgp_dp_np1
112/112 Test #112: homme_shoc_cld_spa_p3_rrtmgp_dp_np1 .....................   Passed   22.74 sec

100% tests passed, 0 tests failed out of 107


Label Time Summary:

bfbhash             =   0.70 secproc (1 test)

check               =   0.72 secproc (1 test)

cld                 =  83.86 secproc (8 tests)

cld_fraction        =   1.02 secproc (1 test)

cosp                =   1.42 secproc (1 test)

diagnostics         =  15.98 secproc (17 tests)

dp                  =  22.74 secproc (1 test)

driver              = 119.64 secproc (18 tests)

dynamics            =  74.12 secproc (9 tests)

fail                =  71.34 secproc (4 tests)

horiz_remap         =   1.42 secproc (1 test)

io                  =  24.31 secproc (11 tests)

nudging             =  13.51 secproc (1 test)

p3                  = 272.86 secproc (15 tests)

p3_sk               = 180.93 secproc (2 tests)

pg2                 =   9.47 secproc (1 test)

physics             = 548.61 secproc (32 tests)

physics_nudging     =   1.53 secproc (1 test)

remap               =   2.17 secproc (1 test)

rrtmgp              =  90.29 secproc (14 tests)

shoc                = 147.45 secproc (15 tests)

spa                 =  55.73 secproc (6 tests)

surface_coupling    =   1.29 secproc (1 test)

tms                 =   9.47 secproc (1 test)


Total Test time (real) = 814.97 sec


The following tests did not run:

31 - physics_saturation_baseline_fake (Disabled)

39 - p3_baseline_f90_fake (Disabled)

40 - p3_baseline_cxx_fake (Disabled)

47 - shoc_baseline_f90_fake (Disabled)

48 - shoc_baseline_cxx_fake (Disabled)


Testing '''dd2b74d74f957a85c5c8b0af3e78c56214cc562d''' for test '''release'''


RUN: taskset -c 104-155 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5222/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5222/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5222/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5222/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicxx -DCMAKE_C_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release/data" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5222/scream/components/eamxx/ctest-build/release


Testing '''dd2b74d74f957a85c5c8b0af3e78c56214cc562d''' for test '''full_sp_debug'''


RUN: taskset -c 52-103 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5222/scream/components/eamxx/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5222/scream/components/eamxx/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5222/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5222/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicxx -DCMAKE_C_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_sp_debug/data" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5222/scream/components/eamxx/ctest-build/full_sp_debug


Testing '''dd2b74d74f957a85c5c8b0af3e78c56214cc562d''' for test '''full_debug'''


RUN: taskset -c 0-51 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5222/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5222/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5222/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5222/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicxx -DCMAKE_C_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug/data" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5222/scream/components/eamxx/ctest-build/full_debug

OVERALL STATUS: PASS

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5222/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -p -i -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5222/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -p -i -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5222/scream/components/eamxx

Completed analysis on weaver'


    

  • [[ 0 != 0 ]]
    • 

  • [[ 1 == 0 ]]
    • 

  • [[ weaver == \m\a\p\p\y ]]
    • 

  • set +x
    
Performing Post build task...
    
Match found for : : True
    
Logical operation result is TRUE
    
Running script  : #!/bin/bash -le
    • 



cd $WORKSPACE/${BUILD_ID}/


./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins5345555901812543842.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Finished: SUCCESS

@E3SM-Autotester E3SM-Autotester removed the AT: RETEST Force the autotester (AT) to retest the PR label Jan 11, 2024
@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - GitHub reports Mergeable status = None

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 4754
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;AT: AUTOMERGE;bugfix;SCREAMv0;DP-SCREAM
PULLREQUESTNUM 2658
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 4802bc8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA f2f2178
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5225
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;AT: AUTOMERGE;bugfix;SCREAMv0;DP-SCREAM
PULLREQUESTNUM 2658
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 4802bc8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA f2f2178
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)
  • Branch: bogensch/DPv0_coordinatefix
  • SHA: 4802bc8
  • Mode: TEST_REPO

Pull Request Author: bogensch

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 4754
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;AT: AUTOMERGE;bugfix;SCREAMv0;DP-SCREAM
PULLREQUESTNUM 2658
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 4802bc8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA f2f2178
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5225
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;AT: AUTOMERGE;bugfix;SCREAMv0;DP-SCREAM
PULLREQUESTNUM 2658
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 4802bc8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA f2f2178
TEST_REPO_ALIAS SCREAM
Console Output (last 100 lines) : SCREAM_PullRequest_Autotester_Mappy # 4754 (click to expand) 

Finished CREATE_NEWCASE for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 1.161460 seconds (PASS)
Starting XML for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 1 procs
Finished XML for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 0.869594 seconds (PASS)
Starting SETUP for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Finished XML for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 1.098866 seconds (PASS)
Finished XML for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 1.123011 seconds (PASS)
Finished XML for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 1.154897 seconds (PASS)
Finished XML for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 1.176903 seconds (PASS)
Starting SETUP for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 with 1 procs
Starting SETUP for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 procs
Starting SETUP for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Starting SETUP for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Finished XML for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 0.965518 seconds (PASS)
Starting SETUP for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 1 procs
Finished SETUP for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 1.980113 seconds (PASS)
Starting SHAREDLIB_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 procs
Finished SETUP for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 2.410469 seconds (PASS)
Starting SHAREDLIB_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Finished SETUP for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 2.354196 seconds (PASS)
Finished SETUP for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 2.403365 seconds (PASS)
Starting SHAREDLIB_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 with 1 procs
Starting SHAREDLIB_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Finished SETUP for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 2.420230 seconds (PASS)
Finished SETUP for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 2.377327 seconds (PASS)
Starting SHAREDLIB_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 1 procs
Starting SHAREDLIB_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Finished SHAREDLIB_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 46.972681 seconds (PASS)
Finished SHAREDLIB_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 46.794379 seconds (PASS)
Starting MODEL_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 16 procs
Starting MODEL_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 with 16 procs
Finished SHAREDLIB_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 61.020098 seconds (PASS)
Starting MODEL_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 16 procs
Finished SHAREDLIB_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 70.037467 seconds (PASS)
Starting MODEL_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 16 procs
Finished SHAREDLIB_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 127.514812 seconds (PASS)
Finished SHAREDLIB_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 153.948411 seconds (PASS)
Starting MODEL_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 with 16 procs
Finished MODEL_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 139.957553 seconds (PASS)
Finished MODEL_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 139.957583 seconds (PASS)
Starting RUN for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 1 proc on interactive node and 64 procs on compute nodes
Starting RUN for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 0.862594 seconds (PEND). [COMPLETED 1 of 6]
Finished RUN for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 0.880038 seconds (PEND). [COMPLETED 2 of 6]
Starting MODEL_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 16 procs
Finished MODEL_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 209.685010 seconds (PASS)
Starting RUN for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 6.429240 seconds (PEND). [COMPLETED 3 of 6]
Finished MODEL_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 243.020639 seconds (PASS)
Starting RUN for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 5.382295 seconds (PEND). [COMPLETED 4 of 6]
Finished MODEL_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 401.016808 seconds (PASS)
Starting RUN for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 0.913504 seconds (PEND). [COMPLETED 5 of 6]
Finished MODEL_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 470.052643 seconds (PASS)
Starting RUN for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 0.923257 seconds (PEND). [COMPLETED 6 of 6]
Waiting for tests to finish
PASS ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9.C.20240111_160914_y52r8s
PASS ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.C.20240111_160914_y52r8s
PASS ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2.C.20240111_160914_y52r8s
PASS ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9.C.20240111_160914_y52r8s
PASS PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.C.20240111_160914_y52r8s
PASS SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9.C.20240111_160914_y52r8s
test-scheduler took 733.786838054657 seconds'
+ [[ 0 != 0 ]]
+ set +x
######################################################
FAILS DETECTED:
  SCREAM V0 TESTING FAILED!
######################################################
Build step 'Execute shell' marked build as failure
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 109158 killed;
[ssh-agent] Stopped.
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/


./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh


We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs


squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs scancel


[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins4793445875529696749.sh

scancel: error: No job identification provided

POST BUILD TASK : FAILURE

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE
Console Output (last 100 lines) : SCREAM_PullRequest_Autotester_Weaver # 5225 (click to expand) 

        Start 100: shoc_p3_source
100/112 Test #100: shoc_p3_source ..........................................   Passed    2.01 sec
        Start 101: shoc_p3_nudged
101/112 Test #101: shoc_p3_nudged ..........................................   Passed    2.63 sec
        Start 102: shoc_p3_nudged_remapped
102/112 Test #102: shoc_p3_nudged_remapped .................................   Passed    1.75 sec
        Start 103: homme_shoc_cld_p3_rrtmgp_np1
103/112 Test #103: homme_shoc_cld_p3_rrtmgp_np1 ............................   Passed    3.67 sec
        Start 104: model_baseline
104/112 Test #104: model_baseline ..........................................   Passed    4.73 sec
        Start 105: model_initial
105/112 Test #105: model_initial ...........................................   Passed    4.11 sec
        Start 106: model_restart
106/112 Test #106: model_restart ...........................................   Passed    5.39 sec
        Start 107: restarted_vs_monolithic_check_np1
107/112 Test #107: restarted_vs_monolithic_check_np1 .......................   Passed    0.08 sec
        Start 108: homme_shoc_cld_spa_p3_rrtmgp_np1
108/112 Test #108: homme_shoc_cld_spa_p3_rrtmgp_np1 ........................   Passed   14.66 sec
        Start 109: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1
109/112 Test #109: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1 ..............   Passed   16.50 sec
        Start 110: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1
110/112 Test #110: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1 ...   Passed    1.10 sec
        Start 111: homme_shoc_cld_spa_p3_rrtmgp_pg2_np1
111/112 Test #111: homme_shoc_cld_spa_p3_rrtmgp_pg2_np1 ....................   Passed    9.43 sec
        Start 112: homme_shoc_cld_spa_p3_rrtmgp_dp_np1
112/112 Test #112: homme_shoc_cld_spa_p3_rrtmgp_dp_np1 .....................   Passed   22.87 sec

100% tests passed, 0 tests failed out of 107


Label Time Summary:

bfbhash             =   0.72 secproc (1 test)

check               =   0.74 secproc (1 test)

cld                 =  84.41 secproc (8 tests)

cld_fraction        =   1.14 secproc (1 test)

cosp                =   1.39 secproc (1 test)

diagnostics         =  16.39 secproc (17 tests)

dp                  =  22.87 secproc (1 test)

driver              = 120.44 secproc (18 tests)

dynamics            =  74.87 secproc (9 tests)

fail                =  85.29 secproc (4 tests)

horiz_remap         =   3.95 secproc (1 test)

io                  =  27.26 secproc (11 tests)

nudging             =  13.56 secproc (1 test)

p3                  = 255.60 secproc (15 tests)

p3_sk               = 177.27 secproc (2 tests)

pg2                 =   9.43 secproc (1 test)

physics             = 531.84 secproc (32 tests)

physics_nudging     =   1.30 secproc (1 test)

remap               =   3.68 secproc (1 test)

rrtmgp              =  91.21 secproc (14 tests)

shoc                = 151.59 secproc (15 tests)

spa                 =  56.14 secproc (6 tests)

surface_coupling    =   1.21 secproc (1 test)

tms                 =   9.43 secproc (1 test)


Total Test time (real) = 844.98 sec


The following tests did not run:

31 - physics_saturation_baseline_fake (Disabled)

39 - p3_baseline_f90_fake (Disabled)

40 - p3_baseline_cxx_fake (Disabled)

47 - shoc_baseline_f90_fake (Disabled)

48 - shoc_baseline_cxx_fake (Disabled)


Testing '''3631d9a8270a73b7f34ff4d3e453a30f204d3fa9''' for test '''release'''


RUN: taskset -c 104-155 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5225/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5225/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5225/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5225/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicxx -DCMAKE_C_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release/data" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5225/scream/components/eamxx/ctest-build/release


Testing '''3631d9a8270a73b7f34ff4d3e453a30f204d3fa9''' for test '''full_sp_debug'''


RUN: taskset -c 52-103 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5225/scream/components/eamxx/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5225/scream/components/eamxx/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5225/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5225/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicxx -DCMAKE_C_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_sp_debug/data" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5225/scream/components/eamxx/ctest-build/full_sp_debug


Testing '''3631d9a8270a73b7f34ff4d3e453a30f204d3fa9''' for test '''full_debug'''


RUN: taskset -c 0-51 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5225/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5225/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5225/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5225/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicxx -DCMAKE_C_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug/data" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5225/scream/components/eamxx/ctest-build/full_debug

OVERALL STATUS: PASS

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5225/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -p -i -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5225/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -p -i -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5225/scream/components/eamxx

Completed analysis on weaver'


    

  • [[ 0 != 0 ]]
    • 

  • [[ 1 == 0 ]]
    • 

  • [[ weaver == \m\a\p\p\y ]]
    • 

  • set +x
    
Performing Post build task...
    
Match found for : : True
    
Logical operation result is TRUE
    
Running script  : #!/bin/bash -le
    • 



cd $WORKSPACE/${BUILD_ID}/


./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins5806049007308492535.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: SUCCESS

@bogensch bogensch added the AT: RETEST Force the autotester (AT) to retest the PR label Jan 17, 2024
@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Failure: Timed out waiting for job SCREAM_PullRequest_Autotester_Mappy to start: Total Wait = 1803

  • Other jobs have been previously started - We must stop them...

@E3SM-Autotester E3SM-Autotester removed the AT: RETEST Force the autotester (AT) to retest the PR label Jan 17, 2024
@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 4770
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;AT: AUTOMERGE;bugfix;SCREAMv0;DP-SCREAM
PULLREQUESTNUM 2658
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 4802bc8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA f2f2178
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5241
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;AT: AUTOMERGE;bugfix;SCREAMv0;DP-SCREAM
PULLREQUESTNUM 2658
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 4802bc8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA f2f2178
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)
  • Branch: bogensch/DPv0_coordinatefix
  • SHA: 4802bc8
  • Mode: TEST_REPO

Pull Request Author: bogensch

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 4770
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;AT: AUTOMERGE;bugfix;SCREAMv0;DP-SCREAM
PULLREQUESTNUM 2658
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 4802bc8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA f2f2178
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5241
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;AT: AUTOMERGE;bugfix;SCREAMv0;DP-SCREAM
PULLREQUESTNUM 2658
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 4802bc8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA f2f2178
TEST_REPO_ALIAS SCREAM
Console Output (last 100 lines) : SCREAM_PullRequest_Autotester_Mappy # 4770 (click to expand) 

Finished CREATE_NEWCASE for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 1.104364 seconds (PASS)
Starting XML for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 1 procs
Finished XML for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 1.066472 seconds (PASS)
Finished XML for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 1.105493 seconds (PASS)
Finished XML for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 1.149864 seconds (PASS)
Finished XML for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 1.154419 seconds (PASS)
Finished XML for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 1.210728 seconds (PASS)
Starting SETUP for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 with 1 procs
Starting SETUP for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Starting SETUP for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 procs
Starting SETUP for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Starting SETUP for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Finished XML for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 0.974935 seconds (PASS)
Starting SETUP for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 1 procs
Finished SETUP for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 2.092211 seconds (PASS)
Starting SHAREDLIB_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 procs
Finished SETUP for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 2.506453 seconds (PASS)
Finished SETUP for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 2.512287 seconds (PASS)
Finished SETUP for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 2.515008 seconds (PASS)
Finished SETUP for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 2.520359 seconds (PASS)
Finished SETUP for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 2.391040 seconds (PASS)
Starting SHAREDLIB_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 1 procs
Starting SHAREDLIB_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 with 1 procs
Starting SHAREDLIB_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Starting SHAREDLIB_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Starting SHAREDLIB_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Finished SHAREDLIB_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 47.605553 seconds (PASS)
Finished SHAREDLIB_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 47.624760 seconds (PASS)
Starting MODEL_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 16 procs
Starting MODEL_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 with 16 procs
Finished SHAREDLIB_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 61.532305 seconds (PASS)
Starting MODEL_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 16 procs
Finished SHAREDLIB_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 69.302311 seconds (PASS)
Starting MODEL_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 16 procs
Finished SHAREDLIB_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 127.747330 seconds (PASS)
Finished SHAREDLIB_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 156.510920 seconds (PASS)
Starting MODEL_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 with 16 procs
Finished MODEL_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 141.055557 seconds (PASS)
Starting RUN for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 1 proc on interactive node and 64 procs on compute nodes
Finished MODEL_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 141.181804 seconds (PASS)
Starting RUN for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 0.862145 seconds (PEND). [COMPLETED 1 of 6]
Starting MODEL_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 16 procs
Finished RUN for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 0.852704 seconds (PEND). [COMPLETED 2 of 6]
Finished MODEL_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 212.056017 seconds (PASS)
Starting RUN for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 5.710582 seconds (PEND). [COMPLETED 3 of 6]
Finished MODEL_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 238.839931 seconds (PASS)
Starting RUN for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 5.650649 seconds (PEND). [COMPLETED 4 of 6]
Finished MODEL_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 403.879450 seconds (PASS)
Starting RUN for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 1.050982 seconds (PEND). [COMPLETED 5 of 6]
Finished MODEL_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 465.486621 seconds (PASS)
Starting RUN for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 1.196114 seconds (PEND). [COMPLETED 6 of 6]
Waiting for tests to finish
PASS ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9.C.20240117_155018_cdln6q
PASS ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.C.20240117_155018_cdln6q
PASS ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2.C.20240117_155018_cdln6q
PASS ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9.C.20240117_155018_cdln6q
PASS PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.C.20240117_155018_cdln6q
PASS SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9.C.20240117_155018_cdln6q
test-scheduler took 739.1603269577026 seconds'
+ [[ 0 != 0 ]]
+ set +x
######################################################
FAILS DETECTED:
  SCREAM V0 TESTING FAILED!
######################################################
Build step 'Execute shell' marked build as failure
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 12842 killed;
[ssh-agent] Stopped.
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/


./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh


We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs


squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs scancel


[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins10630468576228336511.sh

scancel: error: No job identification provided

POST BUILD TASK : FAILURE

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE
Console Output (last 100 lines) : SCREAM_PullRequest_Autotester_Weaver # 5241 (click to expand) 

 99/112 Test  #99: create_vert_remap_and_weights ...........................   Passed    4.05 sec
        Start 100: shoc_p3_source
100/112 Test #100: shoc_p3_source ..........................................   Passed   22.20 sec
        Start 101: shoc_p3_nudged
101/112 Test #101: shoc_p3_nudged ..........................................   Passed    8.01 sec
        Start 102: shoc_p3_nudged_remapped
102/112 Test #102: shoc_p3_nudged_remapped .................................   Passed    2.49 sec
        Start 103: homme_shoc_cld_p3_rrtmgp_np1
103/112 Test #103: homme_shoc_cld_p3_rrtmgp_np1 ............................   Passed   33.98 sec
        Start 104: model_baseline
104/112 Test #104: model_baseline ..........................................   Passed   25.01 sec
        Start 105: model_initial
105/112 Test #105: model_initial ...........................................   Passed   10.69 sec
        Start 106: model_restart
106/112 Test #106: model_restart ...........................................   Passed    5.11 sec
        Start 107: restarted_vs_monolithic_check_np1
107/112 Test #107: restarted_vs_monolithic_check_np1 .......................   Passed    0.39 sec
        Start 108: homme_shoc_cld_spa_p3_rrtmgp_np1
108/112 Test #108: homme_shoc_cld_spa_p3_rrtmgp_np1 ........................   Passed   36.17 sec
        Start 109: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1
109/112 Test #109: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1 ..............   Passed   36.47 sec
        Start 110: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1
110/112 Test #110: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1 ...   Passed   15.32 sec
        Start 111: homme_shoc_cld_spa_p3_rrtmgp_pg2_np1
111/112 Test #111: homme_shoc_cld_spa_p3_rrtmgp_pg2_np1 ....................   Passed   26.51 sec
        Start 112: homme_shoc_cld_spa_p3_rrtmgp_dp_np1
112/112 Test #112: homme_shoc_cld_spa_p3_rrtmgp_dp_np1 .....................   Passed   42.72 sec

100% tests passed, 0 tests failed out of 107


Label Time Summary:

bfbhash             =   3.38 secproc (1 test)

check               =   2.66 secproc (1 test)

cld                 = 234.33 secproc (8 tests)

cld_fraction        =   8.94 secproc (1 test)

cosp                =  12.70 secproc (1 test)

diagnostics         = 259.32 secproc (17 tests)

dp                  =  42.72 secproc (1 test)

driver              = 384.60 secproc (18 tests)

dynamics            = 244.92 secproc (9 tests)

fail                =  78.02 secproc (4 tests)

horiz_remap         =  12.14 secproc (1 test)

io                  =  95.42 secproc (11 tests)

nudging             =  27.72 secproc (1 test)

p3                  = 447.28 secproc (15 tests)

p3_sk               = 173.40 secproc (2 tests)

pg2                 =  26.51 secproc (1 test)

physics             = 798.42 secproc (32 tests)

physics_nudging     =   7.94 secproc (1 test)

remap               =   9.22 secproc (1 test)

rrtmgp              = 279.42 secproc (14 tests)

shoc                = 318.90 secproc (15 tests)

spa                 = 113.93 secproc (6 tests)

surface_coupling    =   8.49 secproc (1 test)

tms                 =  26.51 secproc (1 test)


Total Test time (real) = 1560.85 sec


The following tests did not run:

31 - physics_saturation_baseline_fake (Disabled)

39 - p3_baseline_f90_fake (Disabled)

40 - p3_baseline_cxx_fake (Disabled)

47 - shoc_baseline_f90_fake (Disabled)

48 - shoc_baseline_cxx_fake (Disabled)


Testing '''0b86bcda8b11d59a246d68cedfa61f6fe697e762''' for test '''full_sp_debug'''


RUN: taskset -c 52-103 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5241/scream/components/eamxx/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5241/scream/components/eamxx/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5241/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5241/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicxx -DCMAKE_C_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_sp_debug/data" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5241/scream/components/eamxx/ctest-build/full_sp_debug


Testing '''0b86bcda8b11d59a246d68cedfa61f6fe697e762''' for test '''release'''


RUN: taskset -c 104-155 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5241/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5241/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5241/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5241/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicxx -DCMAKE_C_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release/data" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5241/scream/components/eamxx/ctest-build/release


Testing '''0b86bcda8b11d59a246d68cedfa61f6fe697e762''' for test '''full_debug'''


RUN: taskset -c 0-51 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5241/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5241/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5241/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5241/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicxx -DCMAKE_C_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug/data" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5241/scream/components/eamxx/ctest-build/full_debug

OVERALL STATUS: PASS

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5241/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -p -i -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5241/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -p -i -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5241/scream/components/eamxx

Completed analysis on weaver'


    

  • [[ 0 != 0 ]]
    • 

  • [[ 1 == 0 ]]
    • 

  • [[ weaver == \m\a\p\p\y ]]
    • 

  • set +x
    
Performing Post build task...
    
Match found for : : True
    
Logical operation result is TRUE
    
Running script  : #!/bin/bash -le
    • 



cd $WORKSPACE/${BUILD_ID}/


./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins6513343260892464205.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Finished: SUCCESS

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Failure: Timed out waiting for job SCREAM_PullRequest_Autotester_Mappy to start: Total Wait = 1803

  • Other jobs have been previously started - We must stop them...

@bogensch bogensch added the AT: RETEST Force the autotester (AT) to retest the PR label Jan 24, 2024
@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 4796
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;AT: RETEST;AT: AUTOMERGE;bugfix;SCREAMv0;DP-SCREAM
PULLREQUESTNUM 2658
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 4802bc8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA f2f2178
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5261
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;AT: RETEST;AT: AUTOMERGE;bugfix;SCREAMv0;DP-SCREAM
PULLREQUESTNUM 2658
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 4802bc8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA f2f2178
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)
  • Branch: bogensch/DPv0_coordinatefix
  • SHA: 4802bc8
  • Mode: TEST_REPO

Pull Request Author: bogensch

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 4796
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;AT: RETEST;AT: AUTOMERGE;bugfix;SCREAMv0;DP-SCREAM
PULLREQUESTNUM 2658
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 4802bc8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA f2f2178
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5261
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;AT: RETEST;AT: AUTOMERGE;bugfix;SCREAMv0;DP-SCREAM
PULLREQUESTNUM 2658
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 4802bc8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA f2f2178
TEST_REPO_ALIAS SCREAM
Console Output (last 100 lines) : SCREAM_PullRequest_Autotester_Mappy # 4796 (click to expand) 

Finished CREATE_NEWCASE for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 1.079911 seconds (PASS)
Starting XML for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 1 procs
Finished XML for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 1.099069 seconds (PASS)
Finished XML for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 1.101673 seconds (PASS)
Finished XML for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 1.158380 seconds (PASS)
Finished XML for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 1.161043 seconds (PASS)
Finished XML for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 1.186040 seconds (PASS)
Starting SETUP for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 with 1 procs
Starting SETUP for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 procs
Starting SETUP for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Starting SETUP for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Starting SETUP for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Finished XML for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 0.971784 seconds (PASS)
Starting SETUP for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 1 procs
Finished SETUP for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 2.113076 seconds (PASS)
Starting SHAREDLIB_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 procs
Finished SETUP for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 2.478599 seconds (PASS)
Finished SETUP for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 2.490592 seconds (PASS)
Finished SETUP for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 2.504680 seconds (PASS)
Finished SETUP for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 2.504068 seconds (PASS)
Finished SETUP for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 2.396405 seconds (PASS)
Starting SHAREDLIB_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 1 procs
Starting SHAREDLIB_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 with 1 procs
Starting SHAREDLIB_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Starting SHAREDLIB_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Starting SHAREDLIB_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Finished SHAREDLIB_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 47.090537 seconds (PASS)
Finished SHAREDLIB_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 47.234775 seconds (PASS)
Starting MODEL_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 16 procs
Starting MODEL_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 with 16 procs
Finished SHAREDLIB_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 61.208535 seconds (PASS)
Starting MODEL_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 16 procs
Finished SHAREDLIB_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 70.924778 seconds (PASS)
Starting MODEL_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 16 procs
Finished SHAREDLIB_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 132.986688 seconds (PASS)
Finished SHAREDLIB_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 159.447631 seconds (PASS)
Starting MODEL_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 with 16 procs
Finished MODEL_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 148.218131 seconds (PASS)
Starting RUN for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 0.895326 seconds (PEND). [COMPLETED 1 of 6]
Starting MODEL_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 16 procs
Finished MODEL_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 149.938280 seconds (PASS)
Starting RUN for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 0.930801 seconds (PEND). [COMPLETED 2 of 6]
Finished MODEL_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 211.648131 seconds (PASS)
Starting RUN for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 5.734544 seconds (PEND). [COMPLETED 3 of 6]
Finished MODEL_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 242.104548 seconds (PASS)
Starting RUN for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 5.686459 seconds (PEND). [COMPLETED 4 of 6]
Finished MODEL_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 385.863204 seconds (PASS)
Starting RUN for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 1.091711 seconds (PEND). [COMPLETED 5 of 6]
Finished MODEL_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 436.749849 seconds (PASS)
Starting RUN for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 0.896198 seconds (PEND). [COMPLETED 6 of 6]
Waiting for tests to finish
PASS ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9.C.20240124_101337_fzmlhn
PASS ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.C.20240124_101337_fzmlhn
PASS ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2.C.20240124_101337_fzmlhn
PASS ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9.C.20240124_101337_fzmlhn
PASS PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.C.20240124_101337_fzmlhn
PASS SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9.C.20240124_101337_fzmlhn
test-scheduler took 676.0066423416138 seconds'
+ [[ 0 != 0 ]]
+ set +x
######################################################
FAILS DETECTED:
  SCREAM V0 TESTING FAILED!
######################################################
Build step 'Execute shell' marked build as failure
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 12405 killed;
[ssh-agent] Stopped.
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/


./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh


We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs


squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs scancel


[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins3455249987854862850.sh

scancel: error: No job identification provided

POST BUILD TASK : FAILURE

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE
Console Output (last 100 lines) : SCREAM_PullRequest_Autotester_Weaver # 5261 (click to expand) 

        Start  98: create_vert_remap_and_weights
 98/111 Test  #98: create_vert_remap_and_weights ...........................   Passed    4.33 sec
        Start  99: shoc_p3_source
 99/111 Test  #99: shoc_p3_source ..........................................   Passed   10.59 sec
        Start 100: shoc_p3_nudged
100/111 Test #100: shoc_p3_nudged ..........................................   Passed    6.33 sec
        Start 101: shoc_p3_nudged_remapped
101/111 Test #101: shoc_p3_nudged_remapped .................................   Passed    2.46 sec
        Start 102: homme_shoc_cld_p3_rrtmgp_np1
102/111 Test #102: homme_shoc_cld_p3_rrtmgp_np1 ............................   Passed   15.15 sec
        Start 103: homme_shoc_cld_p3_rrtmgp_pg2_np1
103/111 Test #103: homme_shoc_cld_p3_rrtmgp_pg2_np1 ........................   Passed   16.94 sec
        Start 104: model_baseline
104/111 Test #104: model_baseline ..........................................   Passed   15.89 sec
        Start 105: model_initial
105/111 Test #105: model_initial ...........................................   Passed    5.55 sec
        Start 106: model_restart
106/111 Test #106: model_restart ...........................................   Passed    7.91 sec
        Start 107: restarted_vs_monolithic_check_np1
107/111 Test #107: restarted_vs_monolithic_check_np1 .......................   Passed    0.21 sec
        Start 108: homme_shoc_cld_spa_p3_rrtmgp_np1
108/111 Test #108: homme_shoc_cld_spa_p3_rrtmgp_np1 ........................   Passed   17.73 sec
        Start 109: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1
109/111 Test #109: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1 ..............   Passed   18.89 sec
        Start 110: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1
110/111 Test #110: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1 ...   Passed    4.73 sec
        Start 111: homme_shoc_cld_p3_rrtmgp_dp_np1
111/111 Test #111: homme_shoc_cld_p3_rrtmgp_dp_np1 .........................   Passed   31.13 sec

100% tests passed, 0 tests failed out of 106


Label Time Summary:

bfbhash             =   1.98 secproc (1 test)

check               =   1.77 secproc (1 test)

cld                 = 133.62 secproc (8 tests)

cld_fraction        =  11.24 secproc (1 test)

cosp                =   5.48 secproc (1 test)

diagnostics         = 124.95 secproc (17 tests)

dp                  =  31.13 secproc (1 test)

driver              = 221.55 secproc (18 tests)

dynamics            = 141.73 secproc (9 tests)

fail                =  76.55 secproc (4 tests)

io                  =  67.69 secproc (11 tests)

nudging             =  11.43 secproc (1 test)

p3                  = 285.30 secproc (15 tests)

p3_sk               = 147.15 secproc (2 tests)

pg2                 =  16.94 secproc (1 test)

physics             = 570.13 secproc (32 tests)

physics_nudging     =  10.03 secproc (1 test)

remap               =   9.38 secproc (1 test)

rrtmgp              = 162.81 secproc (14 tests)

shoc                = 218.41 secproc (15 tests)

spa                 =  56.53 secproc (6 tests)

surface_coupling    =   7.70 secproc (1 test)

tms                 =  16.94 sec*proc (1 test)


Total Test time (real) = 1175.03 sec


The following tests did not run:

30 - physics_saturation_baseline_fake (Disabled)

38 - p3_baseline_f90_fake (Disabled)

39 - p3_baseline_cxx_fake (Disabled)

46 - shoc_baseline_f90_fake (Disabled)

47 - shoc_baseline_cxx_fake (Disabled)


Testing '''c69bf783d34e8e66302e4de82e98f1d3c01f1bf7''' for test '''release'''


RUN: taskset -c 104-155 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5261/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5261/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5261/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5261/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicxx -DCMAKE_C_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release/data" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5261/scream/components/eamxx/ctest-build/release


Testing '''c69bf783d34e8e66302e4de82e98f1d3c01f1bf7''' for test '''full_sp_debug'''


RUN: taskset -c 52-103 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5261/scream/components/eamxx/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5261/scream/components/eamxx/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5261/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5261/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicxx -DCMAKE_C_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_sp_debug/data" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5261/scream/components/eamxx/ctest-build/full_sp_debug


Testing '''c69bf783d34e8e66302e4de82e98f1d3c01f1bf7''' for test '''full_debug'''


RUN: taskset -c 0-51 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5261/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5261/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5261/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5261/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicxx -DCMAKE_C_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug/data" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5261/scream/components/eamxx/ctest-build/full_debug

OVERALL STATUS: PASS

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5261/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -p -i -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5261/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -p -i -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5261/scream/components/eamxx

Completed analysis on weaver'


    

  • [[ 0 != 0 ]]
    • 

  • [[ 1 == 0 ]]
    • 

  • [[ weaver == \m\a\p\p\y ]]
    • 

  • set +x
    
Performing Post build task...
    
Match found for : : True
    
Logical operation result is TRUE
    
Running script  : #!/bin/bash -le
    • 



cd $WORKSPACE/${BUILD_ID}/


./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins2571563115382182339.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Finished: SUCCESS

@E3SM-Autotester E3SM-Autotester removed the AT: RETEST Force the autotester (AT) to retest the PR label Jan 24, 2024
@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Failure: Timed out waiting for job SCREAM_PullRequest_Autotester_Mappy to start: Total Wait = 1803

  • Other jobs have been previously started - We must stop them...

@tcclevenger
Copy link
Contributor

@bogensch I can't really tell what the mappy fails are from the output here, and I think the build isn't on jenkins dashboard anymore. I'll retest and then make sure to look at the log when it finishes. Odd that it would fail since I don't think there are any V1 F90 tests that set dp_crm=true..

@tcclevenger tcclevenger added the AT: RETEST Force the autotester (AT) to retest the PR label Jan 25, 2024
@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 4806
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;AT: RETEST;AT: AUTOMERGE;bugfix;SCREAMv0;DP-SCREAM
PULLREQUESTNUM 2658
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 4802bc8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA f2f2178
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5270
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;AT: RETEST;AT: AUTOMERGE;bugfix;SCREAMv0;DP-SCREAM
PULLREQUESTNUM 2658
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 4802bc8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA f2f2178
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)
  • Branch: bogensch/DPv0_coordinatefix
  • SHA: 4802bc8
  • Mode: TEST_REPO

Pull Request Author: bogensch

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 4806
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;AT: RETEST;AT: AUTOMERGE;bugfix;SCREAMv0;DP-SCREAM
PULLREQUESTNUM 2658
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 4802bc8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA f2f2178
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5270
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;AT: RETEST;AT: AUTOMERGE;bugfix;SCREAMv0;DP-SCREAM
PULLREQUESTNUM 2658
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 4802bc8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA f2f2178
TEST_REPO_ALIAS SCREAM
Console Output (last 100 lines) : SCREAM_PullRequest_Autotester_Mappy # 4806 (click to expand) 

Finished CREATE_NEWCASE for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 1.252311 seconds (PASS)
Starting XML for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 1 procs
Finished XML for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 1.072602 seconds (PASS)
Finished XML for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 1.160484 seconds (PASS)
Finished XML for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 1.174762 seconds (PASS)
Finished XML for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 1.200448 seconds (PASS)
Finished XML for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 1.209276 seconds (PASS)
Starting SETUP for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 with 1 procs
Starting SETUP for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 procs
Starting SETUP for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Starting SETUP for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Starting SETUP for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Finished XML for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 0.897449 seconds (PASS)
Starting SETUP for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 1 procs
Finished SETUP for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 2.255808 seconds (PASS)
Starting SHAREDLIB_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 procs
Finished SETUP for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 2.693135 seconds (PASS)
Finished SETUP for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 2.739517 seconds (PASS)
Finished SETUP for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 2.737449 seconds (PASS)
Finished SETUP for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 2.741890 seconds (PASS)
Starting SHAREDLIB_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 with 1 procs
Starting SHAREDLIB_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Starting SHAREDLIB_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Starting SHAREDLIB_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Finished SETUP for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 2.526588 seconds (PASS)
Starting SHAREDLIB_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 1 procs
Finished SHAREDLIB_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 52.805924 seconds (PASS)
Starting MODEL_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 16 procs
Finished SHAREDLIB_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 53.242655 seconds (PASS)
Starting MODEL_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 with 16 procs
Finished SHAREDLIB_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 67.597284 seconds (PASS)
Starting MODEL_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 16 procs
Finished SHAREDLIB_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 75.133486 seconds (PASS)
Starting MODEL_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 16 procs
Finished SHAREDLIB_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 147.263360 seconds (PASS)
Finished SHAREDLIB_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 182.530311 seconds (PASS)
Starting MODEL_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 with 16 procs
Finished MODEL_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 164.447149 seconds (PASS)
Starting RUN for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 1 proc on interactive node and 64 procs on compute nodes
Finished MODEL_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 165.127516 seconds (PASS)
Starting RUN for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 1.067470 seconds (PEND). [COMPLETED 1 of 6]
Starting MODEL_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 16 procs
Finished RUN for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 0.929110 seconds (PEND). [COMPLETED 2 of 6]
Finished MODEL_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 229.143662 seconds (PASS)
Starting RUN for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 4.658512 seconds (PEND). [COMPLETED 3 of 6]
Finished MODEL_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 270.296811 seconds (PASS)
Starting RUN for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 5.704180 seconds (PEND). [COMPLETED 4 of 6]
Finished MODEL_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 431.958127 seconds (PASS)
Starting RUN for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 1.285937 seconds (PEND). [COMPLETED 5 of 6]
Finished MODEL_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 481.747353 seconds (PASS)
Starting RUN for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 0.942684 seconds (PEND). [COMPLETED 6 of 6]
Waiting for tests to finish
PASS ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9.C.20240125_114004_6zjar2
PASS ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.C.20240125_114004_6zjar2
PASS ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2.C.20240125_114004_6zjar2
PASS ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9.C.20240125_114004_6zjar2
PASS PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.C.20240125_114004_6zjar2
PASS SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9.C.20240125_114004_6zjar2
test-scheduler took 743.4953818321228 seconds'
+ [[ 0 != 0 ]]
+ set +x
######################################################
FAILS DETECTED:
  SCREAM V0 TESTING FAILED!
######################################################
Build step 'Execute shell' marked build as failure
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 18729 killed;
[ssh-agent] Stopped.
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/


./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh


We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs


squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs scancel


[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins96287942827295389.sh

scancel: error: No job identification provided

POST BUILD TASK : FAILURE

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE
Console Output (last 100 lines) : SCREAM_PullRequest_Autotester_Weaver # 5270 (click to expand) 

dp                  =  23.53 sec*proc (1 test)
driver              = 209.81 sec*proc (18 tests)
dynamics            = 129.42 sec*proc (9 tests)
fail                =  36.58 sec*proc (4 tests)
io                  =  76.35 sec*proc (11 tests)
nudging             =   9.25 sec*proc (1 test)
p3                  = 192.46 sec*proc (15 tests)
p3_sk               =  44.61 sec*proc (2 tests)
pg2                 =  17.57 sec*proc (1 test)
physics             = 364.70 sec*proc (32 tests)
physics_nudging     =   5.06 sec*proc (1 test)
remap               =   7.11 sec*proc (1 test)
rrtmgp              = 144.06 sec*proc (14 tests)
shoc                = 176.55 sec*proc (15 tests)
spa                 =  52.80 sec*proc (6 tests)
surface_coupling    =  11.89 sec*proc (1 test)
tms                 =  17.57 sec*proc (1 test)

Total Test time (real) = 809.61 sec


The following tests did not run:

30 - physics_saturation_baseline_fake (Disabled)

38 - p3_baseline_f90_fake (Disabled)

39 - p3_baseline_cxx_fake (Disabled)

46 - shoc_baseline_f90_fake (Disabled)

47 - shoc_baseline_cxx_fake (Disabled)

111/111 Test #111: homme_shoc_cld_p3_rrtmgp_dp_np1 .........................   Passed   32.66 sec


100% tests passed, 0 tests failed out of 106


Label Time Summary:

bfbhash             =   1.31 secproc (1 test)

check               =   1.02 secproc (1 test)

cld                 = 161.04 secproc (8 tests)

cld_fraction        =  15.50 secproc (1 test)

cosp                =  15.07 secproc (1 test)

diagnostics         = 155.71 secproc (17 tests)

dp                  =  32.66 secproc (1 test)

driver              = 327.40 secproc (18 tests)

dynamics            = 161.95 secproc (9 tests)

fail                =  15.70 secproc (4 tests)

io                  =  87.36 secproc (11 tests)

nudging             =  19.69 secproc (1 test)

p3                  = 206.27 secproc (15 tests)

p3_sk               =  11.28 secproc (2 tests)

pg2                 =  23.67 secproc (1 test)

physics             = 339.83 secproc (32 tests)

physics_nudging     =   9.87 secproc (1 test)

remap               =  13.06 secproc (1 test)

rrtmgp              = 206.82 secproc (14 tests)

shoc                = 188.92 secproc (15 tests)

spa                 =  76.27 secproc (6 tests)

surface_coupling    =  16.40 secproc (1 test)

tms                 =  23.67 sec*proc (1 test)


Total Test time (real) = 835.60 sec


The following tests did not run:

30 - physics_saturation_baseline_fake (Disabled)

38 - p3_baseline_f90_fake (Disabled)

39 - p3_baseline_cxx_fake (Disabled)

46 - shoc_baseline_f90_fake (Disabled)

47 - shoc_baseline_cxx_fake (Disabled)


Testing '''00e078199af78b53c6a7735c541f1a0c8f10a749''' for test '''full_sp_debug'''


RUN: taskset -c 52-103 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5270/scream/components/eamxx/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5270/scream/components/eamxx/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5270/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5270/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicxx -DCMAKE_C_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_sp_debug/data" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5270/scream/components/eamxx/ctest-build/full_sp_debug


Testing '''00e078199af78b53c6a7735c541f1a0c8f10a749''' for test '''full_debug'''


RUN: taskset -c 0-51 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5270/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5270/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5270/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5270/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicxx -DCMAKE_C_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug/data" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5270/scream/components/eamxx/ctest-build/full_debug


Testing '''00e078199af78b53c6a7735c541f1a0c8f10a749''' for test '''release'''


RUN: taskset -c 104-155 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5270/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5270/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5270/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5270/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicxx -DCMAKE_C_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release/data" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5270/scream/components/eamxx/ctest-build/release

OVERALL STATUS: PASS

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5270/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -p -i -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5270/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -p -i -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5270/scream/components/eamxx

Completed analysis on weaver'


    

  • [[ 0 != 0 ]]
    • 

  • [[ 1 == 0 ]]
    • 

  • [[ weaver == \m\a\p\p\y ]]
    • 

  • set +x
    
Performing Post build task...
    
Match found for : : True
    
Logical operation result is TRUE
    
Running script  : #!/bin/bash -le
    • 



cd $WORKSPACE/${BUILD_ID}/


./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins18006844606916346414.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: SUCCESS

@E3SM-Autotester E3SM-Autotester removed the AT: RETEST Force the autotester (AT) to retest the PR label Jan 25, 2024
@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 4807
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;AT: AUTOMERGE;bugfix;SCREAMv0;DP-SCREAM
PULLREQUESTNUM 2658
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 4802bc8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA f2f2178
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5271
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;AT: AUTOMERGE;bugfix;SCREAMv0;DP-SCREAM
PULLREQUESTNUM 2658
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 4802bc8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA f2f2178
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)
  • Branch: bogensch/DPv0_coordinatefix
  • SHA: 4802bc8
  • Mode: TEST_REPO

Pull Request Author: bogensch

@E3SM-Autotester
Copy link
Collaborator

NOTICE: The AutoTester has encountered an internal error (usually a Communications Timeout), testing will be restarted, previous tests may still be running but will be ignored by the AutoTester...

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Failure: Timed out waiting for job SCREAM_PullRequest_Autotester_Weaver to start: Total Wait = 1803

  • Other jobs have been previously started - We must stop them...

@tcclevenger tcclevenger added the AT: RETEST Force the autotester (AT) to retest the PR label Jan 29, 2024
@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 4817
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;AT: RETEST;AT: AUTOMERGE;bugfix;SCREAMv0;DP-SCREAM
PULLREQUESTNUM 2658
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 4802bc8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA f2f2178
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5272
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;AT: RETEST;AT: AUTOMERGE;bugfix;SCREAMv0;DP-SCREAM
PULLREQUESTNUM 2658
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 4802bc8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA f2f2178
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)
  • Branch: bogensch/DPv0_coordinatefix
  • SHA: 4802bc8
  • Mode: TEST_REPO

Pull Request Author: bogensch

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 4817
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;AT: RETEST;AT: AUTOMERGE;bugfix;SCREAMv0;DP-SCREAM
PULLREQUESTNUM 2658
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 4802bc8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA f2f2178
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5272
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;AT: RETEST;AT: AUTOMERGE;bugfix;SCREAMv0;DP-SCREAM
PULLREQUESTNUM 2658
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 4802bc8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA f2f2178
TEST_REPO_ALIAS SCREAM
Console Output (last 100 lines) : SCREAM_PullRequest_Autotester_Mappy # 4817 (click to expand) 

Starting XML for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Finished CREATE_NEWCASE for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 1.213301 seconds (PASS)
Starting XML for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 1 procs
Finished XML for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 1.033202 seconds (PASS)
Finished XML for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 1.072693 seconds (PASS)
Finished XML for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 1.134350 seconds (PASS)
Finished XML for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 1.141568 seconds (PASS)
Finished XML for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 1.169107 seconds (PASS)
Starting SETUP for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 with 1 procs
Starting SETUP for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 procs
Starting SETUP for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Starting SETUP for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Starting SETUP for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Finished XML for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 0.862755 seconds (PASS)
Starting SETUP for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 1 procs
Finished SETUP for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 2.235493 seconds (PASS)
Starting SHAREDLIB_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 procs
Finished SETUP for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 2.696434 seconds (PASS)
Finished SETUP for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 2.710473 seconds (PASS)
Finished SETUP for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 2.704509 seconds (PASS)
Finished SETUP for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 2.708698 seconds (PASS)
Starting SHAREDLIB_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 with 1 procs
Starting SHAREDLIB_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Starting SHAREDLIB_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Starting SHAREDLIB_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Finished SETUP for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 2.553073 seconds (PASS)
Starting SHAREDLIB_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 1 procs
Finished SHAREDLIB_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 58.600557 seconds (PASS)
Finished SHAREDLIB_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 58.803827 seconds (PASS)
Starting MODEL_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 16 procs
Starting MODEL_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 with 16 procs
Finished SHAREDLIB_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 73.020979 seconds (PASS)
Starting MODEL_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 16 procs
Finished SHAREDLIB_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 77.383236 seconds (PASS)
Starting MODEL_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 16 procs
Finished SHAREDLIB_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 149.154412 seconds (PASS)
Finished SHAREDLIB_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 178.466625 seconds (PASS)
Starting MODEL_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 with 16 procs
Finished MODEL_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 160.657223 seconds (PASS)
Starting RUN for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished MODEL_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 160.994654 seconds (PASS)
Starting RUN for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 0.995636 seconds (PEND). [COMPLETED 1 of 6]
Starting MODEL_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 16 procs
Finished RUN for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 0.959359 seconds (PEND). [COMPLETED 2 of 6]
Finished MODEL_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 227.142078 seconds (PASS)
Starting RUN for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 5.988844 seconds (PEND). [COMPLETED 3 of 6]
Finished MODEL_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 259.536213 seconds (PASS)
Starting RUN for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 4.828512 seconds (PEND). [COMPLETED 4 of 6]
Finished MODEL_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 417.098855 seconds (PASS)
Starting RUN for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 1.084737 seconds (PEND). [COMPLETED 5 of 6]
Finished MODEL_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 465.394371 seconds (PASS)
Starting RUN for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 0.945409 seconds (PEND). [COMPLETED 6 of 6]
Waiting for tests to finish
PASS ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9.C.20240129_083737_dolkxx
PASS ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.C.20240129_083737_dolkxx
PASS ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2.C.20240129_083737_dolkxx
PASS ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9.C.20240129_083737_dolkxx
PASS PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.C.20240129_083737_dolkxx
PASS SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9.C.20240129_083737_dolkxx
test-scheduler took 729.1394634246826 seconds'
+ [[ 0 != 0 ]]
+ set +x
######################################################
FAILS DETECTED:
  SCREAM V0 TESTING FAILED!
######################################################
Build step 'Execute shell' marked build as failure
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 46788 killed;
[ssh-agent] Stopped.
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/


./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh


We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs


squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs scancel


[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins8221646484710631776.sh

POST BUILD TASK : FAILURE

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE
Console Output (last 100 lines) : SCREAM_PullRequest_Autotester_Weaver # 5272 (click to expand) 

 98/111 Test  #98: create_vert_remap_and_weights ...........................   Passed    1.25 sec
        Start  99: shoc_p3_source
 99/111 Test  #99: shoc_p3_source ..........................................   Passed    5.80 sec
        Start 100: shoc_p3_nudged
100/111 Test #100: shoc_p3_nudged ..........................................   Passed    1.83 sec
        Start 101: shoc_p3_nudged_remapped
101/111 Test #101: shoc_p3_nudged_remapped .................................   Passed    1.80 sec
        Start 102: homme_shoc_cld_p3_rrtmgp_np1
102/111 Test #102: homme_shoc_cld_p3_rrtmgp_np1 ............................   Passed    7.49 sec
        Start 103: homme_shoc_cld_p3_rrtmgp_pg2_np1
103/111 Test #103: homme_shoc_cld_p3_rrtmgp_pg2_np1 ........................   Passed   12.91 sec
        Start 104: model_baseline
104/111 Test #104: model_baseline ..........................................   Passed   14.56 sec
        Start 105: model_initial
105/111 Test #105: model_initial ...........................................   Passed   41.08 sec
        Start 106: model_restart
106/111 Test #106: model_restart ...........................................   Passed    6.08 sec
        Start 107: restarted_vs_monolithic_check_np1
107/111 Test #107: restarted_vs_monolithic_check_np1 .......................   Passed    0.08 sec
        Start 108: homme_shoc_cld_spa_p3_rrtmgp_np1
108/111 Test #108: homme_shoc_cld_spa_p3_rrtmgp_np1 ........................   Passed   10.33 sec
        Start 109: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1
109/111 Test #109: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1 ..............   Passed   12.98 sec
        Start 110: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1
110/111 Test #110: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1 ...   Passed    1.12 sec
        Start 111: homme_shoc_cld_p3_rrtmgp_dp_np1
111/111 Test #111: homme_shoc_cld_p3_rrtmgp_dp_np1 .........................   Passed   16.82 sec

100% tests passed, 0 tests failed out of 106


Label Time Summary:

bfbhash             =   1.37 secproc (1 test)

check               =   0.96 secproc (1 test)

cld                 = 101.72 secproc (8 tests)

cld_fraction        =   4.80 secproc (1 test)

cosp                =   5.00 secproc (1 test)

diagnostics         =  56.50 secproc (17 tests)

dp                  =  16.82 secproc (1 test)

driver              = 166.94 secproc (18 tests)

dynamics            =  80.60 secproc (9 tests)

fail                =  58.97 secproc (4 tests)

io                  =  58.26 secproc (11 tests)

nudging             =  28.68 secproc (1 test)

p3                  = 293.60 secproc (15 tests)

p3_sk               = 139.47 secproc (2 tests)

pg2                 =  12.91 secproc (1 test)

physics             = 537.86 secproc (32 tests)

physics_nudging     =   3.33 secproc (1 test)

remap               =   9.23 secproc (1 test)

rrtmgp              = 127.70 secproc (14 tests)

shoc                = 166.13 secproc (15 tests)

spa                 =  52.12 secproc (6 tests)

surface_coupling    =   4.62 secproc (1 test)

tms                 =  12.91 sec*proc (1 test)


Total Test time (real) = 920.52 sec


The following tests did not run:

30 - physics_saturation_baseline_fake (Disabled)

38 - p3_baseline_f90_fake (Disabled)

39 - p3_baseline_cxx_fake (Disabled)

46 - shoc_baseline_f90_fake (Disabled)

47 - shoc_baseline_cxx_fake (Disabled)


Testing '''f135f937453b5bbb18a19e0f31216ef9bf4bd4a0''' for test '''release'''


RUN: taskset -c 104-155 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5272/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5272/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5272/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5272/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicxx -DCMAKE_C_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release/data" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5272/scream/components/eamxx/ctest-build/release


Testing '''f135f937453b5bbb18a19e0f31216ef9bf4bd4a0''' for test '''full_sp_debug'''


RUN: taskset -c 52-103 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5272/scream/components/eamxx/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5272/scream/components/eamxx/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5272/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5272/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicxx -DCMAKE_C_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_sp_debug/data" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5272/scream/components/eamxx/ctest-build/full_sp_debug


Testing '''f135f937453b5bbb18a19e0f31216ef9bf4bd4a0''' for test '''full_debug'''


RUN: taskset -c 0-51 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5272/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5272/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5272/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5272/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicxx -DCMAKE_C_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug/data" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5272/scream/components/eamxx/ctest-build/full_debug

OVERALL STATUS: PASS

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5272/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -p -i -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5272/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -p -i -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5272/scream/components/eamxx

Completed analysis on weaver'


    

  • [[ 0 != 0 ]]
    • 

  • [[ 1 == 0 ]]
    • 

  • [[ weaver == \m\a\p\p\y ]]
    • 

  • set +x
    
Performing Post build task...
    
Match found for : : True
    
Logical operation result is TRUE
    
Running script  : #!/bin/bash -le
    • 



cd $WORKSPACE/${BUILD_ID}/


./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins2136173905974686837.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: SUCCESS

@E3SM-Autotester E3SM-Autotester removed the AT: RETEST Force the autotester (AT) to retest the PR label Jan 29, 2024
@tcclevenger
Copy link
Contributor

@bogensch Looks like there is a DP v0 test and that is what is diffing (because you changed the output of 2 fields). This should be expected. If this sounds right to you, I can manually merge.

@bogensch
Copy link
Contributor Author

Thanks @tcclevenger. Yes, that would be expected.

@tcclevenger tcclevenger merged commit 2e551c3 into master Jan 29, 2024
2 of 3 checks passed
@tcclevenger tcclevenger deleted the bogensch/DPv0_coordinatefix branch January 29, 2024 21:02
@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Failure: Timed out waiting for job SCREAM_PullRequest_Autotester_Weaver to start: Total Wait = 1803

  • Other jobs have been previously started - We must stop them...

@tcclevenger tcclevenger removed the BFB Bit for bit label Jan 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tcclevenger tcclevenger tcclevenger approved these changes

Assignees
No one assigned
Labels
AT: AUTOMERGE Inform the autotester (AT) that it can merge this PR if reviewers approved, and tests pass bugfix DP-SCREAM SCREAMv0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@bogensch @E3SM-Autotester @tcclevenger