Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

time-varying inputs for radiation #2650

Closed
wants to merge 15 commits into from
Closed

time-varying inputs for radiation #2650

wants to merge 15 commits into from

Conversation

mahf708
Copy link
Contributor

@mahf708 mahf708 commented Jan 3, 2024 •
edited
Loading

GHG file for SCREAM: atm/cam/ggas/GHG_CMIP-1-2-0_Annual_Global_0000-2014_scream_c20240104.nc

@mahf708 mahf708 added the AT: RETEST Force the autotester (AT) to retest the PR label Jan 3, 2024
@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection is Not Necessary for this Pull Request.

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 4723
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: RETEST
PULLREQUESTNUM 2650
SCREAM_SOURCE_REPO https://github.com/mahf708/scream
SCREAM_SOURCE_SHA 74e2417
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 2f9def8
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5197
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: RETEST
PULLREQUESTNUM 2650
SCREAM_SOURCE_REPO https://github.com/mahf708/scream
SCREAM_SOURCE_SHA 74e2417
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 2f9def8
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (mahf708/scream)
  • Branch: time-varying-amip-inputs
  • SHA: 74e2417
  • Mode: TEST_REPO

Pull Request Author: mahf708

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 4723
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: RETEST
PULLREQUESTNUM 2650
SCREAM_SOURCE_REPO https://github.com/mahf708/scream
SCREAM_SOURCE_SHA 74e2417
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 2f9def8
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5197
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: RETEST
PULLREQUESTNUM 2650
SCREAM_SOURCE_REPO https://github.com/mahf708/scream
SCREAM_SOURCE_SHA 74e2417
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 2f9def8
TEST_REPO_ALIAS SCREAM
Console Output (last 100 lines) : SCREAM_PullRequest_Autotester_Mappy # 4723 (click to expand) 

317:rrtmgp_standalone_not_chunked_np4_vs_np1
318:rrtmgp_standalone_chunked_np4
319:rrtmgp_standalone_chunked_vs_not_chunked
324:shoc_cld_p3_rrtmgp_np1
325:shoc_cld_p3_rrtmgp_np2
326:shoc_cld_p3_rrtmgp_np3
327:shoc_cld_p3_rrtmgp_np4
328:shoc_cld_p3_rrtmgp_np2_vs_np1
329:shoc_cld_p3_rrtmgp_np3_vs_np1
330:shoc_cld_p3_rrtmgp_np4_vs_np1
331:shoc_cld_spa_p3_rrtmgp_np1
332:shoc_cld_spa_p3_rrtmgp_np2
333:shoc_cld_spa_p3_rrtmgp_np3
334:shoc_cld_spa_p3_rrtmgp_np4
335:shoc_cld_spa_p3_rrtmgp_np2_vs_np1
336:shoc_cld_spa_p3_rrtmgp_np3_vs_np1
337:shoc_cld_spa_p3_rrtmgp_np4_vs_np1
346:homme_shoc_cld_p3_rrtmgp_np1
347:homme_shoc_cld_p3_rrtmgp_np2
348:homme_shoc_cld_p3_rrtmgp_np3
349:homme_shoc_cld_p3_rrtmgp_np4
350:homme_shoc_cld_p3_rrtmgp_np2_vs_np1
351:homme_shoc_cld_p3_rrtmgp_np3_vs_np1
352:homme_shoc_cld_p3_rrtmgp_np4_vs_np1
353:model_baseline_np4
354:model_initial_np4
355:model_restart_np4
356:restarted_vs_monolithic_check_np4
357:homme_shoc_cld_spa_p3_rrtmgp_np1
358:homme_shoc_cld_spa_p3_rrtmgp_np2
359:homme_shoc_cld_spa_p3_rrtmgp_np3
360:homme_shoc_cld_spa_p3_rrtmgp_np4
361:homme_shoc_cld_spa_p3_rrtmgp_np2_vs_np1
362:homme_shoc_cld_spa_p3_rrtmgp_np3_vs_np1
363:homme_shoc_cld_spa_p3_rrtmgp_np4_vs_np1
364:homme_shoc_cld_spa_p3_rrtmgp_128levels_np1
365:homme_shoc_cld_spa_p3_rrtmgp_128levels_np2
366:homme_shoc_cld_spa_p3_rrtmgp_128levels_np3
367:homme_shoc_cld_spa_p3_rrtmgp_128levels_np4
368:homme_shoc_cld_spa_p3_rrtmgp_128levels_np2_vs_np1
369:homme_shoc_cld_spa_p3_rrtmgp_128levels_np3_vs_np1
370:homme_shoc_cld_spa_p3_rrtmgp_128levels_np4_vs_np1
371:homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1
372:homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np2
373:homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np3
374:homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np4
375:homme_shoc_cld_spa_p3_rrtmgp_pg2_np1
376:homme_shoc_cld_spa_p3_rrtmgp_pg2_np2
377:homme_shoc_cld_spa_p3_rrtmgp_pg2_np3
378:homme_shoc_cld_spa_p3_rrtmgp_pg2_np4
379:homme_shoc_cld_spa_p3_rrtmgp_pg2_np2_vs_np1
380:homme_shoc_cld_spa_p3_rrtmgp_pg2_np3_vs_np1
381:homme_shoc_cld_spa_p3_rrtmgp_pg2_np4_vs_np1
382:homme_shoc_cld_spa_p3_rrtmgp_dp_np1
383:homme_shoc_cld_spa_p3_rrtmgp_dp_np2
384:homme_shoc_cld_spa_p3_rrtmgp_dp_np3
385:homme_shoc_cld_spa_p3_rrtmgp_dp_np4
386:homme_shoc_cld_spa_p3_rrtmgp_dp_np2_vs_np1
387:homme_shoc_cld_spa_p3_rrtmgp_dp_np3_vs_np1
388:homme_shoc_cld_spa_p3_rrtmgp_dp_np4_vs_np1
  SCREAM V1 TESTING FAILED!
Waiting for tests to finish
FAIL ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9.C.20240103_092354_um85qr
FAIL ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.C.20240103_092354_um85qr
FAIL ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2.C.20240103_092354_um85qr
FAIL ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9.C.20240103_092354_um85qr
FAIL PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.C.20240103_092354_um85qr
FAIL SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9.C.20240103_092354_um85qr
test-scheduler took 536.0744757652283 seconds
######################################################
Build step 'Execute shell' marked build as failure
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 115089 killed;
[ssh-agent] Stopped.
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/


./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh


We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs


squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs scancel


[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins14823176259361297013.sh

scancel: error: No job identification provided

POST BUILD TASK : FAILURE

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE
Console Output (last 100 lines) : SCREAM_PullRequest_Autotester_Weaver # 5197 (click to expand) 

93:shoc_cld_p3_rrtmgp_np1
94:shoc_cld_spa_p3_rrtmgp_np1
103:homme_shoc_cld_p3_rrtmgp_np1
104:model_baseline
105:model_initial
106:model_restart
107:restarted_vs_monolithic_check_np1
108:homme_shoc_cld_spa_p3_rrtmgp_np1
109:homme_shoc_cld_spa_p3_rrtmgp_128levels_np1
110:homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1
111:homme_shoc_cld_spa_p3_rrtmgp_pg2_np1
112:homme_shoc_cld_spa_p3_rrtmgp_dp_np1

Build type release failed at testing time. Here'''s a list of failed tests:

88:rrtmgp_standalone_unit

89:rrtmgp_standalone_not_chunked_np1

90:rrtmgp_standalone_chunked

91:rrtmgp_standalone_chunked_vs_not_chunked

93:shoc_cld_p3_rrtmgp_np1

94:shoc_cld_spa_p3_rrtmgp_np1

103:homme_shoc_cld_p3_rrtmgp_np1

104:model_baseline

105:model_initial

106:model_restart

107:restarted_vs_monolithic_check_np1

108:homme_shoc_cld_spa_p3_rrtmgp_np1

109:homme_shoc_cld_spa_p3_rrtmgp_128levels_np1

110:homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1

111:homme_shoc_cld_spa_p3_rrtmgp_pg2_np1

112:homme_shoc_cld_spa_p3_rrtmgp_dp_np1


Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5197/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -p -i -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5197/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -p -i -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5197/scream/components/eamxx

weaver failed'


    

  • [[ 1 == 0 ]]
    • 

  • [[ weaver == \m\a\p\p\y ]]
    • 

  • set +x
    
######################################################
    
FAILS DETECTED:
    
SCREAM STANDALONE TESTING FAILED!
    
Build type full_debug failed at testing time. Here's a list of failed tests:
    
88:rrtmgp_standalone_unit
    
89:rrtmgp_standalone_not_chunked_np1
    
90:rrtmgp_standalone_chunked
    
91:rrtmgp_standalone_chunked_vs_not_chunked
    
93:shoc_cld_p3_rrtmgp_np1
    
94:shoc_cld_spa_p3_rrtmgp_np1
    
103:homme_shoc_cld_p3_rrtmgp_np1
    
104:model_baseline
    
105:model_initial
    
106:model_restart
    
107:restarted_vs_monolithic_check_np1
    
108:homme_shoc_cld_spa_p3_rrtmgp_np1
    
109:homme_shoc_cld_spa_p3_rrtmgp_128levels_np1
    
110:homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1
    
111:homme_shoc_cld_spa_p3_rrtmgp_pg2_np1
    
112:homme_shoc_cld_spa_p3_rrtmgp_dp_np1
    • 



Build type release failed at testing time. Here's a list of failed tests:

88:rrtmgp_standalone_unit

89:rrtmgp_standalone_not_chunked_np1

90:rrtmgp_standalone_chunked

91:rrtmgp_standalone_chunked_vs_not_chunked

93:shoc_cld_p3_rrtmgp_np1

94:shoc_cld_spa_p3_rrtmgp_np1

103:homme_shoc_cld_p3_rrtmgp_np1

104:model_baseline

105:model_initial

106:model_restart

107:restarted_vs_monolithic_check_np1

108:homme_shoc_cld_spa_p3_rrtmgp_np1

109:homme_shoc_cld_spa_p3_rrtmgp_128levels_np1

110:homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1

111:homme_shoc_cld_spa_p3_rrtmgp_pg2_np1

112:homme_shoc_cld_spa_p3_rrtmgp_dp_np1


Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5197/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -p -i -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5197/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -p -i -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5197/scream/components/eamxx

weaver failed

######################################################

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le


cd $WORKSPACE/${BUILD_ID}/


./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins6737297994873457965.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE

@E3SM-Autotester E3SM-Autotester removed the AT: RETEST Force the autotester (AT) to retest the PR label Jan 3, 2024
@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection is Not Necessary for this Pull Request.

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 4724
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS
PULLREQUESTNUM 2650
SCREAM_SOURCE_REPO https://github.com/mahf708/scream
SCREAM_SOURCE_SHA 8f5beff
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 2f9def8
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5198
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS
PULLREQUESTNUM 2650
SCREAM_SOURCE_REPO https://github.com/mahf708/scream
SCREAM_SOURCE_SHA 8f5beff
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 2f9def8
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (mahf708/scream)
  • Branch: time-varying-amip-inputs
  • SHA: 8f5beff
  • Mode: TEST_REPO

Pull Request Author: mahf708

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 4724
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS
PULLREQUESTNUM 2650
SCREAM_SOURCE_REPO https://github.com/mahf708/scream
SCREAM_SOURCE_SHA 8f5beff
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 2f9def8
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5198
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
PR_LABELS
PULLREQUESTNUM 2650
SCREAM_SOURCE_REPO https://github.com/mahf708/scream
SCREAM_SOURCE_SHA 8f5beff
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 2f9def8
TEST_REPO_ALIAS SCREAM
Console Output (last 100 lines) : SCREAM_PullRequest_Autotester_Mappy # 4724 (click to expand) 

Finished MODEL_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 140.828758 seconds (PASS)
Starting RUN for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 0.856894 seconds (PEND). [COMPLETED 2 of 6]
Finished MODEL_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 173.843418 seconds (PASS)
Starting RUN for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 1.000196 seconds (PEND). [COMPLETED 3 of 6]
Finished MODEL_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 185.337497 seconds (PASS)
Starting RUN for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 0.899181 seconds (PEND). [COMPLETED 4 of 6]
Finished MODEL_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 264.764961 seconds (PASS)
Starting RUN for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 1.388062 seconds (PEND). [COMPLETED 5 of 6]
Finished MODEL_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 309.319601 seconds (PASS)
Starting RUN for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 1.187403 seconds (PEND). [COMPLETED 6 of 6]
Waiting for tests to finish
FAIL ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9.C.20240103_125730_7a3ak5
FAIL ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.C.20240103_125730_7a3ak5
FAIL ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2.C.20240103_125730_7a3ak5
FAIL ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9.C.20240103_125730_7a3ak5
FAIL PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.C.20240103_125730_7a3ak5
FAIL SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9.C.20240103_125730_7a3ak5
test-scheduler took 523.3955919742584 seconds'
+ errors='Waiting for tests to finish
FAIL ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9.C.20240103_125730_7a3ak5
FAIL ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.C.20240103_125730_7a3ak5
FAIL ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2.C.20240103_125730_7a3ak5
FAIL ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9.C.20240103_125730_7a3ak5
FAIL PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.C.20240103_125730_7a3ak5
FAIL SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9.C.20240103_125730_7a3ak5
test-scheduler took 523.3955919742584 seconds'
+ V1_FAILURES_DETAILS+='Waiting for tests to finish
FAIL ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9.C.20240103_125730_7a3ak5
FAIL ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.C.20240103_125730_7a3ak5
FAIL ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2.C.20240103_125730_7a3ak5
FAIL ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9.C.20240103_125730_7a3ak5
FAIL PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.C.20240103_125730_7a3ak5
FAIL SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9.C.20240103_125730_7a3ak5
test-scheduler took 523.3955919742584 seconds'
+ set +x
######################################################
FAILS DETECTED:
  SCREAM V1 TESTING FAILED!
Waiting for tests to finish
FAIL ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9.C.20240103_125730_7a3ak5
FAIL ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.C.20240103_125730_7a3ak5
FAIL ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2.C.20240103_125730_7a3ak5
FAIL ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9.C.20240103_125730_7a3ak5
FAIL PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.C.20240103_125730_7a3ak5
FAIL SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9.C.20240103_125730_7a3ak5
test-scheduler took 523.3955919742584 seconds
######################################################
Build step 'Execute shell' marked build as failure
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 76588 killed;
[ssh-agent] Stopped.
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/


./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh


We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs


squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs scancel


[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins17331795589042793246.sh

scancel: error: No job identification provided

POST BUILD TASK : FAILURE

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE
Console Output (last 100 lines) : SCREAM_PullRequest_Autotester_Weaver # 5198 (click to expand) 

        Start 100: shoc_p3_source
100/112 Test #100: shoc_p3_source ..........................................   Passed   12.40 sec
        Start 101: shoc_p3_nudged
101/112 Test #101: shoc_p3_nudged ..........................................   Passed    2.89 sec
        Start 102: shoc_p3_nudged_remapped
102/112 Test #102: shoc_p3_nudged_remapped .................................   Passed    2.61 sec
        Start 103: homme_shoc_cld_p3_rrtmgp_np1
103/112 Test #103: homme_shoc_cld_p3_rrtmgp_np1 ............................   Passed   23.71 sec
        Start 104: model_baseline
104/112 Test #104: model_baseline ..........................................   Passed   27.61 sec
        Start 105: model_initial
105/112 Test #105: model_initial ...........................................   Passed    8.24 sec
        Start 106: model_restart
106/112 Test #106: model_restart ...........................................   Passed    5.54 sec
        Start 107: restarted_vs_monolithic_check_np1
107/112 Test #107: restarted_vs_monolithic_check_np1 .......................   Passed    0.20 sec
        Start 108: homme_shoc_cld_spa_p3_rrtmgp_np1
108/112 Test #108: homme_shoc_cld_spa_p3_rrtmgp_np1 ........................   Passed   46.21 sec
        Start 109: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1
109/112 Test #109: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1 ..............   Passed   34.32 sec
        Start 110: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1
110/112 Test #110: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1 ...   Passed    5.10 sec
        Start 111: homme_shoc_cld_spa_p3_rrtmgp_pg2_np1
111/112 Test #111: homme_shoc_cld_spa_p3_rrtmgp_pg2_np1 ....................   Passed   36.78 sec
        Start 112: homme_shoc_cld_spa_p3_rrtmgp_dp_np1
112/112 Test #112: homme_shoc_cld_spa_p3_rrtmgp_dp_np1 .....................   Passed   40.75 sec

100% tests passed, 0 tests failed out of 107


Label Time Summary:

bfbhash             =   3.76 secproc (1 test)

check               =   1.03 secproc (1 test)

cld                 = 242.24 secproc (8 tests)

cld_fraction        =   7.78 secproc (1 test)

cosp                =  15.77 secproc (1 test)

diagnostics         = 246.23 secproc (17 tests)

dp                  =  40.75 secproc (1 test)

driver              = 405.52 secproc (18 tests)

dynamics            = 214.90 secproc (9 tests)

fail                =  68.96 secproc (4 tests)

horiz_remap         =   1.62 secproc (1 test)

io                  =  26.05 secproc (11 tests)

nudging             =  30.22 secproc (1 test)

p3                  = 418.08 secproc (15 tests)

p3_sk               = 162.64 secproc (2 tests)

pg2                 =  36.78 secproc (1 test)

physics             = 754.32 secproc (32 tests)

physics_nudging     =  12.93 secproc (1 test)

remap               =   2.35 secproc (1 test)

rrtmgp              = 282.10 secproc (14 tests)

shoc                = 318.96 secproc (15 tests)

spa                 = 126.35 secproc (6 tests)

surface_coupling    =  20.73 secproc (1 test)

tms                 =  36.78 secproc (1 test)


Total Test time (real) = 1475.05 sec


The following tests did not run:

31 - physics_saturation_baseline_fake (Disabled)

39 - p3_baseline_f90_fake (Disabled)

40 - p3_baseline_cxx_fake (Disabled)

47 - shoc_baseline_f90_fake (Disabled)

48 - shoc_baseline_cxx_fake (Disabled)


Testing '''8f5beff791db75484363892c50c126aedffec65d''' for test '''release'''


RUN: taskset -c 104-155 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5198/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5198/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5198/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5198/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicxx -DCMAKE_C_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release/data" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5198/scream/components/eamxx/ctest-build/release


Testing '''8f5beff791db75484363892c50c126aedffec65d''' for test '''full_sp_debug'''


RUN: taskset -c 52-103 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5198/scream/components/eamxx/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5198/scream/components/eamxx/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5198/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5198/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicxx -DCMAKE_C_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_sp_debug/data" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5198/scream/components/eamxx/ctest-build/full_sp_debug


Testing '''8f5beff791db75484363892c50c126aedffec65d''' for test '''full_debug'''


RUN: taskset -c 0-51 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5198/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5198/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5198/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5198/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicxx -DCMAKE_C_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug/data" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5198/scream/components/eamxx/ctest-build/full_debug

OVERALL STATUS: PASS

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5198/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -p -i -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5198/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -p -i -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5198/scream/components/eamxx

Completed analysis on weaver'


    

  • [[ 0 != 0 ]]
    • 

  • [[ 1 == 0 ]]
    • 

  • [[ weaver == \m\a\p\p\y ]]
    • 

  • set +x
    
Performing Post build task...
    
Match found for : : True
    
Logical operation result is TRUE
    
Running script  : #!/bin/bash -le
    • 



cd $WORKSPACE/${BUILD_ID}/


./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins5112203973928354419.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: SUCCESS

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 4725
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS
PULLREQUESTNUM 2650
SCREAM_SOURCE_REPO https://github.com/mahf708/scream
SCREAM_SOURCE_SHA 8f5beff
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 2f9def8
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5199
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS
PULLREQUESTNUM 2650
SCREAM_SOURCE_REPO https://github.com/mahf708/scream
SCREAM_SOURCE_SHA 8f5beff
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 2f9def8
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (mahf708/scream)
  • Branch: time-varying-amip-inputs
  • SHA: 8f5beff
  • Mode: TEST_REPO

Pull Request Author: mahf708

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Error: Jenkins Jobs - A user has pushed a change to the PR before testing completed. NEW EVENT 'committed', ID C_kwDOCEfuetoAKDRmNWZmMDJjYzBkODAxOTFiNTQxNDBhM2QzN2ZjMTM4MzliODQ2NDk... The Jenkins Jobs will be shutdown; Testing of this PR must occur again.

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 4725
  • Status: ERROR

Jenkins Parameters

Parameter Name Value
PR_LABELS
PULLREQUESTNUM 2650
SCREAM_SOURCE_REPO https://github.com/mahf708/scream
SCREAM_SOURCE_SHA 8f5beff
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 2f9def8
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5199
  • Status: ERROR

Jenkins Parameters

Parameter Name Value
PR_LABELS
PULLREQUESTNUM 2650
SCREAM_SOURCE_REPO https://github.com/mahf708/scream
SCREAM_SOURCE_SHA 8f5beff
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 2f9def8
TEST_REPO_ALIAS SCREAM
Console Output (last 100 lines) : SCREAM_PullRequest_Autotester_Mappy # 4725 (click to expand) 

Starting SHAREDLIB_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 1 procs
Starting SHAREDLIB_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 with 1 procs
Starting SHAREDLIB_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Starting SHAREDLIB_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Starting SHAREDLIB_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Finished SHAREDLIB_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 47.638065 seconds (PASS)
Starting MODEL_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 with 16 procs
Finished SHAREDLIB_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 47.732971 seconds (PASS)
Starting MODEL_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 16 procs
Finished SHAREDLIB_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 61.501094 seconds (PASS)
Starting MODEL_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 16 procs
Finished SHAREDLIB_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 69.592919 seconds (PASS)
Starting MODEL_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 16 procs
Finished SHAREDLIB_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 127.445833 seconds (PASS)
Finished SHAREDLIB_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 155.413729 seconds (PASS)
Starting MODEL_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 with 16 procs
Finished MODEL_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 139.695279 seconds (PASS)
Starting RUN for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 0.874613 seconds (PEND). [COMPLETED 1 of 6]
Starting MODEL_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 16 procs
Finished MODEL_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 141.927387 seconds (PASS)
Starting RUN for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 0.867743 seconds (PEND). [COMPLETED 2 of 6]
Finished MODEL_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 175.632218 seconds (PASS)
Starting RUN for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 0.982192 seconds (PEND). [COMPLETED 3 of 6]
Finished MODEL_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 186.641383 seconds (PASS)
Starting RUN for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 0.864634 seconds (PEND). [COMPLETED 4 of 6]
Finished MODEL_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 260.370389 seconds (PASS)
Starting RUN for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 1.266507 seconds (PEND). [COMPLETED 5 of 6]
Finished MODEL_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 314.214690 seconds (PASS)
Starting RUN for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 1.679338 seconds (PEND). [COMPLETED 6 of 6]
Waiting for tests to finish
FAIL ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9.C.20240103_162519_ks1pvq
FAIL ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.C.20240103_162519_ks1pvq
FAIL ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2.C.20240103_162519_ks1pvq
FAIL ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9.C.20240103_162519_ks1pvq
FAIL PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.C.20240103_162519_ks1pvq
FAIL SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9.C.20240103_162519_ks1pvq
test-scheduler took 528.8679020404816 seconds'
+ errors='Waiting for tests to finish
FAIL ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9.C.20240103_162519_ks1pvq
FAIL ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.C.20240103_162519_ks1pvq
FAIL ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2.C.20240103_162519_ks1pvq
FAIL ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9.C.20240103_162519_ks1pvq
FAIL PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.C.20240103_162519_ks1pvq
FAIL SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9.C.20240103_162519_ks1pvq
test-scheduler took 528.8679020404816 seconds'
+ V1_FAILURES_DETAILS+='Waiting for tests to finish
FAIL ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9.C.20240103_162519_ks1pvq
FAIL ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.C.20240103_162519_ks1pvq
FAIL ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2.C.20240103_162519_ks1pvq
FAIL ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9.C.20240103_162519_ks1pvq
FAIL PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.C.20240103_162519_ks1pvq
FAIL SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9.C.20240103_162519_ks1pvq
test-scheduler took 528.8679020404816 seconds'
+ set +x
######################################################
FAILS DETECTED:
  SCREAM V1 TESTING FAILED!
Waiting for tests to finish
FAIL ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9.C.20240103_162519_ks1pvq
FAIL ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.C.20240103_162519_ks1pvq
FAIL ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2.C.20240103_162519_ks1pvq
FAIL ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9.C.20240103_162519_ks1pvq
FAIL PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.C.20240103_162519_ks1pvq
FAIL SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9.C.20240103_162519_ks1pvq
test-scheduler took 528.8679020404816 seconds
######################################################
scancel: error: No job identification provided
POST BUILD TASK : FAILURE
END OF POST BUILD TASK : 0
Finished: ABORTED
Console Output (last 100 lines) : SCREAM_PullRequest_Autotester_Weaver # 5199 (click to expand) 

 -> Test release baselines are valid and do not need to be regenerated
###############################################################################
Running tests!
###############################################################################
Run dashboard with model Experimental
   Source directory: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5199/scream/components/eamxx
   Build directory: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5199/scream/components/eamxx/ctest-build/release
   Group: E3SM_SCREAM
Run dashboard with model Experimental
   Source directory: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5199/scream/components/eamxx
   Build directory: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5199/scream/components/eamxx/ctest-build/full_debug
   Group: E3SM_SCREAM
Run dashboard with model Experimental
   Source directory: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5199/scream/components/eamxx
   Build directory: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5199/scream/components/eamxx/ctest-build/full_sp_debug
   Group: E3SM_SCREAM
   Reading ctest configuration file: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5199/scream/components/eamxx/CTestConfig.cmake
   Reading ctest configuration file: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5199/scream/components/eamxx/CTestConfig.cmake
   Reading ctest configuration file: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5199/scream/components/eamxx/CTestConfig.cmake
   Site: weaver
   Build name: scream_unit_tests_release
   Site: weaver
   Build name: scream_unit_tests_full_sp_debug
   Site: weaver
   Build name: scream_unit_tests_full_debug
   Use E3SM_SCREAM tag: 20240103-2325
   Use E3SM_SCREAM tag: 20240103-2325
   Use E3SM_SCREAM tag: 20240103-2325
Configure project
Configure project
Configure project
   Each . represents 1024 bytes of output
       Each . represents 1024 bytes of output
       Each . represents 1024 bytes of output
    .................................................................... Size of output: 21K
Build project
   Each symbol represents 1024 bytes of output.
   '\''!'\'' represents an error and '\''*'\'' a warning.
    ........*******. Size of output: 22K
Build project
   Each symbol represents 1024 bytes of output.
   '\''!'\'' represents an error and '\''*'\'' a warning.
     Size of output: 22K
Build project
   Each symbol represents 1024 bytes of output.
   '\''!'\'' represents an error and '\''*'\'' a warning.
    ..............******..********....**********..**.*************.**.*****.......................  Size: 50K
    ..........................  Size: 50K
    ...............................................................................  Size: 99K
    .............*.  Size: 100K
    ...................................................  Size: 149K
    ................................................**.....*........*..........*.............  Size: 199K
    ...........................*.....*............**.............  Size: 149K
    ..............  Size: 249K
    .....................  Size: 49K
    ..............................................  Size: 299K
    ....................................................  Size: 349K
    ..........................................................  Size: 400K
    ...................................................................  Size: 449K
    .......................................................  Size: 199K
    ...........  Size: 500K
    ...............................................................................  Size: 250K
    ...................................................  Size: 299K
    ........................................................  Size: 350K
    ............................................................  Size: 399K
    ..................................................  Size: 449K
    ......................................................................  Size: 499K
    ..................................................  Size: 549K
    ..........................................................  Size: 599K
    ....................................................  Size: 649K
    .....................................................  Size: 700K
    ..................................................  Size: 749K
    ...................................................  Size: 800K
    ......................................  Size: 549K
    ............................................................  Size: 599K
    ....................................  Size: 849K
    ............................................  Size: 650K
    ...................................................  Size: 699K
    ......................................................  Size: 749K
    ..........................................................  Size: 800K
    ....................................................  Size: 849K
    ..................................................  Size: 900K
    .....................................................  Size: 950K
    .........................................................  Size: 1000K
    .............................................  Size: 900K
    ...................Terminated
Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5199/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -p -i -m weaver
RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5199/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -p -i -m weaver
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5199/scream/components/eamxx
weaver failed'
+ errors=
+ SA_FAILURES_DETAILS+=
+ [[ 1 == 0 ]]
+ [[ weaver == \m\a\p\p\y ]]
+ set +x
######################################################
FAILS DETECTED:
  SCREAM STANDALONE TESTING FAILED!

######################################################

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection is Not Necessary for this Pull Request.

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 4726
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS
PULLREQUESTNUM 2650
SCREAM_SOURCE_REPO https://github.com/mahf708/scream
SCREAM_SOURCE_SHA 4f5ff02
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 2f9def8
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5200
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS
PULLREQUESTNUM 2650
SCREAM_SOURCE_REPO https://github.com/mahf708/scream
SCREAM_SOURCE_SHA 4f5ff02
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 2f9def8
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (mahf708/scream)
  • Branch: time-varying-amip-inputs
  • SHA: 4f5ff02
  • Mode: TEST_REPO

Pull Request Author: mahf708

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 4726
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
PR_LABELS
PULLREQUESTNUM 2650
SCREAM_SOURCE_REPO https://github.com/mahf708/scream
SCREAM_SOURCE_SHA 4f5ff02
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 2f9def8
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5200
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
PR_LABELS
PULLREQUESTNUM 2650
SCREAM_SOURCE_REPO https://github.com/mahf708/scream
SCREAM_SOURCE_SHA 4f5ff02
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 2f9def8
TEST_REPO_ALIAS SCREAM

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
WARNING: NO REVIEWERS HAVE BEEN REQUESTED FOR THIS PULL REQUEST!

@E3SM-Autotester
Copy link
Collaborator

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

@mahf708
Copy link
Contributor Author

mahf708 commented Jan 4, 2024

@brhillman @AaronDonahue this prototype PR enables (in theory) time-varying CO2 to be used for the AMIP runs (it is off by default and I haven't really tested it). Without much effort, we can also enable O3 or whatever anything else from the active_gases. The implementation was written in haste just to get it out, but it could be refined. In general, two questions must be answered explicitly before we move forward:

  1. Do we want this? If so, we should try to do the same for all active gases as applicable not just CO2.
  2. We should also do the same thing for SPA inputs.

Extending this to O3, other active gases, and SPA inputs isn't much more work, but it is not trivial work and will require some testing that's likely beyond my ability and bandwidth at this moment (someone more familiar with our testing infrastructure can potentially do it pretty quickly; it will likely take me a while to figure out how to test this outside of production).

@AaronDonahue if you have a moment to skim the PR, let me know if you see anything wrong :)

@mahf708 mahf708 added AT: WIP Inform the autotester (AT) that the PR is a work in progress, and should not be tested AT: RETEST Force the autotester (AT) to retest the PR and removed AT: WIP Inform the autotester (AT) that the PR is a work in progress, and should not be tested labels Jan 4, 2024
@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection is Not Necessary for this Pull Request.

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 4727
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: RETEST
PULLREQUESTNUM 2650
SCREAM_SOURCE_REPO https://github.com/mahf708/scream
SCREAM_SOURCE_SHA 2c450ca
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 2f9def8
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5201
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: RETEST
PULLREQUESTNUM 2650
SCREAM_SOURCE_REPO https://github.com/mahf708/scream
SCREAM_SOURCE_SHA 2c450ca
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 2f9def8
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (mahf708/scream)
  • Branch: time-varying-amip-inputs
  • SHA: 2c450ca
  • Mode: TEST_REPO

Pull Request Author: mahf708

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Error: Jenkins Jobs - A user has pushed a change to the PR before testing completed. NEW EVENT 'committed', ID C_kwDOCEfuetoAKGFjYzYwMjNiYTY2MmQ5Y2Y2ODgyNzMxZGMzNmIwMmIzMDM0Mzk0NWQ... The Jenkins Jobs will be shutdown; Testing of this PR must occur again.

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 4727
  • Status: ERROR

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: RETEST
PULLREQUESTNUM 2650
SCREAM_SOURCE_REPO https://github.com/mahf708/scream
SCREAM_SOURCE_SHA 2c450ca
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 2f9def8
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5201
  • Status: ERROR

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: RETEST
PULLREQUESTNUM 2650
SCREAM_SOURCE_REPO https://github.com/mahf708/scream
SCREAM_SOURCE_SHA 2c450ca
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 2f9def8
TEST_REPO_ALIAS SCREAM
Console Output (last 100 lines) : SCREAM_PullRequest_Autotester_Mappy # 4727 (click to expand) 

Finished CREATE_NEWCASE for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 0.642148 seconds (PASS)
Starting XML for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 procs
Starting XML for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Starting XML for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 with 1 procs
Starting XML for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Starting XML for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Finished CREATE_NEWCASE for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 1.091477 seconds (PASS)
Starting XML for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 1 procs
Finished XML for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 1.116369 seconds (PASS)
Finished XML for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 1.164633 seconds (PASS)
Finished XML for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 1.215507 seconds (PASS)
Finished XML for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 1.222353 seconds (PASS)
Finished XML for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 1.226273 seconds (PASS)
Starting SETUP for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 procs
Starting SETUP for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Starting SETUP for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 with 1 procs
Starting SETUP for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Starting SETUP for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Finished XML for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 1.006384 seconds (PASS)
Starting SETUP for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 1 procs
Finished SETUP for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 3.096331 seconds (PASS)
Starting SHAREDLIB_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 procs
Finished SETUP for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 3.494792 seconds (PASS)
Finished SETUP for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 3.498883 seconds (PASS)
Finished SETUP for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 3.500658 seconds (PASS)
Finished SETUP for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 3.320921 seconds (PASS)
Finished SETUP for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 3.551592 seconds (PASS)
Starting SHAREDLIB_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 1 procs
Starting SHAREDLIB_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Starting SHAREDLIB_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 with 1 procs
Starting SHAREDLIB_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Starting SHAREDLIB_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 procs
Finished SHAREDLIB_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 48.301332 seconds (PASS)
Finished SHAREDLIB_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 48.324822 seconds (PASS)
Starting MODEL_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 16 procs
Starting MODEL_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 with 16 procs
Finished SHAREDLIB_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 62.683066 seconds (PASS)
Starting MODEL_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 16 procs
Finished SHAREDLIB_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 69.990462 seconds (PASS)
Starting MODEL_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 16 procs
Finished SHAREDLIB_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 128.645099 seconds (PASS)
Finished SHAREDLIB_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 156.554884 seconds (PASS)
Starting MODEL_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 with 16 procs
Finished MODEL_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 139.304408 seconds (PASS)
Starting RUN for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 in 0.887930 seconds (PEND). [COMPLETED 1 of 6]
Starting MODEL_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 16 procs
Finished MODEL_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 141.656887 seconds (PASS)
Starting RUN for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 in 0.872051 seconds (PEND). [COMPLETED 2 of 6]
Finished MODEL_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 211.279987 seconds (PASS)
Starting RUN for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 5.735573 seconds (PEND). [COMPLETED 3 of 6]
Finished MODEL_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 237.215635 seconds (PASS)
Starting RUN for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 6.011133 seconds (PEND). [COMPLETED 4 of 6]
Finished MODEL_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 396.676387 seconds (PASS)
Starting RUN for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 in 0.963460 seconds (PEND). [COMPLETED 5 of 6]
Finished MODEL_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 465.347065 seconds (PASS)
Starting RUN for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 in 2.245507 seconds (PEND). [COMPLETED 6 of 6]
Waiting for tests to finish
PASS ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9.C.20240104_135543_1fnfrk
PASS ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.C.20240104_135543_1fnfrk
PASS ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2.C.20240104_135543_1fnfrk
PASS ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9.C.20240104_135543_1fnfrk
PASS PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.C.20240104_135543_1fnfrk
PASS SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9.C.20240104_135543_1fnfrk
test-scheduler took 740.2096590995789 seconds'
+ [[ 0 != 0 ]]
+ set +x
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 16349 killed;
[ssh-agent] Stopped.
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/


./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh


We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs


squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs scancel


[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins16224039119623316472.sh

scancel: error: No job identification provided

POST BUILD TASK : FAILURE

END OF POST BUILD TASK : 0

Finished: SUCCESS
Console Output (last 100 lines) : SCREAM_PullRequest_Autotester_Weaver # 5201 (click to expand) 

export __LMOD_REF_COUNT_PKG_CONFIG_PATH;
PKG_CONFIG_PATH=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe/lib/pkgconfig;
export PKG_CONFIG_PATH;
PYTHON_BIN=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe/bin;
export PYTHON_BIN;
PYTHON_INC=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe/include;
export PYTHON_INC;
PYTHON_LIB=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe/lib;
export PYTHON_LIB;
PYTHON_ROOT=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe;
export PYTHON_ROOT;
PYTHON_VERSION=3.10.8;
export PYTHON_VERSION;
_LMFILES_=/projects/ppc64le-pwr9-rhel8/modulefiles/lmod/utilities/linux-rhel8-ppc64le/Core/python/3.10.8.lua;
export _LMFILES_;
_ModuleTable001_=X01vZHVsZVRhYmxlXyA9IHsKTVR2ZXJzaW9uID0gMywKY19yZWJ1aWxkVGltZSA9IGZhbHNlLApjX3Nob3J0VGltZSA9IGZhbHNlLApkZXB0aFQgPSB7fSwKZmFtaWx5ID0ge30sCm1UID0gewpweXRob24gPSB7CmZuID0gIi9wcm9qZWN0cy9wcGM2NGxlLXB3cjktcmhlbDgvbW9kdWxlZmlsZXMvbG1vZC91dGlsaXRpZXMvbGludXgtcmhlbDgtcHBjNjRsZS9Db3JlL3B5dGhvbi8zLjEwLjgubHVhIiwKZnVsbE5hbWUgPSAicHl0aG9uLzMuMTAuOCIsCmxvYWRPcmRlciA9IDEsCnByb3BUID0ge30sCnN0YWNrRGVwdGggPSAwLApzdGF0dXMgPSAiYWN0aXZlIiwKdXNlck5hbWUgPSAicHl0aG9uLzMuMTAuOCIsCndWID0gIjAwMDAwMDAwMy4wMDAwMDAwMTAuMDAwMDAwMDA4Lip6;
export _ModuleTable001_;
_ModuleTable002_=ZmluYWwiLAp9LAp9LAptcGF0aEEgPSB7CiIvcHJvamVjdHMvcHBjNjRsZS1wd3I5LXJoZWw4L21vZHVsZWZpbGVzL2xtb2QvY29tcGlsZXJzIiwgIi9wcm9qZWN0cy9wcGM2NGxlLXB3cjktcmhlbDgvbW9kdWxlZmlsZXMvbG1vZC91dGlsaXRpZXMvbGludXgtcmhlbDgtcHBjNjRsZS9Db3JlIiwKfSwKc3lzdGVtQmFzZU1QQVRIID0gIi9wcm9qZWN0cy9wcGM2NGxlLXB3cjktcmhlbDgvbW9kdWxlZmlsZXMvbG1vZC9jb21waWxlcnM6L3Byb2plY3RzL3BwYzY0bGUtcHdyOS1yaGVsOC9tb2R1bGVmaWxlcy9sbW9kL3V0aWxpdGllcy9saW51eC1yaGVsOC1wcGM2NGxlL0NvcmUiLAp9Cg==;
export _ModuleTable002_;
_ModuleTable_Sz_=2;
export _ModuleTable_Sz_;'
+++ __LMOD_REF_COUNT_CMAKE_PREFIX_PATH=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe:1
+++ export __LMOD_REF_COUNT_CMAKE_PREFIX_PATH
+++ CMAKE_PREFIX_PATH=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe
+++ export CMAKE_PREFIX_PATH
+++ __LMOD_REF_COUNT_LD_LIBRARY_PATH='/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe/lib:1;/opt/lsf/10.1/linux3.10-glibc2.17-ppc64le/lib:1'
+++ export __LMOD_REF_COUNT_LD_LIBRARY_PATH
+++ LD_LIBRARY_PATH=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe/lib:/opt/lsf/10.1/linux3.10-glibc2.17-ppc64le/lib
+++ export LD_LIBRARY_PATH
+++ __LMOD_REF_COUNT_LIBRARY_PATH=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe/lib:1
+++ export __LMOD_REF_COUNT_LIBRARY_PATH
+++ LIBRARY_PATH=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe/lib
+++ export LIBRARY_PATH
+++ LOADEDMODULES=python/3.10.8
+++ export LOADEDMODULES
+++ __LMOD_REF_COUNT_MANPATH='/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe/share/man:1;/opt/lsf/10.1/man:1'
+++ export __LMOD_REF_COUNT_MANPATH
+++ MANPATH=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe/share/man:/opt/lsf/10.1/man::
+++ export MANPATH
+++ __LMOD_REF_COUNT_MODULEPATH='/projects/ppc64le-pwr9-rhel8/modulefiles/lmod/compilers:1;/projects/ppc64le-pwr9-rhel8/modulefiles/lmod/utilities/linux-rhel8-ppc64le/Core:1'
+++ export __LMOD_REF_COUNT_MODULEPATH
+++ MODULEPATH=/projects/ppc64le-pwr9-rhel8/modulefiles/lmod/compilers:/projects/ppc64le-pwr9-rhel8/modulefiles/lmod/utilities/linux-rhel8-ppc64le/Core
+++ export MODULEPATH
+++ __LMOD_REF_COUNT_PATH='/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe/bin:1;/opt/lsf/10.1/linux3.10-glibc2.17-ppc64le/etc:1;/opt/lsf/10.1/linux3.10-glibc2.17-ppc64le/bin:1;/usr/local/bin:1;/usr/bin:1;/usr/local/sbin:1;/usr/sbin:1;/home/e3sm-jenkins/.local/bin:1;/home/e3sm-jenkins/bin:1'
+++ export __LMOD_REF_COUNT_PATH
+++ PATH=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe/bin:/opt/lsf/10.1/linux3.10-glibc2.17-ppc64le/etc:/opt/lsf/10.1/linux3.10-glibc2.17-ppc64le/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/e3sm-jenkins/.local/bin:/home/e3sm-jenkins/bin
+++ export PATH
+++ __LMOD_REF_COUNT_PKG_CONFIG_PATH=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe/lib/pkgconfig:1
+++ export __LMOD_REF_COUNT_PKG_CONFIG_PATH
+++ PKG_CONFIG_PATH=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe/lib/pkgconfig
+++ export PKG_CONFIG_PATH
+++ PYTHON_BIN=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe/bin
+++ export PYTHON_BIN
+++ PYTHON_INC=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe/include
+++ export PYTHON_INC
+++ PYTHON_LIB=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe/lib
+++ export PYTHON_LIB
+++ PYTHON_ROOT=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe
+++ export PYTHON_ROOT
+++ PYTHON_VERSION=3.10.8
+++ export PYTHON_VERSION
+++ _LMFILES_=/projects/ppc64le-pwr9-rhel8/modulefiles/lmod/utilities/linux-rhel8-ppc64le/Core/python/3.10.8.lua
+++ export _LMFILES_
+++ _ModuleTable001_=X01vZHVsZVRhYmxlXyA9IHsKTVR2ZXJzaW9uID0gMywKY19yZWJ1aWxkVGltZSA9IGZhbHNlLApjX3Nob3J0VGltZSA9IGZhbHNlLApkZXB0aFQgPSB7fSwKZmFtaWx5ID0ge30sCm1UID0gewpweXRob24gPSB7CmZuID0gIi9wcm9qZWN0cy9wcGM2NGxlLXB3cjktcmhlbDgvbW9kdWxlZmlsZXMvbG1vZC91dGlsaXRpZXMvbGludXgtcmhlbDgtcHBjNjRsZS9Db3JlL3B5dGhvbi8zLjEwLjgubHVhIiwKZnVsbE5hbWUgPSAicHl0aG9uLzMuMTAuOCIsCmxvYWRPcmRlciA9IDEsCnByb3BUID0ge30sCnN0YWNrRGVwdGggPSAwLApzdGF0dXMgPSAiYWN0aXZlIiwKdXNlck5hbWUgPSAicHl0aG9uLzMuMTAuOCIsCndWID0gIjAwMDAwMDAwMy4wMDAwMDAwMTAuMDAwMDAwMDA4Lip6
+++ export _ModuleTable001_
+++ _ModuleTable002_=ZmluYWwiLAp9LAp9LAptcGF0aEEgPSB7CiIvcHJvamVjdHMvcHBjNjRsZS1wd3I5LXJoZWw4L21vZHVsZWZpbGVzL2xtb2QvY29tcGlsZXJzIiwgIi9wcm9qZWN0cy9wcGM2NGxlLXB3cjktcmhlbDgvbW9kdWxlZmlsZXMvbG1vZC91dGlsaXRpZXMvbGludXgtcmhlbDgtcHBjNjRsZS9Db3JlIiwKfSwKc3lzdGVtQmFzZU1QQVRIID0gIi9wcm9qZWN0cy9wcGM2NGxlLXB3cjktcmhlbDgvbW9kdWxlZmlsZXMvbG1vZC9jb21waWxlcnM6L3Byb2plY3RzL3BwYzY0bGUtcHdyOS1yaGVsOC9tb2R1bGVmaWxlcy9sbW9kL3V0aWxpdGllcy9saW51eC1yaGVsOC1wcGM2NGxlL0NvcmUiLAp9Cg==
+++ export _ModuleTable002_
+++ _ModuleTable_Sz_=2
+++ export _ModuleTable_Sz_
++ SCREAM_MACHINE=weaver
+ [[ 0 == 1 ]]
+ [[ 0 == 1 ]]
+ [[ 0 == 1 ]]
++ whoami
+ [[ e3sm-jenkins == \e\3\s\m\-\j\e\n\k\i\n\s ]]
+ git config --local user.email jenkins@ignore.com
+ git config --local user.name 'Jenkins Jenkins'
+ declare -i fails=0
+ BASELINES_DIR=AUTO
+ TAS_ARGS='--baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -p -i -m $machine'
+ set +e
+ '[' -n 2650 ']'
+ is_at_run=1
+ SA_FAILURES_DETAILS=
+ '[' 1 -eq 1 ']'
++ ./scripts/gather-all-data './scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -p -i -m $machine' -l -m weaver
***Forced exclusive execution
<>
<>
Build was aborted
Aborted by Luca Bertagna
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/


./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins14389375072740138895.sh

@E3SM-Autotester E3SM-Autotester removed the AT: RETEST Force the autotester (AT) to retest the PR label Jan 4, 2024
@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection is Not Necessary for this Pull Request.

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 4728
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS
PULLREQUESTNUM 2650
SCREAM_SOURCE_REPO https://github.com/mahf708/scream
SCREAM_SOURCE_SHA acc6023
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 2f9def8
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5202
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS
PULLREQUESTNUM 2650
SCREAM_SOURCE_REPO https://github.com/mahf708/scream
SCREAM_SOURCE_SHA acc6023
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 2f9def8
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (mahf708/scream)
  • Branch: time-varying-amip-inputs
  • SHA: acc6023
  • Mode: TEST_REPO

Pull Request Author: mahf708

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 4728
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
PR_LABELS
PULLREQUESTNUM 2650
SCREAM_SOURCE_REPO https://github.com/mahf708/scream
SCREAM_SOURCE_SHA acc6023
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 2f9def8
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5202
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
PR_LABELS
PULLREQUESTNUM 2650
SCREAM_SOURCE_REPO https://github.com/mahf708/scream
SCREAM_SOURCE_SHA acc6023
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 2f9def8
TEST_REPO_ALIAS SCREAM

@mahf708 mahf708 added the AT: WIP Inform the autotester (AT) that the PR is a work in progress, and should not be tested label Jan 14, 2024
@mahf708 mahf708 removed the AT: WIP Inform the autotester (AT) that the PR is a work in progress, and should not be tested label Jan 15, 2024
@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection is Not Necessary for this Pull Request.

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 4765
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS
PULLREQUESTNUM 2650
SCREAM_SOURCE_REPO https://github.com/mahf708/scream
SCREAM_SOURCE_SHA 8410151
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 9f3b148
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5236
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS
PULLREQUESTNUM 2650
SCREAM_SOURCE_REPO https://github.com/mahf708/scream
SCREAM_SOURCE_SHA 8410151
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 9f3b148
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (mahf708/scream)
  • Branch: time-varying-amip-inputs
  • SHA: 8410151
  • Mode: TEST_REPO

Pull Request Author: mahf708

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 4765
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
PR_LABELS
PULLREQUESTNUM 2650
SCREAM_SOURCE_REPO https://github.com/mahf708/scream
SCREAM_SOURCE_SHA 8410151
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 9f3b148
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5236
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
PR_LABELS
PULLREQUESTNUM 2650
SCREAM_SOURCE_REPO https://github.com/mahf708/scream
SCREAM_SOURCE_SHA 8410151
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 9f3b148
TEST_REPO_ALIAS SCREAM

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
THE LAST COMMIT TO THIS PULL REQUEST HAS NOT BEEN REVIEWED YET!

@E3SM-Autotester
Copy link
Collaborator

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

@mahf708 mahf708 added the AT: WIP Inform the autotester (AT) that the PR is a work in progress, and should not be tested label Jan 16, 2024
@mahf708
Copy link
Contributor Author

mahf708 commented Jan 18, 2024

Closing since this won't fit with our available machinery at this time. Will revisit in the future.

@mahf708 mahf708 closed this Jan 18, 2024
@brhillman brhillman reopened this Jan 26, 2024
@mahf708
Copy link
Contributor Author

mahf708 commented Jan 26, 2024

Status update: We are going to redo this assuming we have 3D inputs, thus bypassing a critical bottleneck.

@mahf708
Copy link
Contributor Author

mahf708 commented Mar 14, 2024

Will revisit later 😄

@mahf708 mahf708 closed this Mar 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bartgol bartgol bartgol left review comments

@AaronDonahue AaronDonahue AaronDonahue left review comments

@brhillman brhillman brhillman left review comments

Assignees
No one assigned
Labels
AT: WIP Inform the autotester (AT) that the PR is a work in progress, and should not be tested
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@mahf708 @E3SM-Autotester @bartgol @AaronDonahue @brhillman