Case Fail after Submitting with ERROR: (shr_stream_readTCoord) ERROR: nf90_get_att units

[rytam2]

Hello! I am running an AMIP experiment on E3SM on a local machine with compset F2010 and grid ne30pg2_EC30to60E2r2. I have a file that contains the sensitivities of SSTs to ENSO cycle from ERA-5 data, and have implemented the SST pattern to the E3SM ocean initial conditions at 1 degree (sst_ice_CMIP6_DECK_E3SM_1x1_2010_clim_c20190821.nc). Prior to multiplying the two datasets, I have regridded the ERA-5 data to the E3SM grid with ncremap, and have ensured there is no NaN values over land and beyond 78.5S, as the ERA-5 dataset I have are SSTs only and no data beyond 78.5S.

The regridding process I did:

ncatted -a _FillValue,latitude,m,f,1.0e36 ensosensitivity.nc ensosensitivity_edited  #solve warning: NC_DOUBLE "_FillValue" 
ncpdq -a time,-lat,lon ensosensitivity_edited.nc ensosensitivity_correctedlat.nc #solve warning:  2D grids run from south-to-north
ncremap -d sst_ice_CMIP6_DECK_E3SM_1x1_2010_clim_c20190821.nc ensosensitivity_correctedlat.nc regridded.nc
ncatted -a _FillValue,t2m,m,f,1.0e36 regridded.nc regridded_fixed.nc #solve warning: NC_DOUBLE "_FillValue" 

After the regridding, I do the multiplication and called .combine_first, where the NaN values on land and the Antarctic ocean will be identical as the E3SM IC values. After that, I save the dataset. I tried both doing regridding again (as above) and without regridding before loading the dataset to the case. The case set up and built successfully, but it fails around 23seconds after I submit and return the error as follow in the e3sm.log.1760469.230919-101042:

MOSART decomp info proc =         1 begr =      5401 endr =     10800 numr =      5400
MOSART decomp info proc =        23 begr =    124201 endr =    129600 numr =      5400
MOSART decomp info proc =        46 begr =    248401 endr =    253800 numr =      5400
MOSART decomp info proc =        47 begr =    253801 endr =    259200 numr =      5400
 ERROR: (shr_stream_readTCoord)  ERROR: nf90_get_att units
#0  0x1a74335 in ???
#1  0x1a744b3 in ???
#2  0x1b652ba in ???
#3  0x1b677b2 in ???
#4  0x1ae6ee3 in ???
#5  0x1b4e35b in ???
#6  0x1931e11 in ???
#7  0x1937df7 in ???
#8  0x193130b in ???
#9  0x437ecb in ???
#10  0x4277ae in ???
#11  0x4354e0 in ???
#12  0x2b2a79aab554 in ???
#13  0x417e48 in ???
#14  0xffffffffffffffff in ???
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 1001.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------

Is there anything else I should have done when implementing the ERA-5 ENSO pattern or doing regridding to E3SM grids? Or is this error caused by something else?

Any advice and comments are much appreciated! Thank you a lot in advance.

EDIT 1:
Thanks Noel for the suggestion! I added DEBUG=TRUE, and there is a little more info:

MOSART decomp info proc =         1 begr =      5401 endr =     10800 numr =      5400
MOSART decomp info proc =        23 begr =    124201 endr =    129600 numr =      5400
MOSART decomp info proc =        46 begr =    248401 endr =    253800 numr =      5400
MOSART decomp info proc =        47 begr =    253801 endr =    259200 numr =      5400
 ERROR: (shr_stream_readTCoord)  ERROR: nf90_get_att units
#0  0x48017f0 in __shr_abort_mod_MOD_shr_abort_backtrace
        at /data/keeling/a/rytam2/E3SM2.0/E3SM/share/util/shr_abort_mod.F90:104
#1  0x48019d2 in __shr_abort_mod_MOD_shr_abort_abort
        at /data/keeling/a/rytam2/E3SM2.0/E3SM/share/util/shr_abort_mod.F90:61
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 1001.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
#2  0x4a1d429 in shr_stream_readtcoord
        at /data/keeling/a/rytam2/E3SM2.0/E3SM/share/streams/shr_stream_mod.F90:1482
#3  0x4a1f5f5 in __shr_stream_mod_MOD_shr_stream_findbounds
        at /data/keeling/a/rytam2/E3SM2.0/E3SM/share/streams/shr_stream_mod.F90:1056
#4  0x48ff7e0 in __shr_dmodel_mod_MOD_shr_dmodel_readlbub
        at /data/keeling/a/rytam2/E3SM2.0/E3SM/share/streams/shr_dmodel_mod.F90:677
#5  0x49ea659 in __shr_strdata_mod_MOD_shr_strdata_advance
        at /data/keeling/a/rytam2/E3SM2.0/E3SM/share/streams/shr_strdata_mod.F90:897
#6  0x45912c1 in __docn_comp_mod_MOD_docn_comp_run
        at /data/keeling/a/rytam2/E3SM2.0/E3SM/components/data_comps/docn/src/docn_comp_mod.F90:474
#7  0x45a0617 in __docn_comp_mod_MOD_docn_comp_init
        at /data/keeling/a/rytam2/E3SM2.0/E3SM/components/data_comps/docn/src/docn_comp_mod.F90:365
#8  0x458d8d6 in __ocn_comp_mct_MOD_ocn_init_mct
        at /data/keeling/a/rytam2/E3SM2.0/E3SM/components/data_comps/docn/src/ocn_comp_mct.F90:163
#9  0x44ad61 in __component_mod_MOD_component_init_cc
        at /data/keeling/a/rytam2/E3SM2.0/E3SM/driver-mct/main/component_mod.F90:248
#10  0x432934 in __cime_comp_mod_MOD_cime_init
        at /data/keeling/a/rytam2/E3SM2.0/E3SM/driver-mct/main/cime_comp_mod.F90:1443
#11  0x443bcb in cime_driver
        at /data/keeling/a/rytam2/E3SM2.0/E3SM/driver-mct/main/cime_driver.F90:122
#12  0x443d0b in main
        at /data/keeling/a/rytam2/E3SM2.0/E3SM/driver-mct/main/cime_driver.F90:23

[ndkeen]

Someone else may already know the issue, but something you could do quickly is try again with DEBUG=TRUE which might give you more information about the failure. Ah, but I see that you already see error message ERROR: (shr_stream_readTCoord) ERROR: nf90_get_att units, so may not need anything additional from DEBUG.