rewind post (multiple forecst hours) in rocoto workflow

[SarahLu-NOAA]

I conducted a 72-hour UFS exp over the NE US domain at Derecho. The plot_allvars failed (the run directory is /glade/work/clu/ufs/expt_dirs/RRFS_neus_12km_gfsv16). The issue, I believe, is related to POST_OUTPUT_DOMAIN_NAME in this experiment is different from these in COMOUT_REF.

The config.yaml is copied to /glade/work/clu/ufs/ufs-srweather-app/ush/SandBox/config.yaml_smoke_gfsv16.
I then modified config.yaml and var_defns.sh in the run directory, changing POST_OUTPUT_DOMAIN_NAME from rrfs_neus_12km_gfsv16 to rrfs_neus_12km. I then re-do post for f072 (by rocotoboot and rocotorun), and the two experiments now have the same POST_OUTPUT_DOMAIN_NAME.

Since it's not feasible to manually re-run post for all 72 forecast hours, I modify config.yaml in $USHDIR (/glade/work/clu/ufs/ufs-srweather-app/ush) and only include post and plot in rocoto: tasks: taskgroups. When I run generate_FV3LAM_wflow.py, I encounter error msg:
Traceback (most recent call last):
File "/glade/work/clu/ufs/ufs-srweather-app/ush/./generate_FV3LAM_wflow.py", line 768, in
expt_dir = generate_FV3LAM_wflow(USHdir, wflow_logfile, pargs.debug)
File "/glade/work/clu/ufs/ufs-srweather-app/ush/./generate_FV3LAM_wflow.py", line 72, in generate_FV3LAM_wflow
expt_config = setup(ushdir,debug=debug)
File "/glade/work/clu/ufs/ufs-srweather-app/ush/setup.py", line 1425, in setup
res_in_fns = link_fix(
File "/glade/work/clu/ufs/ufs-srweather-app/ush/link_fix.py", line 243, in link_fix
print_err_msg_exit(
File "/glade/work/clu/ufs/ufs-srweather-app/ush/python_utils/print_msg.py", line 24, in print_err_msg_exit
sys.exit(1)
SystemExit: 1

Could you please advise how to re-run post and plot with revised POST_OUTPUT_DOMAIN_NAME and what steps in my approach can be corrected. Many thanks. -Sarah

[SarahLu-NOAA]

BTW - when I run generate_FV3LAM_wflow.py, the run directory has been renamed
/glade/work/clu/ufs/expt_dirs/RRFS_neus_12km_gfsv16_old_20231218_094336

[SarahLu-NOAA]

Some update.
-run directory: I copy RRFS_neus_12km_gfsv16_old_20231218_094336 back to RRFS_neus_12km_gfsv16.
-config: I change PREEXISTING_DIR_METHOD from rename to reuse in config.yaml.
Still can't generate the workflow, with the same error msg. config.yaml (that fails to generate the workflow) is copied to
/glade/work/clu/ufs/ufs-srweather-app/ush/config.yaml_failed

[gspetro-NOAA]

Hi @SarahLu-NOAA ,

It sounds like your experiment initially ran the post tasks without problems, and the main issue is the path in COMOUT_REF. To avoid errors in the plotting, COMOUT_REF needs to be set to a file path that actually exists. Since the file path to the experiment is set automatically as ${EXPT_BASEDIR}/${EXPT_SUBDIR}, COMOUT_REF should be like ${EXPT_BASEDIR}/${EXPT_SUBDIR}/${PDY}${cyc}/postprd. In your case, ${EXPT_SUBDIR} was set to RRFS_neus_12km_gfsv16 (just like POST_OUTPUT_DOMAIN_NAME), but the path you originally gave in COMOUT_REF used RRFS_neus_12km instead of RRFS_neus_12km_gfsv16. The best approach would be to modify the path in COMOUT_REF instead of modifying EXPT_SUBDIR or POST_OUTPUT_DOMAIN_NAME. After modifying the value of COMOUT_REF in all appropriate places (including config.yaml and var_defns.sh), you just need to rocotorewind/rocotorun the plot_allvars task (much easier than rerunning the 72 post tasks again!). I have reached out to @danielabdi-noaa because I believe there may be additional places where the COMOUT_REF value needs to be adjusted before rerunning the task.

Also, keep in mind, if you just want plots for your single experiment, you don't need to set COMOUT_REF at all. If you are trying to do a difference plot between this experiment and some other experiment, you would set COMOUT_REF: ${EXPT_BASEDIR}/EXPT_SUBDIR_FOR_OTHER_EXPT/${PDY}${cyc}/postprd where EXPT_SUBDIR_FOR_OTHER_EXPT is exactly the same as the other experiment's EXPT_SUBDIR value. Otherwise, you can leave it as the default (COMOUT_REF: ''), and it will just do the basic plotting and skip the difference plotting.

Hope this helps!

-- Gillian

[SarahLu-NOAA]

@gspetro-NOAA, Many thanks!!
I have two experiments (gfsv16 and gfsv17p8), both successfully completed all post for 72 forecast hours but failed at plot.
I set COMOUT_REF to "${EXPT_BASEDIR}/RRFS_neus_12km/${PDY}${cyc}/postprd"
and POST_OUTPUT_DOMAIN_NAME: rrfs_neus_12km_gfsv17p8 (or rrfs_neus_12km_gfsv16)

While COMOUT_REF is correct, the plot_allvars failed because my experiment produces files like srw.t00z.prslev.f060.rrfs_neus_12km_gfsv17p8.grib2 but the REF experiment (specified in COMOUT_REF) produces files like
srw.t00z.prslev.f059.rrfs_neus_12km.grib2. My experiment has different POST_OUTPUT_DOMAIN_NAME as the baseline reference experiment. The plot error msg is:
OSError: [Errno could not open %s] /glade/work/clu/ufs/expt_dirs/RRFS_neus_12km/2023120200/postprd/srw.t00z.prslev.f000.rrfs_neus_12km_gfsv17p8.grib2

I have the impression that the new experiment and reference experiment need to have identical POST_OUTPUT_DOMAIN_NAME. Could you please confirm or correct my understanding. Thanks -Sarah

[dshawul]

@SarahLu-NOAA I agree with @gspetro-NOAA that it is best not to manually modify var_defns etc and instead re-run everything from the start. Unfortunately, I don't have access to Cheynenne to look into your runs.

One thing is that changing POST_OUTPUT_DOMAIN_NAME may not have the desired effect. This is because in the plotting scripts, the domain name for the latter is set with PREDEF_GRID_NAME if it is defined, or the experiment subdirectory name otherwise.
Here is the definition in JREGIONAL_PLOT_ALLVARS,

GRID_NAME=${PREDEF_GRID_NAME:-$EXPT_SUBDIR}

If the POST_OUTPUT_DOMAIN_NAME does not match any of these, issues due to inconsistent post file names may arise.
I think the scripts can be improved in this regard, since POST_OUTPUT_DOMAIN_NAME is what the python plotting scripts expect.

[SarahLu-NOAA]

Thanks, @dshawul and @gspetro-NOAA . I do plan to re-run these experiments and conduct additional experiments. It's very helpful to know the dependence of config parameters. In particular, I'll make sure GRID_NAME and POST_OUTPUT_DOMAIN_NAME are consistent among various experiments so plot_allvars can work.