Vertical remapping on restart distorts phis field

[rurata]

Describe the bug
When we restart a simulation with a different vertical grid, the phis field becomes distorted, copying the same section of the field over the whole tile according to the number of processors per tile (picture attached).

This happens when both going up or down in vertical resolution. Normal restarts do not experience this problem. This was confirmed to behave the same in the main branch, commit [29396d3], and FMS tag 2022.02

To Reproduce
Warm start a simulation with a different NPZ.

Expected behavior
We expect the phis field to be initialized correctly, copying what was in the fv_core.tileX.nc file.

System Environment
Describe the system environment, include:

• OS: Operating System: Red Hat Enterprise Linux Server 7.9 (Maipo)
  CPE OS Name: cpe:/o:redhat:enterprise_linux:7.9:GA:server
  Kernel: Linux 3.10.0-1160.66.1.1chaos.ch6.x86_64
  Architecture: x86-64
• Compiler(s): ifort version 19.1.3.304
• MPI type, and version (e.g. MPICH, Cray MPI, openMPI): mpi-hpe/mpt.2.25
• netCDF Version: 4.4.1.1
• Configure options: -DSPMD -Duse_libMPI -Duse_netCDF -DINTERNAL_FILE_NML -DCUBE_CORE -fpp -Wp,-w -fno-alias -ftz -assume byterecl -i4 -r8 -nowarn -O2 -fp-model precise -qoverride-limits -c -qopenmp -traceback -xSSE4.2

Additional context
It seems like an issue in the read_data call for phis in remap_restart in fv_io.F90. When we moved the phis field to fv_srf_wnd.res which uses read_restart instead, we did not see this problem.

[bensonr]

@rurata - was a PR ever put in to address this issue?

[rurata]

@bensonr This issue still persists as of be0a6a6. I have attached a screenshot of a C48 restart changing NPZ

[lharris4]

@rurata I think you are right that the data is being read incorrectly, but it is strange that the 3D fields are fine. Do the fields like delp get read in correctly by read_data()? If so then perhaps the 2D routine has a problem.

https://github.com/NOAA-GFDL/GFDL_atmos_cubed_sphere/blob/main/tools/fv_io.F90#L725-L726

[rurata]

@lharris4 Hi, I just did some checking and it looks like delp and temp are also both corrupted by read_data. I have attached screenshots of these fields just after restart.

[bensonr]

@rurata - I have looked at the code in question (fv_io.F90) and agree there is an issue. I will work to get you a fix.

[laurenchilutti]

I am assigning this issue to myself. Rusty and I have discussed a fix and I will try to have this fixed by the end of September.