get_grid_sizerequested tile index 1 is out of bounds (1:0) when running model config produced with regional-mom6

[croachutas]

Hi,
I recently downloaded the current version of regional-mom6, got it working on my workstation and setup a test high resolution config for south eastern Tasmania. I've looked at the resulting input files and they look correct but when I go to run with a MOM6 program built with sis2 coupling (tested and working on several of the MOM6 examples) I get the following output:

NOTE from PE     0: MPP_DOMAINS_SET_STACK_SIZE: stack size set to    32768.
&MPP_IO_NML
 HEADER_BUFFER_VAL=16384      ,
 GLOBAL_FIELD_ON_ROOT_PE=T,
 IO_CLOCKS_ON=F,
 SHUFFLE=0          ,
 DEFLATE_LEVEL=-1         ,
 CF_COMPLIANCE=F,
 /
NOTE from PE     0: MPP_IO_SET_STACK_SIZE: stack size set to     131072.
NOTE from PE     0: MPP_DOMAINS_SET_STACK_SIZE: stack size set to  1600000.
 Entering coupler_init at 20240904 142103.350
 Starting initializing ensemble_manager at 20240904 142103.351
 Finished initializing ensemble_manager at 20240904 142103.351
NOTE from PE     0: coupler_init: Atmos PE range:      0     8 ens_01
NOTE from PE     0: coupler_init: Ocean PE range:      0     8 ens_01
NOTE from PE     0: coupler_init: Land PE range:      0     8 ens_01
NOTE from PE     0: coupler_init: Ice PE range:      0     8 ens_01
NOTE from PE     0: coupler_init: Running with SERIAL coupling.
NOTE from PE     0: coupler_init: Sending most recent fluxes to ocean.
 Starting to initialize diag_manager at 20240904 142103.351
NOTE from PE     0: diag_manager_mod::diag_manager_init: diag_manager is using fms2_io
 Finished initializing diag_manager at 20240904 142103.356
 Starting to initialize tracer_manager at 20240904 142103.357
NOTE from PE     0: You are using the legacy version of the field_table

 Dumping coupler_mod/types tree
types/
      air_sea_gas_flux_generic/
            implementation/
                  ocmip2/
                        num_parameters = 2
                  duce/
                        num_parameters = 1
                  johnson/
                        num_parameters = 2
            num_flags = 0
            use_atm_pressure = T
            use_10m_wind_speed = T
            pass_through_ice = F
            atm/
                  name[1] = 'pcair'
                  name[2] = 'u10'
                  name[3] = 'psurf'
                  long_name[1] = 'Atmospheric concentration'
                  long_name[2] = 'Wind speed at 10 m'
                  long_name[3] = 'Surface atmospheric pressure'
                  units[1] = 'mol/mol'
                  units[2] = 'm/s'
                  units[3] = 'Pa'
            ice/
                  name[1] = 'alpha'
                  name[2] = 'csurf'
                  name[3] = 'sc_no'
                  long_name[1] = 'Solubility w.r.t. atmosphere'
                  long_name[2] = 'Ocean concentration'
                  long_name[3] = 'Schmidt number'
                  units[1] = 'mol/m^3/atm'
                  units[2] = 'mol/m^3'
                  units[3] = 'dimensionless'
            flux/
                  name[1] = 'flux'
                  name[2] = 'deltap'
                  name[3] = 'kw'
                  name[4] = 'flux0'
                  long_name[1] = 'Surface flux'
                  long_name[2] = 'Ocean-air delta pressure'
                  long_name[3] = 'Piston velocity'
                  long_name[4] = 'Surface flux no atm'
                  units[1] = 'mol/m^2/s'
                  units[2] = 'uatm'
                  units[3] = 'm/s'
                  units[4] = 'mol/m^2/s'
      air_sea_gas_flux/
            implementation/
                  ocmip2/
                        num_parameters = 2
                  ocmip2_data/
                        num_parameters = 2
                  linear/
                        num_parameters = 3
            num_flags = 0
            use_atm_pressure = T
            use_10m_wind_speed = T
            pass_through_ice = F
            atm/
                  name[1] = 'pcair'
                  name[2] = 'u10'
                  name[3] = 'psurf'
                  long_name[1] = 'Atmospheric concentration'
                  long_name[2] = 'Wind speed at 10 m'
                  long_name[3] = 'Surface atmospheric pressure'
                  units[1] = 'mol/mol'
                  units[2] = 'm/s'
                  units[3] = 'Pa'
            ice/
                  name[1] = 'alpha'
                  name[2] = 'csurf'
                  long_name[1] = 'Solubility from atmosphere times Schmidt number term'
                  long_name[2] = 'Ocean concentration times Schmidt number term'
                  units[1] = 'mol/m^3/atm'
                  units[2] = 'mol/m^3'
            flux/
                  name = 'flux'
                  long_name = 'Surface flux'
                  units = 'mol/m^2/s'
      air_sea_deposition/
            implementation/
                  dry/
                        num_parameters = 1
                  wet/
                        num_parameters = 1
            num_flags = 0
            use_atm_pressure = F
            use_10m_wind_speed = F
            pass_through_ice = T
            atm/
                  name = 'deposition'
                  long_name = 'Atmospheric deposition'
                  units = 'kg/m^2/s'
            ice/
                  name = NULL
                  long_name = NULL
                  units = NULL
            flux/
                  name = 'flux'
                  long_name = 'Surface deposition'
                  units = 'mol/m^2/s'
      land_sea_runoff/
            implementation/
                  river/
                        num_parameters = 1
            num_flags = 0
            use_atm_pressure = F
            use_10m_wind_speed = F
            pass_through_ice = T
            atm/
                  name = 'runoff'
                  long_name = 'Concentration in land runoff'
                  units = 'mol/m^3'
            ice/
                  name = NULL
                  long_name = NULL
                  units = NULL
            flux/
                  name = 'flux'
                  long_name = 'Concentration in land runoff'
                  units = 'mol/m^3'
NOTE from PE     0: open_param_file: MOM_input has been opened successfully.
NOTE from PE     0: open_param_file: MOM_layout has been opened successfully.
NOTE from PE     0: open_param_file: MOM_override has been opened successfully.
NOTE from PE     0: close_param_file: MOM_input has been closed successfully.
NOTE from PE     0: close_param_file: MOM_layout has been closed successfully.
NOTE from PE     0: close_param_file: MOM_override has been closed successfully.
 ==>Note from atmos_ocean_fluxes_mod(atmos_ocean_fluxes_init): No gas fluxes
 Finished initializing tracer_manager at 20240904 142103.379
 Beginning to initialize component models at 20240904 142103.379
 Starting to initialize atmospheric model at 20240904 142103.379

FATAL from PE     2: get_grid_sizerequested tile index 1 is out of bounds (1:0)


FATAL from PE     4: get_grid_sizerequested tile index 1 is out of bounds (1:0)


FATAL from PE     6: get_grid_sizerequested tile index 1 is out of bounds (1:0)


FATAL from PE     7: get_grid_sizerequested tile index 1 is out of bounds (1:0)


FATAL from PE     0: get_grid_sizerequested tile index 1 is out of bounds (1:0)


FATAL from PE     1: get_grid_sizerequested tile index 1 is out of bounds (1:0)


FATAL from PE     0: get_grid_sizerequested tile index 1 is out of bounds (1:0)


FATAL from PE     3: get_grid_sizerequested tile index 1 is out of bounds (1:0)


FATAL from PE     5: get_grid_sizerequested tile index 1 is out of bounds (1:0)


FATAL from PE     8: get_grid_sizerequested tile index 1 is out of bounds (1:0)

I'm unclear on what this get_grid_sizerequested error means and if it's something wrong with MOM6 or something wrong with the config produced with regional-mom6.

Any ideas?

[ashjbarnes]

this is a weird one! Never seen it before. My suspicion is that it's to do with the 'ntiles' dimension in the bathymetry file, but that could be totally wrong. I don't understand what ntiles does, just that MOM seemed to want it. @angus-g or @micaeljtoliveira might know.

Which executable are you using? Can you confirm that your executable works with the output of the demo notebook?

[croachutas]

Hi Ash,
Sorry for the slow reply but had other things to deal with over the last little while (AMSA conference prep...).

Tried using the demo notebook config and I'm getting the same error. SO, most probably something to do with my MOM6 installation... Might try recompiling it after checking if some of the cpp flags and similar are wrong (hybrid of your setup-mom6 ninja stuff for NCI combined with what sense I could make from the default MOM6 ubuntu install...).

[ChrisC28]

Hi @croachutas
If you are interested, @reillyja and I have a working executable on the Setonix machine (part of the Pawsey supercomputing center in Perth). I've got plenty of SU, so I'd be happy to set you up with an account for your experiments. We've scaled Mom6 to '000s of cores with reasonable results. We have also got a working version of Payu for run management.

The workflow on Setonix is a little different to Gadi, but we can work you through it.

Happy to chat at AMSA or otherwise. Let me know.

[croachutas]

Hi Chris,
No current experiment in mind, rather still trying to work out if I can get a MOM6 based prac going for either teaching OUC4B or the Polar Obs and Modelling unit. Thus the interest in getting it working on something other than Gadi.
-Chris