Skip to content

Commit

Permalink
Update IO formats and add new IO namelist controls (CICE-Consortium#928)
Browse files Browse the repository at this point in the history
This provides new features for CICE IO both thru netCDF and PIO.  New namelist are added to control history and restart format, hdf5 compression and chunking, the PIO rearranger, and PIO IO task control.  Separate controls are provided for history and restart files.  The namelist changes are for

  _history_format, restart_format
  history_rearranger, restart_rearranger
  history_iotasks, history_root, history_stride, restart_iotasks, restart_root, and restart_stride
  history_chunksize, history_deflate, restart_chunksize, restart_deflate._

In particular,

- Update restart_format and history_format options to 'cdf1', 'cdf2', 'cdf5', 'hdf5', 'pnetcdf1', 'pnetcdf2', 'pnetcdf5', 'default'.  The old options, 'default', 'pio_netcdf', and 'pio_pnetcdf' are still supported and backwards compatible with lcdf64, but are deprecated and no longer documented.  The old options and old namelist lcdf64 are covered by the new options.  Support of the old options should be removed in the future.  Note that some problems were discovered when opening files with hdf5 format but reading non-hdf5 files with a spack built PIO/netCDF.  As a result, the format specified for the restart read is always 'cdf1' which provides flexibility and robustness across software installs, although it may result in serial reads of hdf5 files when a parallel read could be done.
- Deprecate lcdf64 namelist.  This namelist is no longer needed and is covered by the new restart_format and history_format options.  The namelist still exists and is backwards compatible with the old 'default', 'pio_netcdf', and 'pio_pnetcdf' format options, but is no longer documented.  This should be removed in the future.
- Add new namelist to control PIO pe/task setup (iotasks, root, stride) for history and restart.  These settings control the PIO IO tasks.  The root, stride, and iotasks are consistent with the MPI communicator.  root=0 is the first MPI task.  These control PIO IO performance and are usually a function of things like the IO and node hardware.  See PIO for more information.  CICE computes PIO iotask, root, and stride defaults for cases where -99 is passed in for some or all of these namelist.  Those defaults are somewhat constrained by a bug in PIO, NCAR/ParallelIO#1986.  The current implementation avoids the bug by limiting the iotasks for some MPI task counts.  This is noted in ice_pio.F90.
- Add new namelist to control PIO rearranger (rearranger) for history and restart.  Supports 'box', 'subset', and 'default'.  These control how PIO rearrangment is carried out.  default is equivalent to box and the box generally performs better.  See PIO for more information.
- Add new namelist to support hdf5 compression and chunking (deflate, chunksize) for history and restart.  The deflate controls file compression and is an integer between 0 and 9 where 0 means no compression and 9 is maximum compression.   Generally, the higher the number, the slower the IO and the smaller the file, but the optimal setting depends on the contents of the file.  Chunksize provides a performance control for the hdf5 parallel writes.  It is a 2d array and is associated with the size of the piece of the array written by hdf5.  hdf5 can be read and written in parallel, but that depends on how netCDF and PIO are built.  Note that prior version of PIO, including PIO1, do not support the hdf5 compression and chunking thru the PIO interface.
- Add new namelist settings (set_nml files) and update the io_suite to cover the new IO options.  Remove old namelist settings associated with the deprecated format options and the lcdf64 namelist.  These deprecated feature are no longer tested.
- Update documentation to add new namelist and IO features.
- Update the nuopc/cmeps driver code to support the new features.
- Update the default ice_in to add the new namelist.
- Update the derecho netcdf module to a version that supports hdf5.
- Clean up some code formatting (indentation)

---------

Co-authored-by: Anton Steketee <anton.steketee@anu.edu.au>
  • Loading branch information
apcraig and anton-seaice authored Feb 22, 2024
1 parent 9e9e5b3 commit aca8357
Show file tree
Hide file tree
Showing 39 changed files with 1,092 additions and 631 deletions.
13 changes: 9 additions & 4 deletions cicecore/cicedyn/analysis/ice_history_shared.F90
Original file line number Diff line number Diff line change
Expand Up @@ -49,18 +49,23 @@ module ice_history_shared
history_dir , & ! directory name for history file
incond_dir ! directory for snapshot initial conditions

character (len=char_len_long), public :: &
pointer_file ! input pointer file for restarts

character (len=char_len), public :: &
version_name

character (len=char_len), public :: &
history_format
history_format , & ! history format, cdf1, cdf2, cdf5, etc
history_rearranger ! history file rearranger, box or subset for pio

character (len=char_len), public :: &
hist_suffix(max_nstrm) ! appended to 'h' in filename when not 'x'

integer (kind=int_kind), public :: &
history_iotasks , & ! iotasks, root, stride defines io pes for pio
history_root , & ! iotasks, root, stride defines io pes for pio
history_stride , & ! iotasks, root, stride defines io pes for pio
history_deflate , & ! compression level for hdf5/netcdf4
history_chunksize(2) ! chunksize for hdf5/netcdf4

!---------------------------------------------------------------
! Instructions for adding a field: (search for 'example')
! Here or in ice_history_[process].F90:
Expand Down
303 changes: 247 additions & 56 deletions cicecore/cicedyn/general/ice_init.F90

Large diffs are not rendered by default.

12 changes: 6 additions & 6 deletions cicecore/cicedyn/infrastructure/ice_read_write.F90
Original file line number Diff line number Diff line change
Expand Up @@ -1241,7 +1241,7 @@ subroutine ice_read_nc_xy(fid, nrec, varname, work, diag, &
status = nf90_get_att(fid, varid, "_FillValue", missingvalue)
! call ice_check_nc(status, subname//' ERROR: Missing _FillValue', &
! file=__FILE__, line=__LINE__)
write(nu_diag,*) subname,' missingvalue= ',missingvalue
! write(nu_diag,*) subname,' missingvalue= ',missingvalue
amin = minval(work_g1)
amax = maxval(work_g1, mask = work_g1 /= missingvalue)
asum = sum (work_g1, mask = work_g1 /= missingvalue)
Expand Down Expand Up @@ -1442,7 +1442,7 @@ subroutine ice_read_nc_xyz(fid, nrec, varname, work, diag, &
status = nf90_get_att(fid, varid, "_FillValue", missingvalue)
! call ice_check_nc(status, subname//' ERROR: Missing _FillValue', &
! file=__FILE__, line=__LINE__)
write(nu_diag,*) subname,' missingvalue= ',missingvalue
! write(nu_diag,*) subname,' missingvalue= ',missingvalue
do n=1,ncat
amin = minval(work_g1(:,:,n))
amax = maxval(work_g1(:,:,n), mask = work_g1(:,:,n) /= missingvalue)
Expand Down Expand Up @@ -1654,7 +1654,7 @@ subroutine ice_read_nc_xyf(fid, nrec, varname, work, diag, &
status = nf90_get_att(fid, varid, "_FillValue", missingvalue)
! call ice_check_nc(status, subname//' ERROR: Missing _FillValue', &
! file=__FILE__, line=__LINE__)
write(nu_diag,*) subname,' missingvalue= ',missingvalue
! write(nu_diag,*) subname,' missingvalue= ',missingvalue
do n = 1, nfreq
amin = minval(work_g1(:,:,n))
amax = maxval(work_g1(:,:,n), mask = work_g1(:,:,n) /= missingvalue)
Expand Down Expand Up @@ -2589,13 +2589,13 @@ subroutine ice_check_nc(status, abort_msg, file, line)
#ifdef USE_NETCDF
if (status /= nf90_noerr) then
if (present(file) .and. present(line)) then
call abort_ice(subname//trim(nf90_strerror(status))//', '//trim(abort_msg), &
call abort_ice(subname//' '//trim(nf90_strerror(status))//', '//trim(abort_msg), &
file=file, line=line)
elseif (present(file)) then
call abort_ice(subname//trim(nf90_strerror(status))//', '//trim(abort_msg), &
call abort_ice(subname//' '//trim(nf90_strerror(status))//', '//trim(abort_msg), &
file=file)
else
call abort_ice(subname//trim(nf90_strerror(status))//', '//trim(abort_msg))
call abort_ice(subname//' '//trim(nf90_strerror(status))//', '//trim(abort_msg))
endif
endif
#else
Expand Down
347 changes: 160 additions & 187 deletions cicecore/cicedyn/infrastructure/io/io_netcdf/ice_history_write.F90

Large diffs are not rendered by default.

52 changes: 41 additions & 11 deletions cicecore/cicedyn/infrastructure/io/io_netcdf/ice_restart.F90
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,8 @@ module ice_restart
use ice_read_write, only: ice_check_nc
use ice_restart_shared, only: &
restart_ext, restart_dir, restart_file, pointer_file, &
runid, use_restart_time, lcdf64, lenstr, restart_coszen
runid, use_restart_time, lenstr, restart_coszen, restart_format, &
restart_chunksize, restart_deflate
use ice_fileunits, only: nu_diag, nu_rst_pointer
use ice_exit, only: abort_ice
use icepack_intfc, only: icepack_query_parameters
Expand All @@ -29,10 +30,12 @@ module ice_restart
implicit none
private
public :: init_restart_write, init_restart_read, &
read_restart_field, write_restart_field, final_restart, &
query_field
read_restart_field, write_restart_field, final_restart, &
query_field

integer (kind=int_kind) :: ncid
integer (kind=int_kind) :: ncid , &
dimid_ni, & ! netCDF identifiers
dimid_nj

!=======================================================================

Expand Down Expand Up @@ -169,8 +172,7 @@ subroutine init_restart_write(filename_spec)
integer (kind=int_kind), allocatable :: dims(:)

integer (kind=int_kind) :: &
dimid_ni, & ! netCDF identifiers
dimid_nj, & !

dimid_ncat, & !
iflag, & ! netCDF creation flag
status ! status variable from netCDF routine
Expand Down Expand Up @@ -216,8 +218,18 @@ subroutine init_restart_write(filename_spec)
write(nu_rst_pointer,'(a)') filename
close(nu_rst_pointer)

iflag = 0
if (lcdf64) iflag = nf90_64bit_offset
if (restart_format == 'cdf1') then
iflag = nf90_clobber
elseif (restart_format == 'cdf2') then
iflag = ior(nf90_clobber,nf90_64bit_offset)
elseif (restart_format == 'cdf5') then
iflag = ior(nf90_clobber,nf90_64bit_data)
elseif (restart_format == 'hdf5') then
iflag = ior(nf90_clobber,nf90_netcdf4)
else
call abort_ice(subname//' ERROR: restart_format not allowed for '//trim(restart_format), &
file=__FILE__, line=__LINE__)
endif
status = nf90_create(trim(filename), iflag, ncid)
call ice_check_nc(status, subname//' ERROR: creating '//trim(filename), file=__FILE__, line=__LINE__)

Expand Down Expand Up @@ -873,14 +885,32 @@ subroutine define_rest_field(ncid, vname, dims)

integer (kind=int_kind) :: varid

integer (kind=int_kind) :: &
status ! status variable from netCDF routine
integer (kind=int_kind) :: chunks(size(dims)), status, i

character(len=*), parameter :: subname = '(define_rest_field)'

#ifdef USE_NETCDF

status = nf90_def_var(ncid,trim(vname),nf90_double,dims,varid)
call ice_check_nc(status, subname//' ERROR: def var '//trim(vname), file=__FILE__, line=__LINE__)

if (restart_format=='hdf5' .and. size(dims)>1) then
if (dims(1)==dimid_ni .and. dims(2)==dimid_nj) then
chunks(1)=restart_chunksize(1)
chunks(2)=restart_chunksize(2)
do i = 3, size(dims)
chunks(i) = 0
enddo
status = nf90_def_var_chunking(ncid, varid, NF90_CHUNKED, chunksizes=chunks)
call ice_check_nc(status, subname//' ERROR: chunking var '//trim(vname), file=__FILE__, line=__LINE__)
endif
endif

if (restart_format=='hdf5' .and. restart_deflate/=0) then
status=nf90_def_var_deflate(ncid, varid, shuffle=0, deflate=1, deflate_level=restart_deflate)
call ice_check_nc(status, subname//' ERROR deflating var '//trim(vname), file=__FILE__, line=__LINE__)
endif

#else
call abort_ice(subname//' ERROR: USE_NETCDF cpp not defined', &
file=__FILE__, line=__LINE__)
Expand Down Expand Up @@ -911,7 +941,7 @@ logical function query_field(nu,vname)
endif
call broadcast_scalar(query_field,master_task)
#else
call abort_ice(subname//' ERROR: USE_NETCDF cpp not defined for '//trim(ice_ic), &
call abort_ice(subname//' ERROR: USE_NETCDF cpp not defined', &
file=__FILE__, line=__LINE__)
#endif

Expand Down
Loading

0 comments on commit aca8357

Please sign in to comment.