Add MPI-IO collective reads of NetCDF-4 files #1405

dkokron · 2023-11-07T22:12:52Z

NetCDF-4, using the HDF5 file layout, has the ability to do parallel I/O in two different modes. The two modes are referred to as “independent” while the second mode is referred to as “collective”. The collective mode has been tested with a few NOAA workloads and shown to provide substantial improvement in job startup time while reducing negative impact on the underlying Lustre file system. The collective mode is not currently available in FMS (October 2023).

This PR does not address parallel I/O via pNetCDF.

This PR adds an option to enable collective reads. The user controls that option via settings in input.nml. The default behavior is unchanged, the user has to activate collective reads using the settings in input.nml.

Fixes # 1322
#1322

How Has This Been Tested?
I have run a RRFS case on WCOSS2 with and without collective reads activated. The resulting binary restart files are zero-diff.

The compile time environment used to compile FMS was
Currently Loaded Modules:

craype-x86-rome (H) 2) libfabric/1.11.0.0. (H) 3) craype-network-ofi (H) 4) hpc/1.2.0 5) hpc-intel/19.1.3.304 6) hpc-cray-mpich/8.1.19 7) hdf5/1.14.1 8) netcdf/4.9.2

The compile and runtime environment for RRFS was

craype-x86-rome (H) 4) envvar/1.0 7) craype/2.7.13 10) jasper/2.0.25 13) hdf5/1.14.0 16) esmf/8.4.2 19) g2/3.4.5 22) sp/2.3.3 25) mapl/2.35.2-esmf-8.4.2 28) cray-pals/1.2.2
libfabric/1.11.0.0. (H) 5) PrgEnv-intel/8.1.0 8) cray-mpich/8.1.12 11) zlib/1.2.11 14) netcdf/4.9.2 17) bacio/2.4.1 20) g2tmpl/1.10.2 23) w3emc/2.9.2 26) ufs_common
craype-network-ofi (H) 6) intel/19.1.3.304 9) cmake/3.20.2 12) libpng/1.6.37 15) pio/2.5.10 18) crtm/2.4.0 21) ip/3.3.3 24) gftl-shared/v1.5.0 27) ufs_wcoss2.intel

Please describe the tests that you ran to verify your changes. Please also note
any relevant details for your test configuration (e.g. compiler, OS). Include
enough information so someone can reproduce your tests.

Checklist:

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings
Any dependent changes have been merged and published in downstream modules
New check tests, if applicable, are included
The current CMake test for Parallel NetCDF via (nc-config --has-parallel) works in my case. Testing via (nc-config --has-parallel4) is preferred.
make distcheck passes

…tions within NetCDF. This has been tested on Cactus using the RRFS application with up to 24 ensemble members.

….ParallelStartup Merged with GFDL main

…t behavior is unchanged, user has to activate collective reads

bensonr

Besides the inline comment, please remove the commented out debug statements.

bensonr · 2023-11-08T14:46:27Z

fms2_io/netcdf_io.F90

+        if(string_compare(trim(fileobj%path), trim(fn_collective(i)), .true.)) fileobj%use_collective = .true.
+      enddo
+      if(fileobj%use_collective) then
+        err = nf90_open(trim(fileobj%path), ior(NF90_NOWRITE, NF90_MPIIO), fileobj%ncid, comm=mpp_comm_private, info=MPI_INFO_NULL)


Does the use of mpp_comm_private here imply that all ranks associated with reads and writes the the named file will block until all mpi-ranks associated with that communicator have contributed?

I'm only working with reads, but yes those reads are collective.

Debugging prints removed from fms2_io/include/netcdf_read_data.inc and fms2_io/netcdf_io.F90

Of course, the NF90_NOWRITE specifies read-only. My concern is mpp_comm_private is not the correct communicator to be using here. For the general use of libFMS, mpp_comm_private is the MPI_COMM_WORLD global communicator, i.e.all mpi-ranks for the job as a whole. For the UFS, mpp_comm_private in libFMS is initialized with a local communicator that is a subset of the global mpi-ranks. I am trying to ensure the proper communicator will be in effect for all uses of parallel NetCDF within libFMS.

One intended use case involves a GCM with nested domains, thus the communicator used in this nf90_open() call might change. I have not done any testing with that case.

From line 862 of FV3/io/fv3atm_restart_io.F90
call read_restart(Phy_restart, ignore_checksum=ignore_rst_cksum)

The read_restart() routine is from fms2_io_mod

@dkokron Thanks
I am assuming Phy_restart is defined as FmsNetcdfDomainFile_t?
I think the code is missing a call to mpp_set_current_pelist. If you can push what you currently have I will take a better look. I think I can test it out with our unit tests.

Do you get better performance for the HAFS nested grid case?

Yes, Phy_restart is of type FmsNetcdfDomainFile_t.

I am still investigating.

@uramirez8707 Do you have access to the WCOSS2 test system 'acorn'?

dkokron · 2023-11-08T15:52:29Z

Pete Johnsen had these questions.

why do we need to know the number of files being read with collective reads ahead of time? (and why user has to put this in namelist)
how about integer variables and collective reads? there are some integer fields like land / surface masks that could be read collectively
might need to issue a warning and fall back to original read method if the input file is not NetCDF4 format

My responses:

I need to allocate an array to hold the file names.
Integer variables could be added. None of the variables that are read through this code were integer.
I did run into this situation with some of the oro_* files which were originally NetCDF Classic format. Had to convert them to NetCDF4 to get them working with this code. I'll add some testing for this.

…_get_current_pelist_comm to gain access to the appropriate communicator

…communicator associated with the current pelist

dkokron · 2023-11-27T20:24:42Z

I'd need to mimic the following code using only FMS calls.

     color=domain%tile_id
     call MPI_Comm_split(mpp_get_current_pelist_comm(), color(1), mpp_pe(), domain%TileComm, err)

What I need is an MPI communicator that contains the ranks associated with each tile of a domain2d. I think the following should accomplish the task. Is there a better way?

     !allocate(pelist(mpp_get_tile_npes(domain)))
     !call mpp_get_tile_pelist(domain, pelist)
     !call mpp_declare_pelist(pelist)
     !domain%TileComm = peset(get_peset(pelist))%id
     !deallocate(pelist)

…int fallback notice

bensonr · 2023-11-28T19:22:32Z

@dkokron - I'm poking around in FMS and the domains structure to find the info you are asking about. We've been following your commits and discussing them internally. I'd like to propose another meeting to share some of our observations.

dkokron · 2023-11-28T23:15:13Z

@bensonr I'm free 11am to 2pm central every day. Send me an invite.

dkokron · 2023-11-29T21:59:13Z

How does one prepare and run the unit test suite when FMS is compiled using CMake?

rem1776 · 2023-11-30T15:01:41Z

How does one prepare and run the unit test suite when FMS is compiled using CMake?

Unfortunately our test suite is currently only available via the autotools build since it's our main supported build system. We have an open issue for it here: #827.

uramirez8707

Some comments for discussion

uramirez8707 · 2023-12-04T17:49:47Z

fms2_io/include/netcdf_read_data.inc

@@ -362,8 +362,12 @@ subroutine netcdf_read_data_2d(fileobj, variable_name, buf, unlim_dim_level, &
      type is (integer(kind=i8_kind))
        err = nf90_get_var(fileobj%ncid, varid, buf, start=c, count=e)
      type is (real(kind=r4_kind))
+        ! NetCDF does not have the ability to specify collective I/O at the file basis
+        ! so we must activate at the variable level
+        if(fileobj%use_collective) err = nf90_var_par_access(fileobj%ncid, varid, nf90_collective)


This is inside a if (fileobj%is_root) then so only the first rank in the pelist is going to get to the nf90_var_par_access and nf90_get_var calls

uramirez8707 · 2023-12-04T17:51:36Z

fms2_io/include/netcdf_read_data.inc

@@ -362,8 +362,12 @@ subroutine netcdf_read_data_2d(fileobj, variable_name, buf, unlim_dim_level, &
      type is (integer(kind=i8_kind))
        err = nf90_get_var(fileobj%ncid, varid, buf, start=c, count=e)
      type is (real(kind=r4_kind))
+        ! NetCDF does not have the ability to specify collective I/O at the file basis
+        ! so we must activate at the variable level
+        if(fileobj%use_collective) err = nf90_var_par_access(fileobj%ncid, varid, nf90_collective)
        err = nf90_get_var(fileobj%ncid, varid, buf, start=c, count=e)


for domain decomposed files, c and e correspond to the section of the data for all of the PEs in the pelist:

FMS/fms2_io/include/domain_read.inc

Lines 183 to 185 in 7f58528

call netcdf_read_data(fileobj, variable_name, buf_i4_kind, &

unlim_dim_level=unlim_dim_level, &

corner=c, edge_lengths=e, broadcast=.false.)

the netcdf_read_data calls will need to be modified in domain_read_*d so that each ranks reads their own section

@uramirez8707 Those c and e arrays have had the appropriate values during all of my testing so far. We'll see if that changes after removing the io_layout=1,1 'fix'.

diff --git a/tools/fv_mp_mod.F90 b/tools/fv_mp_mod.F90 index 4f2c0f2..f1e6012 100644 --- a/tools/fv_mp_mod.F90 +++ b/tools/fv_mp_mod.F90 @@ -658,12 +658,12 @@ contains !--- if io_layout\=(1,1) then read io_layout=io_layout (no change) l_layout = mpp_get_io_domain_layout(domain) call mpp_copy_domain(domain, domain_for_read) - if (ALL(l_layout == 1)) then - call mpp_get_layout(domain, l_layout) + !if (ALL(l_layout == 1)) then + ! call mpp_get_layout(domain, l_layout) + ! call mpp_define_io_domain(domain_for_read, l_layout) + !else call mpp_define_io_domain(domain_for_read, l_layout) - else - call mpp_define_io_domain(domain_for_read, l_layout) - endif + !endif endif deallocate(pe_start,pe_end)

uramirez8707 · 2023-12-04T17:55:24Z

fms2_io/include/netcdf_read_data.inc

        err = nf90_get_var(fileobj%ncid, varid, buf, start=c, count=e)
      type is (real(kind=r8_kind))
+        if(fileobj%use_collective) err = nf90_var_par_access(fileobj%ncid, varid, nf90_collective)


I think if use_collective is true then bdcast needs to be set to false

FMS/fms2_io/include/netcdf_read_data.inc

Lines 149 to 153 in 7f58528

if (present(broadcast)) then

bcast = broadcast

else

bcast = .true.

endif

Good point. Thank you.

uramirez8707 · 2023-12-04T17:58:25Z

fms2_io/netcdf_io.F90

+      ! so we must activate at the variable level in netcdf_read_data_2d() and netcdf_read_data_3d()
+      if(fileobj%use_collective .and. fileobj%TileComm < 0) then
+        !write(6,'("netcdf_file_open: Open for collective read "A,I4)') trim(fileobj%path), szTile
+        err = nf90_open(trim(fileobj%path), ior(NF90_NOWRITE, NF90_MPIIO), fileobj%ncid, comm=fileobj%TileComm, info=MPI_INFO_NULL)


how will this work for reading metadata? Currently a root rank reads the data and broadcasts to the other ranks. Is this still needed?

@bensonr Do you want to maintain support for the following read scenarios?

fileobj%is_root (single rank) reads then broadcasts. I believe this is the situation when the io_layout=(1,1) WAR isn't active.

all ranks associated with a file read independently (parallel-independent). This is the situation when the io_layout=(1,1) WAR is active)

all ranks associated with a file read collectively (parallel-collective).

Put another way, are you proposing UFS eliminates the io_layout=(1,1) work-around from their code?

@dkokron - I thought I answered this, but see I never hit the comment button. I am proposing the UFS eliminate the workaround as the new path through FMS with NC-4 files should behave the same way. Let me know your thoughts.

I'm only testing for NetCDF-4 format after a failed attempt to open for parallel-collective.

In order to obtain the new FMS path, the user needs to modify their code before each open_file() for which they want the new path (currently 4 file types out of the many 10s).

e.g.

+ Phy_restart%use_collective = .true. + Phy_restart%TileComm = mpp_get_tile_comm(fv_domain) amiopen=open_file(Phy_restart, trim(fname), 'read', domain=fv_domain, is_restart=.true., dont_add_res_to_filename=.true.)

The way things are coded now (after removing the new path from those is_root blocks and removing the io_layout=(1,1) workaround), the default path is still read-broadcast.

Let me think about this some more.

thomas-robinson · 2023-12-05T17:13:03Z

fms2_io/netcdf_io.F90

@@ -32,6 +32,7 @@ module netcdf_io_mod
 use mpp_mod
 use fms_io_utils_mod
 use platform_mod
+use mpi, only: MPI_INFO_NULL


This MPI dependency should be removed. It can be made public through mpp

I will remove this dependency once FMS provides the needed functionality via some other means.

fms2_io/netcdf_io.F90

MatthewPyle-NOAA · 2024-01-26T17:28:52Z

Does this PR still have a path forward? The RRFS development team would love to utilize it within a true FMS release.

bensonr · 2024-01-30T18:56:41Z

@MatthewPyle-NOAA - yes, this still has a path forward. I estimate having something you cana test with by Feb 8th

dkokron · 2024-01-30T19:57:20Z

Matthew, Activating the new FMS feature in RRFS requires some code changes. I should run some testing in my sandbox before you try it out. For my testing to be relevant, I need an updated version of the RRFS code. I have version 0.6.1. Shun Liu has promised to get me the latest. Can you check with Shun on that request? Dan

…

On Tue, Jan 30, 2024 at 12:56 PM Rusty Benson ***@***.***> wrote: @MatthewPyle-NOAA <https://github.com/MatthewPyle-NOAA> - yes, this still has a path forward. I estimate having something you cana test with by Feb 8th — Reply to this email directly, view it on GitHub <#1405 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACODV2FWKXKM57SACPLSIGDYRE67HAVCNFSM6AAAAAA7B46T2GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJXGY4TENZRGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

MatthewPyle-NOAA · 2024-02-01T20:27:10Z

@dkokron I just pinged Shun via e-mail - hopefully one way or another we'll be able to point you at an updated model code by next week.

bensonr · 2024-02-09T04:04:07Z

@dkokron @MatthewPyle-NOAA - see PR #1457

MatthewPyle-NOAA · 2024-02-12T18:10:19Z

Thanks @bensonr. @dkokron do you still need fresh RRFS code to test with?

dkokron · 2024-02-13T04:16:03Z

Haven't heard back from Shun.

…

On Mon, Feb 12, 2024, 12:10 PM MatthewPyle-NOAA ***@***.***> wrote: Thanks @bensonr <https://github.com/bensonr>. @dkokron <https://github.com/dkokron> do you still need fresh RRFS code to test with? — Reply to this email directly, view it on GitHub <#1405 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACODV2FATWLYIO6DYLFS42DYTJLJNAVCNFSM6AAAAAA7B46T2GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZZGI3TMNRUGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

ShunLiu-NOAA · 2024-02-21T18:40:52Z

Haven't heard back from Shun.
…
On Mon, Feb 12, 2024, 12:10 PM MatthewPyle-NOAA @.> wrote: Thanks @bensonr https://github.com/bensonr. @dkokron https://github.com/dkokron do you still need fresh RRFS code to test with? — Reply to this email directly, view it on GitHub <#1405 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACODV2FATWLYIO6DYLFS42DYTJLJNAVCNFSM6AAAAAA7B46T2GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZZGI3TMNRUGQ . You are receiving this because you were mentioned.Message ID: @.>

Dan, We test these modification in RRFS realtime parallel. With these changes, the run time of 1h forecast is reduced from 1350s to 950s. I think these changes should be merged into develop. Thank you so much for your help.

dkokron · 2024-02-22T19:11:46Z

I wouldn't add the changes to rrfs until we get an FMS release with the required modifications. Remember, my FMS build on Dogwood is NOT official. On Wed, Feb 21, 2024, 12:41 PM ShunLiu-NOAA ***@***.***> wrote:

…

bensonr · 2024-02-23T14:29:11Z

@dkokron - there is now an alpha tag with the ability to declare a pelist and get the communicator directly as well as to query a domain to get individual tile communicators or the full domain communicator.

We'd like to include your PR in the release, so let us know how much time you think you'll need to merge/implement within the current working branch. Don't forget to include logic specific unit tests for the MPI-IO capabilities.

dkokron · 2024-02-26T18:40:39Z

As far as I am aware, the only place where I am allowed to do this sort of development is on Acorn. That system is down for software work until at least 1 March.

…equest

dkokron · 2024-03-04T14:43:25Z

@bensonr Do you suggest overlaying the parallel IO work on top of 2024.01-alpha4?

rem1776 · 2024-03-04T15:26:25Z

@dkokron We discussed this a bit with Rusty, it would be best to put the updates in on the main branch instead.

dkokron · 2024-03-04T16:05:52Z

I will overlay on main.

dkokron · 2024-03-04T19:45:12Z

@bensonr @rem1776 The user application needs some way to tell FMS which MPI communicator is associated with which file. I did that in UFS using the following code. I don't see the equivalent of mpp_set_tile_comm() in the current FMS 'main'. Which FMS function should I be using to achieve the functionality of mpp_set_tile_comm()?

    call mpp_get_layout(Atm%domain,read_layout)
    call mpp_copy_domain(Atm%domain, domain_for_read)
    call mpp_define_io_domain(domain_for_read, read_layout)
    call mpp_set_tile_comm(domain_for_read)

    GFS_restart%use_collective = .true.
    GFS_restart%TileComm = mpp_get_tile_comm(domain_for_read)

bensonr · 2024-03-04T20:58:32Z

@dkokron - there are two new interfaces you can use to get each the current tile commID (mpp_get_domain_tile_commid) and the full domain commID (mpp_get_domain_commid). For a single tile domain, the tile and domain commIDs should be the same.

There is also a new optional commID argument to mpp_declare_pelist that will return the commID for any pelist group created.

dkokron · 2024-03-04T21:22:52Z

@bensonr I tried using mpp_get_domain_tile_commid() in the above sequence, but it failed at runtime. I'll investigate some more and report back here.

dkokron · 2024-03-05T19:13:01Z

@bensonr Does FMS provide a way to access MPI_COMM_NULL?

dkokron · 2024-03-05T19:44:14Z

I think I accidentally closed this PR when I renamed my fms.ParallelStartup branch to fms.ParallelStartupLegacy. The fms.ParallelStartup branch is now built on top of recent FMS:main.

Should I open a new PR?

rem1776 · 2024-03-05T20:27:50Z

@dkokron Yeah you'll need to open a new PR from the new branch. With github it seems there's no way to change the branch you're merging in once the PR is created.

And to answer your earlier question, MPI_COMM_NULL can be accessed through MPP_INFO_NULL in mpp_mod.

dkokron · 2024-03-05T21:22:13Z

MPI_INFO_NULL and MPI_COMM_NULL are not the same thing.
/opt/cray/pe/mpich/8.1.12/ofi/intel/19.0/include/mpif.h: INTEGER MPI_COMM_NULL
/opt/cray/pe/mpich/8.1.12/ofi/intel/19.0/include/mpif.h: PARAMETER (MPI_COMM_NULL=67108864)

/opt/cray/pe/mpich/8.1.12/ofi/intel/19.0/include/mpif.h: INTEGER MPI_INFO_NULL
/opt/cray/pe/mpich/8.1.12/ofi/intel/19.0/include/mpif.h: PARAMETER (MPI_INFO_NULL=469762048)

bensonr · 2024-03-05T21:36:38Z

@dkokron - sorry have been busy all day. It doesn't look like we are pushing out MPI_COMM_NULL in FMS. If you need that, we can implement it as part of your PR in a fashion similar to the way MPI_INFO_NULL is exposed as MPP_INFO_NULL.

If you think we need to have a call for anything, let me know and we can set one up for tomorrow early morning or late afternoon.

thomas-robinson · 2024-03-07T10:38:59Z

@dkokron what happened to this PR?

dkokron · 2024-03-09T04:48:25Z

@thomas-robinson It got closed accidentally when I updated the underlying branch to a more recent FMS branch. I will open a new PR once testing of the new code is complete.

dkokron · 2024-03-11T16:51:06Z

New PR at
#1477

Dan Kokron and others added 5 commits August 8, 2023 16:15

Initial working proof-of-concept to enable MPI-IO collective optimiza…

07ea0d7

…tions within NetCDF. This has been tested on Cactus using the RRFS application with up to 24 ensemble members.

Merge branch 'NOAA-GFDL:main' into fms.ParallelStartup

7ba4f38

Use mpp_comm_private instead of MPI_COMM_WORLD for parallel IO

b2e41ad

Merge branch 'fms.ParallelStartup' of github.com:dkokron/FMS into fms…

16ba1e4

….ParallelStartup Merged with GFDL main

Mods to allow user control of collective reads from input.nml. Defaul…

43838ac

…t behavior is unchanged, user has to activate collective reads

dkokron requested review from thomas-robinson, bensonr and rem1776 as code owners November 7, 2023 22:12

bensonr reviewed Nov 8, 2023

View reviewed changes

Remove debugging prints per Rusty Benson

b875ecc

Dan Kokron added 5 commits November 10, 2023 17:38

Revert makeing mpp_comm_private public. Use the new function call mpp…

b46f59b

…_get_current_pelist_comm to gain access to the appropriate communicator

Add function mpp_get_current_pelist_comm to allow user access to the …

5e6137f

…communicator associated with the current pelist

Remove file test functionality

0ad3f73

Add tile communicator get and set utilities

09e5779

Remove references to is_reader

24abffc

Dan Kokron added 2 commits November 28, 2023 15:59

Add fallback to parallel-independent if parallel-collective fails. Pr…

9f4bc7f

…int fallback notice

Improve fallback notice slightly

c45d842

Dan Kokron added 4 commits November 29, 2023 02:13

Revert all changes to netcdf_io_init

495a8bb

i index not needed anymore

4a517ae

minor formatting change

92494ea

Update proposed replacement for MPI_Comm_split

cd2b22a

uramirez8707 reviewed Dec 4, 2023

View reviewed changes

thomas-robinson requested changes Dec 5, 2023

View reviewed changes

Move collective code path from under is_root control

82731db

bensonr mentioned this pull request Feb 9, 2024

store and expose MPI sub-communicators for tiles and domains #1456

Closed

Save in preparation for merge. May not need these in the final pull-r…

4182ea9

…equest

dkokron closed this Mar 5, 2024

dkokron deleted the fms.ParallelStartup branch March 5, 2024 19:40

dkokron mentioned this pull request Mar 11, 2024

Fms.parallel startup #1477

Merged

8 tasks

	call netcdf_read_data(fileobj, variable_name, buf_i4_kind, &
	unlim_dim_level=unlim_dim_level, &
	corner=c, edge_lengths=e, broadcast=.false.)

	if (present(broadcast)) then
	bcast = broadcast
	else
	bcast = .true.
	endif

Add MPI-IO collective reads of NetCDF-4 files #1405

Add MPI-IO collective reads of NetCDF-4 files #1405

Conversation

dkokron commented Nov 7, 2023

bensonr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bensonr Nov 8, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dkokron commented Nov 8, 2023

dkokron commented Nov 27, 2023

bensonr commented Nov 28, 2023

dkokron commented Nov 28, 2023

dkokron commented Nov 29, 2023

rem1776 commented Nov 30, 2023

uramirez8707 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dkokron Dec 11, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MatthewPyle-NOAA commented Jan 26, 2024

bensonr commented Jan 30, 2024

dkokron commented Jan 30, 2024 via email

MatthewPyle-NOAA commented Feb 1, 2024

bensonr commented Feb 9, 2024

MatthewPyle-NOAA commented Feb 12, 2024

dkokron commented Feb 13, 2024 via email

ShunLiu-NOAA commented Feb 21, 2024

dkokron commented Feb 22, 2024 via email

bensonr commented Feb 23, 2024

dkokron commented Feb 26, 2024

dkokron commented Mar 4, 2024

rem1776 commented Mar 4, 2024

dkokron commented Mar 4, 2024

dkokron commented Mar 4, 2024 • edited Loading

bensonr commented Mar 4, 2024

dkokron commented Mar 4, 2024

dkokron commented Mar 5, 2024

dkokron commented Mar 5, 2024 • edited Loading

rem1776 commented Mar 5, 2024 • edited Loading

dkokron commented Mar 5, 2024

bensonr commented Mar 5, 2024

thomas-robinson commented Mar 7, 2024

dkokron commented Mar 9, 2024

dkokron commented Mar 11, 2024

bensonr Nov 8, 2023 •

edited

Loading

dkokron Dec 11, 2023 •

edited

Loading

dkokron commented Mar 4, 2024 •

edited

Loading

dkokron commented Mar 5, 2024 •

edited

Loading

rem1776 commented Mar 5, 2024 •

edited

Loading