Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

netcdf SWMR compatible writing #108

Open
heikoklein opened this issue Jan 3, 2023 · 2 comments
Open

netcdf SWMR compatible writing #108

heikoklein opened this issue Jan 3, 2023 · 2 comments

Comments

@heikoklein
Copy link
Member

SNAP currently opens the output netcdf-file for each time-step in write-append mode and closes it again after the output-timestep is written. With netcdf3 and early netcdf4 implementations this allowed files to be read while the model was still running.

The new hdf5.10 SWMR (single write multiple read) implementation https://support.hdfgroup.org/HDF5/docNewFeatures/SWMR/Design-HDF5-FileLocking.pdf explicitly states that files which are opened for reading can no longer be opened for writing. This leads to the SNAP model crashing when one starts to analyze the model-results while the model is still running.

The updated implementation should keep the write file-handle of the output netcdf-file open for the complete time the model is running. It should also ensure that the output is synced to disk after the output-timestep so all information is available to all readers. The SWMR mode should be the default, while the old writer should still be kept available.

@magnusuMET
Copy link
Collaborator

See progress for this feature in netcdf-c here: Unidata/netcdf-c#1448

@heikoklein
Copy link
Member Author

The implementation of this feature can already start independently of the availability in netcdf-c (though full functionality requires netcdf-c)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants