ValueError when opening cfradial file using fsspec

[aladinor]

• xradar version: 0.6.1
• Python version: 3.12.4
• Operating System: Ubuntu 22.04

Hi all,

I am trying to open cfradial files from an s3 bucket as follows,

import xradar as xd
import fsspec


def main():
    # Set the URL and path for the cloud
    URL = 'https://js2.jetstream-cloud.org:8001/'
    path = f'pythia/radar/erad2024'

    fs = fsspec.filesystem("s3", anon=True, client_kwargs=dict(endpoint_url=URL))
    files = fs.glob("pythia/radar/erad2024/20240522_MeteoSwiss_ARPA_Lombardia/Data/Cband/*.nc")
    radar_files = [f"s3://{i}" for i in files]
    with fs.open(radar_files[0], mode="rb") as fileObj:
        dt2 = xd.io.open_cfradial1_datatree(fileObj)


if __name__ == '__main__':
    main()

And I got the following error.

Traceback (most recent call last):
  File "/media/alfonso/drive/Alfonso/python/raw2zarr/sigmet2zarr/delete.py", line 17, in <module>
    main()
  File "/media/alfonso/drive/Alfonso/python/raw2zarr/sigmet2zarr/delete.py", line 13, in main
    dt2 = xd.io.open_cfradial1_datatree(fs.open(radar_files[0]))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/alfonso/mambaforge/envs/raw2zarr/lib/python3.12/site-packages/xradar/io/backends/cfradial1.py", line 347, in open_cfradial1_datatree
    ds = open_dataset(filename_or_obj, engine="netcdf4", **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/alfonso/mambaforge/envs/raw2zarr/lib/python3.12/site-packages/xarray/backends/api.py", line 571, in open_dataset
    backend_ds = backend.open_dataset(
                 ^^^^^^^^^^^^^^^^^^^^^
  File "/home/alfonso/mambaforge/envs/raw2zarr/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 645, in open_dataset
    store = NetCDF4DataStore.open(
            ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/alfonso/mambaforge/envs/raw2zarr/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 381, in open
    raise ValueError(
ValueError: can only read bytes or file-like objects with engine='scipy' or 'h5netcdf'

Digging around, I found this is happening because the netcdf4 engine does not support fsspec s3.FileSystem object here

xradar/xradar/io/backends/cfradial1.py

Line 345 in 482952b

ds = open_dataset(filename_or_obj, engine="netcdf4", **kwargs)

However, if we change the engine to h5netcdf, it will work and will create exactly the same datatree object.

ds = open_dataset(filename_or_obj, engine="h5netcdf", **kwargs)

@kmuehlbauer, what do you think are the main implications of using h5netcdf instead of netcdf4 engine?

[aladinor]

I guess this error has to do with this fsspec/filesystem_spec#579 (comment)