Open Data using File-Like Object

[Ockenfuss]

• xradar version: 0.6.0
• Python version: 3.11.5
• Operating System: Alma Linux

Is it possible to use a file-like object instead of filenames when using xradar? According to the xarray open_dataset documentation, as well as the xradar Odim Backend Documentation, file objects are supported, but with the 'odim' engine, I get an error:

from open_radar_data import DATASETS
import xarray as xr
import xradar as xd
filename = DATASETS.fetch("71_20181220_060628.pvol.h5")
#works:
ds = xr.open_dataset(filename, group="sweep_0", engine="odim") 
#works as well (at least no exception raised):
with open(filename, 'rb') as f:
    ds = xr.open_dataset(f)
# Raises TypeError: cannot pickle '_io.BufferedReader' object
with open(filename, 'rb') as f:
    ds = xr.open_dataset(f, group="sweep_0", engine="odim")

Thank you for any help and the xradar development in general!
Paul

[kmuehlbauer]

@Ockenfuss

Thanks for raising and the clear description. Could you please also provide the full traceback? Thanks!

Which xarray version are you on?

[Ockenfuss]

I am on the current xarray version 2024.7.0. Can you reproduce it?

Traceback

Traceback (most recent call last):
  File "/scratch/p/Paul.Ockenfuss/Test/test_polardata/xradar_bug.py", line 6, in <module>
    ds = xr.open_dataset(f, group="sweep_0", engine="odim")
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch-local/Paul.Ockenfuss/virtual_environments/radar/lib/python3.11/site-packages/xarray/backends/api.py", line 588, in open_dataset
    backend_ds = backend.open_dataset(
                 ^^^^^^^^^^^^^^^^^^^^^
  File "/scratch-local/Paul.Ockenfuss/virtual_environments/radar/lib/python3.11/site-packages/xradar/io/backends/odim.py", line 825, in open_dataset
    ds = ds.sortby(dim0)
         ^^^^^^^^^^^^^^^
  File "/scratch-local/Paul.Ockenfuss/virtual_environments/radar/lib/python3.11/site-packages/xarray/core/dataset.py", line 8182, in sortby
    aligned_vars = align(self, *arrays, join="left")
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch-local/Paul.Ockenfuss/virtual_environments/radar/lib/python3.11/site-packages/xarray/core/alignment.py", line 882, in align
    aligner.align()
  File "/scratch-local/Paul.Ockenfuss/virtual_environments/radar/lib/python3.11/site-packages/xarray/core/alignment.py", line 582, in align
    self.reindex_all()
  File "/scratch-local/Paul.Ockenfuss/virtual_environments/radar/lib/python3.11/site-packages/xarray/core/alignment.py", line 557, in reindex_all
    self.results = tuple(
                   ^^^^^^
  File "/scratch-local/Paul.Ockenfuss/virtual_environments/radar/lib/python3.11/site-packages/xarray/core/alignment.py", line 558, in <genexpr>
    self._reindex_one(obj, matching_indexes)
  File "/scratch-local/Paul.Ockenfuss/virtual_environments/radar/lib/python3.11/site-packages/xarray/core/alignment.py", line 546, in _reindex_one
    return obj._reindex_callback(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch-local/Paul.Ockenfuss/virtual_environments/radar/lib/python3.11/site-packages/xarray/core/dataset.py", line 3517, in _reindex_callback
    reindexed = self.copy(deep=aligner.copy)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch-local/Paul.Ockenfuss/virtual_environments/radar/lib/python3.11/site-packages/xarray/core/dataset.py", line 1366, in copy
    return self._copy(deep=deep, data=data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch-local/Paul.Ockenfuss/virtual_environments/radar/lib/python3.11/site-packages/xarray/core/dataset.py", line 1402, in _copy
    variables[k] = v._copy(deep=deep, data=data.get(k), memo=memo)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch-local/Paul.Ockenfuss/virtual_environments/radar/lib/python3.11/site-packages/xarray/core/variable.py", line 925, in _copy
    ndata = copy.deepcopy(ndata, memo)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 211, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 211, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
         ^^^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 211, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 211, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
         ^^^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 211, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 211, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
         ^^^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 211, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 211, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
         ^^^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 211, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 211, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
         ^^^^^^^^^^^^^^^^^
  File "/software/opt/el_9/x86_64/python/3.11-2023.09/lib/python3.11/copy.py", line 161, in deepcopy
    rv = reductor(4)
         ^^^^^^^^^^^
TypeError: cannot pickle '_io.BufferedReader' object

[kmuehlbauer]

Yes, and I think I've already hit this, if you look at the tests:

xradar/tests/io/test_io.py

Lines 222 to 232 in bf03ab8

	def test_open_odim_dataset_stream(odim_file):
	with open(odim_file, mode="rb") as fhandle:
	contents = io.BytesIO(fhandle.read())
	xr.open_dataset(contents, group="sweep_0", engine="odim")
	def test_open_odim_dataset_fsspec(odim_file):
	with fsspec.open(odim_file, mode="rb") as fhandle:
	xr.open_dataset(fhandle, group="sweep_0", engine="odim")

The issue is triggered by ds.sortby(), we can strip this down to pure xarray:

import numpy as np
import netCDF4
import xarray as xr
import fsspec
import io

# create file with netCDF4 to have all bits and pieces
filename = "test.h5"
with netCDF4.Dataset(filename,'w') as f:
    f.createDimension('x', 3)
    f.createDimension('y', 6)
    var = f.createVariable('var', 'int8', ('x', 'y'))
    xcoord = f.createVariable('x', 'int8', ('x'))
    ycoord = f.createVariable('y', 'int8', ('y'))
    xcoord[:] = [2, 1, 0]
    ycoord[:] = [0, 1, 2, 3, 4, 5]
    var[:] = np.arange(18).reshape((3, 6))

This works, maybe because CachingFileManager ist used?

with xr.backends.H5NetCDFStore.open(filename) as store:
    store_entrypoint = xr.backends.StoreBackendEntrypoint()
    ds = store_entrypoint.open_dataset(store)
    display(ds["var"].values)
    ds = ds.sortby('x')
    display(ds["var"].values)

array([[ 0,  1,  2,  3,  4,  5],
       [ 6,  7,  8,  9, 10, 11],
       [12, 13, 14, 15, 16, 17]], dtype=int8)

array([[12, 13, 14, 15, 16, 17],
       [ 6,  7,  8,  9, 10, 11],
       [ 0,  1,  2,  3,  4,  5]], dtype=int8)

This works too, with reindex.

with open(filename, 'rb') as f:
    print(type(f))
    store = xr.backends.H5NetCDFStore.open(f)
    store_entrypoint = xr.backends.StoreBackendEntrypoint()
    ds = store_entrypoint.open_dataset(store)
    display(ds["var"].values)
    ds = ds.reindex({"x": np.array([0, 1, 2])})
    display(ds["var"].values)

<class '_io.BufferedReader'>
array([[ 0,  1,  2,  3,  4,  5],
       [ 6,  7,  8,  9, 10, 11],
       [12, 13, 14, 15, 16, 17]], dtype=int8)

array([[12, 13, 14, 15, 16, 17],
       [ 6,  7,  8,  9, 10, 11],
       [ 0,  1,  2,  3,  4,  5]], dtype=int8)

This breaks with sortby.

with open(filename, 'rb') as f:
    print(type(f))
    store = xr.backends.H5NetCDFStore.open(f)
    store_entrypoint = xr.backends.StoreBackendEntrypoint()
    ds = store_entrypoint.open_dataset(store)
    display(ds["var"].values)
    ds = ds.sortby("x")
    display(ds["var"].values)

Output with Traceback

<class '_io.BufferedReader'>
array([[ 0,  1,  2,  3,  4,  5],
       [ 6,  7,  8,  9, 10, 11],
       [12, 13, 14, 15, 16, 17]], dtype=int8)

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[69], line 6
      4 ds = store_entrypoint.open_dataset(store)
      5 display(ds["var"].values)
----> 6 ds = ds.sortby("x")
      7 display(ds["var"].values)

File [/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/site-packages/xarray/core/dataset.py:8182](http://localhost:8888/lab/tree/home/kai/python/gists/xradar/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/site-packages/xarray/core/dataset.py#line=8181), in Dataset.sortby(self, variables, ascending)
   8180     variables = variables
   8181 arrays = [v if isinstance(v, DataArray) else self[v] for v in variables]
-> 8182 aligned_vars = align(self, *arrays, join="left")
   8183 aligned_self = cast("Self", aligned_vars[0])
   8184 aligned_other_vars = cast(tuple[DataArray, ...], aligned_vars[1:])

File [/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/site-packages/xarray/core/alignment.py:882](http://localhost:8888/lab/tree/home/kai/python/gists/xradar/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/site-packages/xarray/core/alignment.py#line=881), in align(join, copy, indexes, exclude, fill_value, *objects)
    686 """
    687 Given any number of Dataset and/or DataArray objects, returns new
    688 objects with aligned indexes and dimension sizes.
   (...)
    872 
    873 """
    874 aligner = Aligner(
    875     objects,
    876     join=join,
   (...)
    880     fill_value=fill_value,
    881 )
--> 882 aligner.align()
    883 return aligner.results

File [/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/site-packages/xarray/core/alignment.py:582](http://localhost:8888/lab/tree/home/kai/python/gists/xradar/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/site-packages/xarray/core/alignment.py#line=581), in Aligner.align(self)
    580     self.results = self.objects
    581 else:
--> 582     self.reindex_all()

File [/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/site-packages/xarray/core/alignment.py:557](http://localhost:8888/lab/tree/home/kai/python/gists/xradar/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/site-packages/xarray/core/alignment.py#line=556), in Aligner.reindex_all(self)
    556 def reindex_all(self) -> None:
--> 557     self.results = tuple(
    558         self._reindex_one(obj, matching_indexes)
    559         for obj, matching_indexes in zip(
    560             self.objects, self.objects_matching_indexes
    561         )
    562     )

File [/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/site-packages/xarray/core/alignment.py:558](http://localhost:8888/lab/tree/home/kai/python/gists/xradar/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/site-packages/xarray/core/alignment.py#line=557), in <genexpr>(.0)
    556 def reindex_all(self) -> None:
    557     self.results = tuple(
--> 558         self._reindex_one(obj, matching_indexes)
    559         for obj, matching_indexes in zip(
    560             self.objects, self.objects_matching_indexes
    561         )
    562     )

File [/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/site-packages/xarray/core/alignment.py:546](http://localhost:8888/lab/tree/home/kai/python/gists/xradar/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/site-packages/xarray/core/alignment.py#line=545), in Aligner._reindex_one(self, obj, matching_indexes)
    543 new_indexes, new_variables = self._get_indexes_and_vars(obj, matching_indexes)
    544 dim_pos_indexers = self._get_dim_pos_indexers(matching_indexes)
--> 546 return obj._reindex_callback(
    547     self,
    548     dim_pos_indexers,
    549     new_variables,
    550     new_indexes,
    551     self.fill_value,
    552     self.exclude_dims,
    553     self.exclude_vars,
    554 )

File [/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/site-packages/xarray/core/dataset.py:3517](http://localhost:8888/lab/tree/home/kai/python/gists/xradar/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/site-packages/xarray/core/dataset.py#line=3516), in Dataset._reindex_callback(self, aligner, dim_pos_indexers, variables, indexes, fill_value, exclude_dims, exclude_vars)
   3515         reindexed = self._overwrite_indexes(new_indexes, new_variables)
   3516     else:
-> 3517         reindexed = self.copy(deep=aligner.copy)
   3518 else:
   3519     to_reindex = {
   3520         k: v
   3521         for k, v in self.variables.items()
   3522         if k not in variables and k not in exclude_vars
   3523     }

File [/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/site-packages/xarray/core/dataset.py:1366](http://localhost:8888/lab/tree/home/kai/python/gists/xradar/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/site-packages/xarray/core/dataset.py#line=1365), in Dataset.copy(self, deep, data)
   1269 def copy(self, deep: bool = False, data: DataVars | None = None) -> Self:
   1270     """Returns a copy of this dataset.
   1271 
   1272     If `deep=True`, a deep copy is made of each of the component variables.
   (...)
   1364     pandas.DataFrame.copy
   1365     """
-> 1366     return self._copy(deep=deep, data=data)

File [/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/site-packages/xarray/core/dataset.py:1402](http://localhost:8888/lab/tree/home/kai/python/gists/xradar/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/site-packages/xarray/core/dataset.py#line=1401), in Dataset._copy(self, deep, data, memo)
   1400         variables[k] = index_vars[k]
   1401     else:
-> 1402         variables[k] = v._copy(deep=deep, data=data.get(k), memo=memo)
   1404 attrs = copy.deepcopy(self._attrs, memo) if deep else copy.copy(self._attrs)
   1405 encoding = (
   1406     copy.deepcopy(self._encoding, memo) if deep else copy.copy(self._encoding)
   1407 )

File [/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/site-packages/xarray/core/variable.py:925](http://localhost:8888/lab/tree/home/kai/python/gists/xradar/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/site-packages/xarray/core/variable.py#line=924), in Variable._copy(self, deep, data, memo)
    922         ndata = indexing.MemoryCachedArray(data_old.array)  # type: ignore[assignment]
    924     if deep:
--> 925         ndata = copy.deepcopy(ndata, memo)
    927 else:
    928     ndata = as_compatible_data(data)

File [/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py:162](http://localhost:8888/lab/tree/home/kai/python/gists/xradar/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py#line=161), in deepcopy(x, memo, _nil)
    160                 y = x
    161             else:
--> 162                 y = _reconstruct(x, memo, *rv)
    164 # If is its own copy, don't memoize.
    165 if y is not x:

File [/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py:259](http://localhost:8888/lab/tree/home/kai/python/gists/xradar/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py#line=258), in _reconstruct(x, memo, func, args, state, listiter, dictiter, deepcopy)
    257 if state is not None:
    258     if deep:
--> 259         state = deepcopy(state, memo)
    260     if hasattr(y, '__setstate__'):
    261         y.__setstate__(state)

File [/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py:136](http://localhost:8888/lab/tree/home/kai/python/gists/xradar/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py#line=135), in deepcopy(x, memo, _nil)
    134 copier = _deepcopy_dispatch.get(cls)
    135 if copier is not None:
--> 136     y = copier(x, memo)
    137 else:
    138     if issubclass(cls, type):

File [/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py:201](http://localhost:8888/lab/tree/home/kai/python/gists/xradar/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py#line=200), in _deepcopy_tuple(x, memo, deepcopy)
    200 def _deepcopy_tuple(x, memo, deepcopy=deepcopy):
--> 201     y = [deepcopy(a, memo) for a in x]
    202     # We're not going to put the tuple in the memo, but it's still important we
    203     # check for it, in case the tuple contains recursive mutable structures.
    204     try:

File [/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py:136](http://localhost:8888/lab/tree/home/kai/python/gists/xradar/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py#line=135), in deepcopy(x, memo, _nil)
    134 copier = _deepcopy_dispatch.get(cls)
    135 if copier is not None:
--> 136     y = copier(x, memo)
    137 else:
    138     if issubclass(cls, type):

File [/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py:221](http://localhost:8888/lab/tree/home/kai/python/gists/xradar/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py#line=220), in _deepcopy_dict(x, memo, deepcopy)
    219 memo[id(x)] = y
    220 for key, value in x.items():
--> 221     y[deepcopy(key, memo)] = deepcopy(value, memo)
    222 return y

File [/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py:162](http://localhost:8888/lab/tree/home/kai/python/gists/xradar/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py#line=161), in deepcopy(x, memo, _nil)
    160                 y = x
    161             else:
--> 162                 y = _reconstruct(x, memo, *rv)
    164 # If is its own copy, don't memoize.
    165 if y is not x:

File [/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py:259](http://localhost:8888/lab/tree/home/kai/python/gists/xradar/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py#line=258), in _reconstruct(x, memo, func, args, state, listiter, dictiter, deepcopy)
    257 if state is not None:
    258     if deep:
--> 259         state = deepcopy(state, memo)
    260     if hasattr(y, '__setstate__'):
    261         y.__setstate__(state)

File [/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py:136](http://localhost:8888/lab/tree/home/kai/python/gists/xradar/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py#line=135), in deepcopy(x, memo, _nil)
    134 copier = _deepcopy_dispatch.get(cls)
    135 if copier is not None:
--> 136     y = copier(x, memo)
    137 else:
    138     if issubclass(cls, type):

File [/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py:201](http://localhost:8888/lab/tree/home/kai/python/gists/xradar/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py#line=200), in _deepcopy_tuple(x, memo, deepcopy)
    200 def _deepcopy_tuple(x, memo, deepcopy=deepcopy):
--> 201     y = [deepcopy(a, memo) for a in x]
    202     # We're not going to put the tuple in the memo, but it's still important we
    203     # check for it, in case the tuple contains recursive mutable structures.
    204     try:

File [/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py:136](http://localhost:8888/lab/tree/home/kai/python/gists/xradar/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py#line=135), in deepcopy(x, memo, _nil)
    134 copier = _deepcopy_dispatch.get(cls)
    135 if copier is not None:
--> 136     y = copier(x, memo)
    137 else:
    138     if issubclass(cls, type):

File [/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py:221](http://localhost:8888/lab/tree/home/kai/python/gists/xradar/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py#line=220), in _deepcopy_dict(x, memo, deepcopy)
    219 memo[id(x)] = y
    220 for key, value in x.items():
--> 221     y[deepcopy(key, memo)] = deepcopy(value, memo)
    222 return y

File [/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py:162](http://localhost:8888/lab/tree/home/kai/python/gists/xradar/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py#line=161), in deepcopy(x, memo, _nil)
    160                 y = x
    161             else:
--> 162                 y = _reconstruct(x, memo, *rv)
    164 # If is its own copy, don't memoize.
    165 if y is not x:

File [/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py:259](http://localhost:8888/lab/tree/home/kai/python/gists/xradar/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py#line=258), in _reconstruct(x, memo, func, args, state, listiter, dictiter, deepcopy)
    257 if state is not None:
    258     if deep:
--> 259         state = deepcopy(state, memo)
    260     if hasattr(y, '__setstate__'):
    261         y.__setstate__(state)

File [/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py:136](http://localhost:8888/lab/tree/home/kai/python/gists/xradar/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py#line=135), in deepcopy(x, memo, _nil)
    134 copier = _deepcopy_dispatch.get(cls)
    135 if copier is not None:
--> 136     y = copier(x, memo)
    137 else:
    138     if issubclass(cls, type):

File [/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py:201](http://localhost:8888/lab/tree/home/kai/python/gists/xradar/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py#line=200), in _deepcopy_tuple(x, memo, deepcopy)
    200 def _deepcopy_tuple(x, memo, deepcopy=deepcopy):
--> 201     y = [deepcopy(a, memo) for a in x]
    202     # We're not going to put the tuple in the memo, but it's still important we
    203     # check for it, in case the tuple contains recursive mutable structures.
    204     try:

File [/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py:136](http://localhost:8888/lab/tree/home/kai/python/gists/xradar/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py#line=135), in deepcopy(x, memo, _nil)
    134 copier = _deepcopy_dispatch.get(cls)
    135 if copier is not None:
--> 136     y = copier(x, memo)
    137 else:
    138     if issubclass(cls, type):

File [/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py:221](http://localhost:8888/lab/tree/home/kai/python/gists/xradar/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py#line=220), in _deepcopy_dict(x, memo, deepcopy)
    219 memo[id(x)] = y
    220 for key, value in x.items():
--> 221     y[deepcopy(key, memo)] = deepcopy(value, memo)
    222 return y

File [/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py:162](http://localhost:8888/lab/tree/home/kai/python/gists/xradar/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py#line=161), in deepcopy(x, memo, _nil)
    160                 y = x
    161             else:
--> 162                 y = _reconstruct(x, memo, *rv)
    164 # If is its own copy, don't memoize.
    165 if y is not x:

File [/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py:259](http://localhost:8888/lab/tree/home/kai/python/gists/xradar/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py#line=258), in _reconstruct(x, memo, func, args, state, listiter, dictiter, deepcopy)
    257 if state is not None:
    258     if deep:
--> 259         state = deepcopy(state, memo)
    260     if hasattr(y, '__setstate__'):
    261         y.__setstate__(state)

File [/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py:136](http://localhost:8888/lab/tree/home/kai/python/gists/xradar/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py#line=135), in deepcopy(x, memo, _nil)
    134 copier = _deepcopy_dispatch.get(cls)
    135 if copier is not None:
--> 136     y = copier(x, memo)
    137 else:
    138     if issubclass(cls, type):

File [/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py:201](http://localhost:8888/lab/tree/home/kai/python/gists/xradar/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py#line=200), in _deepcopy_tuple(x, memo, deepcopy)
    200 def _deepcopy_tuple(x, memo, deepcopy=deepcopy):
--> 201     y = [deepcopy(a, memo) for a in x]
    202     # We're not going to put the tuple in the memo, but it's still important we
    203     # check for it, in case the tuple contains recursive mutable structures.
    204     try:

File [/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py:136](http://localhost:8888/lab/tree/home/kai/python/gists/xradar/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py#line=135), in deepcopy(x, memo, _nil)
    134 copier = _deepcopy_dispatch.get(cls)
    135 if copier is not None:
--> 136     y = copier(x, memo)
    137 else:
    138     if issubclass(cls, type):

File [/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py:201](http://localhost:8888/lab/tree/home/kai/python/gists/xradar/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py#line=200), in _deepcopy_tuple(x, memo, deepcopy)
    200 def _deepcopy_tuple(x, memo, deepcopy=deepcopy):
--> 201     y = [deepcopy(a, memo) for a in x]
    202     # We're not going to put the tuple in the memo, but it's still important we
    203     # check for it, in case the tuple contains recursive mutable structures.
    204     try:

File [/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py:151](http://localhost:8888/lab/tree/home/kai/python/gists/xradar/home/kai/data/mambaforge/envs/xr_312_np2/lib/python3.12/copy.py#line=150), in deepcopy(x, memo, _nil)
    149 reductor = getattr(x, "__reduce_ex__", None)
    150 if reductor is not None:
--> 151     rv = reductor(4)
    152 else:
    153     reductor = getattr(x, "__reduce__", None)

TypeError: cannot pickle 'BufferedReader' instances

There are quite a bunch of GH issues out there, which are discussing that problem. It seems, that io.BufferedReader can't be pickled.

XRef:

• Inconsistent behavior with fsspec.open with mode='r' vs. mode='rb' fsspec/filesystem_spec#579
• Regressions on Python 3.12: TypeError: cannot pickle 'IndexedGzipFile' instances pauldmccarthy/indexed_gzip#125
• tbc

What I'm wondering is why the reindex above works and the sortby fails? @benbovy, can you shed some light here?

[Ockenfuss]

Thanks for the investigation @kmuehlbauer! Interesting that it works with reindex, but not sortby.

[kmuehlbauer]

Interesting that it works with reindex, but not sortby.

There is a deepcopy in the sortby code flow somewhere, but not in the reindex. Let's see if we get some answers from @benbovy (sorry for tagging twice).

[benbovy]

Sorry @kmuehlbauer not sure what is causing the different behavior here, although I haven't inspected further the reindex vs. sortby code.

[kmuehlbauer]

Thanks @benbovy, I'll open an issue on the xarray tracker for further investigation.