Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Canvas.raster should support distributed and out-of-core regridding of dask arrays #553

Closed
philippjfr opened this issue Jan 18, 2018 · 2 comments

Comments

@philippjfr
Copy link
Member

The Canvas.raster method up- or downsamples xarray DataArrays, which themselves may wrap a numpy or dask array. Since dask arrays can often be larger than can be fit into memory at one time it would be great if it could apply the resampling in a distributed and out-of-core manner. This requires loading the chunks in one by one, up- or downsampling them and then stitching them all together again. Technically it should be possible to apply this using the new xr.apply_ufunc helper but there are certain things I haven't yet fully thought through, e.g. during downsampling I believe you need to have some overlap between chunks so that the aggregation at the edges of each chunk aggregates over the edge of the next chunk and a similar solution may be required for correct interpolation during upsampling.

This is a significant chunk of work but a very well defined task and would be very useful for very large arrays.

@jbednar
Copy link
Member

jbednar commented Feb 9, 2018

The notebook in https://anaconda.org/philippjfr/tiling_demo/notebook shows some of how to do this manually, I think...

@philippjfr
Copy link
Member Author

Implemented in #762

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants