Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

getting error when trying to run look <source elided> out = np.full(arr1.shape, np.nan, dtype=np.float32) rows, cols = arr1.shape #802

Open
krishnaglodha opened this issue May 25, 2024 · 3 comments
Assignees
Labels
bug Something isn't working

Comments

@krishnaglodha
Copy link

Describe the bug
I'm trying to get statistical values of my xarray.DataArray in loop , but getting the error

TypingError                               Traceback (most recent call last)
Cell In[18], [line 2](vscode-notebook-cell:?execution_count=18&line=2)
      [1](vscode-notebook-cell:?execution_count=18&line=1) for val in index_xarray:
----> [2](vscode-notebook-cell:?execution_count=18&line=2)     print(val.max().compute().values.item())

File ~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1169, in DataArray.compute(self, **kwargs)
   [1150](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1150) """Manually trigger loading of this array's data from disk or a
   [1151](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1151) remote source into memory and return a new array. The original is
   [1152](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1152) left unaltered.
   (...)
   [1166](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1166) dask.compute
   [1167](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1167) """
   [1168](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1168) new = self.copy(deep=False)
-> [1169](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1169) return new.load(**kwargs)

File ~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1143, in DataArray.load(self, **kwargs)
   [1125](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1125) def load(self, **kwargs) -> Self:
   [1126](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1126)     """Manually trigger loading of this array's data from disk or a
   [1127](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1127)     remote source into memory and return this array.
   [1128](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1128) 
   (...)
   [1141](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1141)     dask.compute
   [1142](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1142)     """
-> [1143](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1143)     ds = self._to_temp_dataset().load(**kwargs)
   [1144](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1144)     new = self._from_temp_dataset(ds)
   [1145](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1145)     self._variable = new._variable

File ~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataset.py:845, in Dataset.load(self, **kwargs)
    [842](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataset.py:842) chunkmanager = get_chunked_array_type(*lazy_data.values())
    [844](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataset.py:844) # evaluate all the chunked arrays simultaneously
--> [845](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataset.py:845) evaluated_data: tuple[np.ndarray[Any, Any], ...] = chunkmanager.compute(
    [846](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataset.py:846)     *lazy_data.values(), **kwargs
    [847](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataset.py:847) )
    [849](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataset.py:849) for k, data in zip(lazy_data, evaluated_data):
    [850](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataset.py:850)     self.variables[k].data = data

File ~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/namedarray/daskmanager.py:86, in DaskManager.compute(self, *data, **kwargs)
     [81](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/namedarray/daskmanager.py:81) def compute(
     [82](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/namedarray/daskmanager.py:82)     self, *data: Any, **kwargs: Any
     [83](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/namedarray/daskmanager.py:83) ) -> tuple[np.ndarray[Any, _DType_co], ...]:
     [84](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/namedarray/daskmanager.py:84)     from dask.array import compute
---> [86](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/namedarray/daskmanager.py:86)     return compute(*data, **kwargs)

File ~/anaconda3/envs/stac/lib/python3.12/site-packages/dask/base.py:661, in compute(traverse, optimize_graph, scheduler, get, *args, **kwargs)
    [658](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/dask/base.py:658)     postcomputes.append(x.__dask_postcompute__())
    [660](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/dask/base.py:660) with shorten_traceback():
--> [661](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/dask/base.py:661)     results = schedule(dsk, keys, **kwargs)
    [663](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/dask/base.py:663) return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)])

File ~/anaconda3/envs/stac/lib/python3.12/site-packages/numba/core/dispatcher.py:468, in _DispatcherBase._compile_for_args(self, *args, **kws)
    [464](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/numba/core/dispatcher.py:464)         msg = (f"{str(e).rstrip()} \n\nThis error may have been caused "
    [465](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/numba/core/dispatcher.py:465)                f"by the following argument(s):\n{args_str}\n")
    [466](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/numba/core/dispatcher.py:466)         e.patch_message(msg)
--> [468](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/numba/core/dispatcher.py:468)     error_rewrite(e, 'typing')
    [469](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/numba/core/dispatcher.py:469) except errors.UnsupportedError as e:
    [470](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/numba/core/dispatcher.py:470)     # Something unsupported is present in the user code, add help info
    [471](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/numba/core/dispatcher.py:471)     error_rewrite(e, 'unsupported_error')

File ~/anaconda3/envs/stac/lib/python3.12/site-packages/numba/core/dispatcher.py:409, in _DispatcherBase._compile_for_args.<locals>.error_rewrite(e, issue_type)
    [407](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/numba/core/dispatcher.py:407)     raise e
    [408](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/numba/core/dispatcher.py:408) else:
--> [409](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/numba/core/dispatcher.py:409)     raise e.with_traceback(None)

TypingError: Failed in nopython mode pipeline (step: nopython frontend)
('wrong tuple length for $130load_attr.11: ', 'expected 2, got 3')
During: typing of exhaust iter at /Users/krishnaglodha/anaconda3/envs/stac/lib/python3.12/site-packages/xrspatial/multispectral.py (772)

File "../../../../anaconda3/envs/stac/lib/python3.12/site-packages/xrspatial/multispectral.py", line 772:
def _normalized_ratio_cpu(arr1, arr2):
    <source elided>
    out = np.full(arr1.shape, np.nan, dtype=np.float32)
    rows, cols = arr1.shape
    ^

Expected behavior
I should be able to print max value
Code

catalog = pystac_client.Client.open('https://earth-search.aws.element84.com/v1')

time_range = f"{start_date}/{end_date}"
collection = "sentinel-2-l2a"
geometry = polygon["features"][0]["geometry"]
search = catalog.search(
    collections=[collection], intersects=geometry, datetime=time_range
) 
print(search.get_all_items())
data = load(
            search.items(),
            geopolygon=geometry,
            chunks={},
            resolution=1,
            projection="epsg:3857",
        )
dc = data.rio.clip([geometry], crs="epsg:4326")
index_xarray = ms.ndvi(dc.nir, dc.red)

for val in index_xarray:
    print(val.max().compute().values.item())

Desktop (please complete the following information):

  • OS: MacOS
  • Browser - chrome

Additional context
Add any other context about the problem here.

@krishnaglodha krishnaglodha added the bug Something isn't working label May 25, 2024
@brendancol brendancol self-assigned this May 27, 2024
@brendancol
Copy link
Contributor

@krishnaglodha thanks for submitting the issue. I'll take a deeper look at NDVI and see if anything changed with recent 3.12 xarray-spatial release.

Can you tell post what type dc.nir.data is...and dtype, and shape?

What happens if you make sure dc.nir etc are 2d arrays by squeezing out any empty dimensions?

@krishnaglodha
Copy link
Author

Screenshot 2024-05-27 at 10 30 32 AM

what baffled me is that is worked when I passed

data = load(
            search.items(),
            geopolygon=geometry,
            chunks={},
            resolution=1,
            projection="epsg:3857",
        ).isel(time=0)

i.e. for time =0

but when I try to get

data = load(
            search.items(),
            geopolygon=geometry,
            chunks={},
            resolution=1,
            projection="epsg:3857",
        )

and then loop through data , I get error

@brendancol
Copy link
Contributor

@krishnaglodha you may be triggering a computation when running isel...I'm not totally sure.

It sounds like there may be one layer in the response that does not have the dimensionality that you expect. If you loop through the layers and print each time step out, do you see that there are only two dimensions?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants