numba issues with 0.6.1

[AaronDavidSchneider]

Hi,

I get some numba issues with 0.6.1 that were not there with 0.6.0.

I am not entirely sure how to make a simple reproduction of the problem. But it appears to happen when I use my cubedsphere package (see cubedsphere.readthedocs.io).

(exorad) schneider@tarf plot % python plot.py
time needed to build regridder: 10.789268970489502
Regridder will use conservative method
Traceback (most recent call last):
  File "/Volumes/EXTERN/Simulations/exorad/paper/paper_runs/plot/plot.py", line 21, in <module>
    ds_reg = regrid()
  File "/Users/schneider/codes/exo/cubedsphere/cubedsphere/regrid.py", line 293, in __call__
    ds[data] = self._regrid_wrapper(interp, **kwargs)
  File "/Users/schneider/codes/exo/cubedsphere/cubedsphere/regrid.py", line 364, in _regrid_wrapper
    data_out += self.regridder[i](ds_in.isel(**{c.FACEDIM: i}), **kwargs)
  File "/Users/schneider/anaconda3/envs/exorad/lib/python3.9/site-packages/xesmf/frontend.py", line 447, in __call__
    return self.regrid_dataarray(
  File "/Users/schneider/anaconda3/envs/exorad/lib/python3.9/site-packages/xesmf/frontend.py", line 522, in regrid_dataarray
    dr_out = xr.apply_ufunc(
  File "/Users/schneider/anaconda3/envs/exorad/lib/python3.9/site-packages/xarray/core/computation.py", line 1174, in apply_ufunc
    return apply_dataarray_vfunc(
  File "/Users/schneider/anaconda3/envs/exorad/lib/python3.9/site-packages/xarray/core/computation.py", line 293, in apply_dataarray_vfunc
    result_var = func(*data_vars)
  File "/Users/schneider/anaconda3/envs/exorad/lib/python3.9/site-packages/xarray/core/computation.py", line 742, in apply_variable_ufunc
    result_data = func(*input_data)
  File "/Users/schneider/anaconda3/envs/exorad/lib/python3.9/site-packages/xesmf/frontend.py", line 469, in _regrid_array
    outdata = apply_weights(weights, indata, shape_in, shape_out)
  File "/Users/schneider/anaconda3/envs/exorad/lib/python3.9/site-packages/xesmf/smm.py", line 150, in apply_weights
    outdata_flat = weights.dot(indata_flat.T).T
  File "/Users/schneider/anaconda3/envs/exorad/lib/python3.9/site-packages/sparse/_coo/core.py", line 901, in dot
    return dot(self, other)
  File "/Users/schneider/anaconda3/envs/exorad/lib/python3.9/site-packages/sparse/_common.py", line 326, in dot
    return tensordot(a, b, axes=(a_axis, b_axis))
  File "/Users/schneider/anaconda3/envs/exorad/lib/python3.9/site-packages/sparse/_common.py", line 198, in tensordot
    res = _dot(at, bt, return_type)
  File "/Users/schneider/anaconda3/envs/exorad/lib/python3.9/site-packages/sparse/_common.py", line 481, in _dot
    return _dot_coo_ndarray_type(a.dtype, b.dtype)(
  File "/Users/schneider/anaconda3/envs/exorad/lib/python3.9/site-packages/numba/core/dispatcher.py", line 420, in _compile_for_args
    error_rewrite(e, 'typing')
  File "/Users/schneider/anaconda3/envs/exorad/lib/python3.9/site-packages/numba/core/dispatcher.py", line 361, in error_rewrite
    raise e.with_traceback(None)
numba.core.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
non-precise type pyobject
During: typing of argument at /Users/schneider/anaconda3/envs/exorad/lib/python3.9/site-packages/sparse/_common.py (1052)

File "../../../../../../../Users/schneider/anaconda3/envs/exorad/lib/python3.9/site-packages/sparse/_common.py", line 1052:
    def _dot_coo_ndarray(coords1, data1, array2, out_shape):  # pragma: no cover
        <source elided>
        """
        out = np.zeros(out_shape, dtype=dtr)
        ^

This error may have been caused by the following argument(s):
- argument 2: Unsupported array dtype: >f8

[aulemahal]

Hi  @AaronDavidSchneider, thanks for bringing this to attention. I fear we (I) might have shot ourselves in the foot with the move to sparse in 0.6.1. I didn't realize it used numba by default and thus removes support for unconventional architectures and unconventional data structures (here "big-endian" float are causing the failure).

To confirm this hypothesis, could you look here : https://numba.readthedocs.io/en/latest/user/installing.html and tell us if you are running xESMF on a non-supported architecture?

I think the best quick workaround would be to re-cast your data to little-endian : ds_in.astype('<f8'). I can look if it makes sense to perform that cast within xESMF and fix the upstream non-implementation that way.

EDIT: As the error is not a complete failure from numba and as it appears cubedsphere and xgcm do not perform any dtype-altering operations, I am guessing your input data comes from a different computer than the one you are running the regridding on, is that correct?
In that case, byte-swapping (casting to '<f8' or 'float64') might be the easiest way out for now, but I can check for a similar solution within xESMF.

[AaronDavidSchneider]

Thanks a lot for the fast answer. I tried it on both Ubuntu and OS X with the same result. Yes indeed we use xesmf to regrid MITgcm data which uses big endian.