xr.open_dataset(remote_url) file not found

[aaronspring]

What happened:

I tried to open a remote url and got OSError, but !wget url works

What you expected to happen:

open the remote netcdf file

Minimal Complete Verifiable Example:

from netCDF4 import Dataset

import netCDF4
netCDF4.__version__

import xarray as xr
xr.__version__

url='https://crudata.uea.ac.uk/cru/data/temperature/HadCRUT.4.6.0.0.median.nc'
# working_url='https://thredds.ucar.edu/thredds/dodsC/grib/NCEP/GFS/Global_0p5deg/GFS_Global_0p5deg_20200923_0000.grib2'
xr.open_dataset(url)
...
netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Dataset.__init__()

netCDF4/_netCDF4.pyx in netCDF4._netCDF4._ensure_nc_success()

OSError: [Errno -90] NetCDF: file not found: b'https://crudata.uea.ac.uk/cru/data/temperature/HadCRUT.4.6.0.0.median.nc'

# seems to be netcdf4 upstream issue
Dataset(url)
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-14-265839034cee> in <module>
----> 1 Dataset(url)

netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Dataset.__init__()

netCDF4/_netCDF4.pyx in netCDF4._netCDF4._ensure_nc_success()

OSError: [Errno -90] NetCDF: file not found: b'https://crudata.uea.ac.uk/cru/data/temperature/HadCRUT.4.6.0.0.median.nc'

Anything else we need to know?:

Environment:

Output of xr.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.7.6 | packaged by conda-forge | (default, Jan 7 2020, 22:33:48)
[GCC 7.3.0]
python-bits: 64
OS: Linux
OS-release: 2.6.32-754.29.2.el6.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: en_US.UTF-8
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
libhdf5: 1.10.5
libnetcdf: 4.6.2

xarray: 0.16.1
pandas: 1.1.2
numpy: 1.19.1
scipy: 1.5.2
netCDF4: 1.5.1.2
pydap: installed
h5netcdf: 0.8.0
h5py: 2.10.0
Nio: 1.5.5
zarr: 2.4.0
cftime: 1.2.1
nc_time_axis: 1.2.0
PseudoNetCDF: None
rasterio: 1.1.0
cfgrib: 0.9.7.6
iris: 2.2.0
bottleneck: 1.3.1
dask: 2.15.0
distributed: 2.20.0
matplotlib: 3.1.2
cartopy: 0.17.0
seaborn: 0.10.1
numbagg: None
pint: 0.11
setuptools: 47.1.1.post20200529
pip: 20.2.3
conda: None
pytest: 5.3.5
IPython: 7.15.0
sphinx: None

[aaronspring]

cross posted: Unidata/netcdf4-python#1043