Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RuntimeError: NetCDF: Filter error: undefined filter encountered #1170

Closed
sebastic opened this issue Jun 25, 2022 · 16 comments
Closed

RuntimeError: NetCDF: Filter error: undefined filter encountered #1170

sebastic opened this issue Jun 25, 2022 · 16 comments

Comments

@sebastic
Copy link
Contributor

Some of the compression tests in 1.6.0 fail when building the Debian package (with NetCDF 4.9.0):

======================================================================
ERROR: runTest (tst_compression_bzip2.CompressionTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/netcdf4-python-1.6.0/test/tst_compression_bzip2.py", line 25, in setUp
    write_netcdf(self.filename2,complevel=4) # with compression
  File "/build/netcdf4-python-1.6.0/test/tst_compression_bzip2.py", line 14, in write_netcdf
    foo = nc.createVariable('data',\
  File "src/netCDF4/_netCDF4.pyx", line 2838, in netCDF4._netCDF4.Dataset.createVariable
  File "src/netCDF4/_netCDF4.pyx", line 4038, in netCDF4._netCDF4.Variable.__init__
  File "src/netCDF4/_netCDF4.pyx", line 1965, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: Filter error: undefined filter encountered

======================================================================
ERROR: runTest (tst_compression_blosc.CompressionTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/netcdf4-python-1.6.0/test/tst_compression_blosc.py", line 38, in setUp
    write_netcdf(self.filename,complevel=iblosc_complevel,blosc_shuffle=iblosc_shuffle)
  File "/build/netcdf4-python-1.6.0/test/tst_compression_blosc.py", line 17, in write_netcdf
    foo_lz = nc.createVariable('data_lz',\
  File "src/netCDF4/_netCDF4.pyx", line 2838, in netCDF4._netCDF4.Dataset.createVariable
  File "src/netCDF4/_netCDF4.pyx", line 4056, in netCDF4._netCDF4.Variable.__init__
  File "src/netCDF4/_netCDF4.pyx", line 1965, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: Filter error: undefined filter encountered

======================================================================
ERROR: runTest (tst_compression_zstd.CompressionTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/netcdf4-python-1.6.0/test/tst_compression_zstd.py", line 25, in setUp
    write_netcdf(self.filename2,complevel=4) # with compression
  File "/build/netcdf4-python-1.6.0/test/tst_compression_zstd.py", line 14, in write_netcdf
    foo = nc.createVariable('data',\
  File "src/netCDF4/_netCDF4.pyx", line 2838, in netCDF4._netCDF4.Dataset.createVariable
  File "src/netCDF4/_netCDF4.pyx", line 4026, in netCDF4._netCDF4.Variable.__init__
  File "src/netCDF4/_netCDF4.pyx", line 1965, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: Filter error: undefined filter encountered

----------------------------------------------------------------------
Ran 100 tests in 50.665s

FAILED (errors=3)

blosc, bz2, zstd, and zlib compression were enabled in netcdf (1:4.9.0-1):

# NetCDF C Configuration Summary
==============================

# General
-------
NetCDF Version:         4.9.0
Dispatch Version:       5
Install Prefix:         /usr
Plugin Install Prefix:  N.A.

# Compiling Options
-----------------
C Compiler:             /usr/bin/cc
CFLAGS:                 -g -O2 -ffile-prefix-map=/build/netcdf-l9kbMS/netcdf-4.9.0=. -fstack-protector-strong -Wformat -Werror=format-security -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2 -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 
CPPFLAGS:                
LDFLAGS:                -Wl,-z,relro -Wl,-z,now 
AM_CFLAGS:
AM_CPPFLAGS:
AM_LDFLAGS:
Shared Library:         yes
Static Library:         no
Extra libraries:        -lhdf5_hl -lhdf5 -lcrypto -lcurl -lpthread -lsz -lz -ldl -lm -lblosc -lzstd -lbz2 -lxml2
XML Parser:             libxml2

# Features
--------
NetCDF-2 API:           yes
HDF4 Support:           no
HDF5 Support:           yes
NetCDF-4 API:           yes
NC-4 Parallel Support:  no
PnetCDF Support:        no
DAP2 Support:           yes
DAP4 Support:           yes
Byte-Range Support:     no
Diskless Support:       yes
MMap Support:           yes
JNA Support:            no
CDF5 Support:           yes
ERANGE Fill Support:    yes
Relaxed Boundary Check: yes
Parallel Filters:       yes
NCZarr Support:         yes
Multi-Filter Support:   yes
Quantization:           yes
Logging:                no
SZIP Write Support:     yes
Standard Filters:       deflate szip blosc zstd bz2
ZSTD Support:           yes
Benchmarks:             no
@jswhit
Copy link
Collaborator

jswhit commented Jun 25, 2022

You have to set the HDF5_PLUGIN_PATH env var to point to where the plugins were installed by netcdf-c

@sebastic
Copy link
Contributor Author

The NetCDF build doesn't specify a HDF5 plugin path, how is one supposed to find out what it is?

@jswhit
Copy link
Collaborator

jswhit commented Jun 25, 2022 •
edited
Loading

From the 'Developer Install' section of the docs:

if the env var NETCDF_PLUGIN_DIR is set to point to the location of the netcdf-c compression plugins built by netcdf >= 4.9.0, they will be installed inside the package. In this case HDF5_PLUGIN_PATH will be set to the package installation path on import, so the extra compression algorithms available in netcdf-c >= 4.9.0 will automatically be available. Otherwise, the user will have to set HDF5_PLUGIN_PATH explicitly to have access to the extra compression plugins.

@jswhit
Copy link
Collaborator

jswhit commented Jun 25, 2022

When building netcdf-c (using autotools) set --with-plugin-dir=$HDF5_PLUGIN_PATH.
With cmake, if HDF5_PLUGIN_PATH is set, and -DPLUGIN_INSTALL_DIR=YES then the plugins will be installed.

@sebastic
Copy link
Contributor Author

NETCDF_PLUGIN_DIR isn't set either. The netcdf package is just built while having the compression libraries available and hence picked up by CMake at configure time:

-- Found Math library: /usr/lib/x86_64-linux-gnu/libm.so
-- Found Szip: headers at /usr/include, libraries at /usr/lib
--      library is /usr/lib/x86_64-linux-gnu/libsz.so
-- Found Bz2: headers at /usr/include, libraries at /usr/lib
--      library is /usr/lib/x86_64-linux-gnu/libbz2.so
-- Found Blosc: headers at /usr/include, libraries at /usr/lib
--      library is /usr/lib/x86_64-linux-gnu/libblosc.so
-- Found Zstd: headers at /usr/include, libraries at /usr/lib
--      library is /usr/lib/x86_64-linux-gnu/libzstd.so
>>> Standard Filter: szip
>>> Standard Filter: blosc
>>> Standard Filter: zstd
>>> Standard Filter: bz2

It does not result in any plugins being installed.

See also the settings:

Plugin Install Prefix:  N.A.

@sebastic
Copy link
Contributor Author

With cmake, if HDF5_PLUGIN_PATH is set, and -DPLUGIN_INSTALL_DIR=YES then the plugins will be installed.

CMake is wrong:

# If user wants, then install selected plugins (default on)
SET(PLUGIN_INSTALL_DIR "NO" CACHE STRING "Whether and where we should install plugins; defaults to yes")

Default is not YES/ON.

@sebastic
Copy link
Contributor Author

NetCDF build fails with -DPLUGIN_INSTALL_DIR:BOOL=ON:

Defaulting to -DPLUGIN_INSTALL_DIR=NOTFOUND

@jswhit
Copy link
Collaborator

jswhit commented Jun 25, 2022

setting HDF5_PLUGIN_PATH to where you want the plugins installed and then using -DPLUGIN_INSTALL_DIR=YES works for me. If you then set NETCDF_PLUGIN_DIR to HDF5_PLUGIN_PATH when building netcdf4-python, the plugins will be copied inside the netcdf4-python installation. Otherwise, you have to have HDF5_PLUGIN_PATH set for the python module to find them.

@jswhit
Copy link
Collaborator

jswhit commented Jun 25, 2022

@sebastic please consider creating a netcdf-c issue with the issues you are finding with plugin installation. There will soon be a 4.9.1 update, and it would be nice for these issues to be fixed in that release.

@sebastic sebastic mentioned this issue Jun 25, 2022
Closed
@jswhit
Copy link
Collaborator

jswhit commented Jun 25, 2022

thanks @sebastic. If you set HDF5_PLUGIN_PATH and set -DPLUGIN_INSTALL_DIR=YES do the netcfd4-python tests pass for you?

@sebastic
Copy link
Contributor Author

This works:

--- a/debian/rules
+++ b/debian/rules
@@ -13,6 +13,10 @@ DEB_HOST_MULTIARCH ?= $(shell dpkg-architecture -qDEB_HOST_MULTIARCH)
 HDF5_INCLUDE_PATH=/usr/include/hdf5/serial
 HDF5_LIBRARY_PATH=/usr/lib/$(DEB_HOST_MULTIARCH)/hdf5/serial
 
+# Use H5_DEFAULT_PLUGINDIR from /usr/include/hdf5/serial/H5pubconf.h
+# https://github.com/Unidata/netcdf-c/issues/2428
+export HDF5_PLUGIN_PATH=$(HDF5_LIBRARY_PATH)/plugins
+
 include /usr/share/dpkg/pkg-info.mk
 
 UPSTREAM_VERSION = $(shell echo $(DEB_VERSION_UPSTREAM) | sed -e 's/\+.*//')
@@ -30,7 +34,8 @@ override_dh_auto_configure:
                -DENABLE_TESTS:BOOL=ON \
                -DENABLE_DOXYGEN:BOOL=ON \
                -DHDF5_INCLUDE_PATH=$(HDF5_INCLUDE_PATH) \
-               -DHDF5_LIBRARY_PATH=$(HDF5_LIBRARY_PATH)
+               -DHDF5_LIBRARY_PATH=$(HDF5_LIBRARY_PATH) \
+               -DPLUGIN_INSTALL_DIR=YES
 
 override_dh_auto_test:
        ARGS="--output-on-failure" dh_auto_test --max-parallel=1 || echo "Ignoring test failures"

But is not sufficient, the hdf5 plugins are not included in any package. The filenames look questionable:

-- Installing: /build/netcdf-4.9.0/debian/tmp/usr/lib/x86_64-linux-gnu/hdf5/serial/plugins
-- Installing: /build/netcdf-4.9.0/debian/tmp/usr/lib/x86_64-linux-gnu/hdf5/serial/plugins/lib__nch5zstd.so
-- Installing: /build/netcdf-4.9.0/debian/tmp/usr/lib/x86_64-linux-gnu/hdf5/serial/plugins/lib__nch5blosc.so
-- Installing: /build/netcdf-4.9.0/debian/tmp/usr/lib/x86_64-linux-gnu/hdf5/serial/plugins/lib__nch5fletcher32.so
-- Installing: /build/netcdf-4.9.0/debian/tmp/usr/lib/x86_64-linux-gnu/hdf5/serial/plugins/lib__nch5shuffle.so
-- Installing: /build/netcdf-4.9.0/debian/tmp/usr/lib/x86_64-linux-gnu/hdf5/serial/plugins/lib__nch5deflate.so
-- Installing: /build/netcdf-4.9.0/debian/tmp/usr/lib/x86_64-linux-gnu/hdf5/serial/plugins/lib__nczhdf5filters.so
-- Installing: /build/netcdf-4.9.0/debian/tmp/usr/lib/x86_64-linux-gnu/hdf5/serial/plugins/lib__nczstdfilters.so
-- Installing: /build/netcdf-4.9.0/debian/tmp/usr/lib/x86_64-linux-gnu/hdf5/serial/plugins/lib__nch5szip.so

It's not clear what their relation is to libnetcdf or the executables, despite reading https://github.com/Unidata/netcdf-c/blob/main/docs/filters.md.

Based on the filenames the hdf5 plugins cannot be included in the libnetcdf19 binary package as there is no SOVERSION, so a new binary package (e.g. libnetcdf-hdf5-plugins) will be required for these files and have that be a Recommended dependency of libnetcdf19 as there doesn't seem to be a hard dependency. Or maybe something else is more appropriate.

Not including the plugins in the Debian package makes my life easier, I only maintain the NetCDF package because its a dependency of GDAL, I don't actually use them. So I think I'm going to update the netcdf package to explicitly not build the plugins and remove the related compression library dependencies.

@sebastic
Copy link
Contributor Author

Disabling plugins with -DENABLE_PLUGINS:BOOL=OFF only helps a little:

======================================================================
ERROR: runTest (tst_compression_bzip2.CompressionTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/netcdf4-python-1.6.0/test/tst_compression_bzip2.py", line 25, in setUp
    write_netcdf(self.filename2,complevel=4) # with compression
  File "/build/netcdf4-python-1.6.0/test/tst_compression_bzip2.py", line 14, in write_netcdf
    foo = nc.createVariable('data',\
  File "src/netCDF4/_netCDF4.pyx", line 2838, in netCDF4._netCDF4.Dataset.createVariable
  File "src/netCDF4/_netCDF4.pyx", line 4038, in netCDF4._netCDF4.Variable.__init__
  File "src/netCDF4/_netCDF4.pyx", line 1965, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: Filter error: undefined filter encountered

======================================================================
FAIL: runTest (tst_compression_blosc.CompressionTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/netcdf4-python-1.6.0/test/tst_compression_blosc.py", line 53, in runTest
    assert f.variables['data_lz'].filters() == dtest
AssertionError

======================================================================
FAIL: runTest (tst_compression_zstd.CompressionTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/netcdf4-python-1.6.0/test/tst_compression_zstd.py", line 46, in runTest
    assert f.variables['data'].filters() ==\
AssertionError

----------------------------------------------------------------------
Ran 100 tests in 52.982s

FAILED (failures=2, errors=1)

@jswhit
Copy link
Collaborator

jswhit commented Jun 25, 2022

You can disable the netcdf4-python plugin tests by setting the env var NO_PLUGINS.

@jswhit
Copy link
Collaborator

jswhit commented Jun 25, 2022

Those are the correct filenames (lib__nc*so) for the netcdf-c plugins. They are dynamically loaded at runtime. You are probably right that they should be in a separate package.

@sebastic
Copy link
Contributor Author

You can disable the netcdf4-python plugin tests by setting the env var NO_PLUGINS.

Thanks works, thanks.

@opoplawski
Copy link
Contributor

@sebastic - you might want to checkout Unidata/netcdf-c#2431 - although we are still using autotools on Fedora.

@rstoneback rstoneback mentioned this issue Jun 28, 2022
Merged
18 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jswhit @opoplawski @sebastic