libcublas on aarch64 requires glibc 2.27

[leofang]

(test) [conda@8e09d4a629ed ~]$ ldd -v /opt/conda/envs/test/lib/python3.10/site-packages/cupy_backends/cuda/libs/../../../../../libcublasLt.so.12
/opt/conda/envs/test/lib/python3.10/site-packages/cupy_backends/cuda/libs/../../../../../libcublasLt.so.12: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /opt/conda/envs/test/lib/python3.10/site-packages/cupy_backends/cuda/libs/../../../../../libcublasLt.so.12)
	linux-vdso.so.1 =>  (0x0000007fa8019000)
	librt.so.1 => /lib64/librt.so.1 (0x0000007f8a7a6000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x0000007f8a771000)
	libdl.so.2 => /lib64/libdl.so.2 (0x0000007f8a750000)
	libm.so.6 => /lib64/libm.so.6 (0x0000007f8a69f000)
	libgcc_s.so.1 => /opt/conda/envs/test/lib/python3.10/site-packages/cupy_backends/cuda/libs/../../../../../libgcc_s.so.1 (0x0000007f8a66e000)
	libc.so.6 => /lib64/libc.so.6 (0x0000007f8a4e8000)
	/lib/ld-linux-aarch64.so.1 (0x0000007fa7feb000)

	Version information:
	/opt/conda/envs/test/lib/python3.10/site-packages/cupy_backends/cuda/libs/../../../../../libcublasLt.so.12:
		librt.so.1 (GLIBC_2.17) => /lib64/librt.so.1
		libm.so.6 (GLIBC_2.27) => not found
		libdl.so.2 (GLIBC_2.17) => /lib64/libdl.so.2
		libgcc_s.so.1 (GCC_4.2.0) => /opt/conda/envs/test/lib/python3.10/site-packages/cupy_backends/cuda/libs/../../../../../libgcc_s.so.1
		libgcc_s.so.1 (GCC_3.3) => /opt/conda/envs/test/lib/python3.10/site-packages/cupy_backends/cuda/libs/../../../../../libgcc_s.so.1
		libgcc_s.so.1 (GCC_4.5.0) => /opt/conda/envs/test/lib/python3.10/site-packages/cupy_backends/cuda/libs/../../../../../libgcc_s.so.1
		libgcc_s.so.1 (GCC_3.0) => /opt/conda/envs/test/lib/python3.10/site-packages/cupy_backends/cuda/libs/../../../../../libgcc_s.so.1
		libpthread.so.0 (GLIBC_2.17) => /lib64/libpthread.so.0
		libc.so.6 (GLIBC_2.17) => /lib64/libc.so.6
	/lib64/librt.so.1:
		libpthread.so.0 (GLIBC_PRIVATE) => /lib64/libpthread.so.0
		libpthread.so.0 (GLIBC_2.17) => /lib64/libpthread.so.0
		libc.so.6 (GLIBC_PRIVATE) => /lib64/libc.so.6
		libc.so.6 (GLIBC_2.17) => /lib64/libc.so.6
	/lib64/libpthread.so.0:
		ld-linux-aarch64.so.1 (GLIBC_2.17) => /lib/ld-linux-aarch64.so.1
		ld-linux-aarch64.so.1 (GLIBC_PRIVATE) => /lib/ld-linux-aarch64.so.1
		libc.so.6 (GLIBC_PRIVATE) => /lib64/libc.so.6
		libc.so.6 (GLIBC_2.17) => /lib64/libc.so.6
	/lib64/libdl.so.2:
		ld-linux-aarch64.so.1 (GLIBC_PRIVATE) => /lib/ld-linux-aarch64.so.1
		libc.so.6 (GLIBC_PRIVATE) => /lib64/libc.so.6
		libc.so.6 (GLIBC_2.17) => /lib64/libc.so.6
	/lib64/libm.so.6:
		libc.so.6 (GLIBC_PRIVATE) => /lib64/libc.so.6
		libc.so.6 (GLIBC_2.17) => /lib64/libc.so.6
	/opt/conda/envs/test/lib/python3.10/site-packages/cupy_backends/cuda/libs/../../../../../libgcc_s.so.1:
		libc.so.6 (GLIBC_2.17) => /lib64/libc.so.6
	/lib64/libc.so.6:
		ld-linux-aarch64.so.1 (GLIBC_2.17) => /lib/ld-linux-aarch64.so.1
		ld-linux-aarch64.so.1 (GLIBC_PRIVATE) => /lib/ld-linux-aarch64.so.1

Note the two lines

/opt/conda/envs/test/lib/python3.10/site-packages/cupy_backends/cuda/libs/../../../../../libcublasLt.so.12: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /opt/conda/envs/test/lib/python3.10/site-packages/cupy_backends/cuda/libs/../../../../../libcublasLt.so.12)

and

		libm.so.6 (GLIBC_2.27) => not found

Essentially, we are once again hitting the same issue as in cuSPARSELt v0.4.0 (conda-forge/cusparselt-feedstock#23 (comment)) and cuDNN v8.8.0 (conda-forge/cudnn-feedstock#58), but this went totally unnoticed until now when I work on conda-forge/cupy-feedstock#199, because we haven't added the dlopen test to all new CUDA packages.

cc: @jakirkham

[leofang]

$ conda list
# packages in environment at /opt/conda/envs/test:
#
# Name                    Version                   Build  Channel
_openmp_mutex             4.5                       2_gnu    conda-forge
bzip2                     1.0.8                hf897c2e_4    conda-forge
ca-certificates           2023.5.7             hcefe29a_0    conda-forge
cuda-cudart               12.0.107             hac28a21_1    conda-forge
cuda-nvrtc                12.0.76              hac28a21_1    conda-forge
cuda-version              12.0                 hffde075_2    conda-forge
cupy                      12.0.0          py310h18013f7_2    <unknown>
fastrlock                 0.8             py310h16c9dca_3    conda-forge
ld_impl_linux-aarch64     2.40                 h2d8c526_0    conda-forge
libblas                   3.9.0           16_linuxaarch64_openblas    conda-forge
libcblas                  3.9.0           16_linuxaarch64_openblas    conda-forge
libcublas                 12.0.1.189           ha21bd82_2    conda-forge
libcufft                  11.0.0.21            ha21bd82_1    conda-forge
libcurand                 10.3.1.50            ha21bd82_0    conda-forge
libcusolver               11.4.2.57            ha21bd82_1    conda-forge
libffi                    3.4.2                h3557bc0_5    conda-forge
libgcc-ng                 12.2.0              h607ecd0_19    conda-forge
libgfortran-ng            12.2.0              he9431aa_19    conda-forge
libgfortran5              12.2.0              hf695500_19    conda-forge
libgomp                   12.2.0              h607ecd0_19    conda-forge
liblapack                 3.9.0           16_linuxaarch64_openblas    conda-forge
libnsl                    2.0.0                hf897c2e_0    conda-forge
libnvjitlink              12.0.76              ha21bd82_1    conda-forge
libopenblas               0.3.21          pthreads_h6cb6f83_3    conda-forge
libsqlite                 3.41.2               h194ca79_1    conda-forge
libstdcxx-ng              12.2.0              hc13a102_19    conda-forge
libuuid                   2.38.1               hb4cce97_0    conda-forge
libzlib                   1.2.13               h4e544f5_4    conda-forge
ncurses                   6.3                  headf329_1    conda-forge
numpy                     1.24.3          py310h7f380f4_0    conda-forge
openssl                   3.1.0                h31becfc_3    conda-forge
pip                       23.1.2             pyhd8ed1ab_0    conda-forge
python                    3.10.11         ha43d526_0_cpython    conda-forge
python_abi                3.10                    3_cp310    conda-forge
readline                  8.2                  h8fc344f_1    conda-forge
setuptools                67.7.2             pyhd8ed1ab_0    conda-forge
tk                        8.6.12               hd8af866_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
wheel                     0.40.0             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h9cdd2b7_0    conda-forge

[jakirkham]

Am curious why we didn't see an issue in the ARM build here? Would think this would have detected a new version of GLIBC is needed and not provided for (as seen elsewhere)

[bdice]

@jakirkham I think only the libraries in $PREFIX/lib/ are checked for symbol compatibility. I believe the files in $PREFIX/targets/sbsa-linux/lib/ are "just files" as far as conda-build cares.

[leofang]

Am curious why we didn't see an issue in the ARM build here? Would think this would have detected a new version of GLIBC is needed and not provided for (as seen elsewhere)

I think what happens is that conda-build correctly found the dependency on libm.so

   INFO (libcublas,targets/sbsa-linux/lib/libcublasLt.so.12.0.1.189): Needed DSO aarch64-conda-linux-gnu/sysroot/lib64/libm.so.6 found in CDT/compiler package conda-forge::sysroot_linux-aarch64-2.17-h43d7e78_13

but it didn't check if the found DSO also satisfies the glibc requirement (i.e. containing all the versioned symbols needed by libcublasLt.so).