Reinstate aarch64+CUDA; add ppc64le+CUDA

[h-vetinari]

A left-over from #875, this is blocked on the slow-moving¹ conda-forge/conda-forge-ci-setup-feedstock#210 (more discussion on the why cross-compilation is a de facto necessity here).

Closes #859
Closes #659

Footnotes

1. This might be further impacted by nvidia deps moving to conda-forge ↩

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

[h-vetinari]

After conda-forge/conda-forge-ci-setup-feedstock#210 got solved, this is now blocked on conda-forge/nvcc-feedstock#95

[isuruf]

@conda-forge-admin, rerender

[h-vetinari]

I'm rebasing this, but need to wait til conda-forge/conda-forge-pinning-feedstock#3624 makes it through the CDN

[conda-forge-webservices]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

[h-vetinari]

Ah, I hadn't seen conda-forge/nvcc-feedstock#96, that's amazing, thanks so much! 🤩

Seems this got through the resolver and now started building! 🥳

Failure is unrelated should be resolvable:

$PREFIX/bin/grpc_cpp_plugin: program not found or is not executable

[h-vetinari]

It seems this is looking for the wrong gnu stubs?

# include <gnu/stubs-32.h>
          ^~~~~~~~~~~~~~~~
1 error generated.

AFAICT they are named differently in our linux-sysroot? Probably needs a patch?

[h-vetinari]

Thanks a lot for the help here Isuru! Checking out where the test failures are coming from in #1015. Sidenote, I saw that arrow wants to find libevent, got an open PR to make that happen (I had included it in host here before already, but rebased that out after it wasn't found even then - it needs the cmake config files to be found without further intervention)

[h-vetinari]

An interesting error while testing pyarrow on the new CUDA builds...

aarch:

export SRC_DIR=/home/conda/feedstock_root/build_artifacts/apache-arrow_1681520079529/test_tmp
$SRC_DIR/conda_test_runner.sh: $PREFIX/bin/python: /lib/ld-linux-aarch64.so.1: bad ELF interpreter: No such file or directory
Tests failed for pyarrow-11.0.0-py38h220f70f_14_cuda.conda - moving package to /home/conda/feedstock_root/build_artifacts/broken

ppc:

export SRC_DIR=/home/conda/feedstock_root/build_artifacts/apache-arrow_1681520086495/test_tmp
$SRC_DIR/conda_test_runner.sh: $PREFIX/bin/python: /lib64/ld64.so.2: bad ELF interpreter: No such file or directory
Tests failed for pyarrow-11.0.0-py311h6081924_14_cuda.conda - moving package to /home/conda/feedstock_root/build_artifacts/broken

[h-vetinari]

An interesting error while testing pyarrow on the new CUDA builds...

The reason this is weird to me is that the error appears on bin/python invocation, and doesn't seem to have anything to do with arrow. It sounds sysroot-related, but I don't know these bits well enough to understand what's going on yet.

[isuruf]

Should be fixed by conda-forge/docker-images#233

[h-vetinari]

Should be fixed by conda-forge/docker-images#233

Ah, right, no QEMU in the new cross-images. Thanks a lot! :)

[h-vetinari]

Something with $CUDA_HOME (or so?) seems to not yet be set up 100% correctly:

import: 'pyarrow.cuda'
Traceback (most recent call last):
  File "/home/conda/feedstock_root/build_artifacts/apache-arrow_1681808893494/test_tmp/run_test.py", line 29, in <module>
    import pyarrow.cuda
  File "$PREFIX/lib/python3.9/site-packages/pyarrow/cuda.py", line 21, in <module>
    from pyarrow._cuda import (Context, IpcMemHandle, CudaBuffer,
ImportError: libcuda.so.1: cannot open shared object file: No such file or directory

[h-vetinari]

Should be fixed by conda-forge/conda-forge-ci-setup-feedstock#234

[h-vetinari]

Cross-compiling CUDA ✅🥳

This PR is now ready for primetime, and I couldn't be more thrilled that we finally got this working. 🤩

Context

Before arrow 10.x, each python version had a separate CI job, which was already regularly running into the 6h time limit when running in emulation (I was constantly restarting these jobs after merging to the 9.0.x and 8.0.x branches, taking up to 13(!) runs to pass). This was also why we never added support for CUDA-on-ppc, because it would have made that restarting pain many times worse still.

With arrow 10, libarrow finally became independent of the python version; allowing us to build it only once, and collapse the huge build matrix¹ to do all python versions in one job per arch. That was a big benefit, but made it impossible to compile CUDA support in emulation, so we ended up dropping support for CUDA-on-aarch at the time, pending cross-compilation capabilities.

IOW, we can now add back CUDA-on-aarch, and add support for CUDA-on-ppc!

I've also prepared backports of this PR for all our maintenance branches (10.0.x, 9.0.x, 8.0.x). Since we had dropped CUDA-on-aarch for arrow 10, backporting there should be obvious, and since cross-compilation hugely cuts down our CI times (and gets rid of the necessity to restart!) on the 9.0.x and 8.0.x branches, it should be a no-brainer there as well². In order to get green CI, those PRs also fix #1016 on all branches.

PTAL @conda-forge/arrow-cpp
CC @conda-forge/core (just for the good news 😊)

PS: It took quite a bit of effort to get this working (wouldn't have happened without @isuruf; a huge thanks! 🙏), here's some indication of what was necessary (no claim of exhaustiveness; just what I'm aware of)...

• expand cuda_112_ppc64le_aarch64 migrator for cross-compilation conda-forge-pinning-feedstock#3624
• cross compile cuda support (cnt'd) conda-forge-ci-setup-feedstock#210
• give sudo permission to copy the cuda headers and libraries for cross compilation docker-images#229
• Support cross compilation nvcc-feedstock#96
• Install qemu in linux-anvil-cuda docker-images#233
• Install libcuda.so.1 to QEMU sysroot conda-forge-ci-setup-feedstock#234

Footnotes

1. 72 jobs at a time when we still had to build for both OpenSSL 1.1.1 & 3.0; would have been 80 had we included CUDA-on-ppc. ↩

2. An argument could be made for leaving the "feature" of CUDA-on-ppc out of the maintenance branches, but since it costs ~nothing and gets us consistency across all branches, I think we should do it. ↩

[h-vetinari]

Seems people are less excited about this than I am? 🤔 😅
(but then, I guess it's me benefitting the most from not having to restart CI all the time)

Unless there are comments to the contrary, I'll merge this (and the maintenance PRs) in 24h.