Update conda recipes for Enhanced Compatibility effort
#9456
Conversation
ajschmidt8
added
improvement
Codecov Report
@@ Coverage Diff @@
## branch-21.12 #9456 +/- ##
================================================
+ Coverage 10.79% 10.83% +0.04%
================================================
Files 116 117 +1
Lines 18869 19442 +573
================================================
+ Hits 2036 2106 +70
- Misses 16833 17336 +503
Continue to review full report at Codecov.
|
|
rerun tests |
It would be good to get confirmation that this is safe to do while including PTX in the libcudf binary. I believe that if we compile with CUDA 11.4 it embeds PTX with a version that matches 11.4 and if you that PTX needs to get loaded or JITed on say an 11.0 machine the driver will error. I believe we decided to do the reverse and only compile with CUDA 11.0 in conda-forge for this same reason. cc @jakirkham |
|
In conda-forge we had to start with 11.2, but yes this was one issue There are a few conversations happening internally about how exactly CUDA Enhanced Compatibility (CEC) can be used |
|
rerun tests |
From the C++ side we either emit SASS for all the necessary architectures so they are not concerned about PTX issues, or we have projects like jitify convert to SASS at runtime and therefore don't run into this problem. In some cases, when using a new runtime with an older driver, Numba will currently emit PTX not understood by the driver. We are working with the Numba team to provide a fix for this before the 21.12 release finalizes. In the interim, users of nightlies will be need to run on a driver new enough to run the code emitted by their runtime or temporarily downgrade to an older runtime. |
JohnZed
dismissed
kkraus14’s stale review
Sorry to dismiss, but we need this PR to get through in order to proceed with the 11.5 rollout, so we are going to merge it in. Please see Rob's comment above for more info. Numba changes will take a few more days to come out.
|
@gpucibot merge |
|
No worries on dismissing the review. I wonder if at a certain point we should just stop shipping PTX in the libcudf binaries because one of three things will happen if the driver needs to interact with the libcudf PTX:
All of these feel like a distinctly worse user experience than just not shipping the PTX and having the driver error from failing to find SASS for the current device architecture. |
|
@kkraus14 #2 is an interesting point in particular.... That's basically the specific usecase that PTX is meant to support, but I agree that the JIT times can be very surprising. I think this is the norm for most shipping CUDA software, though, since when new architectures come out, it's also very unpleasant if there is no ability for them to use popular OSS CUDA code at all. I think the ideal would be if we could detect the need to JIT for a new version and output a clear message saying "first time JIT, this will take a while." I don't know of a way to intercept that JIT and issue a message though. |
|
Asked around, and it sounds like there is not currently an API to intercept when JIT'ing happens and share a message, unfortunately. |
The `dask-cudf` recipe was changed after rapidsai#9456 was opened, so the Enhanced Compatibility changes never made it into this recipe. This PR fixes that.
Similar to rapidsai/cudf#9456. This PR updates the `conda` recipe build strings and `cudatoolkit` version specifications as part of the Enhanced Compatibility efforts. The build strings have been updated to only include the major CUDA version (i.e. `librmm-21.12.00a-cuda11_gc781527_12.tar.bz2`) and the `cudatoolkit` version specifications will now be formatted like `cudatoolkit >=x,<y.0a0` (i.e. `cudatoolkit >=11,<12.0a0`). Moving forward, we'll build the packages with a single CUDA version (i.e. `11.5`) and test them in environments with different CUDA versions (i.e. `11.0`, `11.2`, `11.4`, `11.5` etc.).
This PR updates the
condarecipe build strings andcudatoolkitversion specifications as part of the Enhanced Compatibility efforts.The build strings have been updated to only include the major CUDA version (i.e.
librmm-21.12.00a-cuda11_gc781527_12.tar.bz2) and thecudatoolkitversion specifications will now be formatted likecudatoolkit >=x,<y.0a0(i.e.cudatoolkit >=11,<12.0a0).Moving forward, we'll build the packages with a single CUDA version (i.e.
11.4) and test them in environments with different CUDA versions (i.e.11.0,11.2,11.4, etc.).