-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Build Python packages using the limited API #42
Comments
It is worth noting that the Python Buffer Protocol C API landed in Python 3.11 (additional ref). So think that is a minimum for us Also find this listing of functions in the Limited and Stable API quite helpful |
Yes. I have been able to build most of RAPIDS using Cython's limited API supported (along with some additional changes I have locally) in Python 3.11. Python 3.11 is definitely a must. But as I said above in the "intermediate vs long-term" bullet, we could still benefit before dropping Python<3.11 support by building one wheel for each older Python version and then build an abi3 wheel to be used for Python 3.11+. |
I've made PRs ro rmm, raft, and cuml that address the issues in those repos. I've also taken steps to remove ucxx's usage of the the numpy C API (#41), which in turn removes one of its primary incompatibilities. The last major issue in RAPIDS code that I see is the usage of the array module in the Array class that is vendored by both kvikio and ucxx (and ucx-py). If that can be removed, then I think we'll be in good shape on the RAPIDS end, and we'll just be waiting on support for this feature in Cython itself. @jakirkham expressed interest in helping out with that in the process of making that Array class more broadly usable. |
A small warning here: There's definitely places where Cython is substituting private C API for private Python API, so future compatibility definitely isn't guaranteed (it'll just be a runtime failure rather than a compile-time failure). We'll see how that evolves - I hope to be able to make some of these warnings rather than failures (since it's largely just non-essential introspection support). We're also having to build a few more runtime version-checks into our code. Which is obviously a little risky because although you're compiling the same thing, you're taking different paths on different Python versions. So the upshot is that your testing matrix probably doesn't reduce to a single version. (From Cython's point of view the testing matrix probably expands, because we really should be testing combinations like |
Thanks for chiming in here @da-woods! I appreciate your comments. I agree that there is more complexity around testing here than simply a set and forget single version. At present, RAPIDS typically supports 2 or 3 Python versions at a time. We tend to lag a bit behind NEP 29/SPEC 0 timelines, so we support older versions a bit longer at the expense of not supporting new ones until they've been out for a bit. A significant part of the resource constraint equation for us is certainly on the testing side since running our full test suites on multiple Python versions adds up quickly. The way that I had envisioned this working, if we did move forward, would be that we built on the oldest supported Python (e.g. |
This PR removes usage of the only method in raft's Cython that is not part of the Python limited API. Contributes to rapidsai/build-planning#42 Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Dante Gama Dessavre (https://github.com/dantegd) URL: #5871
This PR removes usage of the only method in rmm's Cython that is not part of the Python limited API. Contributes to rapidsai/build-planning#42 Authors: - Vyas Ramasubramani (https://github.com/vyasr) - https://github.com/jakirkham Approvers: - https://github.com/jakirkham URL: #1545
This PR removes usage of the only method in raft's Cython that is not part of the Python limited API. Contributes to rapidsai/build-planning#42 Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Dante Gama Dessavre (https://github.com/dantegd) URL: #2282
This PR removes usage of the only method in raft's Cython that is not part of the Python limited API. Contributes to rapidsai/build-planning#42 Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Dante Gama Dessavre (https://github.com/dantegd) URL: rapidsai#2282
With the latest versions of branch-24.10, which contain a number of changes I made over the past few months for limited API compatibility along with the removal of pyarrow and numpy as build requirements in cudf and cuspatial, most of RAPIDS now builds with the limited API flag on. I have run some smoke tests and things generally work OK, but I haven't done anything extensive. ucxx and kvikio remain outstanding since we need to rewrite the Array class to not use the Python array's C API since that does not support the limited API. The latest tests can be seen in rapidsai/devcontainers#278. |
I don't know if it's any help, but the quickest non-
It isn't as good, but it's surprisingly close given how much it actually does. "Only" 70% slower. You probably have to replace
with |
Thanks for the tip David! That could be helpful, but I'll have to look at the Array class more closely to be sure. I suspect that there are larger refactorings of our code base that could be done to make this unnecessary. |
Python has a limited API that is guaranteed to be stable across minor releases. Any code using the Python C API that limits itself to using code in the limited API is guaranteed to also compile on future minor versions of Python within the same major family. More importantly, all symbols in the current (and some historical) version of the limited API are part of Python's stable ABI, which also does not change between Python minor versions and allows extensions compiled against one Python version to continue working on future versions of Python.
Currently RAPIDS builds a single wheel per Python version. If we were to compile using the Python stable ABI, we would be able to instead build a single wheel that works for all Python versions that we support. There would be a number of benefits here:
Here are the tasks (some ours, some external) that need to be accomplished to make this possible:
At this stage, it is not yet clear whether the tradeoffs required will be worthwhile, or at what point the ecosystem's support for the limited API will be reliable enough for us to use in production. However, it shouldn't be too much work to get us to the point of at least being able to experiment with limited API builds, so we can start answering questions around performance and complexity fairly soon. I expect that we can pretty easily remove explicit reliance on any APIs that are not part of the stable ABI, at which point this really becomes a question of the level of support our binding tools provide and if/when we're comfortable with those.
The text was updated successfully, but these errors were encountered: