Add support for the limited C API to Argument Clinic

[vstinner]

Feature or enhancement

Has this already been discussed elsewhere?

I have already discussed this feature proposal on Discourse

Links to previous discussion of this feature:

• [C API] Convert a few stdlib extensions to the limited C API (PEP 384) #85283
• gh-85283: Extending Argument Clinic to support to use the limited C API. #26080

Proposal:

I propose adding support for the limited C API to Argument Clinic. Since Argument Clinic supports a wide diversity of parameter types and object types, I propose to add the support incrementally (step by step).

The final goal is to restrict some stdlib C extensions to the limited C API.

Linked PRs

• gh-108494: Argument Clinic partial supports of Limited C API #108495
• gh-108494: AC supports pos-only args in limited C API #108498
• gh-108494: AC: change parse_file() API #108504
• gh-108494: Fix AC limited C API for kwargs #108516
• gh-108494: Argument Clinic: fix support of Limited C API #108536
• gh-108494: Argument Clinic: fix option group for Limited C API #108574
• gh-108494: Argument clinic: Improve the parse_file() API #108575
• gh-108494: Argument Clinic: Document how to generate code that uses the limited C API #108584
• gh-108494: Build _testclinic_limited on Windows #108589
• gh-108494: Don't build _testclinic_limited with TraceRefs #108608
• gh-108494: Argument Clinic: inline parsing code for positional-only parameters in the limited C API #108622
• gh-108494: Document how to add a project in PCbuild/readme.txt #110077
• [3.12] gh-108494: Document how to add a project in PCbuild/readme.txt (GH-110077) #110230
• [3.11] gh-108494: Document how to add a project in PCbuild/readme.txt (#110077) #110232
• gh-108494: Fix Argument Clinic LIMITED_CAPI_REGEX #116610

[vstinner]

My first change adds this code to build _testclinic_limited.c with the limited C API:

    # XXX Temporary solution
    if os.path.basename(filename) == '_testclinic_limited.c':
        print(f"{filename} uses limited C API")
        limited_capi = True

Maybe we could add a marker using Argument Clinic syntax to announce "hey, I want the limited C API"!

Or AC could parse the C code and look for #define Py_LIMITED_API?

[vstinner]

My change added support for limited C API version 3.13. If we consider supporting older Python versions, this issue should be carefully reviewed: #71686 "PY_SSIZE_T_CLEAN conflicts with Py_LIMITED_API". Maybe it has been fixed in the meanwhile, I'm not sure. The goal here would be to be able build a C extension with Python 3.13 which works on Python 3.2 to 3.12.

[vstinner]

Something else to consider, @shihai1991 wrote:

METH_FASTCALL is the part of the stable ABI in #23009.

I don't in which Python version it was usable as part of the stable ABI. My current implementation is limited to METH_VARARGS and METH_VARARGS | METH_KEYWORDS to focus on backward compatibility first.

[vstinner]

Well, see issue #85283 for the interesting discussion in 2020-2021.

[vstinner]

Or AC could parse the C code and look for #define Py_LIMITED_API?

I implemented that in my second PR, see LIMITED_CAPI_REGEX.

[vstinner]

When I try to convert the _bisect module to the limited C API, Argument Clinic generates code using _Py_convert_optional_to_ssize_t() which is not part of the limited C API for Py_ssize_t parameters (ex: _bisect.bisect_right()) :-(

[vstinner]

Thanks to @serhiy-storchaka, Argument Clinic now supports all formats for the limited C API. I'm sure that we will find code which doesn't work and room for improvement. It can be done later. I close the issue. Thanks to everybody who helped me to make it possible :-)

[vstinner]

Follow-up: PR #110964 "[C API] Move undocumented private _PyArg C API to pycore_modsupport.h internal C API".