[subinterpreters] Static types incorrectly share some objects between interpreters

[ericsnowcurrently]

While static types (PyTypeObject values) don't themselves ever change (once PyType_Ready() has run), they do hold mutable data. This means they cannot be safely shared between multiple interpreters without a common GIL (and without state leaking between).

Mutable data:

• the object header (e.g. refcount), as with all objects
• otherwise immutable objects:
  • ob_type (__class__)
  • tp_base (__base__) - set by PyType_Ready() if not set
  • tp_bases (__bases__) - always set by PyType_Ready()
  • tp_mro (__mro__) - always set by PyType_Ready()
  • tp_dict (__dict__) - set by PyType_Ready() if not set
• mutable containers:
  • tp_subclasses (__subclasses__)
  • tp_weaklist

(See https://docs.python.org/3/c-api/typeobj.html#tp-slots.)
(Note that tp_cache is no longer used.)

For the object header, if PEP 683 (immortal objects) is accepted then we can make static types immortal.

For the otherwise immutable objects, we can make sure each is immortal and then we're good. Even tp_dict is fine since it gets hidden behind types.MappingProxyType. We'd also need to either make sure each contained item is immortal.

For tp_subclasses we will need a per-interpreter copy, and do the proper lookup in the __subclasses__ getter. The cache could be stored on PyInterpreterState or even as a dict in tp_subclasses.

For tp_weaklist it's a similar story as for tp_subclasses. Note that tp_weaklist isn't very important for static types since they are never deallocated.

(The above is also discussed in PEP 684.)

CC @kumaraditya303

Linked PRs

• gh-94673: Isolate the _io module to Each Interpreter #102663
• gh-94673: More Per-Interpreter Fields for Builtin Static Types #103912
• gh-94673: Ensure Builtin Static Types are Readied Properly #103940
• gh-94673: Fix _PyTypes_InitTypes() and get_type_attr_as_size() #103961
• gh-94673: Properly Initialize and Finalize Static Builtin Types for Each Interpreter #104072
• gh-94673: Hide Objects in PyTypeObject Behind Accessors #104074
• gh-94673: Ensure subtypes are readied only once in math.trunc() #105465
• [3.12] gh-94673: Ensure subtypes are readied only once in math.trunc() (gh-105465) #105471
• gh-94673: Clarify About Runtime State Related to Static Builtin Types #117761
• gh-94673: Fix compiler warning in typeobject.c #117980

[ericsnowcurrently]

See https://peps.python.org/pep-0684/#global-objects.

[kumaraditya303]

The current _PyInterpreterState_GET (which would be used here) static inline function is quite inefficient as there is a lot indirection and involves atomic reads to get the "current" thread and then get the interpreter state. To improve this we can use thread local ¹ variables on supported platform such as Linux, Windows and macOS. On Linux with glibc, the thread local implementation uses fs segment register and on x86-64 it is very efficient compared to the current implementation.
This isn't required here but would be nice to investigate separately.

Footnotes

1. https://en.cppreference.com/w/c/thread/thread_local ↩

[encukou]

This means they cannot be safely shared between multiple interpreters without a common GIL

Python currently has a common GIL.
Shouldn't this be part of a PEP, rather than an issue?

[ericsnowcurrently]

I have the changes ready for this:

• gh-94673: Add Per-Interpreter Storage for Static Builtin Types #94995 - adds (basically) empty per-interpreter state for the static builtin types.
• based on top of that one:
  • ericsnowcurrently/cpython@init-static-builtin...shareable-static-types-tp_subclasses - add tp_subclasses to that state
  • ericsnowcurrently/cpython@init-static-builtin...shareable-static-types-tp_weaklist - add tp_weaklist to that state

[ericsnowcurrently]

Shouldn't this be part of a PEP, rather than an issue?

This is covered by PEP 684. The issue exists for implementation discussion and to anchor PRs.

[ericsnowcurrently]

This isn't required here but would be nice to investigate separately.

This has been discussed elsewhere. Also see #29228.

FYI, for this PR (and in most cases) the performance penalty is insignificant.

[ericsnowcurrently]

I've split up that first PR:

• gh-94673: Add _PyStaticType_InitBuiltin() #95152
• gh-94673: Always Finalize Static Builtin Types #95153
• gh-94673: Add Per-Interpreter Storage for Static Builtin Types #95255
• gh-94673: Add Per-Interpreter tp_subclasses for Static Builtin Types #95301
• gh-94673: Add Per-Interpreter tp_weaklist for Static Builtin Types #95302

[encukou]

This is covered by PEP 684. The issue exists for implementation discussion and to anchor PRs.

Please get the PEP accepted before merging anything you can't easily revert.

[ericsnowcurrently]

gh-95302 slows things down ~1% so I'll need to address that. (See #95302 (comment).)

[ericsnowcurrently]

FYR, #44470 is where we dealt with leaking memory during finalization. It looks like we missed some cases there (relative to both tp_subclasses and tp_weaklist), which my PRs help to address.

[ericsnowcurrently]

Also FTR, a per-interpreter GIL isn't the only motivator here. It's a good idea to keep a per-interpreter tp_subclasses, etc. Otherwise cls.__subclasses__ and weakref.getweakrefs(cls) will show the objects from all interpreters, which is a problem for users.

[gvanrossum]

We now get a warning when building without pydebug:

Objects/typeobject.c:120:1: warning: unused function 'static_builtin_index_is_set' [-Wunused-function]
static_builtin_index_is_set(PyTypeObject *self)
^
1 warning generated.

[ericsnowcurrently]

I'll take a look.