Improve Py_mod_multiple_interpreters and Py_mod_gil Usability

[ericsnowcurrently]

Feature or enhancement

Proposal:

Here are some deficiencies:

• the slot name "Py_mod_multiple_interpreters" is a bit long
• the "Py_MOD_MULTIPLE_INTERPRETERS_*" slot value names are a bit long
• the slot name "Py_mod_gil" is overly coupled to the GIL (it's really about thread safety)
• the difference between Py_MOD_MULTIPLE_INTERPRETERS_SUPPORTED and Py_MOD_PER_INTERPRETER_GIL_SUPPORTED is unclear
• Py_MOD_PER_INTERPRETER_GIL_SUPPORTED applies to the vast majority of extensions and should be the default
• Py_mod_gil doesn't have an option that explicitly covers the case where an external dependency is not thread-safe
• currently Py_MOD_MULTIPLE_INTERPRETERS_SUPPORTED implies an external dependency is not thread-safe and/or isolated to each interpreter; it should be the other way around (and be a synonym for Py_MOD_PER_INTERPRETER_GIL_SUPPORTED), which would introduce a need for a Py_MOD_PER_INTERPRETER_GIL_NOT_SUPPORTED

We could take a simple approach, working with the existing slots:

• (maybe) shorten the names
• add Py_MOD_PER_INTERPRETER_GIL_NOT_SUPPORTED
• add something like Py_MOD_GIL_USED_FOR_EXTERNAL

I think we could do better, by replacing the existing slots with new ones that focus explicitly on what we care about:

• new module def slot: Py_mod_isolation
  • Py_MOD_NOT_ISOLATED - the module has process-global state and/or objects (e.g. static types)
  • Py_MOD_ISOLATED (default) - the module's state/objects have been isolated to the module (i.e interpreter)
• new module def slot: Py_mod_threadsafe
  • Py_MOD_NOT_THREADSAFE (default) - the module's own state/code is not thread-safe
  • Py_MOD_THREADSAFE - the module's state/code are thread-safe; external dependencies are covered by Py_mod_external_libs
• new module def slot: Py_mod_external_libs
  • Py_MOD_EXTERN_NOT_THREADSAFE - at least one external dependency is not thread-safe (may be used without the GIL held)
  • Py_MOD_EXTERN_NOT_ISOLATED - at least one external dependency is not isolated to the module object (i.e. interpreter)
  • Py_MOD_EXTERN_ISOLATED (default) - all external dependencies are isolated and using them is thread-safe
• for Py_mod_isolation and Py_mod_threadsafe, external dependencies are covered by Py_mod_external_libs
• deprecate Py_mod_multiple_interpreters slot
• deprecate Py_mod_gil slot

Regarding the defaults, for Py_mod_isolation, multi-phase init implies isolation (per PEP 489). For Py_mod_external_libs, we assume that the vast majority of modules will not have problematic dependencies.

Equivalents to current usage:

Py_mod_threadsafe	Py_mod_external_libs		Py_mod_gil
Py_MOD_NOT_THREADSAFE	*		Py_MOD_GIL_USED
Py_MOD_THREADSAFE	Py_MOD_EXTERNAL_NOT_THREADSAFE		Py_MOD_GIL_USED
Py_MOD_THREADSAFE	Py_MOD_EXTERNAL_NOT_ISOLATED		Py_MOD_GIL_NOT_USED
Py_MOD_THREADSAFE	Py_MOD_EXTERNAL_ISOLATED		Py_MOD_GIL_NOT_USED

Py_mod_isolation	Py_mod_external_libs		Py_mod_multiple_interpreters
Py_MOD_NOT_ISOLATED	*		Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED
Py_MOD_ISOLATED	Py_MOD_EXTERNAL_NOT_THREADSAFE		Py_MOD_MULTIPLE_INTERPRETERS_SUPPORTED
Py_MOD_ISOLATED	Py_MOD_EXTERNAL_NOT_ISOLATED		Py_MOD_MULTIPLE_INTERPRETERS_SUPPORTED
Py_MOD_ISOLATED	Py_MOD_EXTERNAL_ISOLATED		Py_MOD_PER_INTERPRETER_GIL_SUPPORTED

CC @encukou

Has this already been discussed elsewhere?

No response given

Links to previous discussion of this feature:

No response

[encukou]

I'd love to combine this with a proposal in https://discuss.python.org/t/stable-abi-limited-api-for-free-threaded-builds/86458:

I’d like to add a module slot that indicates the version of Python used to build the extension, the Py_LIMITED_API value used (if any), and some flags [...]. This would be checked for compatibility, so we no longer rely only on wheel/.soname tags, and so we can add deprecation warnings for upcoming incompatibilities.

Can it wait until 3.15, or are we in a rush?