Doc/howto/abi3t-migration.rst

.. highlight:: c

.. _abi3t-migration-howto:

******************************************************
Migrating to Stable ABI for free threading (``abi3t``)
******************************************************

Starting with the 3.15 release, CPython supports a variant of the Stable ABI
that supports :term:`free-threaded <free threading>` Python:
Stable ABI for Free-Threaded Builds, or ``abi3t`` for short.
This document describes how to adapt C API extensions to support free threading.

Why do this
===========

The typical reason to use Stable ABI is to reduce the number of artifacts that
you need to build and distribute for each version of your library.

Without the Stable ABI, you must build a separate shared library, and typically
a *wheel* distribution, for each feature version of CPython you wish
to support.
For example, each tag in the following table represents a separate
library/wheel:

+-----------------+-----------------------+------------------------+
| CPython version | Non-free-threaded     | Free-threaded          |
+=================+=======================+========================+
| 3.12            | ``cpython-312``       | ---                    |
+-----------------+-----------------------+------------------------+
| 3.13            | ``cpython-313``       | ``cpython-313t``       |
+-----------------+-----------------------+------------------------+
| 3.14            | ``cpython-314``       | ``cpython-314t``       |
+-----------------+-----------------------+------------------------+
| 3.15            | ``cpython-315``       | ``cpython-315t``       |
+-----------------+-----------------------+------------------------+
| 3.16            | ``cpython-316``       | ``cpython-316t``       |
+-----------------+-----------------------+------------------------+
| Later versions  | :samp:`cpython-3{XX}` | :samp:`cpython-3{XX}t` |
+-----------------+-----------------------+------------------------+

That's a lot of builds, especially when multiplied by the number
of supported platforms.

With the Stable ABI (``abi3``, introduced in CPython 3.2), a single extension
(per platform) can cover all *non-free-threaded* builds of CPython:

+-----------------+-------------------+------------------------+
| CPython version | Non-free-threaded | Free-threaded          |
+=================+===================+========================+
| 3.12            | ``abi3``          | ---                    |
+-----------------+                   +------------------------+
| 3.13            |                   | ``cpython-313t``       |
+-----------------+                   +------------------------+
| 3.14            |                   | ``cpython-314t``       |
+-----------------+                   +------------------------+
| 3.15            |                   | ``cpython-315t``       |
+-----------------+                   +------------------------+
| 3.16            |                   | ``cpython-316t``       |
+-----------------+                   +------------------------+
| Later versions  |                   | :samp:`cpython-3{XX}t` |
+-----------------+-------------------+------------------------+

The Stable ABI for free-threaded builds (``abi3t``), introduced in
CPython 3.15, does the same for free-threaded builds.
And it's compatible with non-free-threaded ones as well:

+-----------------+-------------------+------------------+
| CPython version | Non-free-threaded | Free-threaded    |
+=================+===================+==================+
| 3.12            | ``abi3`` *        | ---              |
+-----------------+                   +------------------+
| 3.13            |                   | ``cpython-313t`` |
+-----------------+                   +------------------+
| 3.14            |                   | ``cpython-314t`` |
+-----------------+-------------------+------------------+
| 3.15            | ``abi3t``                            |
+-----------------+                                      +
| 3.16            |                                      |
+-----------------+                                      +
| Later versions  |                                      |
+-----------------+-------------------+------------------+

\* (As above, the ``abi3`` extension is compatible with all non-free-threaded
builds; even the 3.15+ ones that this table "attributes" to ``abi3t``.)

Why *not* do this
-----------------

There are two main downsides to Stable ABI.

First, you extension may become slower, since Stable ABI prioritizes
compatibility over performance.
The difference is usually not noticeable, and often can be mitigated by
using the same source to build both a Stable ABI build and a few
version-specific ones for "tier 1" CPython versions.

Second, not all of the C API is available.
Extensions need to be ported to build for Stable ABI, which may be difficult
or, in rare cases, impossible.

Specifically, ``abi3t`` requires APIs added in CPython 3.15.
If you want to build your extension for older versions of CPython from the
same source, you have two main options:

- Use preprocessor conditionals.

  When following this guide, use ``#ifdef Py_TARGET_ABI3T`` blocks whenever
  you are told to do a change that breaks the build on CPython versions you
  care about. Keep the pre-existing code in ``#else`` blocks.

  For hand-written C extensions, this approach is reasonable down to
  CPython 3.12, due to additions introduced in :pep:`697`.
  Keeping compatibility with 3.11 and below may be worth it for code
  generators (for example, Cython).

- Do not port to ``abi3t``, and continue building separate extensions for
  each version of CPython, until you can drop support for the older versions.

  This is a valid approach. Not all extensions need to switch to ``abi3t``
  right now.


Prerequisites
=============

This guide assumes that you have an extension written directly in C (or C++),
which you want to port to ``abi3t``.

If your extenstion uses a code generator (like Cython) or language binding
(like PyO3), it's best to wait until that tool has support for ``abi3t``.
If you maintain such a tool, you might be able to adapt the instructions
here for your tool.

Non-free-threaded Stable ABI
----------------------------

Your extension should support the Stable ABI (``abi3t``).
If not, either port it first, or follow this guide but be prepared to fix
issues it does not mention.

Free-threading support
----------------------

While it's technically not a hard prerequisite, you will most likely want to
prepare your extension for free threading before you port it to ``abi3t``.
See :ref:`freethreading-extensions-howto` for instructions.

.. seealso::

   `Porting Extension Modules to Support Free-Threading
   <https://py-free-threading.github.io/porting/>`__:
   A community-maintained porting guide for extension authors.

Isolating extension modules
---------------------------

Your module should use :ref:`multi-phase initialization <multi-phase-initialization>`,
and it should either be isolated or limit itself to be loaded at most once
per process.
If it is not your case, follow :ref:`isolating-extensions-howto` first.
(See the :ref:`opt-out section <isolating-extensions-optout>` for a shortcut.)

Avoiding variable-sized types
-----------------------------

If your extension defines variable-sized types (using :c:macro:`Py_tp_itemsize`
or :c:member:`PyTypeObject.tp_itemsize`), it cannot be ported to
``abi3t`` 3.15.


Setting up the build
====================

If you use a build tool (such as setuptools, meson-python, scikit-build-core),
search its documentation for a way to select ``abi3t``.
At the time of writing, not all of them have this; but if your tool does,
use it.
You may want to verify that it set the right flag by temporarily adding the
following just after ``#include <Python.h>``::

   #if Py_TARGET_ABI3T+0 <= 0x30f0000
   #error "abi3t define is not set!"
   #endif

This should result in a different error than "``abt3t`` define is not set".

.. note::

   If your build tool doesn't support ``abi3t`` yet, set the following macro
   before including ``Python.h``::

      #define Py_TARGET_ABI3T 0x30f0000

   or specify it as a compiler flag, for example::

      -DPy_TARGET_ABI3T=0x30f0000

   Once your extension builds with this setting, it will be compatible with
   CPython 3.15 and above.

   If you set this macro manually, you will later need to name and tag the
   resulting extension manually as well.
   This is covered in :ref:`abi3t-migration-tagging` below.

This guide will ask you to make a series of changes.
After each one, verify that your extension still builds in the original
(non-``abi3t``) configuration, and ideally run tests on all Python
versions you support.
This will ensure that nothing breaks as you are porting.


Module export hook
==================

Unless you've done this step already, your extension module defines a
:ref:`module initialization function <extension-pyinit>`
named :samp:`PyInit_{<module_name>}`.
You will need to port it to a :ref:`module export hook <extension-export-hook>`,
:samp:`PyModExport_{<module name>}`, a feature added in CPython 3.15 in
:pep:`793`.

Your existing init function should look like this (with your own names
for ``<modname>`` and ``<moddef>``):

.. code-block::
   :class: bad

   PyMODINIT_FUNC
   PyInit_<modname>(void)
   {
       return PyModuleDef_Init(&<moddef>);
   }

If there is some code before the ``return``, move it to
a :c:macro:`Py_mod_create` or :c:macro:`Py_mod_exec` slot function.
See the :ref:`PyInit documentation <extension-pyinit>` for related information.

The function references a ``PyModuleDef`` object (``<moddef>`` in the code
above).
Its definition should be similar to the following, with different values
and perhaps some fields unnnamed or left out:

.. code-block::
   :class: bad

   static PyModuleDef <moddef> = {
       PyModuleDef_HEAD_INIT,
       .m_name = "my_module",
       .m_doc = "my docstring",
       .m_size = sizeof(my_state_struct),
       .m_methods = my_methods,
       .m_slots = my_slots,
       .m_traverse = my_traverse,
       .m_clear = my_clear,
       .m_free = my_free,
   };

Remove this definition and the ``PyInit`` function (or put them in
an ``#ifndef Py_TARGET_ABI3T`` block, to retain backwards compatibility),
and replace them with the following:

.. code-block::
   :class: good

   PyABIInfo_VAR(abi_info);

   static PySlot my_slot_array[] = {
       PySlot_STATIC_DATA(Py_mod_abi, &abi_info),
       PySlot_STATIC_DATA(Py_mod_name, "my_module"),
       PySlot_STATIC_DATA(Py_mod_doc, "my docstring"),
       PySlot_SIZE(Py_mod_state_size, sizeof(my_state_struct)),
       PySlot_STATIC_DATA(Py_mod_methods, my_methods),
       PySlot_STATIC_DATA(Py_mod_slots, my_slots),
       PySlot_FUNC(Py_mod_state_traverse, my_traverse),
       PySlot_FUNC(Py_mod_state_clear, my_clear),
       PySlot_FUNC(Py_mod_state_free, my_free),
       PySlot_END
   };

   PyMODEXPORT_FUNC
   PyModExport_<modname>(void)
   {
       return my_slot_array;
   }

Leave out any fields that were missing (except the new :c:macro:`Py_mod_abi`),
and substitute your own values.

See the :c:type:`PySlot` and :c:ref:`export hook <extension-export-hook>`
documentation for details on this API.

Associated ``PyModuleDef``
--------------------------

Since the new API does not use a :c:type:`!PyModuleDef` structure, a definition
will not be associated with the resulting module.
This changes the behavior of the following functions:

- :c:func:`PyModule_GetDef`
- :c:func:`PyType_GetModuleByDef`

Check your code for these.
If you do not use them, you can skip this section.

These functions are typically used for two purposes:

1. To get the definition the module was created with.
   This is no longer possible using the new API.
   Modules no longer keep a reference to the definition, so you will need to
   figure out a different way to pass the relevant data around.

.. _abi3t-migration-module-token:

2. To check if a given module object is “yours”.
   This use case is now served by :ref:`module tokens <ext-module-token>` --
   opaque pointers that identify a module.
   To use a token, declare (or reuse) a unique static variable, for example:

   .. code-block::
      :class: good

      static char my_token;

   and add a pointer to it in a new entry to your module's ``PySlot`` array:

   .. code-block::
      :class: good
      :emphasize-lines: 3

      static PySlot my_slot_array[] = {
         ...
         PySlot_STATIC_DATA(Py_mod_token, &my_token),
         PySlot_END
      }

   Then, switch from :c:func:`PyModule_GetDef` calls such as:

   .. code-block::
      :class: bad

      PyModuleDef *def = PyModule_GetDef(module);

   to :c:func:`PyModule_GetToken` (which uses an output argument and may fail
   with an exception):

   .. code-block::
      :class: good

      void *token;
      if (PyModule_GetToken(module, &token) < 0) {
          /* handle error */
      }

   and from :c:func:`PyType_GetModuleByDef` calls such as:

   .. code-block::
      :class: bad

      PyObject *module = PyType_GetModuleByDef(type, my_def);
      /* handle error; use module */

   to :c:func:`PyType_GetModuleByToken` (which returns a strong reference):

   .. code-block::
      :class: good

      PyObject *module = PyType_GetModuleByToken(type, my_token);
      /* handle error; use module */
      Py_XDECREF(module);

``PyObject`` opaqueness
=======================

The :c:type:`PyObject` and :c:type:`PyVarObject` structures are opaque
in ``abi3t``.

Accessing their members is prohibited.
If you do this, switch to getter/setter functions mentioned in
their documentation:

- :c:member:`PyObject.ob_type`
- :c:member:`PyObject.ob_refcnt`
- :c:member:`PyVarObject.ob_size`

Also, the *size* of the :c:type:`PyObject` structures is
unknown to the compiler.
It can -- and *does* -- change between different CPython builds.

.. note::

   While the size is available at runtime (for example as
   ``sys.getsizeof(object())`` in Python code), you should resist the
   temptation to calculate pointer offsets from it.
   The object memory layout is subject to change in future
   ``abi3t`` implementations.


Custom type definitions
-----------------------

Since :c:type:`!PyObject` is opaque, the traditional way of defining
custom types no longer works:

.. code-block::
   :class: bad

   typedef struct {
      PyObject_HEAD  // expands to `PyObject ob_base;` which has unknown size

      int my_data;
   } CustomObject;

   static PyType_Spec CustomType_spec = {
      ...
      .basicsize = sizeof(CustomObject),
      ...
   };

Most likely, all your class definitions, *and* all code that accesses
your classes' data, will need to be rewritten.
This will probably be the biggest change you need to support ``abi3t``.

For each such type, instead of defining a ``struct`` for the entire instance,
define one with only the “additional” fields -- ones specific to your class,
not its superclasses:

.. code-block::
   :class: good

   typedef struct {
      int my_data;
   } CustomObjectData;

Change the name.
Almost all code that uses the struct will need to change
(notably, pointers to the new structure cannot be cast to/from ``PyObject*``),
and changing the name will highlight the usages as compiler errors.
(If you use ``typeof``, C++ ``auto``, or similar ways to avoid
typing the type name, this won't work. Be extra careful, and consider running
tools to detect undefined behavior.)

Then, to create the class, use *negative* ``basicsize`` to indicate
“extra” storage space rather than *total* instance size:

.. code-block::
   :class: good

   static PyType_Spec CustomType_spec = {
      ...
      .basicsize = -sizeof(CustomObjectData), /* note the minus sign */
      ...
   };

If you use :c:macro:`Py_tp_members`, set the :c:macro:`Py_RELATIVE_OFFSET`
flag on each member and specify the :c:member:`~PyMemberDef.offset`
relative to your new struct.


Custom type data access
-----------------------

Then comes the hard part: in all code that needs to access this struct,
you will need an additional :c:func:`PyObject_GetTypeData` call to
retrieve a ``CustomObjectData *`` pointer from ``PyObject *``:

.. code-block::
   :class: good

   PyObject *obj = ...;
   CustomObjectData *data = PyObject_GetTypeData(obj, cls);

Note that this call requires the *type object* for your class (``cls``).

If your class is not subclassable (that is, it does not use the
:c:macro:`Py_TPFLAGS_BASETYPE` flag), ``cls`` will be ``Py_TYPE(obj)``.
Otherwise, **DO NOT USE** ``Py_TYPE`` with :c:func:`!PyObject_GetTypeData`:
it might return memory reserved to an unrelated subclass!
For example, if a user makes a subclass like this:

.. code-block:: python

   class Sub(YourCustomClass):
      __slots__ = ('a', 'b')

then ``Py_TYPE(obj)`` is ``YourCustomClass``, and the underlying memory may
look like this:

.. code-block:: text

   ╭─ PyObject *obj
   │              ╭─ the pointer you want
   │              │                    ╭─ PyObject_GetTypeData(obj, Py_TYPE(obj))
   ▼              ▼                    ▼
   ┌──────────┬───┬────────────────┬───┬─────────────┬───┬─────────────┐
   │ PyObject │...│ CustomTypeData │...│ PyObject *a │...│ PyObject *b │
   └──────────┴───┴────────────────┴───┴─────────────┴───┴─────────────┘

(Ellipses indicate possible padding.
Note that this memory layout is not guaranteed: future versions of Python may
add different padding or even switch the order of the structures.)

There are two main ways to get the right class:

- In instance methods, your implementation may use the :c:type:`PyCMethod`
  signature (and the :c:macro:`METH_METHOD` bit in
  :c:member:`PyMethodDef.ml_flags`),
  and get the class as the ``defining_class`` argument.
- Otherwise, give your class a unique static token using the
  :c:macro:`Py_tp_token` slot, and use:

  .. code-block::
     :class: good

     PyTypeObject cls;
     if (PyType_GetBaseByToken(Py_TYPE(obj), my_tp_token, &cls) < 0) {
         /* handle error */
     }
     CustomObjectData *data = PyObject_GetTypeData(obj, cls);

  Type tokens work similarly to module tokens covered :ref:`earlier in this
  guide <abi3t-migration-module-token>`.



Avoid build-time conditionals
=============================

Check your code for API that identifies the version of Python used to
*build* your extension.
This no longer corresponds to the Python your extension runs on, so code
that uses this information often needs changing.
The macros to check for are:

- :c:macro:`PY_VERSION_HEX`, :c:macro:`PY_MAJOR_VERSION`,
  :c:macro:`PY_MINOR_VERSION`:

  - to get the run-time version, use :c:data:`Py_Version`;
  - to determine what C API is available, use :c:macro:`Py_TARGET_ABI3T`.
    This macro is set to the minimum supported version.

- :c:macro:`Py_GIL_DISABLED`: under ``abi3t``, this macro is always defined.
  Code that works with free-threaded Python *should* also work with
  the GIL enabled (since the GIL can be enabled at run time),
  and usually *does* (unless it, for some reason, requires more than one
  :term:`attached thread state <attached thread state>` at one time).


Further code changes
====================

If you are still left with compiler errors or warnings, find a way to fix them.
Alas, this guide is limited, and cannot cover all possible code
changes extensions may need.

If you find a problem that other extension authors might run into,
consider :ref:`reporting an issue <reporting-documentation-bugs>` (or sending
a pull request) for this guide.

It is possible your issue cannot be fixed for the current version of ``abi3t``.
In that case, reporting it may help it get prioritized for the next version
of CPython.


.. _abi3t-migration-tagging:

Tagging and distribution
========================

If you are using a build tool with ``abi3t`` support, your extension is ready,
but you might want to check that it was built correctly.

Extensions built with ``abi3t`` should have the following extension:

- On Windows: ``.pyd`` (like any other extension);
- Linux, macOS, and other systems that use the ``.so`` suffix: ``.abi3t.so``
  (**not** ``.cpython-315t.so`` or ``.abi3.so``).
  Note that both free-threaded and non-free-threaded builds will
  load ``.abi3t.so`` extensions;
- Other systems: consult your distributor, and perhaps update this guide.

If you distribute the extension as a *wheel*, use the following tags:

* Python tag: :samp:`cp3{XX}`, where *XX* is the minimum Python version
  the extension is built for.
  (For example, ``cp315`` if you set ``Py_TARGET_ABI3T`` to ``0x30f0000``.
  See :ref:`abi3-compiling` for more values.)
* ABI tag: ``abi3.abi3t``. This is a *compressed tag set* that indicates
  support for both non-free-threaded and free-threaded builds.

For example, the wheel filename may look like this:

.. code-block:: text

   myproject-1.0-cp315-abi3.abi3t-macosx_11_0_arm64.whl

.. seealso:: `Platform Compatibility Tags <https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/>`__ in the PyPA package distribution metadata.

If the filename or tags are incorrect, fix them.


Testing
=======

Note that when you build an extension compatible with multiple versions of
CPython, you should always *test* it with each version it supports (for
example, 3.15, 3.16, and so on).
Stable ABI only guarantees *ABI* compatibility; there may also be behavior
changes -- both intentional ones (covered by :pep:`387`) and bugs.

Be sure to run tests on both free-threaded and non-free-threaded builds
of CPython.

If they pass, congratulations! You have an ``abi3t`` extension.