Skip to content

feat: Embeded sub-interpreters #5666

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 31 commits into from
May 20, 2025
Merged
Changes from 1 commit
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
6f5b38d
First draft a subinterpreter embedding API
b-pass May 15, 2025
ee42fe5
Move subinterpreter tests to their own file
b-pass May 18, 2025
45159fd
Migrate subinterpreter tests to use the new embedded class.
b-pass May 16, 2025
5b5a1e8
Add a test for moving subinterpreters across threads for destruction
b-pass May 18, 2025
74080eb
Code organization
b-pass May 16, 2025
51764f0
Add a test which shows demostrates how gil_scoped interacts with sub-…
b-pass May 16, 2025
68543fa
Add documentation for embeded sub-interpreters
b-pass May 18, 2025
7a00f32
Some additional docs work
b-pass May 16, 2025
e903406
Add some convenience accessors
b-pass May 16, 2025
70c24e8
Add some docs cross references
b-pass May 18, 2025
3c78e4b
Sync some things that were split out into #5665
b-pass May 18, 2025
ca44bfe
Update subinterpreter docs example to not use the CPython api
b-pass May 17, 2025
e9b2a57
Fix pip test
b-pass May 17, 2025
096afd9
style: pre-commit fixes
pre-commit-ci[bot] May 17, 2025
9db4f32
Fix MSVC warnings
b-pass May 17, 2025
a536fdc
Add some sub-headings to the docs
b-pass May 17, 2025
29e3171
Oops, make_unique is C++14 so remove it from the tests.
b-pass May 17, 2025
d4b1c44
I think this fixes the EndInterpreter issues on all versions.
b-pass May 18, 2025
eda11c1
Add a note about exceptions.
b-pass May 18, 2025
cd12019
style: pre-commit fixes
pre-commit-ci[bot] May 18, 2025
bddcfb5
Add try/catch to docs examples to match the tips
b-pass May 18, 2025
8770bfa
Python 3.12 is very picky about this first PyThreadState
b-pass May 19, 2025
ae097c8
style: pre-commit fixes
pre-commit-ci[bot] May 19, 2025
6062230
Missed a rename in a ifdef block
b-pass May 19, 2025
2aae3b8
I think this test is causing problems in 3.12, so try ifdefing it to …
b-pass May 19, 2025
dc57729
style: pre-commit fixes
pre-commit-ci[bot] May 19, 2025
ae0da30
Document the 3.12 constraints with a warning
b-pass May 19, 2025
68c68d7
Apply suggestions from code review
henryiii May 20, 2025
30c520b
ci: add cpptest to the clang-tidy job
henryiii May 20, 2025
babf12a
noexcept move operations
b-pass May 20, 2025
59f82a2
Update include/pybind11/subinterpreter.h
b-pass May 20, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Some additional docs work
  • Loading branch information
b-pass authored and henryiii committed May 20, 2025
commit 7a00f322a8dbda2c43d885ecde1929110da6e004
24 changes: 17 additions & 7 deletions docs/advanced/embedding.rst
Original file line number Diff line number Diff line change
Expand Up @@ -256,17 +256,19 @@ interpreter. Sub-interpreters are created and managed with a separate API from
the main interpreter. Beginning in Python 3.12, sub-interpreters each have
their own Global Interpreter Lock (GIL), which means that running a
sub-interpreter in a separate thread from the main interpreter can achieve true
concurrency.
concurrency.

Managing multiple threads and the lifetimes of multiple interpreters and their
GILs can be challenging. Proceed with caution (and lots of testing)!
pybind11's sub-interpreter API can be found in ``pybind11/subinterpreter.h``.

pybind11 :class:`subinterpreter` instances can be safely moved and shared between
threads as needed. However, managing multiple threads and the lifetimes of multiple
interpreters and their GILs can be challenging.
Proceed with caution (and lots of testing)!

The main interpreter must be initialized before creating a sub-interpreter, and
the main interpreter must outlive all sub-interpreters. Sub-interpreters are
managed through a different API than the main interpreter.

The sub-interpreter API can be found in ``pybind11/subinterpreter.h``.

The :class:`subinterpreter` class manages the lifetime of sub-interpreters.
Instances are movable, but not copyable. Default constructing this class does
*not* create a sub-interpreter (it creates an empty holder). To create a
Expand Down Expand Up @@ -391,6 +393,10 @@ it when it goes out of scope.

Best Practices for sub-interpreter safety:

- Avoid moving or disarming RAII objects managing GIL and sub-interpreter lifetimes. Doing so can
lead to confusion about lifetimes. (For example, accidentally extending a
:class:`subinterpreter_scoped_activate` past the lifetime of it's :class:`subinterpreter`.)

- Never share Python objects across different interpreters.

- Avoid global/static state whenever possible. Instead, keep state within each interpreter,
Expand All @@ -404,5 +410,9 @@ Best Practices for sub-interpreter safety:
resulting Python object when the wrong interpreter was active.

- While sub-interpreters each have their own GIL, there can now be multiple independent GILs in one
program, so your code needs to consider thread safety of within the C++ code, and the possibility
of deadlocks caused by multiple GILs and/or the interactions of the GIL(s) and C++'s own locking.
program you need to consider the possibility of deadlocks caused by multiple GILs and/or the
interactions of the GIL(s) and your C++ code's own locking.

- When using multiple threads to run independent sub-interpreters, the independent GILs allow
concurrent calls from different interpreters into the same C++ code from different threads.
So you must still consider the thread safety of your C++ code.