Trouble when reloading extension modules.

[christophwiedemann]

BPO	34309
Nosy	@ncoghlan, @scoder, @vstinner, @encukou, @ericsnowcurrently, @cschramm, @corona10, @shakfu

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2018-08-01.17:36:13.403>
labels = ['extension-modules', 'interpreter-core', '3.9', 'type-crash']
title = 'Trouble when reloading extension modules.'
updated_at = <Date 2021-04-12.10:02:51.999>
user = 'https://bugs.python.org/christophwiedemann'

bugs.python.org fields:

activity = <Date 2021-04-12.10:02:51.999>
actor = 'corona10'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Extension Modules', 'Interpreter Core']
creation = <Date 2018-08-01.17:36:13.403>
creator = 'christoph.wiedemann'
dependencies = []
files = []
hgrepos = []
issue_num = 34309
keywords = []
message_count = 15.0
messages = ['322876', '322881', '322882', '322886', '322890', '322894', '322896', '322931', '322943', '322970', '322972', '322978', '323112', '323265', '390811']
nosy_count = 9.0
nosy_names = ['ncoghlan', 'scoder', 'vstinner', 'petr.viktorin', 'eric.snow', 'cschramm', 'corona10', 'christoph.wiedemann', 'shakfu']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue34309'
versions = ['Python 3.9']

[christophwiedemann]

I'm linking an issue from numpy here: numpy/numpy#8097

Embedding python suffers from a possibility to reliably reset the state of the python interpreter. For my use case, I noticed that when using numpy with Py_Initialize() and Py_Finalize():

Py_Initialize()
// call code importing numpy
Py_Finalize()
Py_Initialize()
// call same code again

The above code will result in a crash.

One of the comments in the referenced issue is that Py_Finalize doesn't unload loaded DLL's or shared objects. Doing that would probably fix the issues.

As of now, embedding python is fundamentally broken for applications which want to embed non-trivial scientific python scripts involving user-adapted python code, because

a) Py_Finalize cannot be used reliably
b) There is no possibility to reliably reset the python interpreter otherwise (because the sub-interpreters are also not working reliably, which is stated in the documentation)
c) manually reloading changed modules via importlib.reload is not a feasible solution

The possibility to reset an embedded python interpreter to an initial state is a strong requirement for such applications.

[ericsnowcurrently]

The matter of unloading extension modules is partly covered in bpo-401713. However, note that a few things have changed in the last 18 years. :) I think it would be worth revisiting the decision in that issue at this point.

If that were sorted out would there be other issues to address?

Regardless, If I understood right, your only objective here is to completely reset Python in the current process to the initial state. You mention "embedded python interpreter", but it sounds more like you mean "embedded python runtime" (particularly since you are using Py_Initialize and Py_Finalize). Why is completely resetting Python "a strong requirement"?

[ericsnowcurrently]

Regarding your 3 points:

a) Py_Finalize cannot be used reliably

Note that unloading extension modules is not the only thing that Py_Finalize isn't doing that it probably should be. I expect that you would find a number of memory leaks and potentially cases that crash the process when calling Py_Initialize() after Py_Finalize().

As bpo-401713 indicates, calling Py_Initialize/Py_Finalize multiple times isn't a well supported pattern. The docs indicate likewise (see https://docs.python.org/3/c-api/init.html#c.Py_FinalizeEx). That said, I'm sure everyone agrees that it *should* work and that we should fix issues there when we can. It's mostly a matter of finding folks motivated enough to work on it. :)

b) There is no possibility to reliably reset the python interpreter otherwise

At the level of the interpreter (main or otherwise), the C-API doesn't really provide anything like that, though I suppose it could. However, it's mostly simpler to just make a new interpreter. So I doubt we'd introduce an explicit concept of resetting an interpreter. What would be the advantage?

At the level of the *runtime* the C-API likewise doesn't offer anything for reset, other than what you've already tried. It would really help to understand why resetting is needed.

(because the sub-interpreters are also not working reliably,
which is stated in the documentation)

This is something I'm actively working on, including improving isolation between interpreters (minimizing global state) and improving the extension module experience when using subinterpreters. The latter is tricky, particularly with extension modules that use libraries that have their own C globals.

Note that isolation between interpreters will never be perfect. If you need perfect isolation then use multiple processes. All interpreters in the same process will always share the runtime state, as well as process-wide data.

FWIW, even if we were to completely isolate the Python runtime, allowing more than one runtime in a process, the runtimes would still share the process-wide data.

c) manually reloading changed modules via importlib.reload is not a
feasible solution

Yeah, while I suppose it could work for some extensions, I expect it would not work for many.

[christophwiedemann]

Thanks for your comments and the link to the issue from the year 2000.

You mention "embedded python interpreter", but it sounds more like you mean "embedded python runtime"

Yes that's right. Sorry for imprecise wording.

Why is completely resetting Python "a strong requirement"?

Because otherwise, if this is not an option, we need to restart the embedding C/C++ application whenever a python module is changed and need to be reloaded. This is occurring frequently in our use case and it is the main reason we want to embed python for rapid development purposes. (The application itself is a development environment for computer vision algorithms, where python is only one of multiple interacting, configurable plugins.)

Being forced to restart the whole application on code changes compromises the advantage of using a scripting language without edit/compile/link steps to a degree which questions the whole idea. I'm not very confident in hacking reload(...) stuff in the modules.

Interestingly enough, our use case matches exactly the one which has been described as unlikely in the original issue (https://bugs.python.org/issue401713#msg34524): the python runtime is dynamically loaded at runtime as a plugin :) I haven't tried it, but I suppose that we get the very same issues when we reload the plugin, because the dynamic libraries of extension modules are still loaded with an invalid state.

Maybe using processes and some kind of socket / shared memory communication would suit our needs better, but this is also much more complicated and error-prone to implement than simply embedding python into the main process.

[ericsnowcurrently]

Ah, thanks for clarifying. So which of these is the main thing you really want:

1. reload extension modules
2. completely restart Python

It sounds like #1. If that's the case then there are a number of issues to resolve to make it work. However, there are some serious technical challenges to overcome. :/

So if it's #1 the problem space is relatively focused so a solution (if possible) would be tractable in the "short" term, so it *could* happen. :) However, if it's #2 then a lot of things will have to fixed and so realistically it might never happen.

FYI, either way none of it will be backported, so the functionality would not be available on Python 3.7 or earlier.

---------------

FWIW, in VS Code they run their plugins (extensions) in a separate process. Their docs give some insight into plugin system design. :)

https://code.visualstudio.com/docs/extensionAPI/patterns-and-principles#_stability-extension-isolation

[ericsnowcurrently]

Also, part of the motivation for PEP-489 (extension module initialization) was to help with reloading extension modules.

[ericsnowcurrently]

Also, PEP-3121 provides a good summary of some of the issues at hand.

[christophwiedemann]

Okay, completely restarting python is not really necessary. Being able to reliably unload and later on re-import python modules (extension modules as well as pure python modules) in an embedded python runtime would solve my problems.

One way to achieve that is currently Py_Initialize / Py_Finalize, but there are the drawbacks already mentioned. Another possibility is using sub-interpreters.

If either of these could be fixed for extension modules (possibly with unloading the shared objects / DLL's :) ) I'd be fine.

I completely understand your point about backporting and it is not an issue.

[encukou]

PEP-489 (Multi-phase extension module initialization) makes it possible/easy to unload/reimport extension modules, in the sense of destroying/recreating the module object. The problem is that the modules needs to opt-in to supporting this, which is not always easy (e.g. the module needs to not use C globals, or use them carefully), and sometimes it's still nearly impossible (see the in-progress PEP-573).

Unloading the actual shared library is another matter, though. That's not currently planned. There's no good way to ensure that there no remaining objects that could reference the shared library's code.
Instead, your best bet is probably to name the new .so/DLL differently, and load an extra copy. (PEP-489 makes it possible to make the .so/DLL contain a module with a different name.) If you do go this way, I'd welcome feedback.

[ericsnowcurrently]

@chris, I can't promise that anything will happen right away, but I'll be sure to look into this further when I work on improving extension module usage in subinterpreters in the next few months.

[ericsnowcurrently]

I've changed the issue title to reflect where things stand.

Hmm, doing so reminded me of an important consideration here. A module object is effectively a fairly light wrapper around a dict. When you call importlib.reload() the loader from the module's spec is used to re-execute the module's existing dict. [1][2] A new module is not created and the existing module namespace is not reset. So during reload the module is responsible for deleting anything in its namespace that wouldn't get replaced when re-executed (including attributes that were added to the namespace externally). For most modules this isn't an issue. However, it's something to consider when reloading a module. See the docs for more explanation and caveats. [3]

[1] https://github.com/python/cpython/blob/master/Lib/importlib/__init__.py#L169
[2] https://github.com/python/cpython/blob/master/Lib/importlib/_bootstrap.py#L610
[3] https://docs.python.org/3/library/importlib.html#importlib.reload