gh-133485: Use interpreters.Interpreter in InterpreterPoolExecutor

[ericsnowcurrently]

Most importantly, this resolves the issues with functions and types defined in __main__.
It also expands the number of supported objects.

• Issue: Minor InterpreterPoolExecutor Issues #133485

[ericsnowcurrently]

@markshannon, any ideas on why this is happening? It smells like a ceval bug, but it certainly could be something I've done wrong.

[ericsnowcurrently]

ping @markshannon

[neonene]

There seem to be related changes in inspect.getclosurevars() since 83ba8c2:

before:
ClosureVars(nonlocals={},
            globals={'INITIALIZER_STATUS': 'uninitialized'},
            builtins={}, unbound=set())
after:
ClosureVars(nonlocals={},
            globals={},
            builtins={}, unbound=set())

• init() on main (without L26):

  3           RESUME                   0

  5           LOAD_FAST_BORROW         0 (x)
              STORE_GLOBAL             0 (INITIALIZER_STATUS)
              LOAD_CONST               0 (None)
              RETURN_VALUE

• 3.3.5 (2014):

  5           0 LOAD_FAST                0 (x)
              3 STORE_GLOBAL             0 (INITIALIZER_STATUS)
              6 LOAD_CONST               0 (None)
              9 RETURN_VALUE

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @ericsnowcurrently for commit ccc135c 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F133957%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

[neonene]

The wasm32-wasi Non-Debug buildbot seems to be unused on merged commits and out of order on PRs. For example: https://buildbot.python.org/#/builders/1373/builds/507

[neonene]

Is the following usage invalid?

INITIALIZER_STATUS = 'uninitialized'

def init(x):
    global INITIALIZER_STATUS
    INITIALIZER_STATUS = x
    INITIALIZER_STATUS  # for now

def get_init_status():
    return INITIALIZER_STATUS

if __name__ == "__main__":
    from concurrent.futures import InterpreterPoolExecutor
    exe = InterpreterPoolExecutor(initializer=init, initargs=('initialized',))
    fut = exe.submit(get_init_status)
    assert fut.result() == 'initialized'  # fails
    exe.shutdown(wait=True)
    assert INITIALIZER_STATUS == 'uninitialized'

[neonene]

cpython/Python/crossinterp.c

Line 601 in ec12559

res.loaded = runpy_run_path(filename, run_modname);

I guess the failure case in my previous comment can be resolved if the runpy_run_path is called only once for __main__ by using/keeping ctx->main.cached->loaded or something across xi-sessions without being replaced. (My experiment was just putting a static local PyObject* var here ignoring the leak.)

[neonene]

self.executor_type(max_workers=numtasks) or some better way not to get stuck?

[ericsnowcurrently]

FWIW, I haven't been able to reproduce the failures locally. It certainly is consistent across all the CI workers though.

[ericsnowcurrently]

Well, I figured out the problem. Apparently ThreadPoolExecutor only starts new workers if there aren't any idle ones (and max_workers hasn't been reached), which is normally reasonable.

However, when a task is submitted, there's a narrow window from the time the task is queued up to when a worker picks up the job. If a second task is submitted and the code reaches ThreadPoolExecutor._adjust_thread_count() before the worker created for the first task picks it up then a second worker will not be started in response to the second submit() call.

The new test (test_blocking), which was hanging, was written with the assumption that there would be one worker for each task. This lead to an effective deadlock when there were fewer workers started. As noted by @neonene, the same situation was reachable by restricting the number of workers.

The solution was to unblock the tasks that did get picked up by workers, so they could finish and the workers could pick up the remaining queued tasks.

[miss-islington-app]

Thanks @ericsnowcurrently for the PR 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

[bedevere-app]

GH-135695 is a backport of this pull request to the 3.14 branch.