Multiprocessing queue doesn't make the program exit properly

[albertino87]

Bug report

Bug description:

Hi, I'm observing a strange behaviour with python multiprocessing Queue object.

My environment:

OS: Windows 10
python: 3.13.1

but I observed the same with:

OS: Windows 10
python: 3.12.7

and:

OS: Windows 10
python: 3.10.14

I have this short script:

from multiprocessing import Queue

a = list(range(716))
queue: Queue = Queue()
for item in a:
    queue.put(item)
raise ValueError(f"my len: {len(a)}")

If I run it, everything is ok, it raises the error and exits:

Traceback (most recent call last):
  File "C:\Users\uXXXXXX\AppData\Roaming\JetBrains\PyCharmCE2024.3\scratches\scratch_1.py", line 7, in <module>
    raise ValueError(f"my len: {len(a)}")
ValueError: my len: 716

Process finished with exit code 1

but if i change the number from 716 to 717 or any other number above it, it raises the error but doesn't exits, the scripts hangs there. and when i forcefully stop the scripts it exits with code -1

Traceback (most recent call last):
  File "C:\Users\uXXXXXX\AppData\Roaming\JetBrains\PyCharmCE2024.3\scratches\scratch_1.py", line 7, in <module>
    raise ValueError(f"my len: {len(a)}")
ValueError: my len: 717

Process finished with exit code -1

CPython versions tested on:

3.13

Operating systems tested on:

Windows

[ZeroIntensity]

I would assume it's because the operating system has a preset limit on how much data can be put into a pipe at once, but as such, there's nothing we can really do to fix it. You should read your queue before trying to shut down the program.

[duaneg]

I think this is a duplicate of #128186. FWIW I also agree with @ZeroIntensity that there isn't anything that we can realistically do about it.

[tim-one]

The docs warn about this already, and give two workarounds:

Warning As mentioned above, if a child process has put items on a queue (and it has not used JoinableQueue.cancel_join_thread), then that process will not terminate until all buffered items have been flushed to the pipe.

This means that if you try joining that process you may get a deadlock unless you are sure that all items which have been put on the queue have been consumed. Similarly, if the child process is non-daemonic then the parent process may hang on exit when it tries to join all its non-daemonic children.

Note that a queue created using a manager does not have this issue. See Programming guidelines.

So closing as "not planned". Python is not the OS, and has no control over how much data an OS pipe can buffer before it refuses to accept more. That's why behavior varies depending on much stuff you put on the queue. "It depends" then on the OS you're using, and how big the queued pickles are. Either arrange to drain (read) the queue before trying to quit, or use one of the other workarounds.