-
-
Notifications
You must be signed in to change notification settings - Fork 2.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Test using multiprocessing only hangs when ran with test suite, works alone #11174
Comments
do the hangups also happen with |
yup. Traceback is not very helpful, since it's just the timeout one, but I'll still share for reference Traceback: Stack of Threads
FAILURES
|
Not sure either, but perhaps using a different timeout method might yield different results? |
The timeout was to avoid the specific test hanging forever, so the test suite can finish. I actually installed a timeout module (pytest-timeout: With this in mind, I honestly don't understand how the timeout method would be related to the bug :/ |
Out of insights then, sorry. |
Same is happening for me with |
@lorinkoz Otherwise, make sure the following code happens before any import multiprocessing as mp
mp.set_start_method("spawn", force=True) Those are the only two things that come to mind, hope it helps. |
Thanks you this helped me |
@rabyj thanks for sharing this. I had a similar problem using the |
EDIT: I found that if I make sure to explicitly close all files that are opened in a sub-process, I no longer need to set I ran into this problem as well. However, |
Closing this for now then. For the record: pytest just runs each test function in sequence, if the test function is blocking somehow, probably the code being tested needs some adjustment -- my point is that pytest does not have a "known issue" with multiprocessing at all, it all depends on the user code running inside the tests. If this issue comes up again, it is probably best to work on a MWE so a proper solution can be discussed. |
… proposed solution from pytest-dev/pytest#11174
for future visitors: I ran into this issue and the |
@pierrebarbera freezegun was not part of my modules, seems like a package specific problem. Then again, maybe MY problem was package specific but I never found out the culprit |
TL;DR:
If you're experiencing a hanging issue during tests with a program using multiprocessing, try adding the following code at the beginning of your problematic test function:
Description
I've encountered an odd test hanging issue with a program that uses multiprocessing.
The program completes:
The program hangs indefinitely when run as part of a full suite of tests.
I wanted to at least document what I experienced even if I can't create a simple minimal example, since I found a workaround that could help others, and no other issue mentioning something like this. All issues are for single tests.
Environment
python:
3.8.10
pytest:
7.2.0
pytest-mock:
3.10.0
OS:
Ubuntu 20.04.6 LTS
Details
The problem occurs with a test that runs a program to completion, which internally uses multiprocessing (via pool.starmap on a generator). When the test is run individually, it completes without problems within roughly 5 seconds. However, when the test is included in a full test suite and run with pytest, it hangs indefinitely (for over 6 minutes at least).
The issue persists even when using
concurrent.futures
in place ofmultiprocessing
(without apartial
as seen below).My workaround is using
mp.set_start_method("spawn", force=True)
at the start of the test.Fix in immediate context:
While the fix makes the test run slower (about 20 seconds), at least it doesn't hang indefinitely.
I'm still uncertain as to why this issue occurs and why the fix works, so I would greatly appreciate any insights!
Complete pip Environment
pip list output - click me
The text was updated successfully, but these errors were encountered: