pytest-xdist workers crashing

[github-actions]

Workflow Run URL

[jrbourbeau]

tests/benchmarks/test_csv.py::test_csv_basic and tests/benchmarks/test_dataframe.py::test_dataframe_align caused pytest-xdist workers to crash for some reason. We've started seeing this with other tests on PRs as well. It's not clear why this is happening. It's not always these tests that cause pytest worker crashes -- other tests do as well.

[gw2] node down: Not properly terminated
[gw2] [  4%] FAILED tests/benchmarks/test_csv.py::test_csv_basic 

replacing crashed worker gw2

[gw3] node down: Not properly terminated
[gw3] [  4%] FAILED tests/benchmarks/test_dataframe.py::test_dataframe_align 

replacing crashed worker gw3

[jrbourbeau]

We saw something similar in #412 with tests/benchmarks/test_array.py::test_anom_mean

[gw0] node down: Not properly terminated
[gw0] [  4%] FAILED tests/benchmarks/test_array.py::test_anom_mean 

replacing crashed worker gw0

[jrbourbeau]

We saw something similar in #413 with tests/benchmarks/test_zarr.py::test_select_scalar

[gw1] node down: Not properly terminated
[gw1] [  4%] FAILED tests/benchmarks/test_zarr.py::test_select_scalar 

replacing crashed worker gw1

[ncclementi]

Looking at it quickly, It seems it is a known issue and is still open pytest-dev/pytest-xdist#466
someone else reported here too pytest-dev/pytest-xdist#714 and didn't get much attention either.

Not quite sure how to proceed here. There are other issues opened or closed with no answer related to this.

• Slaves crash on win64 with error "Not properly terminated" pytest-dev/pytest-xdist#70
• Another potential helpful issue Running tests fails with "node down: Not properly terminated", maybe execnet related? pytest-dev/pytest#3216 that

[ian-r-rose]

It may just be that we have too many concurrent xdist workers. The theory was there should not be much work done on the client, so 8 workers is fine. But that theory might not be correct. In particular, I think that package_sync might be kind of expensive for the client. Some support for package_sync being expensive is that in #429 all of the worker crashes happen on the first test of the given module, which would be when the cluster is being spun up.

Two possible ways to alleviate this:

1. Reduce the number of xdist workers (try six or four?)
2. Revert Use single job for all test categories #370 and distribute the CI across more runners again.

Thoughts?

[ncclementi]

Reduce the number of xdist workers (try six or four?)

We can test this on a branch and run it every day for a few days and see if it fixes it.