Parallelize pytest runs

[webknjaz]

$sbj. This change makes a single pytest run almost 4x faster on my machine:

command	time
$ tox -e py -- -n auto	17.90s
$ tox -e py -- -n 0	68.54s (0:01:08)

I'm pretty sure it'll have a positive impact on the CI completion time too. UPD: in GHA it's about 2x faster in this mode.

Changelog-friendly one-liner: (misc/non user facing?) Enabled pytest to run tests in parallel.

Contributor checklist

• (not exactly but...)Provided the tests for the changes.
• (likely irrelevant)Gave a clear one-line description in the PR (that the maintainers can add to CHANGELOG.md on release).
• (irrelevant)Assign the PR to an existing or new milestone for the target version (following Semantic Versioning).

[codecov]

Codecov Report

Merging #1357 (f6cdc6b) into master (ed58917) will not change coverage.
The diff coverage is n/a.

❗ Current head f6cdc6b differs from pull request most recent head 8a52f62. Consider uploading reports for the commit 8a52f62 to get more accurate results

@@           Coverage Diff           @@
##           master    #1357   +/-   ##
=======================================
  Coverage   99.65%   99.65%           
=======================================
  Files          33       33           
  Lines        2915     2915           
  Branches      308      308           
=======================================
  Hits         2905     2905           
  Misses          5        5           
  Partials        5        5           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ed58917...8a52f62. Read the comment docs.

[atugushev]

Thanks for bringing this! I remember we made a while ago some preparations to make this possible. See #1022 (review).

[atugushev]

Thanks @webknjaz!

[AndydeCleyre]

Hey, I wasn't invited to the review party, but don't think this should have gone through without setting unique cache dirs for parallel pip-compile invocations, via --cache-dir.

---

EDIT: We already had unique cache dirs implemented, and that behavior should indeed have been expected, it just happened to have broken unnoticed, independently of this PR.

[AndydeCleyre]

All the pip-compile tests are unreliable in this state. I also notice this, which may be a problem:

class CliRunner(object):
    """The CLI runner provides functionality to invoke a Click command line
    script for unittesting purposes in a isolated environment.  This only
    works in single-threaded systems without any concurrency as it changes the
    global interpreter state.
    """

[AndydeCleyre]

And I see here we have a no-longer-helpful fixture for this:

@pytest.fixture(autouse=True)
def _temp_dep_cache(tmpdir, monkeypatch):
    monkeypatch.setenv("PIP_TOOLS_CACHE_DIR", str(tmpdir / "cache"))

But that env var was removed in d786232, merged in #1238, which silently broke this fixture.

[webknjaz]

I think it's good that this got merged because it reveals problems with tests that may depend on the shared state/order. This gives us a chance to identify the flaky tests and actually fix them over time. Plus, going forward it'll make the test authors more cautious and will encourage everyone to consider the test stability...

[webknjaz]

    works in single-threaded systems without any concurrency as it changes the
    global interpreter state.

I'm pretty sure pytest-xdist distributes tests across subprocesses, not threads. So this shouldn't be a problem.

[AndydeCleyre]

I'm thinking the real bug here is in #1238, where the intent seemed to be only to remove env vars handled by pip itself, and PIP_TOOLS_CACHE_DIR's removal was an oopsie. I propose we just re-enable the env var, and then I think there's nothing wrong with this parallel business here.