Primer crashes with an unfriendly traceback if requirements for a project can't be installed

[AlexWaygood]

When run on Python 3.11, primer currently crashes with the following traceback (see python/typeshed#9336):

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/mypy_primer.py", line 996, in main
    retcode = inner()
              ^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/mypy_primer.py", line 989, in inner
    retcode = asyncio.run(coro)
              ^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/asyncio/base_events.py", line 650, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/mypy_primer.py", line 768, in primer
    result = await result_fut
             ^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/asyncio/tasks.py", line 605, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/mypy_primer.py", line 425, in primer_result
    await self.setup()
  File "/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/mypy_primer.py", line 364, in setup
    await run(
  File "/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/mypy_primer.py", line 112, in run
    raise subprocess.CalledProcessError(proc.returncode, cmd, output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '/tmp/mypy_primer/projects/_materialize_venv/bin/pip install -r ci/builder/requirements.txt' returned non-zero exit status 1.

This traceback arises from the fact that the requirements for the Materialize project can't be installed on Python 3.11. However, there's nothing in the traceback to indicate that it's the Materialize project that's the problem here (though admittedly it's not too hard to figure it out if you do a CTRL-F for ci/builder/requirements.txt in the mypy_primer source code).

Would you be open to a PR improving the error message if the requirements for a project can't be installed? Alternatively we could skip projects where the requirements can't be installed, but that seems like it could lead to errors going unnoticed, which would be bad.

[JelleZijlstra]

As a co-maintainer I'm going to say we definitely should make the error message better :)

We shouldn't silently skip projects that can't be installed, because then we may never notice when projects stop working. However, it might be reasonable to add some mechanism that allows us to skip Materialize if running under 3.11, if as typeshed maintainers we think the benefit of getting 3.11 coverage outweighs the cost of dropping this project.

[hauntsaninja]

Note if you were to run with 3.11, you may be affected by python/cpython#100133