pip wheel not working on macOS / Python 3.10

[wimglenn]

Description

pip fails loading the build backend (pip._vendor.pep517.wrappers.BackendUnavailable)

Expected behavior

python3 -m pip wheel . should create a myproject-0.1-py3-none-any.whl in the cwd.

pip version

22.0.4

Python version

3.10

OS

macOS Montery 12.3.1

How to Reproduce

Note: the failure is only reproducible outside of a venv. In a venv the build works 🤷

I have a minimal source tree:

$ ls                
pyproject.toml
$ cat pyproject.toml
[build-system]
requires = ["setuptools-ext"]
build-backend = "setuptools_ext"

[project]
name = "myproject"
version = "0.1"

Python 3.10 runtime from brew install python@3.10 and a clean pip installation

$ python3 -m pip --version           
pip 22.0.4 from /usr/local/lib/python3.10/site-packages/pip (python 3.10)
$ python3 -m pip freeze --all
pip==22.0.4
setuptools==62.1.0
$ python3 -m pip config list
$

But pip wheel fails:

$ python3 -m pip wheel .
Processing /private/tmp/s
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
ERROR: Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
    status = run_func(*args)
  File "/usr/local/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
    return func(self, options, args)
  File "/usr/local/lib/python3.10/site-packages/pip/_internal/commands/wheel.py", line 145, in run
    requirement_set = resolver.resolve(reqs, check_supported_wheels=True)
  File "/usr/local/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 75, in resolve
    collected = self.factory.collect_root_requirements(root_reqs)
  File "/usr/local/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 506, in collect_root_requirements
    req = self._make_requirement_from_install_req(
  File "/usr/local/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 468, in _make_requirement_from_install_req
    cand = self._make_candidate_from_link(
  File "/usr/local/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 215, in _make_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
  File "/usr/local/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 288, in __init__
    super().__init__(
  File "/usr/local/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 158, in __init__
    self.dist = self._prepare()
  File "/usr/local/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 227, in _prepare
    dist = self._prepare_distribution()
  File "/usr/local/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 299, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
  File "/usr/local/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 487, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
  File "/usr/local/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 556, in _prepare_linked_requirement
    dist = _get_prepared_distribution(
  File "/usr/local/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 58, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
  File "/usr/local/lib/python3.10/site-packages/pip/_internal/distributions/sdist.py", line 45, in prepare_distribution_metadata
    self._install_build_reqs(finder)
  File "/usr/local/lib/python3.10/site-packages/pip/_internal/distributions/sdist.py", line 104, in _install_build_reqs
    build_reqs = self._get_build_requires_wheel()
  File "/usr/local/lib/python3.10/site-packages/pip/_internal/distributions/sdist.py", line 81, in _get_build_requires_wheel
    return backend.get_requires_for_build_wheel()
  File "/usr/local/lib/python3.10/site-packages/pip/_vendor/pep517/wrappers.py", line 172, in get_requires_for_build_wheel
    return self._call_hook('get_requires_for_build_wheel', {
  File "/usr/local/lib/python3.10/site-packages/pip/_vendor/pep517/wrappers.py", line 332, in _call_hook
    raise BackendUnavailable(data.get('traceback', ''))
pip._vendor.pep517.wrappers.BackendUnavailable: Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 89, in _build_backend
    obj = import_module(mod_path)
  File "/usr/local/Cellar/python@3.10/3.10.2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'setuptools_ext'

Interestingly, the same call succeeds when done in a venv:

$ python3 -m venv .venv
$ source .venv/bin/activate
(.venv) $ python3 -m pip install -qU pip setuptools
(.venv) $ python3 -m pip freeze --all
pip==22.0.4
setuptools==62.1.0
(.venv) $ python3 -m pip --version
pip 22.0.4 from /private/tmp/s/.venv/lib/python3.10/site-packages/pip (python 3.10)
(.venv) $ python3 -m pip wheel .
Processing /private/tmp/s
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: myproject
  Building wheel for myproject (pyproject.toml) ... done
  Created wheel for myproject: filename=myproject-0.1-py3-none-any.whl size=965 sha256=a9471f72e821bcfbc1a8361a2a0f828c99270b1af73849f9f9b6d2af17369dda
  Stored in directory: /private/var/folders/l9/gsl6q67557s4vd8gh4pkqt2c0000gn/T/pip-ephem-wheel-cache-e0ligcse/wheels/91/4d/e6/5c1a7ca4b3201d662a4885a203f45ea99565680187670918b4
Successfully built myproject

Why is that?

Output

I've attached a file with increased verbosity enabled

PYTHONVERBOSE=1 python3 -m pip wheel -vvv . &> out.txt

--> out.txt

Note: There appears to be an assertion firing from this line, however even if I comment out that line the build fails the same way, so it might be a red herring..

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[uranusjr]

This is due to Homebrew using distutils.cfg to configure environment paths, which breaks how pip builds the isolated build environment. The config file is not effective when you're in a virtual environment.

[wimglenn]

crosslink python/cpython#88142 and Homebrew/homebrew-core#76621 (it says "as of 3.10.2, this issue no longer affects Python 3.10 and later" but I am on 3.10.2?)

[uranusjr]

They say that in the sense that users will no longer see warnings about it. It does not mean there are no problems 🙂

[wimglenn]

OK :) do you know how it is even finding a distutils.cfg? There don't seem to be any lying around for python@3.10

$ find /usr/local -name distutils.cfg    
/usr/local/Cellar/pypy/7.3.8/libexec/lib-python/2.7/distutils/distutils.cfg
/usr/local/Cellar/pypy3/7.3.8/libexec/lib-python/3/distutils/distutils.cfg
/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/distutils.cfg

(or perhaps you meant they were using it in < 3.10, and its non-existence is the cause of problems?)

[uranusjr]

Huh interesting. Last time I checked they were using distutils.cfg, maybe they changed something and now the cause is different.

[pradyunsg]

https://github.com/Homebrew/homebrew-core/blob/master/Formula/python@3.10.rb#L62

This is what they’re doing now.

[uranusjr]

Ah so basically The Debian Approach™

The situation is probably similar to pypa/setuptools#3269 I guess. I seem to recall we have an issue on using virtual environment for build isolation (or is it using build for PEP 517 in general?) and it seems like we increasingly need to do that soon.

[pradyunsg]

#10720 is what you might be thinking of.

[uranusjr]

This is likely related to #11539. You can try applying #11598 to see if whether it also fixed this.

[jaraco]

I did confirm that #11598 fixes the issue when I encountered it (https://gist.github.com/jaraco/c1301b607507706fc276df134374e30e).