[BUG] new release (57) breaks piptools pytest

[nicoa]

First things first: please let me know if there is any information missing - It's kind of hard to me to trace this down to the actual bug or to an minimum example of what is failing unfortunately.

setuptools version

setuptools==57.0.0

Python version

3.6,3.7,3.8,3.9

OS

Ubuntu, Windows, MacOS

Additional environment information

Hey, please see this issue where it was discovered: jazzband/pip-tools#1403

First, it was wrongly attributed to be caused by pip version deprecation and feature flag, but it seems to be due to new setuptools (details in description). So currently all CI runners are broken there in that project.

• Last successful run: https://github.com/jazzband/pip-tools/actions/runs/870122483
• First failing: https://github.com/jazzband/pip-tools/actions/runs/873544759
• run which was pinned to setuptools<57 and ran through: https://github.com/jazzband/pip-tools/actions/runs/885789425

Description

Unfortunately, I'm not exactly aware of what happens behind the scenes on piptools testing suite. I'd like to share the code I could reproduce locally so far by running some of the tests manually in the fields below.

Expected behavior

There shouldn't be anything failing because it worked on the old version before.

How to Reproduce

1. in piptools package open tests/test_cli_compile.py and copy the imports from above and the code from the function test_realistic_complex_sub_dependencies, done in the following in detail:
2. navigate to tests folder and run the following:

import os
import subprocess
import sys
from textwrap import dedent
from unittest import mock
import pytest
from pip._internal.utils.urls import path_to_url
from piptools.scripts.compile import cli
from constants import MINIMAL_WHEELS_PATH, PACKAGES_PATH
from click.testing import CliRunner

3. create runner = CliRunner(mix_stderr=False)
4. run

    wheels_dir = "wheels"
    # make a temporary wheel of a fake package
    subprocess.run(
        [
            "pip",
            "wheel",
            "--no-deps",
            "-w",
            wheels_dir,
            os.path.join(PACKAGES_PATH, "fake_with_deps", "."),
        ],
        check=True,
    )
    with open("requirements.in", "w") as req_in:
        req_in.write("fake_with_deps")  # require fake package

    out = runner.invoke(cli, ["-n", "--rebuild", "-f", wheels_dir])

which fails.

Output

expected

In [39]: out.stderr
Out[39]: '#\n# This file is autogenerated by pip-compile\n# To update, run:\n#\n#    pip-compile --find-links=wheels\n#\n--find-links wheels\n\ncolorama==0.3.9\n    # via fake-with-deps\ncornice==1.0.0\n    # via fake-with-deps\nenum34==1.1.6\n    # via fake-with-deps\nfake-with-deps==0.1\n    # via -r requirements.in\nfutures==3.0.5\n    # via fake-with-deps\ngreenlet==1.1.0\n    # via sqlalchemy\nipaddress==1.0.23\n    # via fake-with-deps\njsonschema==2.6.0\n    # via fake-with-deps\npastedeploy==2.1.1\n    # via pyramid\npyramid==1.5.8\n    # via\n    #   cornice\n    #   fake-with-deps\npython-dateutil==2.4.2\n    # via fake-with-deps\npython-memcached==1.59\n    # via fake-with-deps\npyzmq==14.7.0\n    # via fake-with-deps\nrepoze.lru==0.7\n    # via pyramid\nsimplejson==3.17.2\n    # via\n    #   cornice\n    #   fake-with-deps\nsix==1.8.0\n    # via\n    #   fake-with-deps\n    #   python-dateutil\n    #   python-memcached\nsqlalchemy==1.4.15\n    # via fake-with-deps\ntranslationstring==1.4\n    # via pyramid\nvenusian==3.0.0\n    # via pyramid\nwebob==1.8.7\n    # via pyramid\nxmltodict==0.11.0\n    # via fake-with-deps\nzope.deprecation==4.4.0\n    # via pyramid\nzope.interface==5.4.0\n    # via pyramid\n\n# The following packages are considered to be unsafe in a requirements file:\n# setuptools\nDry-run, so nothing updated.\n'

actual (failing) output

      ERROR: Command errored out with exit status 1:
       command: /Users/nalbers/git/pip-tools/thisvirtualenv/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/pd/gtn_zdjj2gn1rsbkhvtckhjc0000gp/T/pip-resolver-x2nbtsp4/futures/setup.py'"'"'; __file__='"'"'/private/var/folders/pd/gtn_zdjj2gn1rsbkhvtckhjc0000gp/T/pip-resolver-x2nbtsp4/futures/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/pd/gtn_zdjj2gn1rsbkhvtckhjc0000gp/T/pip-pip-egg-info-nkzz3eb8
           cwd: /private/var/folders/pd/gtn_zdjj2gn1rsbkhvtckhjc0000gp/T/pip-resolver-x2nbtsp4/futures/
      Complete output (29 lines):
      Traceback (most recent call last):
        File "<string>", line 1, in <module>
        File "/Users/nalbers/git/pip-tools/thisvirtualenv/lib/python3.9/site-packages/setuptools/__init__.py", line 18, in <module>
          from setuptools.dist import Distribution
        File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
        File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
        File "/Users/nalbers/git/pip-tools/thisvirtualenv/lib/python3.9/site-packages/_virtualenv.py", line 89, in exec_module
          old(module)
        File "/Users/nalbers/git/pip-tools/thisvirtualenv/lib/python3.9/site-packages/setuptools/dist.py", line 32, in <module>
          from setuptools.extern.more_itertools import unique_everseen
        File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
        File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 666, in _load_unlocked
        File "<frozen importlib._bootstrap>", line 565, in module_from_spec
        File "/Users/nalbers/git/pip-tools/thisvirtualenv/lib/python3.9/site-packages/setuptools/extern/__init__.py", line 52, in create_module
          return self.load_module(spec.name)
        File "/Users/nalbers/git/pip-tools/thisvirtualenv/lib/python3.9/site-packages/setuptools/extern/__init__.py", line 37, in load_module
          __import__(extant)
        File "/Users/nalbers/git/pip-tools/thisvirtualenv/lib/python3.9/site-packages/setuptools/_vendor/more_itertools/__init__.py", line 1, in <module>
          from .more import *  # noqa
        File "/Users/nalbers/git/pip-tools/thisvirtualenv/lib/python3.9/site-packages/setuptools/_vendor/more_itertools/more.py", line 5, in <module>
          from concurrent.futures import ThreadPoolExecutor
        File "/private/var/folders/pd/gtn_zdjj2gn1rsbkhvtckhjc0000gp/T/pip-resolver-x2nbtsp4/futures/concurrent/futures/__init__.py", line 8, in <module>
          from concurrent.futures._base import (FIRST_COMPLETED,
        File "/private/var/folders/pd/gtn_zdjj2gn1rsbkhvtckhjc0000gp/T/pip-resolver-x2nbtsp4/futures/concurrent/futures/_base.py", line 357
          raise type(self._exception), self._exception, self._traceback
                                     ^
      SyntaxError: invalid syntax
      ----------------------------------------

Code of Conduct

• I agree to follow the PSF Code of Conduct

[atugushev]

git bisect detects 3544de7 commit. Minimal reproducer:

# setup.py
from setuptools import setup

setup(
    name="foo",
    version=0.1,
    install_requires=[
        "futures<3.1,>=3.0.3",
    ],
)

$ pip install -e . --no-cache --force-reinstall

[atugushev]

Ah, finally got it. Since 57 version setuptools uses more-itertools package, which uses/imports concurrent.futures module, which happens to be broken after pip install futures, since futures backports concurrent.futures to Python 2.

File "/usr/lib/setuptools/_vendor/more_itertools/more.py", line 5, in <module>
  from concurrent.futures import ThreadPoolExecutor

@jaraco Looks like it's more like a bug on https://github.com/agronholm/pythonfutures side. It probalby should use python_requires<3 in setup.py (agronholm/pythonfutures#95).

[jaraco]

Thanks for the investigation. Sounds like there's not much for Setuptools to do here. Feel free to comment if I'm mistaken and we can re-open.

[agronholm]

As pointed out on my side, futures already declares a maximum Python version.

[jaraco]

It sounds as if something in the build chain is not honoring the python-requires then.

[agronholm]

There are earlier releases that don't have this declaration, maybe that's why. @nicoa did you check which version was installed?

Maybe I should yank all those obsolete releases.

[atugushev]

It tries to install futures<3.1,>=3.0.3, but python_requires introduced in 3.2.0. It works fine with the latest futures version.

[agronholm]

It tries to install futures<3.1,>=3.0.3, but python_requires introduced in 3.2.0. It works fine with the latest futures version.

So what exactly has this dependency? pip-tools doesn't, setuptools doesn't, more-itertools doesn't.

[agronholm]

I made a post-release that retroactively adds python-requires, and yanked 3.1.1. The earlier releases should be syntax compatible at least.

[PatTheMav]

I still get the reported issue with Python 3.9 and a package that specifies futures>=3.1.0 in its requirements.txt - with setuptools v57+ installed, all versions down to futures v2.2.0 are iterated through (each one failing with SyntaxError: invalid syntax), which also leads to an older version of the original package being installed.

I couldn't figure out how >= 3.1.0 which should yield at least the new 3.1.1 release or any more recent release that introduced the python-requires statement.

The only way to fix this for me was to downgrade setuptools to a version older than 57 - futures will not be installed and the correct version of the original package is installed.

[agronholm]

What's the simplest way to repro this?

[PatTheMav]

Try pip install modoboa-dmarc with pip 21.3 and setup tools 58.2.0, I run this in a Python 3.9 virtualenv.

[agronholm]

Ok so...if I was to yank 3.1.0, what would happen? The installation would fail because there is no matching release.

[agronholm]

What also greatly confuses me is that 3rd party packages aren't supposed to override stdlib imports. Why is that happening here?

[PatTheMav]

What also greatly confuses me is that 3rd party packages aren't supposed to override stdlib imports. Why is that happening here?

From what I could tell that seems like the main issue - if I'm not mistaken, specifying >=3.1.0 will resolve to 3.1.0 as the only viable candidate under Python 3. Under normal circumstances import concurrent.futures seems to work fine, even with futures being installed (probably due to stdlib imports taking precedence as you mentioned before), so somehow this doesn't apply to setuptools/more_itertools.

EDIT: Turns out that the current working directory takes precedence even over the standard library. So if I create a module called math.py and run import math in an interpreter in the same directory, my own module is imported.

What seems to happen here is that setuptools runs in the temporary install directory (pip-resolver-x2nbtsp4/futures/ in the example above) which contains the actual local path "concurrent/futures". The import statement of more_itertools will thus pick up the futures implementation of futures itself and lead to the SyntaxError.

[jaraco]

I can more readily replicate the issue by simply attempting to install futures with the latest setuptools present:

~ $ pip-run -q setuptools -- -m pip-run futures==3.1
Collecting futures==3.1
  Using cached futures-3.1.0.tar.gz (25 kB)
  Preparing metadata (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /Library/Frameworks/Python.framework/Versions/3.10/bin/python3.10 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-ordanrh8/futures_5a939d1c0bd04c8dbe3984bd609d13ce/setup.py'"'"'; __file__='"'"'/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-ordanrh8/futures_5a939d1c0bd04c8dbe3984bd609d13ce/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-pip-egg-info-nmdijp9m
       cwd: /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-ordanrh8/futures_5a939d1c0bd04c8dbe3984bd609d13ce/
  Complete output (24 lines):
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-rqsqhjcq/setuptools/__init__.py", line 18, in <module>
      from setuptools.dist import Distribution
    File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-rqsqhjcq/setuptools/dist.py", line 32, in <module>
      from setuptools.extern.more_itertools import unique_everseen
    File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
    File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 674, in _load_unlocked
    File "<frozen importlib._bootstrap>", line 571, in module_from_spec
    File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-rqsqhjcq/setuptools/extern/__init__.py", line 52, in create_module
      return self.load_module(spec.name)
    File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-rqsqhjcq/setuptools/extern/__init__.py", line 37, in load_module
      __import__(extant)
    File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-rqsqhjcq/setuptools/_vendor/more_itertools/__init__.py", line 1, in <module>
      from .more import *  # noqa
    File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-rqsqhjcq/setuptools/_vendor/more_itertools/more.py", line 5, in <module>
      from concurrent.futures import ThreadPoolExecutor
    File "/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-ordanrh8/futures_5a939d1c0bd04c8dbe3984bd609d13ce/concurrent/futures/__init__.py", line 8, in <module>
      from concurrent.futures._base import (FIRST_COMPLETED,
    File "/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-ordanrh8/futures_5a939d1c0bd04c8dbe3984bd609d13ce/concurrent/futures/_base.py", line 357
      raise type(self._exception), self._exception, self._traceback
                                 ^
  SyntaxError: invalid syntax
  ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/25/83/113da7c7832a4efa80ec7fd5d25b2ea431d9a847cd78f1fa98ddd99d591a/futures-3.1.0.tar.gz#sha256=90a8c71762d53e4ebdb4f7893460761f4ce2f076df08bb4934f39e90464eb889 (from https://pypi.org/simple/futures/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement futures==3.1 (from versions: 0.2.python3, 0.1, 0.2, 1.0, 2.0, 2.1, 2.1.1, 2.1.2, 2.1.3, 2.1.4, 2.1.5, 2.1.6, 2.2.0, 3.0.0, 3.0.1, 3.0.2, 3.0.3, 3.0.4, 3.0.5, 3.1.0, 3.1.1)
ERROR: No matching distribution found for futures==3.1
...

[jaraco]

I note that employing --use-pep517 for the install of futures works around the issue:

~ $ pip-run -q setuptools -- -m pip-run --use-pep517 futures==3.1 -- -c "print('worked')"
Collecting futures==3.1
  Using cached futures-3.1.0-py3-none-any.whl
Installing collected packages: futures
Successfully installed futures-3.1.0
worked

Here's what I think is happening:

• The futures distribution does not supply a pyproject.toml, so builds fallback to the legacy installer.
• The legacy installer retains the legacy behavior that during install of a package, that package's root is added to sys.path.
• During the install of futures, more_itertools gets concurrent.futures from the local build of futures.

Next, I'll attempt to replicate the failure without pip by simulating what pip is doing.

[jaraco]

Indeed, I can replicate the issue without pip thus:

draft $ http https://files.pythonhosted.org/packages/25/83/113da7c7832a4efa80ec7fd5d25b2ea431d9a847cd78f1fa98ddd99d591a/futures-3.1.0.tar.gz | tar xz
draft $ cd futures-3.1.0/
futures-3.1.0 $ pip-run -q setuptools -- setup.py egg_info
/Users/jaraco/draft/futures-3.1.0/setup.py:6: UserWarning: This backport is meant only for Python 2.
Python 3 users do not need it, as the concurrent.futures package is available in the standard library.
  warn('This backport is meant only for Python 2.\n'
Traceback (most recent call last):
  File "/Users/jaraco/draft/futures-3.1.0/setup.py", line 12, in <module>
    from setuptools import setup
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-ug2uyft1/setuptools/__init__.py", line 18, in <module>
    from setuptools.dist import Distribution
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-ug2uyft1/setuptools/dist.py", line 32, in <module>
    from setuptools.extern.more_itertools import unique_everseen
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 674, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 571, in module_from_spec
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-ug2uyft1/setuptools/extern/__init__.py", line 52, in create_module
    return self.load_module(spec.name)
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-ug2uyft1/setuptools/extern/__init__.py", line 37, in load_module
    __import__(extant)
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-ug2uyft1/setuptools/_vendor/more_itertools/__init__.py", line 1, in <module>
    from .more import *  # noqa
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-ug2uyft1/setuptools/_vendor/more_itertools/more.py", line 5, in <module>
    from concurrent.futures import ThreadPoolExecutor
  File "/Users/jaraco/draft/futures-3.1.0/concurrent/futures/__init__.py", line 8, in <module>
    from concurrent.futures._base import (FIRST_COMPLETED,
  File "/Users/jaraco/draft/futures-3.1.0/concurrent/futures/_base.py", line 357
    raise type(self._exception), self._exception, self._traceback
                               ^
SyntaxError: invalid syntax
futures-3.1.0 $ python -c "import concurrent; print(concurrent.__file__)"
/Users/jaraco/draft/futures-3.1.0/concurrent/__init__.py

[jaraco]

So if futures is installed, it won't take precedence to the stdlib (site-packages occurs after builtins), but if futures is in ., it will take precedence.

[jaraco]

And the only reason this issue was not an issue before was because nothing in Setuptools attempted to import concurrent.futures until more_itertools was added, so the failing code path wasn't triggered. Based on my reading of the package, it should never be installed on Python 3, so the packages that are depending on it should stop doing so for Python 3.

[PatTheMav]

Yep exactly, it's the perfect storm of setuptools using a dependency that imports concurrent.futures since v57 and root imports having higher priority than stdlib imports (which "protected" Py3 against this incompatible package so far).

Only installing the futures package itself could lead to this issue, as it's the only one containing an actual concurrent.futures module (or more precisely, any package that has such a module with code that isn't Py3 compatible) available at the project root.