pip behaves poorly when legacy package does not generate a record file

[chrahunt]

Environment

• pip version: 19.3.1
• Python version: 3.8.0
• OS: Ubuntu 18.04

Should apply across all pip, Python, and OS versions.

Description

When pip installs a legacy package (calling setup.py install directly) it passes --record /tmp/path/to/install-record.txt. setuptools should be populating this with the list of installed files.

If a file is not generated, then pip silently continues processing the next requirement to install. This leads to:

1. If any pre-install uninstall occurred for this package then the backed-up files will be left behind in the site-packages directory
2. pip cannot uninstall the "installed" distribution if it got to the point of installing anything
3. users are not informed of the issue
4. really confusing behavior

Expected behavior

1. If pip installs a package that does not generate a record file, then it should fail installation, not continue.
2. Pip should roll back the uninstalled package.

In all likelihood this would result from user error, as demonstrated in the script below. In case it is not - both distutils and setuptools have supported this option for at least 10 years, so it's reasonable to expect legacy-installable packages to treat it correctly.

How to Reproduce

Run the following script

t.sh

#!/bin/sh

cd "$(mktemp -d)"

python -m venv env
./env/bin/python -V
./env/bin/python -m pip install --upgrade pip

site_packages="$(./env/bin/python -c 'import site; print(site.getsitepackages()[0])')"

ls -latr "$site_packages"

cat <<EOF > setup.py
from setuptools import setup

setup(
    name="hello",
    version="0.1.0",
)
EOF

./env/bin/python -m pip install .
ls -latr "$site_packages"

./env/bin/python -m pip install --install-option=--help .
ls -latr "$site_packages"
./env/bin/python -m pip freeze

Output

Output

Python 3.8.0
Collecting pip
  Using cached https://files.pythonhosted.org/packages/00/b6/9cfa56b4081ad13874b0c6f96af8ce16cfbc1cb06bedf8e9164ce5551ec1/pip-19.3.1-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 19.2.3
    Uninstalling pip-19.2.3:
      Successfully uninstalled pip-19.2.3
Successfully installed pip-19.3.1
total 36
drwxrwxr-x 3 chris chris 4096 Dec  7 13:14 ..
-rw-rw-r-- 1 chris chris  126 Dec  7 13:14 easy_install.py
drwxrwxr-x 5 chris chris 4096 Dec  7 13:14 pkg_resources
drwxrwxr-x 6 chris chris 4096 Dec  7 13:14 setuptools
drwxrwxr-x 2 chris chris 4096 Dec  7 13:14 __pycache__
drwxrwxr-x 2 chris chris 4096 Dec  7 13:14 setuptools-41.2.0.dist-info
drwxrwxr-x 5 chris chris 4096 Dec  7 13:14 pip
drwxrwxr-x 2 chris chris 4096 Dec  7 13:14 pip-19.3.1.dist-info
drwxrwxr-x 8 chris chris 4096 Dec  7 13:14 .
Processing /tmp/user/1000/tmp.iMwXry00bF
Installing collected packages: hello
    Running setup.py install for hello ... done
Successfully installed hello-0.1.0
total 40
drwxrwxr-x 3 chris chris 4096 Dec  7 13:14 ..
-rw-rw-r-- 1 chris chris  126 Dec  7 13:14 easy_install.py
drwxrwxr-x 5 chris chris 4096 Dec  7 13:14 pkg_resources
drwxrwxr-x 6 chris chris 4096 Dec  7 13:14 setuptools
drwxrwxr-x 2 chris chris 4096 Dec  7 13:14 __pycache__
drwxrwxr-x 2 chris chris 4096 Dec  7 13:14 setuptools-41.2.0.dist-info
drwxrwxr-x 5 chris chris 4096 Dec  7 13:14 pip
drwxrwxr-x 2 chris chris 4096 Dec  7 13:14 pip-19.3.1.dist-info
drwxrwxr-x 9 chris chris 4096 Dec  7 13:14 .
drwxrwxr-x 2 chris chris 4096 Dec  7 13:14 hello-0.1.0-py3.8.egg-info
/tmp/user/1000/tmp.iMwXry00bF/env/lib/python3.8/site-packages/pip/_internal/commands/install.py:283: UserWarning: Disabling all use of wheels due to the use of --build-options / --global-options / --install-options.
  cmdoptions.check_install_build_global(options)
Processing /tmp/user/1000/tmp.iMwXry00bF
Installing collected packages: hello
  Found existing installation: hello 0.1.0
    Uninstalling hello-0.1.0:
      Successfully uninstalled hello-0.1.0
    Running setup.py install for hello ... done
Successfully installed hello
total 40
drwxrwxr-x 3 chris chris 4096 Dec  7 13:14  ..
-rw-rw-r-- 1 chris chris  126 Dec  7 13:14  easy_install.py
drwxrwxr-x 5 chris chris 4096 Dec  7 13:14  pkg_resources
drwxrwxr-x 6 chris chris 4096 Dec  7 13:14  setuptools
drwxrwxr-x 2 chris chris 4096 Dec  7 13:14  __pycache__
drwxrwxr-x 2 chris chris 4096 Dec  7 13:14  setuptools-41.2.0.dist-info
drwxrwxr-x 5 chris chris 4096 Dec  7 13:14  pip
drwxrwxr-x 2 chris chris 4096 Dec  7 13:14  pip-19.3.1.dist-info
drwxrwxr-x 2 chris chris 4096 Dec  7 13:14 '~ello-0.1.0-py3.8.egg-info'
drwxrwxr-x 9 chris chris 4096 Dec  7 13:14  .
WARNING: Could not generate requirement for distribution -ello 0.1.0 (/tmp/user/1000/tmp.iMwXry00bF/env/lib/python3.8/site-packages): Parse error at "'-ello==0'": Expected W:(abcd...)

Notice the presence of '~ello-0.1.0-py3.8.egg-info' in site-packages and the warning on pip freeze as described in #7269.

[winsonluk]

I think the cause of this is most often req_uninstall calling misc::renames. If a user doesn't have permission to the package in site-packages (e.g., package was installed with sudo pip but the user is running pip), the renames function creates a temp directory (e.g., '~ello-0.1.0-py3.8.egg-info'), then raises an exception without deleting it.

#9393 fixes this by raising the exception before the temp directory is created.

[uranusjr]

Note that setup.py install is currently being deprecated in its entirety, and will be removed in the future (timeline undetermined). A PR to detect this is still welcomed, however, so feel free to send one in if you are interested in seeing this resolved.