pip download --no-deps --no-binary does some unwanted build steps

[altendky]

Environment

• pip version: 20.0.2
• Python version: 3.8.2
• OS: Linux

• Ubuntu eoan
• pyenv installed cpython

Description
I would like to download an sdist from PyPI without any other activity. pip download --no-deps --no-binary :all: cryptography ends up 'installing' build dependencies and more. The example below is done with the cryptography package because it is faster than my real case of PyQt5.

Expected behavior
Pip should exclusively download the sdist.

How to Reproduce
Run pip download --no-deps --no-binary :all: cryptography.

Output

altendky@p1:~/blue$ ll
total 8
drwxrwxr-x  2 altendky altendky 4096 Apr  7 10:04 ./
drwxr-xr-x 55 altendky altendky 4096 Apr  7 09:59 ../

altendky@p1:~/blue$ virtualenv -p python3 venv
created virtual environment CPython3.8.2.final.0-64 in 124ms
  creator CPython3Posix(dest=/home/altendky/blue/venv, clear=False, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/home/altendky/.local/share/virtualenv/seed-app-data/v1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

altendky@p1:~/blue$ venv/bin/pip download --no-deps --no-binary :all: cryptography
Collecting cryptography
  Using cached cryptography-2.9.tar.gz (517 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
  Saved ./cryptography-2.9.tar.gz
Successfully downloaded cryptography

altendky@p1:~/blue$ ll
total 520
drwxrwxr-x  3 altendky altendky   4096 Apr  7 10:05 ./
drwxr-xr-x 55 altendky altendky   4096 Apr  7 09:59 ../
-rw-rw-r--  1 altendky altendky 517211 Apr  7 10:05 cryptography-2.9.tar.gz
drwxrwxr-x  4 altendky altendky   4096 Apr  7 10:04 venv/

altendky@p1:~/blue$ venv/bin/pip download --no-deps --no-binary cryptography cryptography
Collecting cryptography
  File was already downloaded /home/altendky/blue/cryptography-2.9.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Successfully downloaded cryptography

altendky@p1:~/blue$ venv/bin/pip --version
pip 20.0.2 from /home/altendky/blue/venv/lib/python3.8/site-packages/pip (python 3.8)

altendky@p1:~/blue$ venv/bin/python --version --version
Python 3.8.2 (default, Mar  7 2020, 20:18:24) 
[GCC 9.2.1 20191008]

altendky@p1:~/blue$ venv/bin/pip download --no-deps --no-binary=:all: cryptography
Collecting cryptography
  File was already downloaded /home/altendky/blue/cryptography-2.9.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Successfully downloaded cryptography```

[uranusjr]

The issue title seems a bit inaccurate. The build process is necessary to discover dependencies, so it is not unwanted for pip download --no-binary. It’s --no-deps that makes it unwanted.

[altendky]

Thanks for the interest and correction.

[techalchemy]

From a UX standpoint, it may seem confusing that pip responds to the download command by informing you that it is building something -- as a user, if I requested pip to download a package, it may not be completely clear to me why a build would take place. I haven't directly requested anything to be built -- perhaps that is a UX issue and the messaging could be made clearer (e.g. pip could inform the user that it is computing dependencies to download rather than that it is building something, unless it is run in verbose mode?).

I agree with the technical constraints behind computing dependencies here, and I don't really want to get into the behavioral expectations a user might have (i.e. whether you would actually expect to get dependencies along with the main package in the first place) -- I only want to add that I, too, have found this confusing in the past. I often need to only acquire the bits behind a package and there is no clear way to accomplish this.

[altendky]

I don't know if I would want the 'build' output totally hidden but I could see it being useful to add a tier for the dependency collection. At which point --no-deps -> Collecting dependencies looks even more specifically weird. :]

Is there any case where it is useful to collect dependencies when --no-deps is specified? How about a case where it is useful to build a package when not collecting dependencies?

[uranusjr]

Is there any case where it is useful to collect dependencies when --no-deps is specified?

No, pip is just not smart enough to not do it. The “problem” here is that pip download simply reuses code from pip install and just skips the actual install part. For pip install you would always want to build the package even with --no-deps, and the output makes perfect sense there. I’m not quite sure how to fix this either, since it would break the abstraction if we tell whether the underlying implementation whether the message needs to be shown based on whether we are downloading and installating. I wonder if this can be solve with clever logger hierarchy, but that could be over-engineering things as well.

[pradyunsg]

I'm pretty sure there's an issue already for this and I think that's marked blocked on #988.

Apologies if that turns out to be not-correct, but I'm fairly certain that we'd end up fixing this as part of fixing #988 (which is ongoing work).

/cc @pfmoore and I see that @uranusjr has commented already.

[uranusjr]

we'd end up fixing this as part of fixing #988

I would hope so, but we might need more than the new resolver to fix this. To avoid showing the message, the resolver needs to not prepare the candidate. That’s not too hard, but then how do we ensure the candidate gets prepared for install? I’m under the impression that the current code base relies on the resolver to do that.

[pfmoore]

I would have thought that (if it's not already happening that way) the new resolver code will only prepare what it needs (to get dependencies - we originally eagerly prepared to get version/name, but we've changed that now). That means that the candidates that get selected for install, particularly when --no-deps is specified¹, might not be prepared yet. We may want to consider preparing the rest during the "build the list of InstallRequirements to hand back" phase of the new resolver.

¹ Have we implemented --no-deps yet, or is that another TODO?

[uranusjr]

#8387 contains some disussion on this as well. A discussion also too place in Discourse, which led to a PEP submission that may improve the situation.

[pradyunsg]

I'm gonna consolidate this into #1884 by expanding that from just setup.py-based builds to covering this whole topic -- this is basically that but for pyproject.toml based builds.