Feature request: keep track of how packages are installed (manually, automatically, ...)

[bersbersbers]

What's the problem this feature will solve?
Large requirement files with many packages are a pain to install from. I am just migrating from Python 3.8 to 3.9 using a requirements file with 280 entries, and it's not going well. I have already relaxed or removed some requirements manually, and at one point, pip tried to downgrade itself to some version 6.x.

I think the problem is due to the new resolver and the fact that finding a working combination of 280 packages is hard. In addition, there are cases (such as when upgrading all installed packages) when it is unclear if A should be upgraded at the cost of downgrading (or not upgrading) B, or B should be upgraded at the cost of downgrading A. All of this will become more complicated once pip considers installed packages.

What I am missing in pip is some prioritization of packages. One manual way of doing this (in my case) is to include only packages at the top level of pipdeptree in the requirements file. This way, pip could focus on installing (new) versions of packages I am actually interested in using.

However, pipdeptree is only one indication - I may still be using some of the dependent packages directly. A better way would be to keep track of what the user actually installed themselves. Hence, when issuing pip install tensorflow, pip should remember tensorflow as a high-priority/high-interest package, while six is an automatically installed dependency.

How is this useful? When considering package upgrades, the following may happen:
We have A==2 and B==2 installed. When upgrading both A and B, we may have A==3 available which requires B==2, and B==3 which requires A==2. [I notice this is a tricky example due to potential cyclic dependencies, but one can easily construct a similar example using two incompatible packages C and D on which A==3 and B==3 depend, respectively. The point is that ...] ...installing A==3 and B==3 is thus impossible, and the resolver needs to determine which package to upgrade. I have no idea how this is currently done, but knowing that A (tensorflow) was installed manually and B (six) is only a dependency should allow the resolver to prioritize the upgrade of A.

Excuse me if this has been discussed before, but I have not found any such discussion here yet.

Describe the solution you'd like
apt list --installed shows something like this:

x11-utils/focal,now 7.7+5 amd64 [installed]
xauth/focal,now 1:1.1-0ubuntu1 amd64 [installed,automatic]

While I installed x11-utils myself, xauth was installed automatically.

This information should be tracked by pip in my opinion, and used in later operations.

Users should then also have tools to (un)mark an installed package as relevant/prioritized/... (or just "automatic" is that is functionally the same thing).

[uranusjr]

pip already does keep track of this, see #8026. It turns out the idea of “manually installed” is not really that useful due to how installation works, but feel free to experiement with it and see what can be done.

[bersbersbers]

Thank you - https://www.python.org/dev/peps/pep-0376/#requested is a valuable resource. I'll use that for some further reading.

[pradyunsg]

Closing since I don't think we have anything actionable here.

[bersbersbers]

@uranusjr

It turns out the idea of “manually installed” is not really that useful due to how installation works

What do you mean by "due to how installation works"? I understand that once you install from a pip freeze requirements file, the whole concept is useless since every single dependency counts as requested. (I for one have some 150 REQUESTED files in my fresh Python installation, probably for that reason.) Is that the problem you are referring to?

If so, maybe pip freeze (and/or pip list) could be amended to, optionally, output only requested packages? pip list --not-required does something similar already, but by a another mechanism (similar to what I described using pipdeptree above).

Maybe pip freeze could even output the requested information, and pip install could use it, when generating and installing from a requirements file.

but feel free to experiement with it and see what can be done.

I will, thank you. Ultimately, I hope this can speed up the resolver, or at least give it some idea from which direction best to approach the vast space of version combination.

For my reference, are there any features which use the requested thing already?

[uranusjr]

Sorry I was probably too brief in the previous comment. What I meant to say was the feature was implemented for something else, but that thing ended up requiring much more information than simply “did the user specify the package from the command line”, so the REQUESTED file ended up not being unused anywhere, and therefore not really well-tested in practical environments. So yeah, feel free to experiement on it and maybe show the information in pip list etc., but don’t be surprised if you run into some inaccuracies. You should anticipate users complaining about issues once the feature is released, and be ready to either explain to them the subtle difference between your understanding of “manually installed” to theirs (for example, are packages installed from a requirements.txt user-requested or transitive?)