Update dependencies installed during build

[ericholscher]

Currently we have really out of date dependencies in our build process:

https://github.com/rtfd/readthedocs.org/blob/07dc26b1b7f7c1252e423ea7d5e904a007dd2f30/readthedocs/doc_builder/python_environments.py#L281-L295

We should figure out a better way to keep these up to date, similar to the dependencies we update in our requirements file. Perhaps a file in the repo that we read in python, and keep up to date similar to our requirements?

[humitos]

I think the best way for this is to have a requirements file that we can update manually or via pur. Although, this way can't contains any package based on a Feature flag.

Then, we can

1. open it and split line by line to load as a list and continue working as now or
2. do pip install -r but we will need to do cat reqs to show the requirements that we install so the user knows about them.

I think 1) is cleaner and simpler.

NOTE: this issue is about two different things: upgrade current dependencies installed and find a better way to store them.

[agjohnson]

Yeah, the addition of special logic makes this hard. I sort of like the requirements file idea. We might be able to use pip environment variable expansion in the requirements.txt to special case our feature flag packages.

[agjohnson]

Also, moving this to refactoring level. It's not an immediate priority for core.

[humitos]

I'm closing this issue in favor of #8103 --We are creating a new builder that won't install any dependency.

Also, we are talking about a way to modularize our current (magical) builder and version it. That way, old users could stick with the version that installs the Python library versions they are expecting and new users could use the newer version of the builder.