Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support equivalent of pip install --target=... to export distributions and source to flat directory #2110

Closed
huonw opened this issue Apr 2, 2023 · 3 comments · Fixed by #2140
Assignees

Comments

@huonw
Copy link
Collaborator

huonw commented Apr 2, 2023

pip install has a --target/-t option that installs the requested requirements directly to a specific directory, rather than into the usual structure with site-packages etc. It would be nice if pexes could support this too. Currently, there's two related PEX_TOOLS=1 commands, but neither quite do exactly this:

  • PEX_TOOLS=1 ./some-pex venv ...: constructs the full venv structure
  • PEX_TOOLS=1 ./some-pex repository extract ...: only exports the distributions, not the source

Motivation: Direct installation like this is handy for building artefacts for certain cases, such as FaaS platforms like Lambda: their docs recommend just splatting all dependencies into the top level of a zip file, along with whatever first party code: https://docs.aws.amazon.com/lambda/latest/dg/python-package.html#python-package-create-package-with-dependency (steps 4-6).

More discussion: https://pantsbuild.slack.com/archives/C046T6T9U/p1680247328906249

@huonw
Copy link
Collaborator Author

huonw commented Apr 25, 2023

Looking at the code that handles this in pip suggests they do a 'normal' install to a temporary directory, and then move some of the files from that install into the target directory: https://github.com/pypa/pip/blob/51afe45cda2854d810d9714e3891e256b1509737/src/pip/_internal/commands/install.py#L514-L566

This would suggest to me that, for a lot of use cases, just doing a venv export and then copying out the lib/.../site-packages directory might be enough, without having to implement this dedicated feature? I'm not sure if this generalises to #2111, though.

@jsirois
Copy link
Member

jsirois commented Apr 26, 2023

Just noting a correction to the claims above. I think the feature still makes sense.

PEX_TOOLS=1 ./some-pex repository extract ...: only exports the distributions, not the source

This is not true. There is a --sources option for the repository extract tool sub-command that does create and export an sdist of all the 1st party PEX code with proper install_requires metadata such that you can just grab that sdist alone and then copy it somewhere and pip install the-sdist.tar.gz and get everything back (non-reproducibly of course since install_requires entries may not be pinned, and even if they are they will almost never include the full transitive set; so transitive deps may be unpinned).

@jsirois jsirois self-assigned this Apr 27, 2023
@jsirois
Copy link
Member

jsirois commented Apr 27, 2023

This is getting tacked on to the feature work for #1752. There will be a --flat (not sure of spelling yet) flag you can tack on to pex3 venv -d right/here {--python,--platform,--complete-platform}* reqs* {--pex PEX|--lock LOCK}? --scope {all,deps,srcs} to just export to a flat dir instead of a venv site packages dir.

jsirois added a commit to jsirois/pex that referenced this issue Apr 28, 2023
This command allows inspecting venvs created by Pex as well as those
created by other tools.

Work towards pex-tool#1752 and pex-tool#2110
jsirois added a commit that referenced this issue Apr 28, 2023
This command allows inspecting venvs created by Pex as well as those
created by other tools.

Work towards #1752 and #2110
jsirois added a commit to jsirois/pex that referenced this issue Apr 28, 2023
This code as well as selection of a unique target to resolve for will
be re-used by the forthcoming `pex3 venv create` command.

Along the way, clean up confusion noted in pex-tool#2135 in the introduction
of the `pex3 venv` subcommand surrounding `pyvenv.cfg` files generated
by `Virtualenv`.

Work towards pex-tool#1752 and pex-tool#2110.
jsirois added a commit that referenced this issue Apr 28, 2023
This code as well as selection of a unique target to resolve for will be
re-used by the forthcoming `pex3 venv create` command.

Along the way, clean up confusion noted in #2135 in the introduction of
the `pex3 venv` sub-command surrounding `pyvenv.cfg` files generated by
`Virtualenv`.

Work towards #1752 and #2110.
jsirois added a commit to jsirois/pex that referenced this issue Apr 30, 2023
This new subcommand can create either a venv or just populate a flat
`sys.path` directory entry (ala `pip install --target`) given a set of
requirements to resolve, potentially from a lock or an existing PEX
file, but otherwise from indexes and find links repos. Unlike the
sibling `venv` `pex-tool` subcommand, the target can be selected and,
in the flat `sys.path` directory entry case, it can be a foreign
platform.

Fixes pex-tool#1752
Fixes pex-tool#2110
jsirois added a commit to jsirois/pex that referenced this issue May 1, 2023
This new subcommand can create either a venv or just populate a flat
`sys.path` directory entry (ala `pip install --target`) given a set of
requirements to resolve, potentially from a lock or an existing PEX
file, but otherwise from indexes and find links repos. Unlike the
sibling `venv` `pex-tool` subcommand, the target can be selected and,
in the flat `sys.path` directory entry case, it can be a foreign
platform.

Fixes pex-tool#1752
Fixes pex-tool#2110
jsirois added a commit that referenced this issue May 1, 2023
This new sub-command can create either a venv or just populate a flat
`sys.path` directory entry (ala `pip install --target`) given a set of
requirements to resolve, potentially from a lock or an existing PEX
file, but otherwise from indexes and find links repos. Unlike the
sibling `venv` `pex-tool` sub-command, the target can be selected and, 
in the flat `sys.path` directory entry case, it can be a foreign 
platform.

Fixes #1752
Fixes #2110
Fixes #2111
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants