pex without pypi - workaround for not supporting --hashes?

[electroniceagle]

Is it possible to compile from an already fetched .venv without pex reaching out to pypi? I've tried --no-index and ---inherit-path but not having a lot of luck. Since pex doesn't appear to support a requirements.txt file with hashes, how does one know the pex file created contains the packages you scanned and tested?

[jsirois]

So, Pex does support anything Pip supports in a requirements file including hashes:

^jsirois@gill ~ $ cd /tmp/
^jsirois@gill /tmp $ mkdir pex-hash
^jsirois@gill /tmp $ cd pex-hash/
^jsirois@gill /tmp/pex-hash $ python -mvenv venv
^jsirois@gill /tmp/pex-hash $ source venv/bin/activate
(venv) ^jsirois@gill /tmp/pex-hash $ pip -q install -U pip
(venv) ^jsirois@gill /tmp/pex-hash $ pip -q install -U pex
(venv) ^jsirois@gill /tmp/pex-hash $ pex -V
2.1.42
^jsirois@gill /tmp/pex-hash $ pip download -d dists ansicolors
Collecting ansicolors
  Using cached ansicolors-1.1.8-py2.py3-none-any.whl (13 kB)
Saved ./dists/ansicolors-1.1.8-py2.py3-none-any.whl
Successfully downloaded ansicolors
^jsirois@gill /tmp/pex-hash $ pip hash dists/ansicolors-1.1.8-py2.py3-none-any.whl 
dists/ansicolors-1.1.8-py2.py3-none-any.whl:
--hash=sha256:00d2dde5a675579325902536738dd27e4fac1fd68f773fe36c21044eb559e187
^jsirois@gill /tmp/pex-hash $ cat <<EOF > requirements.txt
> ansicolors==1.1.8 --hash=sha256:00d2dde5a675579325902536738dd27e4fac1fd68f773fe36c21044eb559e187
> EOF
^jsirois@gill /tmp/pex-hash $ pex -r requirements.txt -o colors.pex

Then if I edit requirements.txt to give it a bad hash:

^jsirois@gill /tmp/pex-hash $ vi requirements.txt 
^jsirois@gill /tmp/pex-hash $ cat requirements.txt 
ansicolors==1.1.8 --hash=sha256:00d2dde5a675579325902536738dd27e4fac1fd68f773fe36c21044eb559e186
^jsirois@gill /tmp/pex-hash $ pex -r requirements.txt -o colors.pex
ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    ansicolors==1.1.8 from https://files.pythonhosted.org/packages/53/18/a56e2fe47b259bb52201093a3a9d4a32014f9d85071ad07e9d60600890ca/ansicolors-1.1.8-py2.py3-none-any.whl#sha256=00d2dde5a675579325902536738dd27e4fac1fd68f773fe36c21044eb559e187 (from -r requirements.txt (line 1)):
        Expected sha256 00d2dde5a675579325902536738dd27e4fac1fd68f773fe36c21044eb559e186
             Got        00d2dde5a675579325902536738dd27e4fac1fd68f773fe36c21044eb559e187

pid 10913 -> /home/jsirois/.pex/venvs/7fbdc9731d96d42f0de338b2dfbc6bbed1c938e7/832e95f85413646e8b7b056ab4e60414f83d7397/pex --disable-pip-version-check --no-python-version-warning --exists-action a --use-deprecated legacy-resolver --isolated -q --cache-dir /home/jsirois/.pex download --dest /tmp/tmpb8uzmhwg/home.jsirois..venv.pex.bin.python3 --requirement requirements.txt --index-url https://pypi.org/simple --retries 5 --timeout 15 exited with 1 and STDERR:
None

[jsirois]

That bit aside, your original question was:

Is it possible to compile from an already fetched .venv without pex reaching out to pypi?

No. Pex currently has no way to say resolve from a venv.

Pex supports creating a venv from a PEX file. So you could invert all this and first create the pex and then create a venv from it.

1. pex ... --include-tools my.pex
2. PEX_TOOLS=1 my.pex venv create/the.venv/here/please

You also can resolve from another PEX file with --pex-repository. That turns off all resolution sources except the --pex-repository you point to.

[electroniceagle]

Thanks for catching a basic assumption that I totally missed. I'm here looking at a make pex target that seds hashes from the requirements.txt file and the commit message claims hashes aren't supported.

[jsirois]

Aha, yeah - that message was probably from Pex 1.6.9 or earlier when Pex used its own resolver. Ever since Pex 2.0.0 it has used Pip to do resolves.

You did mention resolving from a venv and that is a good idea. Filed: #1361