Add resource / source bundling to pex cli

[baroquebobcat]

Pex supports adding sources and resources directly via the PEXBuilder API, but not the CLI. This adds flags to allow adding sources and resources via directories.

Fixes #490
Fixes #491

I paired on this with @dotordogh!

Two questions came up while working on this

• what should happen if the output location is inside the input path?
• it looks like if the entry point doesn't exist, pex won't complain. Maybe it should?

[kwlzn]

looks great! thanks @dotordogh and @baroquebobcat!

[kwlzn]

nit: two newlines between functions

[kwlzn]

what should happen if the output location is inside the input path?

assuming that we read and add_{source,resource}() all of those files prior to writing the pex, I'm not sure it ultimately matters? tho I wouldn't be opposed to pre-emptively safeguarding this case either way.

it looks like if the entry point doesn't exist, pex won't complain. Maybe it should?

seems reasonable to me. filed: #508

[kwlzn]

not used?

[kwlzn]

@baroquebobcat @dotordogh looks like there's one failure remaining on the 3.6 shard here:

https://travis-ci.org/pantsbuild/pex/jobs/388099378#L1324

looks like a unicode handling thing in python3.6. you should be able to get to a local repro by running:

$ tox -e py36-integration -- -k test_pex_resource_bundling

[baroquebobcat]

Oddly enough, that didn't get me a repro. But, my local python 3.6 is 3.6.5. Could there be a difference between 3.6.3 and 3.6.5?

[kwlzn]

Could there be a difference between 3.6.3 and 3.6.5?

I wouldn't think so.. I'm also on python 3.6.5 and I repro locally:

[omerta pex (pr507)]$ tox -e py36-integration -- -k test_pex_resource_bundling
GLOB sdist-make: /Users/kwilson/dev/pex/setup.py
py36-integration inst-nodeps: /Users/kwilson/dev/pex/.tox/dist/pex-1.4.3.zip
py36-integration installed: atomicwrites==1.1.5,attrs==18.1.0,more-itertools==4.2.0,packaging==16.8,pex==1.4.3,pluggy==0.6.0,py==1.5.3,pyparsing==2.2.0,pytest==3.6.0,six==1.11.0,twitter.common.contextutil==0.3.9,twitter.common.dirutil==0.3.9,twitter.common.lang==0.3.9,twitter.common.testing==0.3.9
py36-integration runtests: PYTHONHASHSEED='1789156364'
py36-integration runtests: commands[0] | py.test tests/test_integration.py -k test_pex_resource_bundling
================================================= test session starts ==================================================
platform darwin -- Python 3.6.5, pytest-3.6.0, py-1.5.3, pluggy-0.6.0
rootdir: /Users/kwilson/dev/pex, inifile:
collected 39 items / 38 deselected                                                                                     

tests/test_integration.py F                                                                                      [100%]

======================================================= FAILURES =======================================================
______________________________________________ test_pex_resource_bundling ______________________________________________

    def test_pex_resource_bundling():
      with temporary_dir() as output_dir:
        with temporary_dir() as input_dir, temporary_dir() as resources_input_dir:
          with open(os.path.join(resources_input_dir, 'greeting'), 'w') as fh:
            fh.write('hello')
          pex_path = os.path.join(output_dir, 'pex1.pex')
    
          with open(os.path.join(input_dir, 'exe.py'), 'w') as fh:
            fh.write(dedent('''
              import pkg_resources
              with pkg_resources.resource_stream('__main__', 'greeting') as f:
                print(f.read())
              '''))
    
          res = run_pex_command([
            '-o', pex_path,
            '-D', input_dir,
            '-R', resources_input_dir,
            '-e', 'exe',
          ])
          res.assert_success()
    
          stdout, rc = run_simple_pex(pex_path)
    
          assert rc == 0
>         assert stdout == b'hello\n'
E         assert b"b'hello'\n" == b'hello\n'
E           At index 0 diff: 98 != 104
E           Left contains more items, first extra item: 111
E           Use -v to get the full diff

tests/test_integration.py:962: AssertionError
------------------------------------------------- Captured stdout call -------------------------------------------------
b'hello'

------------------------------------------------- Captured stderr call -------------------------------------------------
pex: Building pex
pex: Building pex :: Resolving interpreters
pex: Building pex :: Resolving interpreters :: Setting up interpreter /Users/kwilson/dev/pex/.tox/py36-integration/bin/python3.6
pex: Building pex :: Resolving distributions
pex: Warning, using a UrllibContext which is known to be flaky.
pex: Please build pex with the requests module for more reliable downloads.
pex: Constructed UrllibContext context <pex.http.UrllibContext object at 0x10e134550>
pex: Building pex: 3.4ms
pex:   Resolving interpreters: 1.5ms
pex:     Setting up interpreter /Users/kwilson/dev/pex/.tox/py36-integration/bin/python3.6: 1.5ms
pex:   Resolving distributions: 0.9ms
======================================= 1 failed, 38 deselected in 1.05 seconds ========================================
ERROR: InvocationError: '/Users/kwilson/dev/pex/.tox/py36-integration/bin/py.test tests/test_integration.py -k test_pex_resource_bundling'
_______________________________________________________ summary ________________________________________________________
ERROR:   py36-integration: commands failed

[kwlzn]

@baroquebobcat @dotordogh looks like this is still failing the same test on the 3.6 shard here - but that should be the last remaining blocker to landing this.

noting that this patch fixes that locally for me:

diff --git a/tests/test_integration.py b/tests/test_integration.py
index 0a7a6c9..6df25cf 100644
--- a/tests/test_integration.py
+++ b/tests/test_integration.py
@@ -945,7 +945,7 @@ def test_pex_resource_bundling():
         fh.write(dedent('''
           import pkg_resources
           with pkg_resources.resource_stream('__main__', 'greeting') as f:
-            print(f.read())
+            print(f.read().decode('utf-8'))
           '''))
 
       res = run_pex_command([

[kwlzn]

looks good mod the expected pypy shard failure tracked in #497. thanks @dotordogh and @baroquebobcat!

will cut pex 1.4.4 in #512 later today to unblock consumption of these changes in pants.