UnicodeDecodeError 'ascii' codec can't decode byte 0xe3 in position 66: ordinal not in range(128)

[oppianmatt]

Running pip-compile causes a UnicodeDecodeError

Environment Versions

1. Debian Jessie Docker
2. Python version: Python 2.7.14
3. pip version: pip 9.0.1 from /usr/local/lib/python2.7/site-packages (python 2.7)
4. pip-tools version: pip-compile, version 1.9.0

Steps to replicate

1. pip-compile -U

Expected result

requirements.txt updated

Actual result

Traceback (most recent call last):
  File "/usr/local/bin/pip-compile", line 11, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/piptools/scripts/compile.py", line 184, in cli
    results = resolver.resolve(max_rounds=max_rounds)
  File "/usr/local/lib/python2.7/site-packages/piptools/resolver.py", line 107, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/usr/local/lib/python2.7/site-packages/piptools/resolver.py", line 195, in _resolve_one_round
    for dep in self._iter_dependencies(best_match):
  File "/usr/local/lib/python2.7/site-packages/piptools/resolver.py", line 274, in _iter_dependencies
    dependencies = self.repository.get_dependencies(ireq)
  File "/usr/local/lib/python2.7/site-packages/piptools/repositories/pypi.py", line 145, in get_dependencies
    self._dependencies_cache[ireq] = reqset._prepare_file(self.finder, ireq)
  File "/usr/local/lib/python2.7/site-packages/pip/req/req_set.py", line 620, in _prepare_file
    session=self.session, hashes=hashes)
  File "/usr/local/lib/python2.7/site-packages/pip/download.py", line 821, in unpack_url
    hashes=hashes
  File "/usr/local/lib/python2.7/site-packages/pip/download.py", line 663, in unpack_http_url
    unpack_file(from_path, location, content_type, link)
  File "/usr/local/lib/python2.7/site-packages/pip/utils/__init__.py", line 605, in unpack_file
    untar_file(filename, location)
  File "/usr/local/lib/python2.7/site-packages/pip/utils/__init__.py", line 551, in untar_file
    path = os.path.join(location, fn)
  File "/usr/local/lib/python2.7/posixpath.py", line 73, in join
    path += '/' + b
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe3 in position 66: ordinal not in range(128)

[ghost]

Same here

[oppianmatt]

in my case the error I found is coming from ansible, putting a print just before the line shows these filenames:

/tmp/tmpf9NBhMbuild/ansible test/integration/targets/unarchive/meta
/tmp/tmpf9NBhMbuild/ansible test/integration/targets/unarchive/meta/main.yml
/tmp/tmpf9NBhMbuild/ansible test/integration/targets/unarchive/tasks
/tmp/tmpf9NBhMbuild/ansible test/integration/targets/unarchive/tasks/main.yml
/tmp/tmpf9NBhMbuild/ansible test/integration/targets/unarchive/files
/tmp/tmpf9NBhMbuild/ansible test/integration/targets/unarchive/files/foo.txt
/tmp/tmpf9NBhMbuild/ansible test/integration/targets/unarchive/files/test-unarchive-nonascii-くくららととみみ.tar.gz
Traceback (most recent call last):
  File "/usr/local/bin/pip-compile", line 11, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/piptools/scripts/compile.py", line 184, in cli
    results = resolver.resolve(max_rounds=max_rounds)
  File "/usr/local/lib/python2.7/site-packages/piptools/resolver.py", line 107, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/usr/local/lib/python2.7/site-packages/piptools/resolver.py", line 195, in _resolve_one_round
    for dep in self._iter_dependencies(best_match):
  File "/usr/local/lib/python2.7/site-packages/piptools/resolver.py", line 274, in _iter_dependencies
    dependencies = self.repository.get_dependencies(ireq)
  File "/usr/local/lib/python2.7/site-packages/piptools/repositories/pypi.py", line 145, in get_dependencies
    self._dependencies_cache[ireq] = reqset._prepare_file(self.finder, ireq)
  File "/usr/local/lib/python2.7/site-packages/pip/req/req_set.py", line 620, in _prepare_file
    session=self.session, hashes=hashes)
  File "/usr/local/lib/python2.7/site-packages/pip/download.py", line 821, in unpack_url
    hashes=hashes
  File "/usr/local/lib/python2.7/site-packages/pip/download.py", line 663, in unpack_http_url
    unpack_file(from_path, location, content_type, link)
  File "/usr/local/lib/python2.7/site-packages/pip/utils/__init__.py", line 606, in unpack_file
    untar_file(filename, location)
  File "/usr/local/lib/python2.7/site-packages/pip/utils/__init__.py", line 552, in untar_file
    path = os.path.join(location, fn)
  File "/usr/local/lib/python2.7/posixpath.py", line 73, in join
    path += '/' + b
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe3 in position 66: ordinal not in range(128)

• it's non ascii filenames in the package causing it to error
• unicode filenames should be valid
• it's definitely pip-tools issue since pip install -U ansible works
• seems to be similar to Pip compile breaks with scandir==1.4 #433 but the fix provided there does not fix this

[oppianmatt]

work around was to upgrade ansible manually and then continue using pip-tools

[ghost]

I confirm that this problem can be reproduced by installing ansible via pip-compile. If ansible is already installed manually the problem does not happen.