Merge branch 'master' of github.com:jazzband/pip-tools into pip-tools…

…-integration

* 'master' of github.com:jazzband/pip-tools:
  Add comment, fix tests
  Update for compatibility with pip 8-10
  Drop pip 9 vendor dir
  Added 2.0.2 changelog (after actual release).
  Added clearer error reporting when skipping pre-releases (jazzband#655)
  Added a missing (important) changelog entry in 2.0.0
  Updated changelog for 2.0.1 (after release)
  Add missing package_data from vendored pip 9.0.3
  Changelog for v2.0.0
  Change a misleading example about version pinning
  Filter pip environment markers
  Remove setuptools dependency
  Remove the pip version "dimension" from the test matrix
  Remove obsolete Makefile - Using Jazzband release process
  Update and add mention in README.rst about pip vendoring
  Omited piptools/_vendored and piptools/_compat from tests
  Remove now obsolete pip version check due to vendoring
  Add piptools/vendored to sys.path on start
  Add pipools/_vendored/README.rst
  Add vendored pip 9.0.3
  Take CLI parameter as --user, pass through to pip install
  Option to restrict attention to user directory
  cache hashes for --generate-hashes
  Fix capitalization: PyPi -> PyPI
  Pass python_requires argument to setuptools
  Add dev-requirements.txt file to install packages for testing

.appveyor.yml

@@ -3,22 +3,46 @@ branches: {only: [master]}
 environment:
 
     matrix:
-        - TOXENV: py27-pip8
         - TOXENV: py27-pip8.1.1
-        - TOXENV: py27-pip9
+          PIP: 8.1.1
+        - TOXENV: py27-pip9.0.1
+          PIP: 9.0.1
+        - TOXENV: py27-pip9.0.3
+          PIP: 9.0.3
+        - TOXENV: py27-pipmaster
+          PIP: master
         - TOXENV: py27-piplatest
-        - TOXENV: py34-pip8
+          PIP: latest
         - TOXENV: py34-pip8.1.1
-        - TOXENV: py34-pip9
+          PIP: 8.1.1
+        - TOXENV: py34-pip9.0.1
+          PIP: 9.0.1
+        - TOXENV: py34-pip9.0.3
+          PIP: 9.0.3
+        - TOXENV: py34-pipmaster
+          PIP: master
         - TOXENV: py34-piplatest
-        - TOXENV: py35-pip8
+          PIP: latest
         - TOXENV: py35-pip8.1.1
-        - TOXENV: py35-pip9
+          PIP: 8.1.1
+        - TOXENV: py35-pip9.0.1
+          PIP: 9.0.1
+        - TOXENV: py35-pip9.0.3
+          PIP: 9.0.3
+        - TOXENV: py35-pipmaster
+          PIP: master
         - TOXENV: py35-piplatest
-        - TOXENV: py36-pip8
+          PIP: latest
         - TOXENV: py36-pip8.1.1
-        - TOXENV: py36-pip9
+          PIP: 8.1.1
+        - TOXENV: py36-pip9.0.1
+          PIP: 9.0.1
+        - TOXENV: py36-pip9.0.3
+          PIP: 9.0.3
+        - TOXENV: py36-pipmaster
+          PIP: master
         - TOXENV: py36-piplatest
+          PIP: latest
 
 install:
     - C:\Python36\python -m pip install tox

.coveragerc

@@ -0,0 +1,6 @@
+[run]
+branch = True
+source = piptools
+omit =
+    piptools/_compat/*
+    piptools/_vendored/*

.flake8

@@ -1,3 +1,3 @@
 [flake8]
 max-line-length = 120
-exclude = build/*, dist/*, pip_tools.egg-info/*, piptools/_compat/*
+exclude = build/*, dist/*, pip_tools.egg-info/*, piptools/_compat/*, piptools/_vendored/*

.travis.yml

@@ -8,10 +8,12 @@ python:
   - "3.5"
   - "3.6"
   - "pypy"
+
 env:
-  - PIP=8
   - PIP=8.1.1
-  - PIP=9
+  - PIP=9.0.1
+  - PIP=9.0.3
+  - PIP=master
   - PIP=latest
 
 cache: pip

CHANGELOG.md

@@ -8,6 +8,36 @@
 Features:
 - Made local, editable, and relative requirements resolve to relative paths instead of absolute ones ([#507](https://github.com/jazzband/pip-tools/pull/507). Thanks to @orf)
 
+# 2.0.2 (2018-04-28)
+
+Bug Fixes:
+- Added clearer error reporting when skipping pre-releases
+([#655](https://github.com/jazzband/pip-tools/pull/655)). Thanks @WoLpH
+
+# 2.0.1 (2018-04-15)
+
+Bug Fixes:
+- Added missing package data from vendored pip, such as missing cacert.pem file. Thanks @vphilippon
+
+# 2.0.0 (2018-04-15)
+
+Major changes:
+- Vendored `pip` 9.0.3 to keep compatibility for users with `pip` 10.0.0
+([#644](https://github.com/jazzband/pip-tools/pull/644)). Thanks @vphilippon
+
+Features:
+- Improved the speed of pip-compile --generate-hashes by caching the hashes from an existing output file
+([#641](https://github.com/jazzband/pip-tools/pull/641)). Thanks @justicz
+- Added a `pip-sync --user` option to restrict attention to user-local directory
+([#642](https://github.com/jazzband/pip-tools/pull/642)). Thanks @jbergknoff-10e
+- Removed the hard dependency on setuptools
+([#645](https://github.com/jazzband/pip-tools/pull/645)). Thanks @vphilippon
+
+Bug fixes:
+- The pip environment markers on top-level requirements in the source file (requirements.in)
+are now properly handled and will only be processed in the right environment
+([#647](https://github.com/jazzband/pip-tools/pull/647)). Thanks @JoergRittinger
+
 # 1.11.0 (2017-11-30)
 
 Features:
@@ -25,7 +55,7 @@ Bug Fixes:
 Bug Fixes:
 - Fixed bug causing dependencies from invalid wheels for the current platform to be included ([#571](https://github.com/jazzband/pip-tools/pull/571)).
 - `pip-sync` will respect environment markers in the requirements.txt ([600](https://github.com/jazzband/pip-tools/pull/600)). Thanks @hazmat345
-- Converted the ReadMe to have a nice description rendering on PyPi. Thanks @bittner
+- Converted the ReadMe to have a nice description rendering on PyPI. Thanks @bittner
 
 # 1.10.1 (2017-09-27)
 

README.rst

@@ -20,7 +20,7 @@ even when you've pinned them.  `You do pin them, right?`_
    :alt: Jazzband
    :target: https://jazzband.co/
 .. |pypi| image:: https://img.shields.io/pypi/v/pip-tools.svg
-   :alt: PyPi
+   :alt: PyPI
    :target: https://pypi.python.org/pypi/pip-tools/
 .. _You do pin them, right?: http://nvie.com/posts/pin-your-packages/
 
@@ -34,7 +34,6 @@ recommended to install ``pip-tools`` in each project's `virtual environment`_:
 .. code-block:: bash
 
     $ source /path/to/venv/bin/activate
-    (venv)$ pip install --upgrade pip  # pip-tools needs pip==8.0 or higher (!)
     (venv)$ pip install pip-tools
 
 **Note**: all of the remaining example commands assume you've activated your
@@ -156,7 +155,7 @@ To update a specific package to the latest or a specific version use the
 
     $ pip-compile --upgrade-package flask  # only update the flask package
     $ pip-compile --upgrade-package flask --upgrade-package requests  # update both the flask and requests packages
-    $ pip-compile -P flask -P requests==2.0.0  # update the flask package to the latest, and requests to v2.0.0
+    $ pip-compile -P flask -P requests  # same as above, but shorter
 
 If you use multiple Python versions, you can run ``pip-compile`` as
 ``py -X.Y -m piptools compile ...`` on Windows and
@@ -226,6 +225,17 @@ If you use multiple Python versions, you can run ``pip-sync`` as
 ``setuptools``, ``pip``, or ``pip-tools`` itself. Use ``pip install --upgrade``
 to upgrade those packages.
 
+
+Note about ``pip``
+==================
+
+As of v2.0.0, pip-tools has vendored ``pip v9.0.3`` in its codebase.
+This was done to keep ``pip-tools`` working for users with ``pip v10.0.0``.
+Only the ``pip-sync`` command still use the ``pip`` found in the ``virtualenv``,
+and only for the ``pip install`` and ``pip uninstall`` commands issued as shell-outs
+by ``pip-sync``.
+
+
 Other useful tools
 ==================
 

dev-requirements.txt

@@ -0,0 +1,4 @@
+-e .
+mock
+pytest
+wheel

piptools/_compat/__init__.py

@@ -11,3 +11,20 @@
 else:
     from tempfile import TemporaryDirectory
     from contextlib import ExitStack
+
+from .pip_compat import (
+    InstallRequirement,
+    parse_requirements,
+    RequirementSet,
+    user_cache_dir,
+    FAVORITE_HASH,
+    is_file_url,
+    url_to_path,
+    PackageFinder,
+    FormatControl,
+    Wheel,
+    Command,
+    cmdoptions,
+    get_installed_distributions,
+    PyPI,
+)

piptools/_compat/pip_compat.py

@@ -0,0 +1,30 @@
+# -*- coding=utf-8 -*-
+import importlib
+
+def do_import(module_path, subimport=None, old_path=None):
+    internal = 'pip._internal.{0}'.format(module_path)
+    old_path = old_path or module_path
+    pip9 = 'pip.{0}'.format(old_path)
+    try:
+        _tmp = importlib.import_module(internal)
+    except ImportError:
+        _tmp = importlib.import_module(pip9)
+    if subimport:
+        return getattr(_tmp, subimport, _tmp)
+    return _tmp
+
+
+InstallRequirement = do_import('req.req_install', 'InstallRequirement')
+parse_requirements = do_import('req.req_file', 'parse_requirements')
+RequirementSet = do_import('req.req_set', 'RequirementSet')
+user_cache_dir = do_import('utils.appdirs', 'user_cache_dir')
+FAVORITE_HASH = do_import('utils.hashes', 'FAVORITE_HASH')
+is_file_url = do_import('download', 'is_file_url')
+url_to_path = do_import('download', 'url_to_path')
+PackageFinder = do_import('index', 'PackageFinder')
+FormatControl = do_import('index', 'FormatControl')
+Wheel = do_import('wheel', 'Wheel')
+Command = do_import('basecommand', 'Command')
+cmdoptions = do_import('cmdoptions')
+get_installed_distributions = do_import('utils.misc', 'get_installed_distributions', old_path='utils')
+PyPI = do_import('models.index', 'PyPI')

piptools/cache.py

@@ -6,7 +6,7 @@
 import os
 import sys
 
-from pkg_resources import Requirement
+from pip._vendor.packaging.requirements import Requirement
 
 from .exceptions import PipToolsError
 from .locations import CACHE_DIR
@@ -159,6 +159,6 @@ def _reverse_dependencies(self, cache_keys):
         """
         # First, collect all the dependencies into a sequence of (parent, child) tuples, like [('flake8', 'pep8'),
         # ('flake8', 'mccabe'), ...]
-        return lookup_table((key_from_req(Requirement.parse(dep_name)), name)
+        return lookup_table((key_from_req(Requirement(dep_name)), name)
                             for name, version_and_extras in cache_keys
                             for dep_name in self.cache[name][version_and_extras])

piptools/exceptions.py

@@ -3,22 +3,44 @@ class PipToolsError(Exception):
 
 
 class NoCandidateFound(PipToolsError):
-    def __init__(self, ireq, candidates_tried, index_urls):
+    def __init__(self, ireq, candidates_tried, finder):
         self.ireq = ireq
         self.candidates_tried = candidates_tried
-        self.index_urls = index_urls
+        self.finder = finder
 
     def __str__(self):
-        sorted_versions = sorted(c.version for c in self.candidates_tried)
+        versions = []
+        pre_versions = []
+
+        for candidate in sorted(self.candidates_tried):
+            version = str(candidate.version)
+            if candidate.version.is_prerelease:
+                pre_versions.append(version)
+            else:
+                versions.append(version)
+
         lines = [
             'Could not find a version that matches {}'.format(self.ireq),
-            'Tried: {}'.format(', '.join(str(version) for version in sorted_versions) or '(no version found at all)')
         ]
-        if sorted_versions:
+
+        if versions:
+            lines.append('Tried: {}'.format(', '.join(versions)))
+
+        if pre_versions:
+            if self.finder.allow_all_prereleases:
+                line = 'Tried'
+            else:
+                line = 'Skipped'
+
+            line += ' pre-versions: {}'.format(', '.join(pre_versions))
+            lines.append(line)
+
+        if versions or pre_versions:
             lines.append('There are incompatible versions in the resolved dependencies.')
         else:
+            lines.append('No versions found')
             lines.append('{} {} reachable?'.format(
-                'Were' if len(self.index_urls) > 1 else 'Was', ' or '.join(self.index_urls))
+                'Were' if len(self.finder.index_urls) > 1 else 'Was', ' or '.join(self.finder.index_urls))
             )
         return '\n'.join(lines)
 

piptools/locations.py

@@ -2,7 +2,7 @@
 from shutil import rmtree
 
 from .click import secho
-from pip.utils.appdirs import user_cache_dir
+from ._compat import user_cache_dir
 
 # The user_cache_dir helper comes straight from pip itself
 CACHE_DIR = user_cache_dir('pip-tools')

piptools/repositories/local.py

@@ -6,6 +6,7 @@
 
 from piptools.utils import as_tuple, key_from_req, make_install_requirement
 from .base import BaseRepository
+from .._compat import FAVORITE_HASH
 
 
 def ireq_satisfied_by_existing_pin(ireq, existing_pin):
@@ -64,6 +65,16 @@ def get_dependencies(self, ireq):
         return self.repository.get_dependencies(ireq)
 
     def get_hashes(self, ireq):
+        key = key_from_req(ireq.req)
+        existing_pin = self.existing_pins.get(key)
+        if existing_pin and ireq_satisfied_by_existing_pin(ireq, existing_pin):
+            hashes = existing_pin.options.get('hashes', {})
+            hexdigests = hashes.get(FAVORITE_HASH)
+            if hexdigests:
+                return {
+                    ':'.join([FAVORITE_HASH, hexdigest])
+                    for hexdigest in hexdigests
+                }
         return self.repository.get_hashes(ireq)
 
     @contextmanager