Issues with requests != 2.12

[reaperhulk]

In #58 I see you added requests != 2.12 to the requirements of adal. This is sadly not a workable restriction in practice due to the way pip currently does version dependency resolution. To illustrate, consider this simple Python package setup.py:

from setuptools import setup, find_packages

version = '0.1'

setup(
    name='depfail',
    version=version,
    packages=find_packages(),
    entry_points={
        'console_scripts': [
            'run-me = depfail:fail',
        ]
    },
    install_requires=[
        'requests[security]',
        'adal',
    ],
)

We'll also have a depfail/__init__.py that looks like this:

def fail():
    print("No failure!")

This package has two dependencies, requests[security] and adal. You might rationally think that pip would recursively look through dependencies to build a full set of requirements and then use something like a SAT solver to satisfy the constraints imposed on individual packages (and, of course, fail if it can't satisfy them). Unfortunately, that is not the case. In reality here's what your virtualenv will look like if you do pip install .:

adal (0.4.3)
cffi (1.9.1)
cryptography (1.6)
depfail (0.1)
enum34 (1.1.6)
idna (2.1)
ipaddress (1.0.17)
pip (9.0.1)
pyasn1 (0.1.9)
pycparser (2.17)
PyJWT (1.4.2)
pyOpenSSL (16.2.0)
python-dateutil (2.6.0)
requests (2.12.3)
setuptools (30.2.0)
six (1.10.0)
wheel (0.30.0a0)

As you can see, requests 2.12.3 is installed despite the explicit != 2.12. Of course if we do a python -c 'import depfail;depfail.fail()' we get No failure!. So what's the problem?

Well, when you define a console_script it uses setuptools entry points to invoke your script. The installed script is available in your PATH and looks something like this:

# EASY-INSTALL-ENTRY-SCRIPT: 'depfail==0.1','console_scripts','run-me'
__requires__ = 'depfail==0.1'
import re
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(
        load_entry_point('depfail==0.1', 'console_scripts', 'run-me')()
    )

load_entry_point is a bit more conscientious about whether or not dependencies are satisfied so it proceeds to check to see if adal's requirements are met and...

pkg_resources.ContextualVersionConflict: (requests 2.12.3 (/path/to/lib/python2.7/site-packages/requests-2.12.3-py2.7.egg), Requirement.parse('requests!=2.12.*,>=2.0.0'), set(['adal']))

This interaction means that adal can't effectively block requests 2.12 for users who have requests listed as a dependency that is processed before adal (a presumably common scenario). Additionally, this directive causes major breakage for users who invoke their application via console scripts (not an uncommon path).

[fxfitz]

+1 ... this is happening exactly for us.

[yugangw-msft]

@reaperhulk, thanks for the insights. #58 is not a long term solution, and will not work all the time like you outlined. Like I later commented in that PR, I am thinking of just letting adal depend on requests[security] which will straighten out the dependency relationship explicitly. Do you feel there are any problem with that?
adal needs the rsa from 'cryptography` anyway to sign the JWT token request, for client app to authenticate through service principals with certificates, which is a major flow.

[reaperhulk]

@yugangw-msft I've submitted urllib3/urllib3#1063, which should be in an upcoming requests release and will resolve the issues around outdated pyOpenSSL being loaded by requests. That should make it so you don't have to do requests[security], which does add quite a few dependencies. That said, requests[security] does have the advantage of guaranteeing better TLS even on Python < 2.7.9.

[yugangw-msft]

@reaperhulk, thank you! I should revert the change once a new requests version gets released.

[rayluo]

Thanks @reaperhulk . your demo illustrates that our previous workaround #58 doesn't work quite well, "due to the way pip currently does version dependency resolution", and even brings some unexpected side effects. We will revert it after your urllib3/urllib3#1063 gets merged and released in urllib3 and then in requests.

For what its worth, I now understand why requests 2.12 causes the problem in the first place, and why (manually) downgrading requests to 2.11.x can make it work, documented in #58.

@yugangw-msft After we revert #58 and requests has a new release, the affected developers or end users will see this exception message in their console (if they have access to one): "'pyOpenSSL' module missing required functionality. Try upgrading to v0.14 or newer.". Hopefully this would be enough to guide the affected users to take proper action.

[reaperhulk]

@rayluo While the code you linked does raise an ImportError it won't actually bubble all the way to the end user. Instead, it's caught when requests initializes (https://github.com/kennethreitz/requests/blob/master/requests/__init__.py#L51-L55) and pyopenssl is simply not injected if the version is too old. So the effect of urllib3/urllib3#1063 is to not use pyopenssl if it's too old, but also not fail in that case.

[avaranovich]

I am getting

File "D:\Python27\lib\site-packages\pip\_vendor\pkg_resources.py", line 2583, in scan_list

    "Expected ',' or end-of-list in",line,"at",line[p:]
ValueError: ("Expected ',' or end-of-list in", 'requests >=2.0.0,!=2.12.*', 'at', '*')

while python -m pip install adal

[rayluo]

@avaranovich What is the version of your pip? Perhaps this Q&A in stackoverflow could help you.

[aarsan]

@avaranovich any luck with this? I am using this in Azure Web Apps where I have no control of the pip version.

[yugangw-msft]

@reaperhulk, do you know when your change to utllib3 will be included in a new release of request?
If it will take a while, @rayluo, guess we just back out this condition first at adal

[yugangw-msft]

@aarsan, you can use the previous release of adal, unless you need particular fixes

[rayluo]

@yugangw-msft
It looks like @reaperhulk 's patch urllib3/urllib3#1063 will not be released in urllib3 and requests in next month. I'll send out a PR to revert ADAL's PR #58 now. So for those customers who was bothered by that https://github.com/kennethreitz/requests/issues/3701 in the first place, they will have to either upgrade pyopenssl, or (if they still prefer) downgrade requests to 2.11.*