arbitrary code execution "feature" still exists

[leif]

It is nice that with version 1.3 pip finally began validating SSL certificates when HTTPS is used, but I don't think adding that feature alone was sufficient to close #425 ("pip should not execute arbitrary code from the Internet") because pip is still happy to download and execute arbitrary code via HTTP and does so quite frequently.

Many packages in PyPI have HTTP download URLs listed, despite also being available via HTTPS. (The one that just motivated me to file this issue is the "fs" package which is distributed via Google Code yet has an HTTP URL in PyPI).
#425 is very lengthy and was closed for having "no clear goal" so rather than comment there I am opening this new ticket with a clear goal:

pip's default behavior should be to refuse to download anything via HTTP. Downloading packages via HTTP should require the user to specify '--insecure'.

To make this transition less painful (painless for the majority of packages, I think), pip should automatically use HTTPS when it encounters HTTP URLs pointing to well-known HTTPS-capable sites like Google Code (just like the "HTTPS Everywhere" browser extension does).

As a bonus, it would be nice if pip pinned the certificate authorities used by the handful of sites which distribute the majority of the packages on PyPI.

(It would also be nice if pip included the functionality of peep (pinning package versions and verifying their hashes). And, of course, best of all would be if package authors would sign their $#&%@$ releases so that dependency lists could include the dependency authors' public keys... but given that much of the Python community is apparently still OK with blindly executing code they've just fetched via HTTP I'm not going to hold my breath for everyone to start doing that.)

[dstufft]

Hey there,

In pip 1.4 we introduced two new concepts (and options to go with them). One is "external" files which are files hosted elsewhere besides PyPI and are linked directly from the /simple/ index with a hash associated that can verify the downloaded file. The other is "insecure" files which are either directly linked files without a hash or are files linked from a page that the PyPI page links too. In pip 1.4 these types are enabled by default but you can disable them using --no-allow-external and --no-allow-insecure. However in pip 1.5 these types of files are disabled by default and can be renabled on a case by case basis using --allow-external [PROJECT] and --allow-insecure [PROJECT]. Because external urls do not represent an additional security risk (given that they are linked from a page fetched via HTTPS and they include a hash) there is also an option to allow all external files which is --allow-all-external. However since there is an additional security risk for the other category there is no such "allow all" option in pip 1.5 for them.

Additionally in pip 1.5 users will be given warnings when using HTTP urls that shows up brightly colored in order to make it stand up.

Certificate pinning is something I would like to get into pip (although I'd like to investigate using something like the header that google chrome introduced so it can be used on site other than PyPI) and haven't yet merely becaus of time constraints.

The functionality of peep is another thing I'd like to get added into pip and again is mostly waiting on time to design how it actually would work since I don't believe it is ok for pip itself to place meaning in comments (but works really well for peep because then pip itself ignores it.

As far as signing goes, that is also a goal of mine :) Again time constraints and trying to knock out the low hanging fruit like HTTP access as well as general improvements to the entire infrastructure.

[qwcode]

note this in the 1.5 changelog http://www.pip-installer.org/en/latest/news.html#changelog

"pip no longer will scrape insecure external urls by default nor will it install externally hosted files by default. Users may opt into installing externally hosted or insecure files or urls using --allow-external PROJECT and --allow-insecure PROJECT. (Pull #1055)"

[qwcode]

crossed mid-air with @dstufft

[dstufft]

Here are the related pull requests #1121, #1098, #1055, #985.

[dstufft]

Oh yea, pip 1.5 also kills --use-mirrors which didn't verify the authenticity of the downloaded files and downloaded everything over HTTP.

[dstufft]

Although as I think about it more, it might be interesting to straight out disallow non HTTPS urls without passing a flag. This will however kill explicit use of the current mirrors barring the acceptance of PEP449 and the mirrors getting their own domain name and certificates without also including a flag. This however doesn't sound unreasonable.

Thoughts on disallowing HTTP w/o a flag @pypa? (Not sure if mentioning all of us actually works :V)

[nejucomo]

I just created #1168 specifically about certificate pinning, to pull it out of this ticket. I did not create a ticket for code signing, because it's a giant can of worms and I'm not sure how to write a ticket that won't bog down into a 5 year flame war. ;-)

I propose we remove certificate pinning and code signing from this ticket's criteria to limit scope creep, and keep this ticket focused on requiring HTTPS.

Does the 1.5 feature set fulfill @leif 's original proposal?

What criteria does pip use to close tickets? Do they close once a satisfactory implementation / bugfix is in a release's codebase?

[dstufft]

@nejucomo Typically we close them once they've been fixed in the develop branch (or the release branch if it only affects an older release).

I agree on keeping this ticket focused on forced HTTPS and it'll be closed if we either decide not to force it or when it actually lands. Thanks for creating a ticket for certificate pinning :)

[pfmoore]

I'm against a blanket ban on http. For example, I have a local repository of stuff that I might want to get to via -i (at the moment, I use --find-links because it's easier, but I won't always have mapped drive support). Having to set that up as https, or having to add some form of --use-insecure-links flag (which always feels like "you're being naughty" to me) isn't really appropriate there. I'd probably end up adding --use-insecure-links to my pip.ini along with the local cache's -i option, which of course makes things worse, not better :-(

[dstufft]

Well if we implemented a flag for using insecure links it'd be a per domain option not a global option. That being said I'm +0 on making HTTP require a flag. At somepoint there's only so much we can do to protect people and it comes down to where is the line.

[zookoatleastauthoritycom]

+1 for turning off insecure downloads by default. If people want to be vulnerable to network attackers injecting code, that's fine! There may be cases where that is appropriate, but for those cases people can turn on the flag. It should be off by default.

[dstufft]

As it is now in pip 1.5 it'll download from HTTPS by default and won't download anything it can't verify without an --allow-insecure flag. So the question would be if someone does --index-url http://example.com/ should we require them to also do something like -allow-http example.com or is the user typing out http:// enough for "consent". Also important to remember is that pip will give a colored warning to the output if somebody does use an insecure transport.

[zookoatleastauthoritycom]

In my opinion if they explicitly wrote "http://" then that means they want to allow network-based attackers the option of taking over their computer. Now, perhaps they don't reallize that, in which case maybe we could warn them, such as with docs or the coloration that you mention, or even more verbose warnings in the output. But, IMHO, if they wrote "http://" then we shouldn't require them to write anything else to mean "Yes, download unauthenticated code."

[dstufft]

Yea I tend to agree now that I think about it more.

Here's the warning message if you use http FWIW.

[dstufft]

Going to close this issue then since pip 1.4 includes the tools to do what @leif wanted and 1.5 does it by default and the outstanding issue of requiring something besides typing the actual url to confirm you wanted http seems to have 1 core maintainer with a -1, one with a -0, and a external party against it as well.

[dstufft]

Oh and also @nejucomo has opened up tickets for the other bonus issues.

[pfmoore]

I don't want to perpetuate this thread unnecessarily, but can I ask for some clarification on the statement "if they explicitly wrote "http://" then that means they want to allow network-based attackers the option of taking over their computer". I'm assuming that the "consenting adults" principle means that saying you want HTTP means you understand the risks, but the statement I quote implies that someone thinks that I might not...

Suppose I use "-i http://server/xxx", where server is either an ip address of a machine on my company's internal network, or it's a name that is served from the company DNS or via my HOSTS file. Suppose also that I trust my company internal IT security (or at least I'm willing to blame them if there's an issue :-)). And assume that the machine I expect to contact is known secure. Then is there a way that I'm missing whereby an attacker can redirect my connection to a compromised machine? And if there is, what access would they need to my company's internal network to execute the attack?

This is the crux of my -1 to the idea of a blanket ban. That's the situation I expect to be the most common, and I don't see the attack vector. (Even if I'm on my home PC, I don't see how an attacker can inject a compromised copy of a specific IP address into my network, to be honest).

I know this is a complex area. I'm only after the basics of why I should trust the sort of statements I see being made. But equally I don't want to have to be an expert to understand the basics of the risks. I'd be happy with a pointer to a "beginner's overview" type of document that explained why everyone is so strongly against http without any consideration of the environment it's used in.

[dstufft]

I don't think the statement you quoted was directed towards you and was instead a general "they", as in the person who is specifying http.

That being said even on a IT network it's still true that using http opens you up to network attackers, it just so happens that the pool of people who could attack you are much lower (requiring them to have access to your internal network). As far as your home network goes if you have an unsecured (or poorly secured) wireless network it becomes pretty easy.

Pip in general has no idea what the network connection looks like so it needs to treat all networks as untrusted by default. However as you, zooko and then me said that if someone explicitly uses http they can be assumed to understand the risks (the risks being an attacker can take control over your computer) and there doesn't need to be yet another option beyond that. How big of a risk the risk is depends entirely on the network between you and the server you're installing from (which is something pip can't know).

[pfmoore]

I wasn't taking it personally. But I am concerned that I could take a naive view when expressing an opinion on issues like this. It sounds like I'm not, and that's reassuring. (maybe I'm more trusting of categories of people such as "people with access to my company's internal network" than the average security professional, but I'm OK with that).

Thanks for taking the time to clarify for me.

[dstufft]

No problem! There's a reason security folks are also called professional paranoids. We tend to look at things in terms of attack surfaces. :D