Pkg.init() fails with https: Name or service not known

[petercolberg]

Using julia 0.5.0-rc0 with libgit2 0.24.1 on Debian sid:

julia> Pkg.init()
INFO: Initializing package repository /home/peter/.julia/v0.5
INFO: Cloning METADATA from https://github.com/JuliaLang/METADATA.jl
ERROR: GitError(Code:ERROR, Class:Net, Failed to resolve address for https: Name or service not known)
 in init() at ./pkg/pkg.jl:70

julia> versioninfo()
Julia Version 0.5.0-rc0+0
Commit 0030eec (2016-07-26 20:22 UTC)
Platform Info:
  System: Linux (x86_64-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E3-1245 V2 @ 3.40GHz
  WORD_SIZE: 64
  BLAS: libopenblas (NO_LAPACKE DYNAMIC_ARCH NO_AFFINITY Sandybridge)
  LAPACK: libopenblas
  LIBM: libopenlibm
  LLVM: libLLVM-3.8.1 (ORCJIT, ivybridge)

The libgit2 Debian package is built with libcurl3-gnutls 7.47.0.

[tkelman]

Don't think libcurl gives libgit2 the capability of using https, it needs to link to openssl for that.

[petercolberg]

libcurl3-gnutls is linked to GNU TLS, hence the package name. I tried recompiling the libgit2 Debian package with libcurl3 linked against OpenSSL, but the error persists.

[tkelman]

based on my understanding libgit2 uses libcurl for proxy support, but not for https transport, so it might not matter what curl is built against. check the libgit2 feature flags like we do in our tests.

[wildart]

currently, proxy is disabled in libgit2 build, because of libcurl is not an official dependency. Using system libs, if they built with libcurl then proxy should work.

[tkelman]

Right, the issue here is this is the debian build where they're using system libraries for everything, and that system copy of libgit2 is not built directly against openssl (I think) so https isn't working. Report this to the debian libgit2 maintainer I'd say. They're welcome to try @wildart's patch set to link libgit2 against mbedtls if that's preferable to them from a license perspective, but I suspect they might not want to do that since it was declined upstream. Maybe you can package https://github.com/wildart/mbedtlsstream in debian.

[wildart]

mbedtlsstream aren't working without patching libgit2, see libgit2/libgit2#3850

but #17471 will fix this for sure

[tkelman]

As a temporary non-recommended hack that should be removed as soon as possible, another option for debian packaging of 0.5.0 might be patching the default metadata url back to the git:// protocol and calling setprotocol! in juliarc.

[petercolberg]

Then #17471 is a release blocker for 0.5.0, at least from the point of Debian/Debian derivatives.

Debian ships both libgit2 (without OpenSSL) and libmbedtls, but we cannot expect the libgit2 maintainer to add a patch that has not been accepted upstream; especially considering it is security-related.

[tkelman]

Not our fault that debian ships libgit2 without linking to openssl. That's the only supported way upstream with libgit2 that https can work on Linux right now.

[wildart]

we can check libgit2 features and create appropriate urls for https and git+ssh protocols at least for the METADATA location.

[tkelman]

Makes sense. Maybe fall back to git:// though rather than git+ssh, since git:// was what the default metadata url used on 0.4 and earlier, right?

[petercolberg]

The reason libgit2 cannot be linked to OpenSSL is the license of the latter. Software linking to OpenSSL must include its advertising clause; this is incompatible with the GPL license of libgit2. The way around this is for the libgit2 authors to add an explicit linking exception for OpenSSL.

[petercolberg]

git:// is very much deprecated at this point. Fortunately TLS is no longer considered optional nowadays.

[wildart]

@tkelman yes
@petercolberg #17471 should finally resolve license discussion, but it would happen in next iteration of julia and libgit2

[Keno]

Why not rewrite urls to use ssh instead for the debian distribution?

[petercolberg]

I understand the constraints of the release schedule, but the package manager is an integral part of Julia.

If #17471 is not feasible for 0.5.0, would it be for 0.5.1?

[Keno]

Actually does debian link libgit2 with ssh support even? Our patch for the mbedtls crypto backend isn't upstream yet, so I suspect the answer may be no.

[wildart]

yes, it does. see libgcrypt

[Keno]

Ok, but until my patch to libssh that does not support encrypted ssh keys.

[wildart]

#17471 would require libgit2 version bump cause PR still in review

[Keno]

I've also pinged the libgit2 maintainers in libgit2/libgit2#3462 to reconsider including the support for mbedtls by default. Their reasoning for rejection was that they only want the support for whatever the platform-default crypto library is in-tree, but if Debian cannot link libgit2 with openssl due to license incompatibilities, then having the mbedtls support in-tree seems prudent for them.

[tkelman]

For short-term workarounds, is libressl in debian? That should theoretically be a drop-in replacement for most consumers of openssl, I believe, but I don't know whether libressl has finished excising the problematic GPL-incompatibly-licensed parts of openssl yet. The license compatibility issue is also open here as #10763, some interpretations say openssl is considered "part of the OS" so excepted from the GPL copyleft requirements, but I guess debian does not do it that way.

[petercolberg]

@Keno Indeed, neither Debian nor any other distribution is able to distribute libgit2 linked to OpenSSL without violating the GPL due to OpenSSL’s advertising clause. Unfortunately libressl has not been packaged so far. Debian/Ubuntu have GnuTLS as an alternative though.

[tkelman]

I don't think libgit2 supports gnutls.

edit: without writing external custom code / patches, similar to how Art has done for mbedtls

[wildart]

From my own experience, it's not a big issue to support libgit2 custom TLS streams based on any cyptolib. Unfortunately, whole custom TLS stream implementation is not carefully designed. One outstanding problem with custom TLS streams is a proxy support. But proxy stuff never was a strong side of libgit2.

[Keno]

@petercolberg If you haven't seen already, note the discussion on that issue that the libgit2 maintainers believe linking against openssl should not be a problem. Could you contact the libgit2 maintainer or whoever is appropriate on the Debian side to see if we can figure this out?

[petercolberg]

@Keno Yes, thank you for pointing me to the discussion. The libgit2 license includes a generalized linking exception; I will contact debian-legal to see if it applies to OpenSSL.

[ViralBShah]

Our libgit2 now uses curl so that we can support proxies as well. I suspect it is probably doing that on debian too.

[tkelman]

That wasn't the problem here. Debian likely needs to make a second separate libgit2 package that links against openssl for this.

[DemiMarie]

@petercolberg correct. I have git configured to automatically use https:// instead of git:// or http:// everywhere.

[DemiMarie]

I suspect that Debian will not package Julia linked to OpenSSL (even indirectly) unless Julia is built with USE_GPL_LIBS=0.

[KristofferC]

Stale, please open at the Pkg.jl repo for further Pkg problems.