Libraries loaded via _jll packages are visible inside PythonCall environment.

[lassepe]

Disclaimer: I am not sure whether this issue should be posted here or in CondaPkg.jl.

Related issues:
JuliaPackaging/Yggdrasil#9028
avik-pal/Wandb.jl#39

Affects: PythonCall

Describe the bug
Over at Wandb.jl we have observed that, when OpenSSL_jll@v3 is loaded before Wandb.jl (which wraps the Wandb python client via PythonCall), the package fails. When OpenSSL_jll@v3 is loaded after Wandb.jl or when we downgrade to OpenSSL_jll@v1, everything works as expected. This suggests that the libraries loaded via julia _jll's propagate into the python environment.

Expected Behavior:

I would hope that those two environments are isolated. If that is not possible, package authors using PythonCall should be made aware of having to pin JLL's at a version consistent with their dependencies (which is annoying but better than "hoping" that the right dependency gets loaded).

Reproducer

using OpenSSL_jll
using Wandb: Wandb

# trigger ssl error
function provoke_error(logger)
    logging_dict = Dict{String,Any}()
    logging_dict["samples"] = Wandb.Image(rand(2, 2))
    Wandb.log(logger, logging_dict; step=1)
end

logger = Wandb.WandbLogger(; project="test")
provoke_error(logger)

System Info

• Ubuntu 22.04
• Julia 1.10.4
• PythonCall 0.9.20

[cjdoris]

Cross-referencing this discussion on discourse (same issue): https://discourse.julialang.org/t/pythoncall-ssl-verify-error-when-calling-pd-read-csv/115922

With PythonCall, the Python and Julia interpreters are running in the same process, so any libraries loaded by both of them need to be compatible, which can be a bit of a headache sometimes.

In your case, the version of libopenssl used by Python is different from that used by Julia. For now you will need to pin OpenSSL_jll to whatever version Python uses (or conversely pin openssl in your Python environment to whatever Julia uses).

There is an existing mechanism in CondaPkg to ensure the versions of libstdc++ are compatible between Julia and Python. We should be able to do a similar thing for libopenssl.

[lassepe]

Thank you for the response. If I understand correctly, the code linked on discourse adjusts the dependencies for the conda environment, not for Julia, right? Is that even possible in this case? I thought Python just generally doesn't support OpenSSL 3.x and one would have to downgrade on the Julia side instead. Or is that conclusion incorrect?

[cjdoris]

Pretty sure Python works with openssl 3 these days.