Precompilation error on NixOS

[rikhuijzer]

In GiovineItalia/Compose.jl#407, I was asked to file an issue, so here it is. When, I'm trying to test Compose.jl or Gadfly.jl, I get an error on NixOS. The stacktrace is

ERROR: LoadError: InitError: could not load library "/home/rik/.julia/artifacts/079e2f87254bb2465a44de255cd5dde50d4d0b62/lib/libpangocairo-1.0.so"
/usr/lib/libpangoft2-1.0.so.0: undefined symbol: pango_coverage_get_type
Stacktrace:
 [1] dlopen(::String, ::UInt32; throw_error::Bool) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Libdl/src/Libdl.jl:109
 [2] dlopen(::String, ::UInt32) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Libdl/src/Libdl.jl:109
 [3] macro expansion at /home/rik/.julia/packages/JLLWrappers/KuIwt/src/products/library_generators.jl:61 [inlined]
 [4] __init__() at /home/rik/.julia/packages/Pango_jll/XQBSg/src/wrappers/x86_64-linux-gnu.jl:22
 [5] _include_from_serialized(::String, ::Array{Any,1}) at ./loading.jl:697
 [6] _require_search_from_serialized(::Base.PkgId, ::String) at ./loading.jl:782
 [7] _require(::Base.PkgId) at ./loading.jl:1007
 [8] require(::Base.PkgId) at ./loading.jl:928
 [9] require(::Module, ::Symbol) at ./loading.jl:923
 [10] include(::Function, ::Module, ::String) at ./Base.jl:380
 [11] include(::Module, ::String) at ./Base.jl:368
 [12] top-level scope at none:2
 [13] eval at ./boot.jl:331 [inlined]
 [14] eval(::Expr) at ./client.jl:467
 [15] top-level scope at ./none:3
during initialization of module Pango_jll
in expression starting at /home/rik/.julia/packages/Cairo/smWIA/src/Cairo.jl:8
ERROR: LoadError: LoadError: Failed to precompile Cairo [159f3aea-2a34-519c-b102-8c37f9878175] to /home/rik/.julia/compiled/v1.5/Cairo/l6vnT_LSJHI.ji.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] compilecache(::Base.PkgId, ::String) at ./loading.jl:1305
 [3] _require(::Base.PkgId) at ./loading.jl:1030
 [4] require(::Base.PkgId) at ./loading.jl:928
 [5] require(::Module, ::Symbol) at ./loading.jl:923
 [6] include(::String) at ./client.jl:457
 [7] top-level scope at /home/rik/git/Compose.jl/test/runtests.jl:3
 [8] include(::String) at ./client.jl:457
 [9] top-level scope at none:6
in expression starting at /home/rik/git/Compose.jl/test/misc.jl:16
in expression starting at /home/rik/git/Compose.jl/test/runtests.jl:3
ERROR: Package Compose errored during testing

Indeed, it is a precompilation error because

julia> using Cairo
[ Info: Precompiling Cairo [159f3aea-2a34-519c-b102-8c37f9878175]
ERROR: LoadError: InitError: could not load library "/home/rik/.julia/artifacts/079e2f87254bb2465a44de255cd5dde50d4d0b62/lib/libpangocairo-1.0.so"
/usr/lib/libpangoft2-1.0.so.0: undefined symbol: pango_coverage_get_type
Stacktrace:
 [1] dlopen(::String, ::UInt32; throw_error::Bool) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Libdl/src/Libdl.jl:109
 [2] dlopen(::String, ::UInt32) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Libdl/src/Libdl.jl:109
 [3] macro expansion at /home/rik/.julia/packages/JLLWrappers/KuIwt/src/products/library_generators.jl:61 [inlined]
 [4] __init__() at /home/rik/.julia/packages/Pango_jll/XQBSg/src/wrappers/x86_64-linux-gnu.jl:22
 [5] _include_from_serialized(::String, ::Array{Any,1}) at ./loading.jl:697
 [6] _require_search_from_serialized(::Base.PkgId, ::String) at ./loading.jl:782
 [7] _require(::Base.PkgId) at ./loading.jl:1007
 [8] require(::Base.PkgId) at ./loading.jl:928
 [9] require(::Module, ::Symbol) at ./loading.jl:923
 [10] include(::Function, ::Module, ::String) at ./Base.jl:380
 [11] include(::Module, ::String) at ./Base.jl:368
 [12] top-level scope at none:2
 [13] eval at ./boot.jl:331 [inlined]
 [14] eval(::Expr) at ./client.jl:467
 [15] top-level scope at ./none:3
during initialization of module Pango_jll
in expression starting at /home/rik/.julia/packages/Cairo/smWIA/src/Cairo.jl:8
ERROR: Failed to precompile Cairo [159f3aea-2a34-519c-b102-8c37f9878175] to /home/rik/.julia/compiled/v1.5/Cairo/l6vnT_oYsNI.ji.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] compilecache(::Base.PkgId, ::String) at ./loading.jl:1305
 [3] _require(::Base.PkgId) at ./loading.jl:1030
 [4] require(::Base.PkgId) at ./loading.jl:928
 [5] require(::Module, ::Symbol) at ./loading.jl:923

My version of Julia is quite deterministic; defined as follows

{ pkgs, ... }:

let
  papajaBuildInputs = with pkgs.rPackages; [
    afex
    base64enc
    beeswarm
    bookdown
    broom
    knitr
    rlang
    rmarkdown
    rmdfiltr
    yaml
  ];
  papaja = with pkgs.rPackages; buildRPackage {
    name = "papaja";
    src = pkgs.fetchFromGitHub {
      owner = "crsh";
      repo = "papaja";
      rev = "b0a224a5e67e1afff084c46c2854ac6f82b12179";
      sha256 = "14pxnlgg7pzazpyx0hbv9mlvqdylylpb7p4yhh4w2wlcw6sn3rwj";
    };
    # Do not add propagatedBuildInputs = papajaBuildInputs since
    # it might cause a buffer overflow when calling `devtools::document`.
    nativeBuildInputs = papajaBuildInputs;
  };
  rethinkingBuildInputs = with pkgs.rPackages; [
    coda
    mvtnorm
    shape
  ];
  rethinking-pkg = with pkgs.rPackages; buildRPackage {
    name = "rethinking";
    src = pkgs.fetchFromGitHub {
      owner = "rmcelreath";
      repo = "rethinking";
      rev = "3b48ec8dfda4840b9dce096d0cb9406589ef7923";
      sha256 = "1cinz87q0z9vxpmz98r5y2vby3z8av9p8w283ihzvy4pc1fz0r9b";
    };
    nativeBuildInputs = rethinkingBuildInputs;
    propagatedBuildInputs = rethinkingBuildInputs;
  };
  my-r-packages = with pkgs.rPackages; [
      devtools
      dplyr
      ggplot2
      git2r
      latex2exp
      pkgbuild # Used by RStan.
      # rmdfiltr
      # rethinking-pkg
      # rstan
      svglite
      tidyverse
      zip
  ];
  R-with-my-packages = pkgs.rWrapper.override{
    packages = my-r-packages;
  };

  julia_15 = pkgs.stdenv.mkDerivation {
    name = "julia_15";
    src = pkgs.fetchurl {
      url = "https://julialang-s3.julialang.org/bin/linux/x64/1.5/julia-1.5.3-linux-x86_64.tar.gz";
      sha256 = "1w8masmj96afzm55f2vnlv08wi6vr4iha92k3419gvbgvlwck47i";
    };
    installPhase = ''
      mkdir $out
      cp -R * $out/

      # Patch for https://github.com/JuliaInterop/RCall.jl/issues/339.

      echo "patching $out"
      cp -L ${pkgs.stdenv.cc.cc.lib}/lib/libstdc++.so.6 $out/lib/julia/
    '';
    dontStrip = true;
    ldLibraryPath = with pkgs; stdenv.lib.makeLibraryPath [
      stdenv.cc.cc
      zlib
      glib
      xorg.libXi
      xorg.libxcb
      xorg.libXrender
      xorg.libX11
      xorg.libSM
      xorg.libICE
      xorg.libXext
      dbus
      fontconfig
      freetype
      libGL
    ];
  };
  targetPkgs = pkgs: with pkgs; [
    autoconf
    curl
    gnumake
    utillinux
    m4
    gperf
    unzip
    stdenv.cc
    clang
    binutils
    which
    gmp
    libxml2
    cmake

    fontconfig
    openssl
    which
    ncurses
    gtk2-x11
    atk
    gdk_pixbuf
    cairo
    xorg.libX11
    xorg.xorgproto
    xorg.libXcursor
    xorg.libXrandr
    xorg.libXext
    xorg.libSM
    xorg.libICE
    xorg.libX11
    xorg.libXrandr
    xorg.libXdamage
    xorg.libXrender
    xorg.libXfixes
    xorg.libXcomposite
    xorg.libXcursor
    xorg.libxcb
    xorg.libXi
    xorg.libXScrnSaver
    xorg.libXtst
    xorg.libXt
    xorg.libXxf86vm
    xorg.libXinerama
    nspr
    pdf2svg

    # Nvidia note: may need to change cudnn to match cudatoolkit version
    # cudatoolkit_10_0
    # cudnn_cudatoolkit_10_0
    # linuxPackages.nvidia_x11

    julia_15

    # Arpack.jl
    arpack
    gfortran.cc
    (pkgs.runCommand "openblas64_" {} ''
    mkdir -p "$out"/lib/
    ln -s ${openblasCompat}/lib/libopenblas.so "$out"/lib/libopenblas64_.so.0
    '')

    # Cairo.jl
    cairo
    gettext
    pango.out
    glib.out
    # Gtk.jl
    gtk3
    gtk2
    fontconfig
    gdk_pixbuf
    # GR.jl # Runs even without Xrender and Xext, but cannot save files, so those are required
    qt4
    glfw
    freetype

    conda

    #misc
    xorg.libXxf86vm
    xorg.libSM
    xorg.libXtst
    libpng
    expat
    gnome2.GConf
    nss
    
  ];

  env_vars = ''
    export EXTRA_CCFLAGS="-I/usr/include"
    
    # Points RCall to `libR.so`.
    export LD_LIBRARY_PATH="${pkgs.R}/lib/R/lib:$LD_LIBRARY_PATH"
    # Ensure that RCall uses the same R version as used by `libR.so`.
    # export R_HOME="${pkgs.R}/bin"

    # This does not add dependencies (recursively)!
    # Leaving the code since it was an interesting approach.
    # export R_LIBS_USER="$<removed open curly bracket here>
    #  (lib.concatMapStringsSep ":" (path: path + "/library") my-r-packages)
    # }"
    # export R_LIBS_USER=$R_LIBS_USER:$(R -e 'paste(.libPaths(), collapse=":")')

    # This seems to run after nixos-rebuild-switch, so R knows all the packages.
    # Do not set `R_LIBS_USER` since `using RCall` will overwrite it.
    LIBRARIES=$(Rscript -e 'paste(.libPaths(), collapse=":")')
    export R_LIBS_SITE="$(echo $LIBRARIES | cut -c6- | rev | cut -c2- | rev)"
  '';
  extraOutputsToInstall = ["man" "dev"];
  multiPkgs = pkgs: with pkgs; [ zlib ];

  julia-debug = pkgs.buildFHSUserEnv {
    targetPkgs = targetPkgs;
    name = "julia-debug"; # Name used to start this UserEnv
    multiPkgs = multiPkgs;
    runScript = "bash";
    extraOutputsToInstall = extraOutputsToInstall;
    profile = env_vars;
  };

  julia-fhs = pkgs.buildFHSUserEnv {
    targetPkgs = targetPkgs;
    name = "julia"; # Name used to start this UserEnv
    multiPkgs = multiPkgs;
    runScript = "julia";
    extraOutputsToInstall = extraOutputsToInstall;
    profile = env_vars;
  };
in {
  environment.systemPackages = [ 
    julia-fhs
    R-with-my-packages
  ];
}

Apart from Cairo, this definition of Julia has been working flawlessly for months for Julia 1.4 and 1.5. Even in combination with RCall.

[lobingera]

@giordano Do we test Cairo_jll on something like NixOS?

[giordano]

Testing, no. I was told by a few users that libraries provided by JLLs work out-of-the-box in NixOS, contrary to the old BuildDeps.jl setup. Running executables in JLLs is more problematic because NixOS uses its own interpreter/libc (I don't remember exactly what was the issue), but this isn't a concern for Cairo.

No idea about what's the issue reported by Rik. I can only guess that, given the path /usr/lib/libpangoft2-1.0.so.0, an external pango is being loaded, probably by some other packages. Does this happen in a fresh session?

[rikhuijzer]

Does this happen in a fresh session?

I can make a reproducible Dockerfile based on Ubuntu if you want.

It should probably be fixed upstream in https://github.com/nixos/nixpkgs, but before that I have to switch from my own Julia definition to unstable Julia 1.5 etc etc. All in all, this doesn't score high on my priority list.

[giordano]

I can make a reproducible Dockerfile based on Ubuntu if you want.

That might help, but then I miss the connection with NixOS?

[rikhuijzer]

I can make a reproducible Dockerfile based on Ubuntu if you want.

That might help, but then I miss the connection with NixOS?

The idea of NixOS is that all packages and each version are unique. So, I could define Foo to depend on Bar-0.5, and at the same time have Baz depend on Bar-0.3. This all just works because all files for a package are moved into one folder and that folder is set to read-only. To use these Nix packages, you can use NixOS but also the Nix package manager, which can be installed on most Linux distro's including Ubuntu. Essentially, NixOS is the Nix package manager and a way to describe your entire operating system as code.

I'll add it to my TODO list and plan to have the Dockerfile ready somewhere next week.

[rikhuijzer]

Precompilation succeeds on NixOS unstable with Julia 1.5.3, so it seems this has been fixed upstream! 😄

Closing issue.

[rikhuijzer]

Cairo build problems with NixOS are also discussed at NixOS/nixpkgs#20649 (comment).

[giordano]

That's from May 2019? I don't see how that's relevant at all today, since it's before #292 and #293

[rikhuijzer]

Ok, sorry