Segfault Loading Numba

[xanderdunn]

When I add an import numba to a Python file that I load via PyCall, I get this segmentation fault at runtime:

signal (11): Segmentation fault: 11
_pthread_rwlock_lock at /usr/lib/system/libsystem_pthread.dylib (unknown line)
_ZN4llvm3sys11RWMutexImpl14writer_acquireEv at /usr/local/lib/python2.7/site-packages/llvmlite/binding/libllvmlite.dylib (unknown line)
_ZN4llvm3sys12SmartRWMutexILb1EE4lockEv at /usr/local/lib/python2.7/site-packages/llvmlite/binding/libllvmlite.dylib (unknown line)
_ZN4llvm12PassRegistry23addRegistrationListenerEPNS_24PassRegistrationListenerE at /usr/local/lib/python2.7/site-packages/llvmlite/binding/libllvmlite.dylib (unknown line)
_ZN4llvm2cl4listIPKNS_8PassInfoEbNS_14PassNameParserEEC2IA13_cNS0_4descENS0_12OptionHiddenEEERKT_RKT0_RKT1_ at /usr/local/lib/python2.7/site-packages/llvmlite/binding/libllvmlite.dylib (unknown line)
Segmentation fault: 11

Running Python from the command line and importing numba doesn't cause any errors.

PyCall and command line are using the same Python installation. I rebuilt PyCall to make sure.

llvm being used was built with brew install llvm.

[stevengj]

Possibly there is a conflict between the LLVM version used in Julia and the LLVM version used in Numba? Loading two versions of the same shared library simultaneously is often asking for trouble (see e.g. #65).

[xanderdunn]

@stevengj Thanks for your input.

versioninfo() in Julia shows it's using LLVM 3.3. I installed numba 0.16.0, which also uses llvm 3.3 and an assertion fails when I try to load a Python file with import numba:

Assertion failed: (rwlock != nullptr), function writer_acquire, file RWMutex.cpp, line 100.

signal (6): Abort trap: 6
__pthread_kill at /usr/lib/system/libsystem_kernel.dylib (unknown line)
Abort trap: 6

I also tried both using LLVM 3.6. Back on most recent numba using llvm 3.6, I tried rebuilding Julia so that it uses LLVM 3.6, the same version that numba is using, and got this segmentation fault:

Loading Python module myfile.py
Couldn't import dot_parser, loading of dot files will not be possible.
Segmentation fault: 11

[stevengj]

@Keno, any thoughts on whether Julia's use of LLVM would conflict with Numba's?

[Keno]

Only if either of them use LLVM as a shared library.

[stevengj]

Aren't both of them using LLVM as a shared library?

[stevengj]

@xanderdunn, what is your versioninfo() for Julia?

[Keno]

By default julia statically links LLVM.

[xanderdunn]

versioninfo is what I was referencing above. Back on the default 3.3, here it is:

julia> versioninfo()
Julia Version 0.4.2
Commit bb73f34* (2015-12-06 21:47 UTC)
Platform Info:
  System: Darwin (x86_64-apple-darwin15.0.0)
  CPU: Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz
  WORD_SIZE: 64
  BLAS: libopenblas (DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas
  LIBM: libopenlibm
  LLVM: libLLVM-3.3

[GaZ3ll3]

I used LLVM3.7 for latest numba (self-built llvmlite) on ubuntu, there is no problem importing numba.

[stevengj]

Thanks for the update, @GaZ3ll3. Hopefully the problem has disappeared with recent versions.

[stevengj]

Closing as this seems to be resolved.

[cortner]

I tried to make sure Anaconda uses the correct llvm by installing

Sunrise:~ ortner$ conda install -c anaconda llvm=3.3
Using Anaconda Cloud api site https://api.anaconda.org
Fetching package metadata: ......
Solving package specifications: .........

# All requested packages already installed.
# packages in environment at /Users/ortner/anaconda:
#
llvm                      3.3                           0    anaconda

but @pyimport numba still fails:

julia> using PyCall

julia> @pyimport numba

signal (11): Segmentation fault: 11
_pthread_rwlock_lock at /usr/lib/system/libsystem_pthread.dylib (unknown line)
_ZN4llvm3sys11RWMutexImpl14writer_acquireEv at /Users/ortner/anaconda/lib/python2.7/site-packages/llvmlite/binding/libllvmlite.dylib (unknown line)
_ZN4llvm12PassRegistry23addRegistrationListenerEPNS_24PassRegistrationListenerE at /Users/ortner/anaconda/lib/python2.7/site-packages/llvmlite/binding/libllvmlite.dylib (unknown line)
Segmentation fault: 11

julia> versioninfo()
Julia Version 0.4.5
Commit 2ac304d (2016-03-18 00:58 UTC)
Platform Info:
  System: Darwin (x86_64-apple-darwin13.4.0)
  CPU: Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.3

[cortner]

Update:

After fixing the number installation (back to LLVM 3.7 via llvmlite) I tried on julia 0.5, and @pyimport numba fails here as well for me:

julia> @pyimport numba

signal (11): Segmentation fault: 11
while loading no file, in expression starting on line 379
_pthread_rwlock_lock at /usr/lib/system/libsystem_pthread.dylib (unknown line)
_ZN4llvm3sys11RWMutexImpl14writer_acquireEv at /Users/ortner/anaconda/lib/python2.7/site-packages/llvmlite/binding/libllvmlite.dylib (unknown line)
_ZN4llvm12PassRegistry23addRegistrationListenerEPNS_24PassRegistrationListenerE at /Users/ortner/anaconda/lib/python2.7/site-packages/llvmlite/binding/libllvmlite.dylib (unknown line)
Allocations: 2418975 (Pool: 2416729; Big: 2246); GC: 3
Segmentation fault: 11

julia> versioninfo()
Julia Version 0.5.0-dev+4669
Commit 67c818a (2016-06-11 04:37 UTC)
Platform Info:
  System: Darwin (x86_64-apple-darwin13.4.0)
  CPU: Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.7.1 (ORCJIT, has well)

[rofinn]

@cortner Did you ever get this figured out? I can't seem to get numba and llvmlite to work with PyCall without it segfaulting (I even tried building llvmlite with julia's llvm-config).

[cortner]

I think only partially. Eventually I just gave up.

[rofinn]

FWIW, the solution to my problem had 2 parts.

1. Install the specific version of llvmlite that supports the same minor release of llvm as julia. While this is pretty obvious here is the code used in my build.jl.

# We need install a specific version of llvmlite which supports the same minor version of
# llvm used with julia. https://github.com/numba/llvmlite#compatibility
const llvm_version = VersionNumber(Base.libllvm_version)
if llvm_version > v"4.0.0"
    pip_install("llvmlite==0.17.0")
elseif llvm_version > v"3.9.0"
    pip_install("llvmlite==0.16.0")
elseif llvm_version > v"3.8.0"
    pip_install("llvmlite==0.15.0")
elseif llvm_version > v"3.7.0"
    pip_install("llvmlite==0.12.1")
elseif llvm_version > v"3.6.0"
    pip_install("llvmlite==0.8.0")
else
    error("Can't find a version of llvmlite that support $llvm_version")
end

NOTE: pip_install just calls pip install <pkg> with the pip bundled with Conda.jl

2. I needed to disable the numba jit before using PyCall in my module (vs putting it in my __init__() before importing any python packages).

# This needs to be set before `using PyCall` otherwise the numba library won't see it
# at load time ? :/
ENV["NUMBA_DISABLE_JIT"] = 1

using PyCall

[racinmat]

Happened to me too.
I have julia 1.3.0-rc4, which runs on LLVM 6.0.1, so I had to downgrade numba to 0.41 and llvmlite to 0.26, and then it started working.

[thoth291]

None of the above worked for me: Julia 1.4.2
Reproducer: it assumes you have Conda with numba and if needed it all synced in terms of llvm versions

using Conda
using PyCall
#Look here for compatibility https://github.com/numba/llvmlite#compatibility
# with this Base.libllvm_version
#  For my v"8.0.1" it was llvm<=0.32.0 which is numba=0.48
#Conda.add("numba=0.48")

py"""
from numba import njit

@njit
def py_numba_sum(A):
    s = 0.0
    for a in A:
        s += a
    return s
"""

sum_numba_py = py"py_numba_sum"

Any advise or help would be highly appreciated!