missing rpath for libmkl_intel_thread? #6423
Comments
|
From @stevengj on October 4, 2017 18:40
so it looks like If I run Notice that it has both the absolute path of my conda Clearly, there's something I don't understand about how MacOS resolves shared-library search paths. Something is not getting set correctly when we open But what has changed recently in Conda's NumPy package that would cause this to suddenly appear? Are you linking differently than you used to? Did you only recently start using |
|
From @stevengj on October 5, 2017 15:15 It seems like the problem is specific to julia> Libdl.dlopen("/Users/stevenj/.julia/v0.6/Conda//deps/usr/lib/libiomp5.dylib")
Libdl.dlopen("/Users/stevenj/.julia/v0.6/Conda//deps/usr/lib/libmkl_intel_thread.dylib")
julia> .... call numpy.linalg.inv, it works ....(e.g. there is no problem with |
|
From @stevengj on October 5, 2017 15:21 cc @asmeurer, who has worked on |
|
Seems related to #6401 |
|
@msarahan That seems to be related to a Python 3-only change, while this problem occurs with (at least) Python 2.7. |
|
@msarahan, note also that the solution in #6401 was to dlopen libpython with RTLD_GLOBAL, but we already use |
|
@stevengj Could you provide output for |
|
Older builds used this patch: https://github.com/AnacondaRecipes/numpy-recipe/blob/master/recipe/dlopenflags.patch Based on advice from Intel, we dropped that patch for the latest packages. Maybe that's related? Our 2018 MKL packages come more directly from Intel. We take their packages and apply metadata to them to indicate that we repackage them. Additionally, on Intel's advice, we have switched site.cfg from explicit specification: to their central runtime, which is controllable via environment variables: |
|
|
This bug impacts also the pyplot performance: |
|
@msarahan I'm not sure if you're quoting the I'll try to see locally if the observed errors may be due to removal of |
|
Thanks @oleksandr-pavlyk - library_dirs and include_dir are there, but they're added later in our script. Don't pay too much attention to that. The options I see here, dependeing on @oleksandr-pavlyk's findings are:
We do not have time to explore these options. If anyone in the julia world would like to explore these options, our recipes for these are at: https://github.com/AnacondaRecipes/aggregate/tree/master/intel_repack We're definitely open to changes that would make things work better for everyone. |
|
I tried to reproduce the issue on Mac, using Julia 0.6.0 DMG file from https://julialang.org/downloads/ I am unable to even install "PyCall" seeing I am able to clone Since I am unable to reproduce the issue, I can only suggest to try the call with Intel distribution for Python, which you can install into the existing Miniconda installation using then use The point of this experiment is to confirm whether removal of Alternatively, you could edit at the top of the file before Please let us know if this fixes your issue. |
|
@oleksandr-pavlyk, sorry to hear that you ran into an error with I can confirm that the Intel Distribution for Python (via the command you suggested) eliminates the problem. Editing |
|
@stevengj In the interest of reproducibility of the issue, could you specify output of I created with the following output which shows that the recent version of NumPy's linear algebra code is correctly linked against It would be best if the issue was reproducible though. |
|
My output is: |
|
The issue seems to be quite reproducible — multiple people using Anaconda via PyCall almost simultaneously reported the issue after a recent conda upgrade, and I immediately got the same issue on my MacOS machine by updating conda. The issue is not that NumPy isn't linked to It must be something else in how the shared libraries are set up in Anaconda (compared to the Intel distribution). |
|
I know what the problem is, it's because of how we link to Here are two workarounds (you need Horrible (but will survive updates and reinstalls of Much less horrible but will not survive updates to Adjust paths to taste. Cheers! |
|
@mingwandroid, thanks! Is there any chance of some form of the second solution being incorporated into Anaconda's build? What is the Intel Distribution for Python doing differently? |
|
The |
|
@stevengj, if you want I can give you a detailed analysis? |
|
I would love to hear the details of what is going on, though of course updates to the MKL packages are mainly up to you guys. (Is it possible to disable the use of OpenMP in NumPy at runtime, then?) |
|
I can verify that the |
|
Thank you for the feedback @stevengj. Here is my analysis:
.. should output: Ok, great, we've been able to reproduce this.
(observe a mix of old and new software, this is not the cause of this problem, but it is sub-optimal. AFAICT,
.. and here, near the end of the 7500 odd lines of output we see the problem: So Apple's dynamic loader failed to load
.. our .. Intel's
What has happened here is that we has followed the recommended (and I agree, best) advice on how to link and what I suspect is the correct (or at least best) fix: |
|
@mingwandroid, Which version of Intel python did you find that multiarray.so links explicitly to threading implementations other than mkl_rt? Here is what I get after installing intelpython27-2018.0.018. Same with intelpython35-2018.0.018: |
|
Seems I drove IDP badly and the numpy I was looking at was from the old defaults channel: Adding |
|
@mingwandroid The reason why
|
|
The test package has been released to
.. then test with:
.. and hopefully observe a lack of any |
|
Reopening until this is fixed in the |
|
|
|
Packages have been released to |
|
Issue recurs for me. Specifically, I’m using Julia 1.6.0 and the version of Conda I have is 4.1.0. Is this a regression? |
|
I'm now seeing this in Julia 1.6.1, as well, after making a change to get Python package gdstk working. I then deleted trees ~/.julia/conda, ~/.julia/packages/Conda, ~/.julia/packages/PyCall, and ~/.julia/packages/PyPlot. In a new Julia session, did ENV["PYTHON"]="", Pkg.add PyCall, Pkg.add PyPlot. using PyPlot seemed to work fine (successfully plot), but then in another Julia session using PyPlot gives the abortive Intel MKL FATAL ERROR: Cannot load libmkl_intel_thread.dylib. |
From @stevengj on October 4, 2017 18:23
In Julia, which loads
libpythonas a shared library (installed via conda by default), we are suddenly (with a recent conda upgrade) getting MKL failures on MacOS because it can't findlibmkl_intel_thread.dylib. Manually settingLD_LIBRARY_PATHto Conda'susr/libdirectory (where that library is found) fixes the problem, but that didn't used to be required, and isn't required if I run thepythonexecutable directly.My guess is that there is a missing shared-library dependency somewhere — something in numpy should have explicitly linked
-lmkl_intel_threadbut didn't do so? Or maybe a missing-rpathlinker argument?For example, we get this when plotting via matplotlib (JuliaPy/PyPlot.jl#315), but also when calling any numpy linear-algebra function:
(This just calls
numpy.linalg.invon a random 100x100 numpy matrix. Under the hood, it is callingPyImport_ImportModule("numpy.linalg")inlibpython, etcetera, using Python's standard C embedding API.)Copied from original issue: conda/conda#6074
The text was updated successfully, but these errors were encountered: