topic Shared library linked with MKL library fails to load MKL libs in Intel® oneAPI Math Kernel Library

Shared library linked with MKL library fails to load MKL libs

rpls — Mon, 19 Mar 2018 19:39:52 GMT

I'm building a shared library (VST 2.x plugin) on Linux that will be loaded using dlopen() call. I have an own test loader that loads and executes library commands without problems (dlopen(shared_library,RTLD_NOW|RTLD_GLOBAL)).

When I start a DAW program (Renoise Tracker) it loads the shared library without problems but when it tries to execute commands in the shared library / VST 2.x plugin I get the following error message: "Intel MKL FATAL ERROR: Cannot load libmkl_mc3.so or libmkl_def.so."

I tried to start renoise with LD_PRELOAD=/opt/intel/compilers_and_libraries_2017.0.098/linux/mkl/lib/intel64_lin/libmkl_def.so as suggested in some forums but I get an error: "symbol lookup error: /opt/intel/compilers_and_libraries_2017.0.098/linux/mkl/lib/intel64_lin/libmkl_def.so: undefined symbol: mkl_sparse_optimize_bsr_trsm_i8".

I link MKL shared libraries with the following command-line switches: -Wl,--no-as-needed -lmkl_intel_lp64 -lmkl_gnu_thread -lmkl_core -lgomp -lpthread -lm -ldl. And the LD_LIBRARY_PATH is set as follows: LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2017.0.098/linux/compiler/lib/intel64:/opt/intel/compilers_and_libraries_2017.0.098/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.0.098/linux/tbb/lib/intel64/gcc4.4:/opt/intel/compilers_and_libraries_2017.0.098/linux/compiler/lib/intel64:/opt/intel/compilers_and_libraries_2017.0.098/linux/mkl/lib/intel64

Do someone know how to fix this? I've compiled many programs with MKL and they work fine. I think Renoise either uses some special parameters when using dlopen() or sets environment so that libraries cannot be found and/or MKL tries to load some other libraries than normally. Maybe I should link shared library with some other MKL libraries too?

Other libraries in normal locations like /usr/lib seem to load and work without any problems.

Hi Tomas,

Ying_H_Intel — Tue, 20 Mar 2018 01:21:22 GMT

Hi Tomas,

May you pelase try to link as the https://software.intel.com/en-us/articles/intel-mkl-link-line-advisor/

1. using static mkl like -Wl,--start-group ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a ${MKLROOT}/lib/intel64/libmkl_gnu_thread.a ${MKLROOT}/lib/intel64/libmkl_core.a -Wl,--end-group -lgomp -lpthread -lm -ldl

2. or

-lmkl_rt -lpthread -lm -ldl

to replace the library -Wl,--no-as-needed -lmkl_intel_lp64 -lmkl_gnu_thread -lmkl_core -lgomp

and see if it work?

Best Regards,

Ying

Hi,

Anonymous92 — Thu, 22 Mar 2018 23:14:52 GMT

Hi,

I just ran into the same issue as well and was able to debug the problem using LD_DEBUG=all. I have a shared library that is being dlopen'd and contains a dependency to MKL. My shared library is linked with libmkl_core.so, libmkl_sequential.so, and libmkl_intel_lp64.so.

When the MKL function is called, the mkl_core library is dispatching to libmkl_avx.so and libmkl_def.so and proceeding to dlopen these files. However, the scope of the MKL dispatch library is my executable's library list and the MKL dispatch library's list. My executable doesn't contain the MKL libraries since it is a dependency of the library I originally dlopen'd and the dispatch libraries don't contain the main MKL files in its library list so it ends up getting a fatal error and crashes the program

I think there are 3 options for me to handle this now which are not ideal...
- I've been able to get it working is by doing an LD_PRELOAD containing all of the main MKL libraries I need (mkl_core.so, mkl_sequential.so, mkl_intel_lp64.so) when starting the executable. But this is not ideal because then the libraries are being loaded even if the library using MKL isn't used.
- Dynamically link the shared objects into the executable. But this adds MKL dependencies on the executable when they may not be needed
- Statically link these dispatch libraries into the library with the MKL dependency but then that would also defeat the purpose of using the shared libraries by increasing the library size with unused symbols
- Build a custom MKL shared object containing all of the functions needed. Requires knowing what functions will be called now and in the future

Let me know if you need some more information or a test program to show the problem