Intel® Distribution for Python*
Engage in discussions with community peers related to Python* applications and core computational packages.

Cannot load libmkl_avx.so or libmkl_def.so.

Becker__Neal
Beginner
2,713 Views

Any clues how I can debug this?

python test7.py

Intel MKL FATAL ERROR: Cannot load libmkl_avx.so or libmkl_def.so.

I do have these libraries.  I also tried setting LD_LIBRARY_PATH to where these libraries reside, in

../ext/lib

 

0 Kudos
14 Replies
Robert_C_Intel
Employee
2,713 Views

LD_DEBUG="files libs" LD_DEBUG_OUTPUT=ld.log python test7.py

And then looking in ld.log.* might give a hint since it will show libraries being loaded and search paths. Let me know if that doesn't help and I can ask MKL team. MKL might be using dlopen to load this library.

0 Kudos
Becker__Neal
Beginner
2,713 Views
I think the problems are caused by cvxopt. I have cvxopt from conda install (not intel, which lacks it). I had tried to pip install cvxopt, but the build failed, couldn't find lapack and blas. Strange, because mkl supplies lapack and blas, and because I tried installing openblas, which should also supply them.
0 Kudos
Becker__Neal
Beginner
2,713 Views
9993: file=/home/nbecker/anaconda/envs/idp/bin/libmkl_avx.so [0]; dynamically loaded by /home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../../libmkl_core.so [0] 9993: 9993: file=libmkl_avx.so [0]; dynamically loaded by /home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../../libmkl_core.so [0] 9993: find library=libmkl_avx.so [0]; searching 9993: search path=/home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../.././tls/x86_64:/home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../.././tls:/home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../.././x86_64:/home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../../. (RPATH from file /home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../../libmkl_core.so) 9993: trying file=/home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../.././tls/x86_64/libmkl_avx.so 9993: trying file=/home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../.././tls/libmkl_avx.so 9993: trying file=/home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../.././x86_64/libmkl_avx.so 9993: trying file=/home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../.././libmkl_avx.so 9993: 9993: file=libmkl_avx.so [0]; generating link map 9993: dynamic: 0x00007f10c99b81c0 base: 0x00007f10c769e000 size: 0x0000000002365aa0 9993: entry: 0x00007f10c7767400 phdr: 0x00007f10c99b8000 phnum: 8 9993: 9993: /home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../.././libmkl_avx.so: error: symbol lookup error: undefined symbol: mkl_sparse_optimize_bsr_trsm_i8 (fatal) 9993: 9993: file=/home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../.././libmkl_avx.so [0]; destroying link map 9993: /home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../../libmkl_core.so: error: symbol lookup error: undefined symbol: COIProcessLoadSinkLibraryFromFile (fatal) 9993:
0 Kudos
Becker__Neal
Beginner
2,713 Views
pip install -v cvxopt [...] /home/nbecker/anaconda/envs/idp/ext/lib -L/home/nbecker/anaconda/envs/idp/ext/lib -z noexecstack -z relro -z now -fstack-protector build/temp.linux-x86_64-3.5/src/C/base.o build/temp.linux-x86_64-3.5/src/C/dense.o build/temp.linux-x86_64-3.5/src/C/sparse.o -L/usr/lib -L/home/nbecker/anaconda/envs/idp/lib -lm -llapack -lblas -lpython3.5m -o build/lib.linux-x86_64-3.5/cvxopt/base.cpython-35m-x86_64-linux-gnu.so /usr/bin/ld: cannot find -llapack /usr/bin/ld: cannot find -lblas collect2: error: ld returned 1 exit status error: command 'gcc' failed with exit status 1
0 Kudos
Robert_C_Intel
Employee
2,713 Views

Can you send me test7.py so I can try to reproduce the original problem:

 

python test7.py

0 Kudos
Becker__Neal
Beginner
2,713 Views

test7.py has lots of dependencies on my own wrapped c++ code

I think if you can build an intel version of cvxopt maybe that will fix it

0 Kudos
Robert_C_Intel
Employee
2,713 Views

You are probably right. The conda cvxopt depends on mkl and there is probably a conflict. I will try to make a simpler reproducer.

0 Kudos
Islam_A_
Beginner
2,713 Views

I think the problems are caused by cvxopt. I have cvxopt from conda install (not intel, which lacks it).

I had tried to pip install cvxopt, but the build failed, couldn't find lapack and blas. Strange, because mkl supplies lapack and blas, and because I tried installing openblas, which should also supply them.

0 Kudos
VuiSeng_C_Intel
Employee
2,713 Views

Hi guys, we are running into the exact undefined symbols. 

My use case is caffe build with mkl and python bindings. we're using 2018.0.128 .

Caffe command line works fine but the issue came when we import caffe in python.

"pip install cvxopt" doesn't work.

similar thread on this issue: https://github.com/BVLC/caffe/issues/3884

Pls share if we have a solution.

Thanks.

0 Kudos
Christophe_H_Intel2
2,713 Views

Hi Vui Seng,

Thanks for reporting. Are you attempting to build your own Intel Caffe with Intel Python, or are you seeing this issue when using the Caffe that comes bundled with Intel Python?

Thanks,

Chris

0 Kudos
AndrewC
New Contributor III
2,718 Views

I am struggling with the same issues with python calling C code that calls MKL.Was this resolved?

0 Kudos
AndrewC
New Contributor III
2,718 Views

My understanding is that Python sets a default for the loading of dynamic modules that results in this error.

0 Kudos
AndrewC
New Contributor III
2,718 Views

Solved the issue. Needed to make sure that the Makefile that builds the Python module has -lmkl_rt and -lmkl_def on the link line.

0 Kudos
Rohit_J_Intel
Employee
2,718 Views

If you require cvxopt to utilize blas and lapack routines from Intel MKL (i.e. shipped with IDP), you can build cvxopt from its sources. The instructions are as follows:

<IDP_installation>/bin/conda create -n cvxopt_install python=3.6 setuptools mkl
. <IDP_installation>/bin/activate cvxopt_install

#Fetch sources for cvxopt
git clone https://github.com/cvxopt/cvxopt.git
cd cvxopt
git checkout `git describe --abbrev=0 --tags`

#Fetch SuiteSparse (a runtime requirement)
wget http://faculty.cse.tamu.edu/davis/SuiteSparse/SuiteSparse-4.5.4.tar.gz
tar -xf SuiteSparse-4.5.4.tar.gz

#Export variables required for building cvxopt with MKL
export PREFIX_LIB=`which python`/../lib
export CVXOPT_LAPACK_LIB=mkl_rt
export CVXOPT_BLAS_LIB=mkl_rt
export CVXOPT_SUITESPARSE_SRC_DIR=`pwd`/SuiteSparse
export CVXOPT_BLAS_LIB_DIR=${PREFIX_LIB}
export CVXOPT_BLAS_EXTRA_LINK_ARGS="-L${PREFIX_LIB};-Wl,-rpath,${PREFIX_LIB};-lmkl_rt"

python setup.py install

#post-build clean-up:
for i in `env | grep -E "CVXOPT|PREFIX_LIB" | cut -d '=' -f 1`; do unset $i; done
#cd ..; rm -rf cvxopt

Thanks,
Rohit

0 Kudos
Reply