Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Becker__Neal
Beginner
790 Views

Cannot load libmkl_avx.so or libmkl_def.so.

Any clues how I can debug this?

python test7.py

Intel MKL FATAL ERROR: Cannot load libmkl_avx.so or libmkl_def.so.

I do have these libraries.  I also tried setting LD_LIBRARY_PATH to where these libraries reside, in

../ext/lib

 

0 Kudos
14 Replies
Robert_C_Intel
Employee
790 Views

LD_DEBUG="files libs" LD_DEBUG_OUTPUT=ld.log python test7.py

And then looking in ld.log.* might give a hint since it will show libraries being loaded and search paths. Let me know if that doesn't help and I can ask MKL team. MKL might be using dlopen to load this library.

Becker__Neal
Beginner
790 Views

I think the problems are caused by cvxopt. I have cvxopt from conda install (not intel, which lacks it). I had tried to pip install cvxopt, but the build failed, couldn't find lapack and blas. Strange, because mkl supplies lapack and blas, and because I tried installing openblas, which should also supply them.
Becker__Neal
Beginner
790 Views

9993: file=/home/nbecker/anaconda/envs/idp/bin/libmkl_avx.so [0]; dynamically loaded by /home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../../libmkl_core.so [0] 9993: 9993: file=libmkl_avx.so [0]; dynamically loaded by /home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../../libmkl_core.so [0] 9993: find library=libmkl_avx.so [0]; searching 9993: search path=/home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../.././tls/x86_64:/home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../.././tls:/home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../.././x86_64:/home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../../. (RPATH from file /home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../../libmkl_core.so) 9993: trying file=/home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../.././tls/x86_64/libmkl_avx.so 9993: trying file=/home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../.././tls/libmkl_avx.so 9993: trying file=/home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../.././x86_64/libmkl_avx.so 9993: trying file=/home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../.././libmkl_avx.so 9993: 9993: file=libmkl_avx.so [0]; generating link map 9993: dynamic: 0x00007f10c99b81c0 base: 0x00007f10c769e000 size: 0x0000000002365aa0 9993: entry: 0x00007f10c7767400 phdr: 0x00007f10c99b8000 phnum: 8 9993: 9993: /home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../.././libmkl_avx.so: error: symbol lookup error: undefined symbol: mkl_sparse_optimize_bsr_trsm_i8 (fatal) 9993: 9993: file=/home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../.././libmkl_avx.so [0]; destroying link map 9993: /home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../../libmkl_core.so: error: symbol lookup error: undefined symbol: COIProcessLoadSinkLibraryFromFile (fatal) 9993:
Becker__Neal
Beginner
790 Views

pip install -v cvxopt [...] /home/nbecker/anaconda/envs/idp/ext/lib -L/home/nbecker/anaconda/envs/idp/ext/lib -z noexecstack -z relro -z now -fstack-protector build/temp.linux-x86_64-3.5/src/C/base.o build/temp.linux-x86_64-3.5/src/C/dense.o build/temp.linux-x86_64-3.5/src/C/sparse.o -L/usr/lib -L/home/nbecker/anaconda/envs/idp/lib -lm -llapack -lblas -lpython3.5m -o build/lib.linux-x86_64-3.5/cvxopt/base.cpython-35m-x86_64-linux-gnu.so /usr/bin/ld: cannot find -llapack /usr/bin/ld: cannot find -lblas collect2: error: ld returned 1 exit status error: command 'gcc' failed with exit status 1
Robert_C_Intel
Employee
790 Views

Can you send me test7.py so I can try to reproduce the original problem:

 

python test7.py

Becker__Neal
Beginner
790 Views

test7.py has lots of dependencies on my own wrapped c++ code

I think if you can build an intel version of cvxopt maybe that will fix it

Robert_C_Intel
Employee
790 Views

You are probably right. The conda cvxopt depends on mkl and there is probably a conflict. I will try to make a simpler reproducer.

Islam_A_
Beginner
790 Views

I think the problems are caused by cvxopt. I have cvxopt from conda install (not intel, which lacks it).

I had tried to pip install cvxopt, but the build failed, couldn't find lapack and blas. Strange, because mkl supplies lapack and blas, and because I tried installing openblas, which should also supply them.

VuiSeng_C_Intel
Employee
790 Views

Hi guys, we are running into the exact undefined symbols. 

My use case is caffe build with mkl and python bindings. we're using 2018.0.128 .

Caffe command line works fine but the issue came when we import caffe in python.

"pip install cvxopt" doesn't work.

similar thread on this issue: https://github.com/BVLC/caffe/issues/3884

Pls share if we have a solution.

Thanks.

790 Views

Hi Vui Seng,

Thanks for reporting. Are you attempting to build your own Intel Caffe with Intel Python, or are you seeing this issue when using the Caffe that comes bundled with Intel Python?

Thanks,

Chris

AndrewC
New Contributor I
790 Views

I am struggling with the same issues with python calling C code that calls MKL.Was this resolved?

AndrewC
New Contributor I
790 Views

My understanding is that Python sets a default for the loading of dynamic modules that results in this error.

AndrewC
New Contributor I
790 Views

Solved the issue. Needed to make sure that the Makefile that builds the Python module has -lmkl_rt and -lmkl_def on the link line.

Rohit_J_Intel
Employee
790 Views

If you require cvxopt to utilize blas and lapack routines from Intel MKL (i.e. shipped with IDP), you can build cvxopt from its sources. The instructions are as follows:

<IDP_installation>/bin/conda create -n cvxopt_install python=3.6 setuptools mkl
. <IDP_installation>/bin/activate cvxopt_install

#Fetch sources for cvxopt
git clone https://github.com/cvxopt/cvxopt.git
cd cvxopt
git checkout `git describe --abbrev=0 --tags`

#Fetch SuiteSparse (a runtime requirement)
wget http://faculty.cse.tamu.edu/davis/SuiteSparse/SuiteSparse-4.5.4.tar.gz
tar -xf SuiteSparse-4.5.4.tar.gz

#Export variables required for building cvxopt with MKL
export PREFIX_LIB=`which python`/../lib
export CVXOPT_LAPACK_LIB=mkl_rt
export CVXOPT_BLAS_LIB=mkl_rt
export CVXOPT_SUITESPARSE_SRC_DIR=`pwd`/SuiteSparse
export CVXOPT_BLAS_LIB_DIR=${PREFIX_LIB}
export CVXOPT_BLAS_EXTRA_LINK_ARGS="-L${PREFIX_LIB};-Wl,-rpath,${PREFIX_LIB};-lmkl_rt"

python setup.py install

#post-build clean-up:
for i in `env | grep -E "CVXOPT|PREFIX_LIB" | cut -d '=' -f 1`; do unset $i; done
#cd ..; rm -rf cvxopt

Thanks,
Rohit

Reply