topic test7.py has lots of in Intel® Distribution for Python*

Cannot load libmkl_avx.so or libmkl_def.so.

Becker__Neal — Thu, 09 Jun 2016 17:54:28 GMT

Any clues how I can debug this?

python test7.py

Intel MKL FATAL ERROR: Cannot load libmkl_avx.so or libmkl_def.so.

I do have these libraries. I also tried setting LD_LIBRARY_PATH to where these libraries reside, in

../ext/lib

LD_DEBUG="files libs" LD

Robert_C_Intel — Thu, 09 Jun 2016 18:23:58 GMT

LD_DEBUG="files libs" LD_DEBUG_OUTPUT=ld.log python test7.py

And then looking in ld.log.* might give a hint since it will show libraries being loaded and search paths. Let me know if that doesn't help and I can ask MKL team. MKL might be using dlopen to load this library.

I think the problems are

Becker__Neal — Fri, 10 Jun 2016 11:00:41 GMT

I think the problems are caused by cvxopt. I have cvxopt from conda install (not intel, which lacks it). I had tried to pip install cvxopt, but the build failed, couldn't find lapack and blas. Strange, because mkl supplies lapack and blas, and because I tried installing openblas, which should also supply them.

9993: file=/home/nbecker

Becker__Neal — Fri, 10 Jun 2016 11:00:58 GMT

9993: file=/home/nbecker/anaconda/envs/idp/bin/libmkl_avx.so [0]; dynamically loaded by /home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../../libmkl_core.so [0] 9993: 9993: file=libmkl_avx.so [0]; dynamically loaded by /home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../../libmkl_core.so [0] 9993: find library=libmkl_avx.so [0]; searching 9993: search path=/home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../.././tls/x86_64:/home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../.././tls:/home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../.././x86_64:/home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../../. (RPATH from file /home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../../libmkl_core.so) 9993: trying file=/home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../.././tls/x86_64/libmkl_avx.so 9993: trying file=/home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../.././tls/libmkl_avx.so 9993: trying file=/home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../.././x86_64/libmkl_avx.so 9993: trying file=/home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../.././libmkl_avx.so 9993: 9993: file=libmkl_avx.so [0]; generating link map 9993: dynamic: 0x00007f10c99b81c0 base: 0x00007f10c769e000 size: 0x0000000002365aa0 9993: entry: 0x00007f10c7767400 phdr: 0x00007f10c99b8000 phnum: 8 9993: 9993: /home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../.././libmkl_avx.so: error: symbol lookup error: undefined symbol: mkl_sparse_optimize_bsr_trsm_i8 (fatal) 9993: 9993: file=/home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../.././libmkl_avx.so [0]; destroying link map 9993: /home/nbecker/anaconda/envs/idp/lib/python3.5/site-packages/cvxopt/../../../libmkl_core.so: error: symbol lookup error: undefined symbol: COIProcessLoadSinkLibraryFromFile (fatal) 9993:

pip install -v cvxopt

Becker__Neal — Fri, 10 Jun 2016 11:03:50 GMT

pip install -v cvxopt [...] /home/nbecker/anaconda/envs/idp/ext/lib -L/home/nbecker/anaconda/envs/idp/ext/lib -z noexecstack -z relro -z now -fstack-protector build/temp.linux-x86_64-3.5/src/C/base.o build/temp.linux-x86_64-3.5/src/C/dense.o build/temp.linux-x86_64-3.5/src/C/sparse.o -L/usr/lib -L/home/nbecker/anaconda/envs/idp/lib -lm -llapack -lblas -lpython3.5m -o build/lib.linux-x86_64-3.5/cvxopt/base.cpython-35m-x86_64-linux-gnu.so /usr/bin/ld: cannot find -llapack /usr/bin/ld: cannot find -lblas collect2: error: ld returned 1 exit status error: command 'gcc' failed with exit status 1

Can you send me test7.py so I

Robert_C_Intel — Fri, 10 Jun 2016 11:20:02 GMT

Can you send me test7.py so I can try to reproduce the original problem:

python test7.py

test7.py has lots of

Becker__Neal — Fri, 10 Jun 2016 11:22:52 GMT

test7.py has lots of dependencies on my own wrapped c++ code

I think if you can build an intel version of cvxopt maybe that will fix it

You are probably right. The

Robert_C_Intel — Fri, 10 Jun 2016 11:47:11 GMT

You are probably right. The conda cvxopt depends on mkl and there is probably a conflict. I will try to make a simpler reproducer.

I think the problems are

Islam_A_ — Wed, 10 Aug 2016 20:12:42 GMT

I think the problems are caused by cvxopt. I have cvxopt from conda install (not intel, which lacks it).

I had tried to pip install cvxopt, but the build failed, couldn't find lapack and blas. Strange, because mkl supplies lapack and blas, and because I tried installing openblas, which should also supply them.

Hi guys, we are running into

VuiSeng_C_Intel — Fri, 29 Sep 2017 21:06:18 GMT

Hi guys, we are running into the exact undefined symbols.

My use case is caffe build with mkl and python bindings. we're using 2018.0.128 .

Caffe command line works fine but the issue came when we import caffe in python.

"pip install cvxopt" doesn't work.

similar thread on this issue: https://github.com/BVLC/caffe/issues/3884

Pls share if we have a solution.

Thanks.

Hi Vui Seng,

Christophe_H_Intel2 — Mon, 02 Oct 2017 14:40:00 GMT

Hi Vui Seng,

Thanks for reporting. Are you attempting to build your own Intel Caffe with Intel Python, or are you seeing this issue when using the Caffe that comes bundled with Intel Python?

Thanks,

Chris

I am struggling with the same

AndrewC — Tue, 31 Oct 2017 23:43:00 GMT

I am struggling with the same issues with python calling C code that calls MKL.Was this resolved?

My understanding is that

AndrewC — Wed, 01 Nov 2017 01:57:21 GMT

My understanding is that Python sets a default for the loading of dynamic modules that results in this error.

Solved the issue. Needed to

AndrewC — Wed, 01 Nov 2017 03:02:02 GMT

Solved the issue. Needed to make sure that the Makefile that builds the Python module has -lmkl_rt and -lmkl_def on the link line.

If you require cvxopt to

Rohit_J_Intel — Fri, 17 Nov 2017 14:02:04 GMT

If you require cvxopt to utilize blas and lapack routines from Intel MKL (i.e. shipped with IDP), you can build cvxopt from its sources. The instructions are as follows:

<IDP_installation>/bin/conda create -n cvxopt_install python=3.6 setuptools mkl
. <IDP_installation>/bin/activate cvxopt_install

#Fetch sources for cvxopt
git clone https://github.com/cvxopt/cvxopt.git
cd cvxopt
git checkout `git describe --abbrev=0 --tags`

#Fetch SuiteSparse (a runtime requirement)
wget http://faculty.cse.tamu.edu/davis/SuiteSparse/SuiteSparse-4.5.4.tar.gz
tar -xf SuiteSparse-4.5.4.tar.gz

#Export variables required for building cvxopt with MKL
export PREFIX_LIB=`which python`/../lib
export CVXOPT_LAPACK_LIB=mkl_rt
export CVXOPT_BLAS_LIB=mkl_rt
export CVXOPT_SUITESPARSE_SRC_DIR=`pwd`/SuiteSparse
export CVXOPT_BLAS_LIB_DIR=${PREFIX_LIB}
export CVXOPT_BLAS_EXTRA_LINK_ARGS="-L${PREFIX_LIB};-Wl,-rpath,${PREFIX_LIB};-lmkl_rt"

python setup.py install

#post-build clean-up:
for i in `env | grep -E "CVXOPT|PREFIX_LIB" | cut -d '=' -f 1`; do unset $i; done
#cd ..; rm -rf cvxopt

Thanks,
Rohit