Add libmkl_avx2.so to solve Intel MKL FATAL ERROR

[yan_c_]

I compiler numpy with MKL, everything is ok. But I come across a peculiar question. I have three case:

•  case_1(perinfoMKL1): I add only `mkl_rt` lib in site.cfg file
•  case_2(perinfoMKL2): I add `mkl_intel_lp64, mkl_intel_thread, mkl_core, iomp5, mkl_rt` lib in site.cfg file
•  case_3(perinfoMKL3): I and `mkl_intel_lp64, mkl_intel_thread, mkl_core, iomp5, mkl_rt, mkl_avx2` in site.cfg file

And then, I compiler it, build and install process are ok, no problems. But when I use case_2, a error is occured: Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.  I setted LD_LIBRARY_PATH, but when I add mkl_avx2 libs and compiler it again after, test ok. 

Based on LD_DEBUG, I use LD_DEBUG="files libs" LD_DEBUG_OUTPUT=log ./test.py to display process for input file. I see relocation dependency words, so I guess the problem is related to this. I am not clear in underlying computer inspect, someone can explain it, thanks. I am confused of it a little.

My understanding: Because mkl_rtaccording to relocation meaning, I guess that when group_1(mkl_intel_lp64, mkl_intel_thread, mkl_core, iomp5) and group_2(mkl_rt) exist together, executable program will find a new symbol table rather than specific one libs(such as: group_1)， and some aren't associated with the symbol table to link other symbol table. And then, finally executable program will not find a symbols, because function of libs relocataion again, so it's not find. When I add new lib to compiler again, errored symbols link new symbols successfully.

But according to Intel Forums, someone says that it's bug. So, I am confused of it again:-(  [https://software.intel.com/en-us/forums/intel-math-kernel-library/topic/300857]

Environments: 
  hp computer, CPU——Intel Core i5 4 core，Memory——8G, OS——CentOS 7, miniconda3(Python3.6), numpy 1.13(numpy with mkl, I configure some configuration MKL-information to site.cfg file) I compiled from source code.
  
Below followings are my process and attention points:

1. I seted LD_LIBRARY_PATH. My setting: LD_LIBRARY_PATH="/tmp/mkl-nfs/lib" (Since I will make my environment clean and tidy)
2. Same test code: test.py  , and every test-code will be modified suitable Python interpreter path.
3. Compiler source code is ok for every case.

$ cat test.py
#!/home/yancy/miniconda3/envs/perinfoMKL1/bin/python
# -*- coding: utf-8 -*-

import numpy as np
import time

start_time = time.time()
a = 10 ** 4
A = np.random.random((a, a))
B = np.random.random((a, a))
C = A.dot(B)
print("Time: ", time.time() - start_time)

(perinfoMKL1)$ tail numpy/site.cfg
#[fftw]
#libraries = fftw3
#
# For djbfft, numpy.distutils will look for either djbfft.a or libdjbfft.a . 
#[djbfft]
#include_dirs = /usr/local/djbfft/include
#library_dirs = /usr/local/djbfft/lib
[mkl]
library_dirs = /tmp/mkl-nfs/lib
mkl_libs = mkl_rt


# No error, execute successfully
(perinfoMKL1) $ ./test.py 
Time:  35.454288959503174

(perinfoMKL1)$ grep "libmkl_rt.so" log.27250 
     27250:    file=libmkl_rt.so [0];  needed by /home/yancy/miniconda3/envs/perinfoMKL1/lib/python3.6/site-packages/numpy-1.13.0.dev0+4408f74-py3.6-linux-x86_64.egg/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so [0]
     27250:    find library=libmkl_rt.so [0]; searching
     27250:      trying file=/home/yancy/miniconda3/envs/perinfoMKL1/lib/tls/x86_64/libmkl_rt.so
     27250:      trying file=/home/yancy/miniconda3/envs/perinfoMKL1/lib/tls/libmkl_rt.so
     27250:      trying file=/home/yancy/miniconda3/envs/perinfoMKL1/lib/x86_64/libmkl_rt.so
     27250:      trying file=/home/yancy/miniconda3/envs/perinfoMKL1/lib/libmkl_rt.so
     27250:      trying file=/home/yancy/miniconda3/envs/perinfoMKL1/bin/../lib/libmkl_rt.so
     27250:      trying file=/tmp/mkl-nfs/lib/libmkl_rt.so
     27250:    file=libmkl_rt.so [0];  generating link map
     27250:    calling init: /tmp/mkl-nfs/lib/libmkl_rt.so
     27250:    file=/tmp/mkl-nfs/lib/libmkl_core.so [0];  dynamically loaded by /tmp/mkl-nfs/lib/libmkl_rt.so [0]
     27250:    file=/tmp/mkl-nfs/lib/libiomp5.so [0];  dynamically loaded by /tmp/mkl-nfs/lib/libmkl_rt.so [0]
     27250:    file=/tmp/mkl-nfs/lib/libmkl_intel_thread.so [0];  dynamically loaded by /tmp/mkl-nfs/lib/libmkl_rt.so [0]
     27250:    file=/tmp/mkl-nfs/lib/libmkl_intel_lp64.so [0];  dynamically loaded by /tmp/mkl-nfs/lib/libmkl_rt.so [0]
     27250:    calling fini: /tmp/mkl-nfs/lib/libmkl_rt.so [0]
(perinfoMKL1)]$ grep "libmkl_avx2.so" log.27250 
     27250:    file=/tmp/mkl-nfs/lib/libmkl_avx2.so [0];  dynamically loaded by /tmp/mkl-nfs/lib/libmkl_core.so [0]
     27250:    file=/tmp/mkl-nfs/lib/libmkl_avx2.so [0];  generating link map
     27250:    file=/tmp/mkl-nfs/lib/libmkl_core.so [0];  needed by /tmp/mkl-nfs/lib/libmkl_avx2.so [0] (relocation dependency)
     27250:    calling init: /tmp/mkl-nfs/lib/libmkl_avx2.so
     27250:    opening file=/tmp/mkl-nfs/lib/libmkl_avx2.so [0]; direct_opencount=1
     27250:    file=/tmp/mkl-nfs/lib/libmkl_intel_thread.so [0];  needed by /tmp/mkl-nfs/lib/libmkl_avx2.so [0] (relocation dependency)
     27250:    calling fini: /tmp/mkl-nfs/lib/libmkl_avx2.so [0]

(perinfoMKL2)$ tail numpy/site.cfg
#[fftw]
#libraries = fftw3
#
# For djbfft, numpy.distutils will look for either djbfft.a or libdjbfft.a . 
#[djbfft]
#include_dirs = /usr/local/djbfft/include
#library_dirs = /usr/local/djbfft/lib
[mkl]
library_dirs = /tmp/mkl-nfs/lib
mkl_libs = mkl_intel_lp64, mkl_intel_thread, mkl_core, iomp5, mkl_rt
(perinfoMKL2)$ ./test.py 
Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.


# Error occur, LD_LIBRARY_PATH I setted.
(perinfoMKL2)$ ./test.py 
Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.
(perinfoMKL2)$ grep "libmkl_rt.so" test.log.26855 
     26855:    file=libmkl_rt.so [0];  needed by /home/yancy/miniconda3/envs/perinfoMKL2/lib/python3.6/site-packages/numpy-1.13.0.dev0+4408f74-py3.6-linux-x86_64.egg/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so [0]
     26855:    find library=libmkl_rt.so [0]; searching
     26855:      trying file=/home/yancy/miniconda3/envs/perinfoMKL2/lib/libmkl_rt.so
     26855:      trying file=/home/yancy/miniconda3/envs/perinfoMKL2/bin/../lib/libmkl_rt.so
     26855:      trying file=/tmp/mkl-nfs/lib/libmkl_rt.so
     26855:    file=libmkl_rt.so [0];  generating link map
     26855:    file=/tmp/mkl-nfs/lib/libmkl_intel_lp64.so [0];  needed by /tmp/mkl-nfs/lib/libmkl_rt.so [0] (relocation dependency)
     26855:    file=/tmp/mkl-nfs/lib/libmkl_core.so [0];  needed by /tmp/mkl-nfs/lib/libmkl_rt.so [0] (relocation dependency)
     26855:    calling init: /tmp/mkl-nfs/lib/libmkl_rt.so
     26855:    calling fini: /tmp/mkl-nfs/lib/libmkl_rt.so [0]
(perinfoMKL2) $ grep "libmkl_avx2.so" test.log.26855 
     26855:    file=/tmp/mkl-nfs/lib/libmkl_avx2.so [0];  dynamically loaded by /tmp/mkl-nfs/lib/libmkl_core.so [0]
     26855:    file=/tmp/mkl-nfs/lib/libmkl_avx2.so [0];  generating link map
     26855:    /tmp/mkl-nfs/lib/libmkl_avx2.so: error: symbol lookup error: undefined symbol: mkl_dft_fft_fix_twiddle_table_32f (fatal)
     26855:    file=/tmp/mkl-nfs/lib/libmkl_avx2.so [0];  destroying link map
     26855:    file=/home/yancy/miniconda3/envs/perinfoMKL2/bin/libmkl_avx2.so [0];  dynamically loaded by /tmp/mkl-nfs/lib/libmkl_core.so [0]
     26855:    file=libmkl_avx2.so [0];  dynamically loaded by /tmp/mkl-nfs/lib/libmkl_core.so [0]
     26855:    find library=libmkl_avx2.so [0]; searching
     26855:      trying file=/home/yancy/miniconda3/envs/perinfoMKL2/lib/libmkl_avx2.so
     26855:      trying file=/home/yancy/miniconda3/envs/perinfoMKL2/bin/../lib/libmkl_avx2.so
     26855:      trying file=/tmp/mkl-nfs/lib/libmkl_avx2.so
     26855:    file=libmkl_avx2.so [0];  generating link map
     26855:    /tmp/mkl-nfs/lib/libmkl_avx2.so: error: symbol lookup error: undefined symbol: mkl_dft_fft_fix_twiddle_table_32f (fatal)
     26855:    file=/tmp/mkl-nfs/lib/libmkl_avx2.so [0];  destroying link map

(perinfoMKL3)$ tail numpy/site.cfg

#[fftw]
#libraries = fftw3
#
# For djbfft, numpy.distutils will look for either djbfft.a or libdjbfft.a . 
#[djbfft]
#include_dirs = /usr/local/djbfft/include
#library_dirs = /usr/local/djbfft/lib
[mkl]
library_dirs = /tmp/mkl-nfs/lib
mkl_libs = mkl_intel_lp64, mkl_intel_thread, mkl_core, iomp5, mkl_rt, mkl_avx2

# No error when I add mkl_avx2
(perinfoMKL3)$ ./test.py 
Time:  33.996660232543945

(perinfoMKL3) $ grep "libmkl_rt.so" log.27384 
     27384:    file=libmkl_rt.so [0];  needed by /home/yancy/miniconda3/envs/perinfoMKL3/lib/python3.6/site-packages/numpy-1.13.0.dev0+4408f74-py3.6-linux-x86_64.egg/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so [0]
     27384:    find library=libmkl_rt.so [0]; searching
     27384:      trying file=/home/yancy/miniconda3/envs/perinfoMKL3/lib/libmkl_rt.so
     27384:      trying file=/home/yancy/miniconda3/envs/perinfoMKL3/bin/../lib/libmkl_rt.so
     27384:      trying file=/tmp/mkl-nfs/lib/libmkl_rt.so
     27384:    file=libmkl_rt.so [0];  generating link map
     27384:    file=/tmp/mkl-nfs/lib/libmkl_intel_lp64.so [0];  needed by /tmp/mkl-nfs/lib/libmkl_rt.so [0] (relocation dependency)
     27384:    file=/tmp/mkl-nfs/lib/libmkl_core.so [0];  needed by /tmp/mkl-nfs/lib/libmkl_rt.so [0] (relocation dependency)
     27384:    calling init: /tmp/mkl-nfs/lib/libmkl_rt.so
     27384:    calling fini: /tmp/mkl-nfs/lib/libmkl_rt.so [0]
(perinfoMKL3)$ grep "libmkl.avx2.so" log.27384 
     27384:    file=libmkl_avx2.so [0];  needed by /home/yancy/miniconda3/envs/perinfoMKL3/lib/python3.6/site-packages/numpy-1.13.0.dev0+4408f74-py3.6-linux-x86_64.egg/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so [0]
     27384:    find library=libmkl_avx2.so [0]; searching
     27384:      trying file=/home/yancy/miniconda3/envs/perinfoMKL3/lib/libmkl_avx2.so
     27384:      trying file=/home/yancy/miniconda3/envs/perinfoMKL3/bin/../lib/libmkl_avx2.so
     27384:      trying file=/tmp/mkl-nfs/lib/libmkl_avx2.so
     27384:    file=libmkl_avx2.so [0];  generating link map
     27384:    file=/tmp/mkl-nfs/lib/libmkl_core.so [0];  needed by /tmp/mkl-nfs/lib/libmkl_avx2.so [0] (relocation dependency)
     27384:    file=/tmp/mkl-nfs/lib/libmkl_intel_lp64.so [0];  needed by /tmp/mkl-nfs/lib/libmkl_avx2.so [0] (relocation dependency)
     27384:    file=/tmp/mkl-nfs/lib/libmkl_intel_thread.so [0];  needed by /tmp/mkl-nfs/lib/libmkl_avx2.so [0] (relocation dependency)
     27384:    calling init: /tmp/mkl-nfs/lib/libmkl_avx2.so
     27384:    opening file=/tmp/mkl-nfs/lib/libmkl_avx2.so [0]; direct_opencount=1
     27384:    calling fini: /tmp/mkl-nfs/lib/libmkl_avx2.so [0]

[Ying_H_Intel · Accepted solution]

Hi Yan Cy,

Thank you for raising the issue here.  So let me clarify

1) mkl_rt.so  , which is correct.  

2) mkl_intel_lp64, mkl_intel_thread,  mkl_core, iomp5  mkl_rt . the run incorrect.  but the usage is not correct either. 

Please refer to MKL link advisor,    https://software.intel.com/en-us/articles/intel-mkl-link-line-advisor/

as  "mkl_intel_lp64, mkl_intel_thread,  mkl_core, iomp5"   use multi-layer interfaces , it is  same functionality of mkl_rt.  So you should use either of them, not all of them.

3) mkl_intel_lp64, mkl_intel_thread,  mkl_core, iomp5  mkl_rt  mkl_avx2.

which can run, but the link line is not recommended unless the error  Cannot load neither  , please see more discussion below.

I recalled, there are some discussion in forum and other places about  MKL FATAL ERROR: Cannot load neither

https://software.intel.com/en-us/forums/intel-math-kernel-library/topic/300857

https://github.com/BVLC/caffe/issues/3884

http://stackoverflow.com/questions/13951953/linking-intels-mkl-within-python-c-c-fortran-2003-program

As we understand, the issue  do related to the system and mkl libraries.

Regarding the MKL libraries,  the issue may be that  MKL has cascading libraries structure. for example,  mkl_rt.so  will load the libmkl_avx2.so automatically at run time,  while mkl_rt.so is   runtime single dynamic libraries (SDL) library  and libmkl_avx2.so is processor specific libraries

Please see the article : https://software.intel.com/en-us/articles/a-new-linking-model-single-dynamic-library-mkl_rt-since-intel-mkl-103/

The runtime single dynamic libraries (SDL) library was introduced originally for solving the declared issue related to the processor specific libraries not getting loaded when dlopen(RTLD_GLOBAL) was called.  The typical error message is

MKL FATAL ERROR: Cannot load neither libmkl_mc3.so nor libmkl_def.so

The single library is added allowing user to link to a single interface library which loads dependent libraries dynamically at runtime depending on runtime CPU detection and/or library function calls.

 it seems in python and Center OS , the load  processing was blocked anyway.

Thus, it is also related to the system.   We usually run on Redhat Linux in local machine,  although get such issue report, but we  can't reproduce the issue (for example,  we create https://software.intel.com/en-us/articles/numpyscipy-with-intel-mkl  without the issue)

One probable reason was discussed in  the article https://software.intel.com/en-us/articles/symbol-lookup-error-when-linking-intel-mkl-with-gcc-on-ubuntu

So far the solution we can see are 

1) as you did, add  libmkl_avx2.so  explicitly.  but it may be same issue when the run machine don't support the avx2 instructions.

2) there is some problem in FINDLAPACK.cmake processing( OpenCVFindMKL.cmake).  the fix is  CMAKE_SHARED_LINKER_FLAGS_RELEASE:STRING=-Wl,--no-as-needed

3)  Set environment variable >export  LD_PRELOAD=/path/to/libmkl_avx2.so etc.

Best Regards,

Ying

[yan_c_]

@Ying,  I'm very sorry for replying later. :-(  And thanks your best answer and your helping for me.

Ok, I have payed attention to MKL cascading libraries and structure, and even system issue, and some good solve  ways.

In the meantime, I do another research, and maybe find a little suggestions during this period of time.https://gehrcke.de/2014/02/numpy-built-with-recent-intel-compilers-mkl-fatal-error/  Although this article is out of date, I think that this article should serve as reference to help others like me.

Best, thanks again!

[yan_c_]

Hi, Ying

I have another question, mkl_rt(SDL) automatically load three(interface, threading, and computational libraries) or four library(interface, threading, and computational libraries and even RTL).

I have posted early stage，concrete information: https://software.intel.com/en-us/forums/intel-system-studio/topic/733090 I have known this post weren't placed rightly, sorry.

Best, looking forward to your reply. 

[Ying_H_Intel]

Hi Yan,

Yes, mkl_rt(SDL) automatically load three(interface mkl_intel_lp64, threading mkl_intel_thread, and computational libraries mkl_core.  and  mkl_core will load mkl_avx (processor-optimized )  library.

One thing you may note,  we build and maintain the Intel Distribution for Python  https://software.intel.com/en-us/intel-distribution-for-python, which includes an MKL-accelerated numpy. you don't need to build numpy with MKL.  You can get them by Aconda  and Yum/apt-get etc.

https://software.intel.com/en-us/articles/using-intel-distribution-for-python-with-anaconda

https://software.intel.com/en-us/articles/installing-intel-free-libs-and-python-apt-repo

https://software.intel.com/en-us/articles/installing-intel-free-libs-and-python-yum-repo

Best Regards,

Ying

[yan_c_]

Ok ， thanks your advice and your helping .  

Best，yancy

[Bicici__Ergun]

I added mkl_def to site.cfg to solve the mentioned error. Thank you.