Community
cancel
Showing results for 
Search instead for 
Did you mean: 
malik__tania
Beginner
568 Views

Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.

Hi

I have installed Fupermod software with MKL support. It installed correctally. But when I run the program it give me that error

Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.

Can you please let me know what I am doing wrong?

 

0 Kudos
15 Replies
mecej4
Black Belt
568 Views

When you attempted to run it, did you do so in a command shell after sourcing compilervars.sh or one of its siblings? If not, you may need to see to it that the MKL shared libraries are accessible along LD_LIBRARY_PATH (or the equivalent that pertains to your OS). What do you see when you run the following command?

     echo $LD_LIBRARY_PATH

malik__tania
Beginner
568 Views

 

Yes i did source. Also i add this line in my .bashrc file.

source /opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/bin/mklvars.sh intel64

I am running my program on ubuntu OS.

The output of echo $LD_LIBRARY_PATH is:

/opt/intel/compilers_and_libraries_2016.0.109/linux/tbb/lib/intel64/gcc4.4:/opt/intel/compilers_and_libraries_2016.0.109/linux/compiler/lib/intel64:/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64:/home/tania/.openmpi/lib/:/opt/ThirdParty-3.0.1/platforms/linux64Gcc/gperftools-svn/lib:/opt/paraviewopenfoam44/lib/paraview-4.4:/opt/openfoam30/platforms/linux64GccDPInt32Opt/lib/openmpi-system:/opt/ThirdParty-3.0.1/platforms/linux64GccDPInt32Opt/lib/openmpi-system:/home/tania/.openmpi/lib:/home/tania/OpenFOAM/tania-3.0.1/platforms/linux64GccDPInt32Opt/lib:/opt/site/3.0.1/platforms/linux64GccDPInt32Opt/lib:/opt/openfoam30/platforms/linux64GccDPInt32Opt/lib:/opt/ThirdParty-3.0.1/platforms/linux64GccDPInt32Opt/lib:/opt/openfoam30/platforms/linux64GccDPInt32Opt/lib/dummy:/home/tania/.openmpi/lib/:/home/tania/.openmpi/lib//home/tania/.openmpi/lib:/home/tania/.openmpi/lib//home/tania/.openmpi/lib:/home/tania/.openmpi/lib//home/tmalik/intel/composer_xe_2013_sp1.0.080/mkl/lib/intel64

 

mecej4
Black Belt
568 Views

There are only two directories in that list that pertain to MKL, and those are

     /opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64

and

     /home/tania/.openmpi/lib//home/tmalik/intel/composer_xe_2013_sp1.0.080/mkl/lib/intel64

Do you have the libraries libmkl_avx2.so or libmkl_def.so in those directories?

malik__tania
Beginner
568 Views

I I have found these file in this folder:

tania@tania-Latitude-E5540:/opt/intel$ find -name "libmkl_avx2.so"
./compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64_lin/libmkl_avx2.so
find: `./ism/lib': Permission denied
tania@tania-Latitude-E5540:/opt/intel$ find -name "libmkl_def.so"
./compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64_lin/libmkl_def.so
find: `./ism/lib': Permission denied

How can I add this?

The output of echo $LD_LIBRARY_PATH is:

tania@tania-Latitude-E5540:~$ echo $LD_LIBRARY_PATH
/home/tania/.openmpi/lib/:/opt/intel/compilers_and_libraries_2016.0.109/linux/tbb/lib/intel64/gcc4.4:/opt/intel/compilers_and_libraries_2016.0.109/linux/compiler/lib/intel64:/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64:/opt/ThirdParty-3.0.1/platforms/linux64Gcc/gperftools-svn/lib:/opt/paraviewopenfoam44/lib/paraview-4.4:/opt/openfoam30/platforms/linux64GccDPInt32Opt/lib/openmpi-system:/opt/ThirdParty-3.0.1/platforms/linux64GccDPInt32Opt/lib/openmpi-system:/home/tania/.openmpi/lib:/home/tania/OpenFOAM/tania-3.0.1/platforms/linux64GccDPInt32Opt/lib:/opt/site/3.0.1/platforms/linux64GccDPInt32Opt/lib:/opt/openfoam30/platforms/linux64GccDPInt32Opt/lib:/opt/ThirdParty-3.0.1/platforms/linux64GccDPInt32Opt/lib:/opt/openfoam30/platforms/linux64GccDPInt32Opt/lib/dummy:/home/tania/.openmpi/lib/

 

malik__tania
Beginner
568 Views

 

The MIC_LD_LIBRARY_PATH shows me this output:

tania@tania-Latitude-E5540:~$ echo $MIC_LD_LIBRARY_PATH 
/opt/intel/compilers_and_libraries_2016.0.109/linux/tbb/lib/mic:/opt/intel/compilers_and_libraries_2016.0.109/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/mic:/opt/intel/compilers_and_libraries_2016.0.109/linux/tbb/lib/mic:/opt/intel/compilers_and_libraries_2016.0.109/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/mic

mecej4
Black Belt
568 Views

That makes the problem clear. The new version of Parallel Studio uses intel64_lin as <arch>, whereas that used to be <intel64> in previous versions. The MKL team should resolve this but, as a temporary solution, you can send the output of "echo $LD_LIBRARY_PATH" to a file (echo $LD_LIBRARY_PATH > ldlibpath.txt), edit the file to replace "intel64" by "intel64_lin" throughout, and issue the command

     export LD_LIBRARY_PATH=<_insert_here_the_contents_of_ldlibpath.txt_>

The drawback of this is that you have to issue issue this command each time that you open a shell window to use the compiler and MKL. The better solution is for the start-off scripts compilervars.sh, etc., to be fixed with the new directory names, but the MKL team is to advise on this.

 

malik__tania
Beginner
568 Views

Hi 

I did exactly as you wrote

The output of echo $LD_LIBRARY_PATH after that is:

tania@tania-Latitude-E5540:~/openform_fupermod_test$ echo $LD_LIBRARY_PATH /home/tania/.openmpi/lib/:/opt/intel/compilers_and_libraries_2016.0.109/linux/tbb/lib/intel64_lin/gcc4.4:/opt/intel/compilers_and_libraries_2016.0.109/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64_lin:/opt/ThirdParty-3.0.1/platforms/linux64Gcc/gperftools-svn/lib:/opt/paraviewopenfoam44/lib/paraview-4.4:/opt/openfoam30/platforms/linux64GccDPInt32Opt/lib/openmpi-system:/opt/ThirdParty-3.0.1/platforms/linux64GccDPInt32Opt/lib/openmpi-system:/home/tania/.openmpi/lib:/home/tania/OpenFOAM/tania-3.0.1/platforms/linux64GccDPInt32Opt/lib:/opt/site/3.0.1/platforms/linux64GccDPInt32Opt/lib:/opt/openfoam30/platforms/linux64GccDPInt32Opt/lib:/opt/ThirdParty-3.0.1/platforms/linux64GccDPInt32Opt/lib:/opt/openfoam30/platforms/linux64GccDPInt32Opt/lib/dummy:/home/tania/.openmpi/lib/

But still I got the same error:

Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.

 

mecej4
Black Belt
568 Views

There could be a problem with permissions. What do you see from ls -l in the directory that contains the MKL .so files?

malik__tania
Beginner
568 Views

The output is:

tania@tania-Latitude-E5540:/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl$  ls -l
total 28
drwxr-xr-x  4 root root 4096 Dec 21 15:42 benchmarks
drwxr-xr-x  4 root root 4096 Dec 21 15:41 bin
drwxr-xr-x  2 root root 4096 Dec 21 15:42 examples
drwxr-xr-x  5 root root 4096 Dec 21 15:42 include
drwxr-xr-x 11 root root 4096 Dec 21 15:42 interfaces
drwxr-xr-x  4 root root 4096 Dec 21 15:42 lib
drwxr-xr-x  3 root root 4096 Dec 21 15:41 tools

 

malik__tania
Beginner
568 Views

Where as:

tania@tania-Latitude-E5540:/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64_lin$ ls -l
total 1032580
-rwxr-xr-x 1 root root  32686896 Jul 30 14:03 libmkl_avx2.so
-rwxr-xr-x 1 root root  39565233 Jul 30 14:03 libmkl_avx512_mic.so
-rwxr-xr-x 1 root root  34186727 Jul 30 14:03 libmkl_avx512.so
-rwxr-xr-x 1 root root  35565192 Jul 30 14:03 libmkl_avx.so
-rw-r--r-- 1 root root   1221488 Jul 30 14:04 libmkl_blacs_ilp64.a
-rw-r--r-- 1 root root   1222312 Jul 30 14:04 libmkl_blacs_intelmpi_ilp64.a
-rwxr-xr-x 1 root root    490800 Jul 30 14:04 libmkl_blacs_intelmpi_ilp64.so
-rw-r--r-- 1 root root    707234 Jul 30 14:03 libmkl_blacs_intelmpi_lp64.a
-rwxr-xr-x 1 root root    277115 Jul 30 14:03 libmkl_blacs_intelmpi_lp64.so
-rw-r--r-- 1 root root    706410 Jul 30 14:03 libmkl_blacs_lp64.a
-rw-r--r-- 1 root root   1239632 Jul 30 14:04 libmkl_blacs_openmpi_ilp64.a
-rw-r--r-- 1 root root    724554 Jul 30 14:03 libmkl_blacs_openmpi_lp64.a
-rw-r--r-- 1 root root   1221760 Jul 30 14:04 libmkl_blacs_sgimpt_ilp64.a
-rw-r--r-- 1 root root    706682 Jul 30 14:03 libmkl_blacs_sgimpt_lp64.a
-rw-r--r-- 1 root root    678240 Jul 30 13:43 libmkl_blas95_ilp64.a
-rw-r--r-- 1 root root    674352 Jul 30 13:43 libmkl_blas95_lp64.a
-rw-r--r-- 1 root root    213014 Jul 30 14:04 libmkl_cdft_core.a
-rwxr-xr-x 1 root root    166155 Jul 30 14:04 libmkl_cdft_core.so
-rw-r--r-- 1 root root 422816228 Jul 30 13:53 libmkl_core.a
-rwxr-xr-x 1 root root  25694727 Jul 30 13:53 libmkl_core.so
-rwxr-xr-x 1 root root  29552604 Jul 30 14:03 libmkl_def.so
-rw-r--r-- 1 root root  19322966 Jul 30 13:53 libmkl_gf_ilp64.a
-rwxr-xr-x 1 root root   8479034 Jul 30 14:03 libmkl_gf_ilp64.so
-rw-r--r-- 1 root root  19975400 Jul 30 13:53 libmkl_gf_lp64.a
-rwxr-xr-x 1 root root   8867579 Jul 30 14:03 libmkl_gf_lp64.so
-rw-r--r-- 1 root root  18517016 Jul 30 13:53 libmkl_gnu_thread.a
-rwxr-xr-x 1 root root  13034633 Jul 30 14:03 libmkl_gnu_thread.so
-rw-r--r-- 1 root root  19353176 Jul 30 13:53 libmkl_intel_ilp64.a
-rwxr-xr-x 1 root root   8496705 Jul 30 14:03 libmkl_intel_ilp64.so
-rw-r--r-- 1 root root  20005526 Jul 30 13:53 libmkl_intel_lp64.a
-rwxr-xr-x 1 root root   8885218 Jul 30 14:03 libmkl_intel_lp64.so
-rw-r--r-- 1 root root  28704798 Jul 30 13:53 libmkl_intel_thread.a
-rwxr-xr-x 1 root root  20368592 Jul 30 14:03 libmkl_intel_thread.so
-rw-r--r-- 1 root root   8139352 Jul 30 13:44 libmkl_lapack95_ilp64.a
-rw-r--r-- 1 root root   8132352 Jul 30 13:43 libmkl_lapack95_lp64.a
-rwxr-xr-x 1 root root  36894287 Jul 30 14:03 libmkl_mc3.so
-rwxr-xr-x 1 root root  33396687 Jul 30 14:03 libmkl_mc.so
-rwxr-xr-x 1 root root   5147328 Jul 30 14:03 libmkl_rt.so
-rw-r--r-- 1 root root  11778266 Jul 30 14:05 libmkl_scalapack_ilp64.a
-rwxr-xr-x 1 root root   7512849 Jul 30 14:05 libmkl_scalapack_ilp64.so
-rw-r--r-- 1 root root  11772212 Jul 30 14:04 libmkl_scalapack_lp64.a
-rwxr-xr-x 1 root root   7476637 Jul 30 14:05 libmkl_scalapack_lp64.so
-rw-r--r-- 1 root root  10820490 Jul 30 13:53 libmkl_sequential.a
-rwxr-xr-x 1 root root   7732586 Jul 30 14:03 libmkl_sequential.so
-rwxr-xr-x 1 root root  11277446 Jul 30 13:59 libmkl_vml_avx2.so
-rwxr-xr-x 1 root root  11638255 Jul 30 13:59 libmkl_vml_avx512_mic.so
-rwxr-xr-x 1 root root  10434720 Jul 30 14:00 libmkl_vml_avx512.so
-rwxr-xr-x 1 root root  10930771 Jul 30 13:58 libmkl_vml_avx.so
-rwxr-xr-x 1 root root   5393410 Jul 30 13:54 libmkl_vml_cmpt.so
-rwxr-xr-x 1 root root   5472445 Jul 30 13:53 libmkl_vml_def.so
-rwxr-xr-x 1 root root   9569252 Jul 30 13:56 libmkl_vml_mc2.so
-rwxr-xr-x 1 root root   9817468 Jul 30 13:57 libmkl_vml_mc3.so
-rwxr-xr-x 1 root root   9576458 Jul 30 13:55 libmkl_vml_mc.so
drwxr-xr-x 3 root root      4096 Dec 21 15:41 locale

 

mecej4
Black Belt
568 Views

All the shared library files have execute permissions set, and the directory containing them has execute permissions set, so there is no problem there.

Sorry, I have run out of suggestions at this point.

Ying_H_Intel
Employee
568 Views

Hello, 

Not sure if it is related to the environment.  Could you please try

1)  > ldd  your application, what the output? 

2) you mentioned MIC, so have you xeon phi coprocessor in your system?  does the application run on the xeon phi?   

3) there are some example (fortran or C sample) under MKL install directory.  Could you please unzip one of them and run one of them

for example,  

>/mkl_example/cblas

> make libintel64 function=cblas_dgemm

let us know the output of the sample program. 

4)  or is it possible to LD_PRELOAD=/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64_lin/libmkl_def.so, then run your application.

Here is a similiar issue for your reference: https://software.intel.com/en-us/forums/intel-math-kernel-library/topic/600019

Best Regards,

Ying

malik__tania
Beginner
568 Views

 

HI

Here are the results of 4 options that you suggested:

(1) tania@tania-Latitude-E5540:~/openform_fupermod_test$ ldd ~/build_fupermod/mkl/tools/builder
    not a dynamic executable

 

2) you mentioned MIC, so have you xeon phi coprocessor in your system?  does the application run on the xeon phi? 

I have no idea about it. I just install MKL to use in cblas.

(3) Examples are working  

tania@tania-Latitude-E5540:/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/examples/cblas$ sudo make libintel64 function=cblas_dgemm compiler=gnu

make cblas_dgemmx.res  _IA=intel64 EXT=a RES_EXT=lib
make[1]: Entering directory `/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/examples/cblas'
mkdir -p ./_results/gnu_lp64_parallel_intel64_lib
gcc    -m64 -w -I../../include source/cblas_dgemmx.c source/common_func.c  -Wl,--start-group "../../lib/intel64"/libmkl_intel_lp64.a "../../lib/intel64"/libmkl_intel_thread.a "../../lib/intel64"/libmkl_core.a -Wl,--end-group -L"../../../compiler/lib/intel64" -liomp5 -lpthread -ldl -lm -o _results/gnu_lp64_parallel_intel64_lib/cblas_dgemmx.out
export LD_LIBRARY_PATH="../../lib/intel64"::"../../../compiler/lib/intel64"; _results/gnu_lp64_parallel_intel64_lib/cblas_dgemmx.out data/cblas_dgemmx.d >_results/gnu_lp64_parallel_intel64_lib/cblas_dgemmx.res

 

(4) I got the same error while using PRELOAD before:

tania@tania-Latitude-E5540:~/openform_fupermod_test$ LD_PRELOAD=/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64_lin/libmkl_def.so
tania@tania-Latitude-E5540:~/openform_fupermod_test$ mpirun -np 4 -machinefile machinefile ~/build_fupermod/mkl/tools/builder -l ~/build_fupermod/mkl/routines/mxm/.libs/libmxm_1d.so -U2000 -s100
Kernel:/home/tania/build_fupermod/mkl/routines/mxm/.libs/libmxm_1d.so 
Kernel:/home/tania/build_fupermod/mkl/routines/mxm/.libs/libmxm_1d.so 
Kernel:/home/tania/build_fupermod/mkl/routines/mxm/.libs/libmxm_1d.so 
Kernel:/home/tania/build_fupermod/mkl/routines/mxm/.libs/libmxm_1d.so 

 

Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.

Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.
Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.
Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.
-------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code.. Per user-direction, the job has been aborted.
-------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[48793,1],1]
  Exit code:    1
--------------------------------------------------------------------------

 

malik__tania
Beginner
568 Views

 

I am attaching Fupermod makefile.in and configure.ac file

you can check the compile option of MKL in case if you want see.

Ying_H_Intel
Employee
568 Views

Hi Tania,

Thanks for the reply.  so we can get

1) MKL work in simple command environment, but can't work in the mpirun

 >mpirun -np 4 -machinefile machinefile ~/build_fupermod/mkl/tools/builder -l

~/build_fupermod/mkl/routines/mxm/.libs/libmxm_1d.so -U2000 -s100

2) ~/build_fupermod/mkl/routines/mxm/.libs/libmxm_1d.so are linked mkl librarylibintel_libmkl_intel_lp64.so  or

ibmkl_gnu_thread.so  libmkl_core.so. and libintel_libmkl_intel_lp64.so  or ibmkl_gnu_thread.so  or  libmkl_core.so

internally invokes  libmkl_avx2.so or libmkl_def.so. Because in my application I don't need them directaly.

So the solution is either remove the dependency or  to make sure the mpirun environment can invoke the  libmkl_avx2.so or

libmkl_def.so as command line case.

Could you rebuild the Fupermod or  libmxm_1d.so with static mkl library (for example, libmkl_intel_lp64.a, libmkl_gnu_thread.a libmkl_core.a, not .so), which will not ask the mkl dynamic libary (libmkl_avx2.so  and libmkl_def.so)

other ways, may be -rpath option  or add –no-as-needed  as https://software.intel.com/en-us/forums/intel-math-kernel-

library/topic/...

Regards,

Ying

Reply