Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.

malik__tania · ‎12-21-2015

Hi

I have installed Fupermod software with MKL support. It installed correctally. But when I run the program it give me that error

Can you please let me know what I am doing wrong?

mecej4 · ‎12-21-2015

When you attempted to run it, did you do so in a command shell after sourcing compilervars.sh or one of its siblings? If not, you may need to see to it that the MKL shared libraries are accessible along LD_LIBRARY_PATH (or the equivalent that pertains to your OS). What do you see when you run the following command?

echo $LD_LIBRARY_PATH

malik__tania · ‎12-21-2015

Yes i did source. Also i add this line in my .bashrc file.

source /opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/bin/mklvars.sh intel64

I am running my program on ubuntu OS.

The output of echo $LD_LIBRARY_PATH is:

/opt/intel/compilers_and_libraries_2016.0.109/linux/tbb/lib/intel64/gcc4.4:/opt/intel/compilers_and_libraries_2016.0.109/linux/compiler/lib/intel64:/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64:/home/tania/.openmpi/lib/:/opt/ThirdParty-3.0.1/platforms/linux64Gcc/gperftools-svn/lib:/opt/paraviewopenfoam44/lib/paraview-4.4:/opt/openfoam30/platforms/linux64GccDPInt32Opt/lib/openmpi-system:/opt/ThirdParty-3.0.1/platforms/linux64GccDPInt32Opt/lib/openmpi-system:/home/tania/.openmpi/lib:/home/tania/OpenFOAM/tania-3.0.1/platforms/linux64GccDPInt32Opt/lib:/opt/site/3.0.1/platforms/linux64GccDPInt32Opt/lib:/opt/openfoam30/platforms/linux64GccDPInt32Opt/lib:/opt/ThirdParty-3.0.1/platforms/linux64GccDPInt32Opt/lib:/opt/openfoam30/platforms/linux64GccDPInt32Opt/lib/dummy:/home/tania/.openmpi/lib/:/home/tania/.openmpi/lib//home/tania/.openmpi/lib:/home/tania/.openmpi/lib//home/tania/.openmpi/lib:/home/tania/.openmpi/lib//home/tmalik/intel/composer_xe_2013_sp1.0.080/mkl/lib/intel64

mecej4 · ‎12-21-2015

There are only two directories in that list that pertain to MKL, and those are

/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64

and

/home/tania/.openmpi/lib//home/tmalik/intel/composer_xe_2013_sp1.0.080/mkl/lib/intel64

Do you have the libraries libmkl_avx2.so or libmkl_def.so in those directories?

malik__tania · ‎12-21-2015

I I have found these file in this folder:

tania@tania-Latitude-E5540:/opt/intel$ find -name "libmkl_avx2.so"
./compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64_lin/libmkl_avx2.so
find: `./ism/lib': Permission denied
tania@tania-Latitude-E5540:/opt/intel$ find -name "libmkl_def.so"
./compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64_lin/libmkl_def.so
find: `./ism/lib': Permission denied

How can I add this?

The output of echo $LD_LIBRARY_PATH is:

tania@tania-Latitude-E5540:~$ echo $LD_LIBRARY_PATH
/home/tania/.openmpi/lib/:/opt/intel/compilers_and_libraries_2016.0.109/linux/tbb/lib/intel64/gcc4.4:/opt/intel/compilers_and_libraries_2016.0.109/linux/compiler/lib/intel64:/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64:/opt/ThirdParty-3.0.1/platforms/linux64Gcc/gperftools-svn/lib:/opt/paraviewopenfoam44/lib/paraview-4.4:/opt/openfoam30/platforms/linux64GccDPInt32Opt/lib/openmpi-system:/opt/ThirdParty-3.0.1/platforms/linux64GccDPInt32Opt/lib/openmpi-system:/home/tania/.openmpi/lib:/home/tania/OpenFOAM/tania-3.0.1/platforms/linux64GccDPInt32Opt/lib:/opt/site/3.0.1/platforms/linux64GccDPInt32Opt/lib:/opt/openfoam30/platforms/linux64GccDPInt32Opt/lib:/opt/ThirdParty-3.0.1/platforms/linux64GccDPInt32Opt/lib:/opt/openfoam30/platforms/linux64GccDPInt32Opt/lib/dummy:/home/tania/.openmpi/lib/

malik__tania · ‎12-21-2015

The MIC_LD_LIBRARY_PATH shows me this output:

tania@tania-Latitude-E5540:~$ echo $MIC_LD_LIBRARY_PATH
/opt/intel/compilers_and_libraries_2016.0.109/linux/tbb/lib/mic:/opt/intel/compilers_and_libraries_2016.0.109/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/mic:/opt/intel/compilers_and_libraries_2016.0.109/linux/tbb/lib/mic:/opt/intel/compilers_and_libraries_2016.0.109/linux/compiler/lib/mic:/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/mic

mecej4 · ‎12-21-2015

That makes the problem clear. The new version of Parallel Studio uses intel64_lin as <arch>, whereas that used to be <intel64> in previous versions. The MKL team should resolve this but, as a temporary solution, you can send the output of "echo $LD_LIBRARY_PATH" to a file (echo $LD_LIBRARY_PATH > ldlibpath.txt), edit the file to replace "intel64" by "intel64_lin" throughout, and issue the command

export LD_LIBRARY_PATH=<_insert_here_the_contents_of_ldlibpath.txt_>

The drawback of this is that you have to issue issue this command each time that you open a shell window to use the compiler and MKL. The better solution is for the start-off scripts compilervars.sh, etc., to be fixed with the new directory names, but the MKL team is to advise on this.

malik__tania · ‎12-21-2015

Hi

I did exactly as you wrote

The output of echo $LD_LIBRARY_PATH after that is:

tania@tania-Latitude-E5540:~/openform_fupermod_test$ echo $LD_LIBRARY_PATH /home/tania/.openmpi/lib/:/opt/intel/compilers_and_libraries_2016.0.109/linux/tbb/lib/intel64_lin/gcc4.4:/opt/intel/compilers_and_libraries_2016.0.109/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64_lin:/opt/ThirdParty-3.0.1/platforms/linux64Gcc/gperftools-svn/lib:/opt/paraviewopenfoam44/lib/paraview-4.4:/opt/openfoam30/platforms/linux64GccDPInt32Opt/lib/openmpi-system:/opt/ThirdParty-3.0.1/platforms/linux64GccDPInt32Opt/lib/openmpi-system:/home/tania/.openmpi/lib:/home/tania/OpenFOAM/tania-3.0.1/platforms/linux64GccDPInt32Opt/lib:/opt/site/3.0.1/platforms/linux64GccDPInt32Opt/lib:/opt/openfoam30/platforms/linux64GccDPInt32Opt/lib:/opt/ThirdParty-3.0.1/platforms/linux64GccDPInt32Opt/lib:/opt/openfoam30/platforms/linux64GccDPInt32Opt/lib/dummy:/home/tania/.openmpi/lib/

But still I got the same error:

Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.

mecej4 · ‎12-21-2015

There could be a problem with permissions. What do you see from ls -l in the directory that contains the MKL .so files?

malik__tania · ‎12-21-2015

The output is:

tania@tania-Latitude-E5540:/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl$ ls -l
total 28
drwxr-xr-x 4 root root 4096 Dec 21 15:42 benchmarks
drwxr-xr-x 4 root root 4096 Dec 21 15:41 bin
drwxr-xr-x 2 root root 4096 Dec 21 15:42 examples
drwxr-xr-x 5 root root 4096 Dec 21 15:42 include
drwxr-xr-x 11 root root 4096 Dec 21 15:42 interfaces
drwxr-xr-x 4 root root 4096 Dec 21 15:42 lib
drwxr-xr-x 3 root root 4096 Dec 21 15:41 tools

malik__tania · ‎12-21-2015

Where as:

tania@tania-Latitude-E5540:/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64_lin$ ls -l
total 1032580
-rwxr-xr-x 1 root root 32686896 Jul 30 14:03 libmkl_avx2.so
-rwxr-xr-x 1 root root 39565233 Jul 30 14:03 libmkl_avx512_mic.so
-rwxr-xr-x 1 root root 34186727 Jul 30 14:03 libmkl_avx512.so
-rwxr-xr-x 1 root root 35565192 Jul 30 14:03 libmkl_avx.so
-rw-r--r-- 1 root root 1221488 Jul 30 14:04 libmkl_blacs_ilp64.a
-rw-r--r-- 1 root root 1222312 Jul 30 14:04 libmkl_blacs_intelmpi_ilp64.a
-rwxr-xr-x 1 root root 490800 Jul 30 14:04 libmkl_blacs_intelmpi_ilp64.so
-rw-r--r-- 1 root root 707234 Jul 30 14:03 libmkl_blacs_intelmpi_lp64.a
-rwxr-xr-x 1 root root 277115 Jul 30 14:03 libmkl_blacs_intelmpi_lp64.so
-rw-r--r-- 1 root root 706410 Jul 30 14:03 libmkl_blacs_lp64.a
-rw-r--r-- 1 root root 1239632 Jul 30 14:04 libmkl_blacs_openmpi_ilp64.a
-rw-r--r-- 1 root root 724554 Jul 30 14:03 libmkl_blacs_openmpi_lp64.a
-rw-r--r-- 1 root root 1221760 Jul 30 14:04 libmkl_blacs_sgimpt_ilp64.a
-rw-r--r-- 1 root root 706682 Jul 30 14:03 libmkl_blacs_sgimpt_lp64.a
-rw-r--r-- 1 root root 678240 Jul 30 13:43 libmkl_blas95_ilp64.a
-rw-r--r-- 1 root root 674352 Jul 30 13:43 libmkl_blas95_lp64.a
-rw-r--r-- 1 root root 213014 Jul 30 14:04 libmkl_cdft_core.a
-rwxr-xr-x 1 root root 166155 Jul 30 14:04 libmkl_cdft_core.so
-rw-r--r-- 1 root root 422816228 Jul 30 13:53 libmkl_core.a
-rwxr-xr-x 1 root root 25694727 Jul 30 13:53 libmkl_core.so
-rwxr-xr-x 1 root root 29552604 Jul 30 14:03 libmkl_def.so
-rw-r--r-- 1 root root 19322966 Jul 30 13:53 libmkl_gf_ilp64.a
-rwxr-xr-x 1 root root 8479034 Jul 30 14:03 libmkl_gf_ilp64.so
-rw-r--r-- 1 root root 19975400 Jul 30 13:53 libmkl_gf_lp64.a
-rwxr-xr-x 1 root root 8867579 Jul 30 14:03 libmkl_gf_lp64.so
-rw-r--r-- 1 root root 18517016 Jul 30 13:53 libmkl_gnu_thread.a
-rwxr-xr-x 1 root root 13034633 Jul 30 14:03 libmkl_gnu_thread.so
-rw-r--r-- 1 root root 19353176 Jul 30 13:53 libmkl_intel_ilp64.a
-rwxr-xr-x 1 root root 8496705 Jul 30 14:03 libmkl_intel_ilp64.so
-rw-r--r-- 1 root root 20005526 Jul 30 13:53 libmkl_intel_lp64.a
-rwxr-xr-x 1 root root 8885218 Jul 30 14:03 libmkl_intel_lp64.so
-rw-r--r-- 1 root root 28704798 Jul 30 13:53 libmkl_intel_thread.a
-rwxr-xr-x 1 root root 20368592 Jul 30 14:03 libmkl_intel_thread.so
-rw-r--r-- 1 root root 8139352 Jul 30 13:44 libmkl_lapack95_ilp64.a
-rw-r--r-- 1 root root 8132352 Jul 30 13:43 libmkl_lapack95_lp64.a
-rwxr-xr-x 1 root root 36894287 Jul 30 14:03 libmkl_mc3.so
-rwxr-xr-x 1 root root 33396687 Jul 30 14:03 libmkl_mc.so
-rwxr-xr-x 1 root root 5147328 Jul 30 14:03 libmkl_rt.so
-rw-r--r-- 1 root root 11778266 Jul 30 14:05 libmkl_scalapack_ilp64.a
-rwxr-xr-x 1 root root 7512849 Jul 30 14:05 libmkl_scalapack_ilp64.so
-rw-r--r-- 1 root root 11772212 Jul 30 14:04 libmkl_scalapack_lp64.a
-rwxr-xr-x 1 root root 7476637 Jul 30 14:05 libmkl_scalapack_lp64.so
-rw-r--r-- 1 root root 10820490 Jul 30 13:53 libmkl_sequential.a
-rwxr-xr-x 1 root root 7732586 Jul 30 14:03 libmkl_sequential.so
-rwxr-xr-x 1 root root 11277446 Jul 30 13:59 libmkl_vml_avx2.so
-rwxr-xr-x 1 root root 11638255 Jul 30 13:59 libmkl_vml_avx512_mic.so
-rwxr-xr-x 1 root root 10434720 Jul 30 14:00 libmkl_vml_avx512.so
-rwxr-xr-x 1 root root 10930771 Jul 30 13:58 libmkl_vml_avx.so
-rwxr-xr-x 1 root root 5393410 Jul 30 13:54 libmkl_vml_cmpt.so
-rwxr-xr-x 1 root root 5472445 Jul 30 13:53 libmkl_vml_def.so
-rwxr-xr-x 1 root root 9569252 Jul 30 13:56 libmkl_vml_mc2.so
-rwxr-xr-x 1 root root 9817468 Jul 30 13:57 libmkl_vml_mc3.so
-rwxr-xr-x 1 root root 9576458 Jul 30 13:55 libmkl_vml_mc.so
drwxr-xr-x 3 root root 4096 Dec 21 15:41 locale

mecej4 · ‎12-21-2015

All the shared library files have execute permissions set, and the directory containing them has execute permissions set, so there is no problem there.

Sorry, I have run out of suggestions at this point.

Ying_H_Intel · ‎12-21-2015

Hello,

Not sure if it is related to the environment. Could you please try

1) > ldd your application, what the output?

2) you mentioned MIC, so have you xeon phi coprocessor in your system? does the application run on the xeon phi?

3) there are some example (fortran or C sample) under MKL install directory. Could you please unzip one of them and run one of them

for example,

>/mkl_example/cblas

> make libintel64 function=cblas_dgemm

let us know the output of the sample program.

4) or is it possible to LD_PRELOAD=/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64_lin/libmkl_def.so, then run your application.

Here is a similiar issue for your reference: https://software.intel.com/en-us/forums/intel-math-kernel-library/topic/600019

Best Regards,

Ying

malik__tania · ‎12-22-2015

HI

Here are the results of 4 options that you suggested:

(1) tania@tania-Latitude-E5540:~/openform_fupermod_test$ ldd ~/build_fupermod/mkl/tools/builder
not a dynamic executable

2) you mentioned MIC, so have you xeon phi coprocessor in your system? does the application run on the xeon phi?

I have no idea about it. I just install MKL to use in cblas.

(3) Examples are working

tania@tania-Latitude-E5540:/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/examples/cblas$ sudo make libintel64 function=cblas_dgemm compiler=gnu

make cblas_dgemmx.res _IA=intel64 EXT=a RES_EXT=lib
make[1]: Entering directory `/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/examples/cblas'
mkdir -p ./_results/gnu_lp64_parallel_intel64_lib
gcc -m64 -w -I../../include source/cblas_dgemmx.c source/common_func.c -Wl,--start-group "../../lib/intel64"/libmkl_intel_lp64.a "../../lib/intel64"/libmkl_intel_thread.a "../../lib/intel64"/libmkl_core.a -Wl,--end-group -L"../../../compiler/lib/intel64" -liomp5 -lpthread -ldl -lm -o _results/gnu_lp64_parallel_intel64_lib/cblas_dgemmx.out
export LD_LIBRARY_PATH="../../lib/intel64"::"../../../compiler/lib/intel64"; _results/gnu_lp64_parallel_intel64_lib/cblas_dgemmx.out data/cblas_dgemmx.d >_results/gnu_lp64_parallel_intel64_lib/cblas_dgemmx.res

(4) I got the same error while using PRELOAD before:

tania@tania-Latitude-E5540:~/openform_fupermod_test$ LD_PRELOAD=/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64_lin/libmkl_def.so
tania@tania-Latitude-E5540:~/openform_fupermod_test$ mpirun -np 4 -machinefile machinefile ~/build_fupermod/mkl/tools/builder -l ~/build_fupermod/mkl/routines/mxm/.libs/libmxm_1d.so -U2000 -s100
Kernel:/home/tania/build_fupermod/mkl/routines/mxm/.libs/libmxm_1d.so
Kernel:/home/tania/build_fupermod/mkl/routines/mxm/.libs/libmxm_1d.so
Kernel:/home/tania/build_fupermod/mkl/routines/mxm/.libs/libmxm_1d.so
Kernel:/home/tania/build_fupermod/mkl/routines/mxm/.libs/libmxm_1d.so

Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.

Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.
Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.
Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.
-------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code.. Per user-direction, the job has been aborted.
-------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

Process name: [[48793,1],1]
Exit code: 1
--------------------------------------------------------------------------

malik__tania · ‎12-22-2015

I am attaching Fupermod makefile.in and configure.ac file

you can check the compile option of MKL in case if you want see.

Ying_H_Intel · ‎12-22-2015

Hi Tania,

Thanks for the reply. so we can get

1) MKL work in simple command environment, but can't work in the mpirun

>mpirun -np 4 -machinefile machinefile ~/build_fupermod/mkl/tools/builder -l

~/build_fupermod/mkl/routines/mxm/.libs/libmxm_1d.so -U2000 -s100

2) ~/build_fupermod/mkl/routines/mxm/.libs/libmxm_1d.so are linked mkl librarylibintel_libmkl_intel_lp64.so or

ibmkl_gnu_thread.so libmkl_core.so. and libintel_libmkl_intel_lp64.so or ibmkl_gnu_thread.so or libmkl_core.so

internally invokes libmkl_avx2.so or libmkl_def.so. Because in my application I don't need them directaly.

So the solution is either remove the dependency or to make sure the mpirun environment can invoke the libmkl_avx2.so or

libmkl_def.so as command line case.

Could you rebuild the Fupermod or libmxm_1d.so with static mkl library (for example, libmkl_intel_lp64.a, libmkl_gnu_thread.a libmkl_core.a, not .so), which will not ask the mkl dynamic libary (libmkl_avx2.so and libmkl_def.so)

other ways, may be -rpath option or add –no-as-needed as https://software.intel.com/en-us/forums/intel-math-kernel-

library/topic/...

Regards,

Ying