topic in Intel® oneAPI Math Kernel Library

MKL might not support my platform!!

malik__tania — Tue, 22 Dec 2015 16:39:47 GMT

I installed Fupermod software framwork with MKL it installed correctally but when I run my application i got this error:

tania@tania-Latitude-E5540:~/openform_fupermod_test$ mpirun -np 6 -machinefile machinefile ~/build_fupermod/mkl/tools/builder -l ~/build_fupermod/mkl/routines/mxm/.libs/libmxm_1d.so -U2000 -s100
Kernel:/home/tania/build_fupermod/mkl/routines/mxm/.libs/libmxm_1d.so
Kernel:/home/tania/build_fupermod/mkl/routines/mxm/.libs/libmxm_1d.so
Kernel:/home/tania/build_fupermod/mkl/routines/mxm/.libs/libmxm_1d.so
Kernel:/home/tania/build_fupermod/mkl/routines/mxm/.libs/libmxm_1d.so
Kernel:/home/tania/build_fupermod/mkl/routines/mxm/.libs/libmxm_1d.so
Kernel:/home/tania/build_fupermod/mkl/routines/mxm/.libs/libmxm_1d.so
Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.
Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.
-------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code.. Per user-direction, the job has been aborted.
-------------------------------------------------------
Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

Process name: [[32927,1],2]
Exit code: 2

I posted about this error in this thread. But still didn't get any solution.

https://software.intel.com/en-us/comment/1853675#comment-1853675

Here is my platform description.

tania@tania-Latitude-E5540:~/openform_fupermod_test$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 2
Core(s) per socket: 2
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 69
Stepping: 1
CPU MHz: 1399.992
BogoMIPS: 5387.86
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 4096K
NUMA node0 CPU(s): 0-3

I run MKL example cblas it works properly. Here is the output

tania@tania-Latitude-E5540:/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/examples/cblas$ sudo make libintel64 function=cblas_dgemm compiler=gnu
make cblas_dgemmx.res _IA=intel64 EXT=a RES_EXT=lib
make[1]: Entering directory `/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/examples/cblas'
mkdir -p ./_results/gnu_lp64_parallel_intel64_lib
gcc -m64 -w -I../../include source/cblas_dgemmx.c source/common_func.c -Wl,--start-group "../../lib/intel64"/libmkl_intel_lp64.a "../../lib/intel64"/libmkl_intel_thread.a "../../lib/intel64"/libmkl_core.a -Wl,--end-group -L"../../../compiler/lib/intel64" -liomp5 -lpthread -ldl -lm -o _results/gnu_lp64_parallel_intel64_lib/cblas_dgemmx.out

Here you can see with simple c program, I also checked linking it has no issue.

tania@tania-Latitude-E5540:/tmp$ gcc -o test test.c -L/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64_lin -lmkl_avx2 -lmkl_def
tania@tania-Latitude-E5540:/tmp$
tania@tania-Latitude-E5540:/tmp$
tania@tania-Latitude-E5540:/tmp$ ./test
tania@tania-Latitude-E5540:/tmp$
tania@tania-Latitude-E5540:/tmp$
tania@tania-Latitude-E5540:/tmp$ mpicc -o test test.c -L/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64_lin -lmkl_intel_lp64 -lmkl_gnu_thread -lmkl_sequential -lmkl_core
tania@tania-Latitude-E5540:/tmp$
tania@tania-Latitude-E5540:/tmp$
tania@tania-Latitude-E5540:/tmp$ ./test
tania@tania-Latitude-E5540:/tmp$
tania@tania-Latitude-E5540:/tmp$ mpicc -fopenmp -o test test.c -L/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64_lin -lmkl_intel_lp64 -lmkl_gnu_thread -lmkl_sequential -lmkl_core -lgomp
tania@tania-Latitude-E5540:/tmp$
tania@tania-Latitude-E5540:/tmp$ ./test
tania@tania-Latitude-E5540:/tmp$ mpicc -O0 -fopenmp -o test test.c -L/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64_lin -lmkl_intel_lp64 -lmkl_gnu_thread -lmkl_sequential -lmkl_core -lgsl -lgslcblas -ldl -lm -lrt -lgomp -pthread
tania@tania-Latitude-E5540:/tmp$
tania@tania-Latitude-E5540:/tmp$
tania@tania-Latitude-E5540:/tmp$ ./test
tania@tania-Latitude-E5540:/tmp$
tania@tania-Latitude-E5540:/tmp$ mpicc -O0 -fopenmp -o test test.c -L/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64_lin -lmkl_intel_lp64 -lmkl_gnu_thread -lmkl_sequential -lmkl_core -lgsl -lgslcblas -ldl -lm -lrt -lgomp -pthread

Now I am sure there is some problem with MKL with my platform. As it ask for the ibmkl_avx2.so or libmkl_def.so. Although the path includes these files. But I don't know why I am getting this error. Plz help me. And plz let me know, If it is a bug in MKL how can I report it?

Hi Tania,

Roman_D_Intel1 — Tue, 22 Dec 2015 17:12:29 GMT

Hi Tania,

1) Does your app run fine if executed without mpirun?

2) Could you please print the contents of LD_LIBRARY_PATH before running the program?

3) Which mpi do you use? If Intel's, could you please add -genvall to mpirun's options?

malik__tania — Tue, 22 Dec 2015 17:21:53 GMT

(1) I got the same error without Mpirun

tania@tania-Latitude-E5540:~/openform_fupermod_test$ ~/build_fupermod/mkl/tools/builder -l ~/build_fupermod/mkl/routines/mxm/.libs/libmxm_1d.so -U2000 -s100
Kernel:/home/tania/build_fupermod/mkl/routines/mxm/.libs/libmxm_1d.so
Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.

(2) Here is the output of echo $LD_LIBRARY_PATH

tania@tania-Latitude-E5540:~/openform_fupermod_test$ echo $LD_LIBRARY_PATH
/home/tania/.openmpi/lib/:/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64_lin:/opt/ThirdParty-3.0.1/platforms/linux64Gcc/gperftools-svn/lib:/opt/paraviewopenfoam44/lib/paraview-4.4:/opt/openfoam30/platforms/linux64GccDPInt32Opt/lib/openmpi-system:/opt/ThirdParty-3.0.1/platforms/linux64GccDPInt32Opt/lib/openmpi-system:/home/tania/.openmpi/lib:/home/tania/OpenFOAM/tania-3.0.1/platforms/linux64GccDPInt32Opt/lib:/opt/site/3.0.1/platforms/linux64GccDPInt32Opt/lib:/opt/openfoam30/platforms/linux64GccDPInt32Opt/lib:/opt/ThirdParty-3.0.1/platforms/linux64GccDPInt32Opt/lib:/opt/openfoam30/platforms/linux64GccDPInt32Opt/lib/dummy:/home/tania/.openmpi/lib/
tania@tania-Latitude-E5540:~/openform_fupermod_test$ mpirun -np 6 -machinefile machinefile ~/build_fupermod/mkl/tools/builder -l ~/build_fupermod/mkl/routines/mxm/.libs/libmxm_1d.so -U2000 -s100
Kernel:/home/tania/build_fupermod/mkl/routines/mxm/.libs/libmxm_1d.so
Kernel:/home/tania/build_fupermod/mkl/routines/mxm/.libs/libmxm_1d.so
Kernel:/home/tania/build_fupermod/mkl/routines/mxm/.libs/libmxm_1d.so
Kernel:/home/tania/build_fupermod/mkl/routines/mxm/.libs/libmxm_1d.so
Kernel:/home/tania/build_fupermod/mkl/routines/mxm/.libs/libmxm_1d.so
Kernel:/home/tania/build_fupermod/mkl/routines/mxm/.libs/libmxm_1d.so
Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.
Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.
-------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code.. Per user-direction, the job has been aborted.
-------------------------------------------------------
Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.
Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.
Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

Process name: [[56104,1],1]
Exit code: 2

(3) I am using openmpi-1.10.1

Also when I install Fupermod with GSL . My application run perfactlly. Plus on grid5000 platform it works with MKL too, but not working on my laptop.

malik__tania — Tue, 22 Dec 2015 17:41:24 GMT

Here is the ldd output

tania@tania-Latitude-E5540:~/openform_fupermod_test$ ldd ~/build_fupermod/mkl/routines/mxm/.libs/libmxm_1d.so
   linux-vdso.so.1 => (0x00007ffcf9139000)
   libmpi_cxx.so.1 => /home/tania/.openmpi/lib/libmpi_cxx.so.1 (0x00007f2522ba4000)
   libmpi.so.12 => /home/tania/.openmpi/lib/libmpi.so.12 (0x00007f25228cf000)
   libmkl_intel_lp64.so => /opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64_lin/libmkl_intel_lp64.so (0x00007f2521f91000)
   libmkl_gnu_thread.so => /opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64_lin/libmkl_gnu_thread.so (0x00007f252120c000)
   libmkl_core.so => /opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64_lin/libmkl_core.so (0x00007f251f927000)
   libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f251f621000)
   libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f251f412000)
   libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f251f1fc000)
   libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f251efde000)
   libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f251ec19000)
   libopen-pal.so.13 => /home/tania/.openmpi/lib/libopen-pal.so.13 (0x00007f251e93b000)
   libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f251e637000)
   libibverbs.so.1 => /usr/lib/libibverbs.so.1 (0x00007f251e429000)
   libopen-rte.so.12 => /home/tania/.openmpi/lib/libopen-rte.so.12 (0x00007f251e1ad000)
   libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f251dfa9000)
   /lib64/ld-linux-x86-64.so.2 (0x00007f2522fd7000)
   librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f251dda1000)
   libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f251db9e000)

Seems like libmkl_intel_lp64.so or ibmkl_gnu_thread.so or libmkl_core.so internally invokes libmkl_avx2.so or libmkl_def.so. Because in my application I don't need them directaly.

Hi Tania,

Ying_H_Intel — Mon, 28 Dec 2015 01:46:04 GMT

Hi Tania,

The libmxm_1d.so build looks fine. You are exactly right, libmkl_intel_lp64.so or ibmkl_gnu_thread.so or libmkl_core.so internally invokes libmkl_avx2.so or libmkl_def.so( they are processor-optimized library). So the problem is the calling environment can't recognize the following Link Path. Could you rebuild the libmxm_1d.so with static library? for example, you can link the libmkl_intel_lp64.a libmkl_gnu_thread.a libmkl_core.a in your link line.

Then when ldd libmxm_1d.so, no mkl library needed.

Best Regards,
Ying

malik__tania — Mon, 28 Dec 2015 10:21:02 GMT

Hi Ying

I have to build the libmxm_1d.so with dynamic library. Because I have to use MKL laibrary threading with this builder application. Is their any other option to make it fix please?

Hi Tania,

Ying_H_Intel — Tue, 29 Dec 2015 01:52:18 GMT

Hi Tania,

YOu can build the libmxm_1d.so with threading MKL library, whatever link with dynamic library and static library. For example, libmkl_intel_lp64.a libmkl_gnu_thread.a libmkl_core.a + dynamic gomp library are static library and GNU openmp threading.

If have to use threading MKL library, you may need to add -rpath option and -Wl -as-no-needed option in your build link. as I mentioned last time.

Best Regards,

Ying