Threading issue with MKL

[postaquestion]

I get a segfault in an application linked to MKL, whenever more than one thread is used. The code works fine with OMP_NUM_THREADS=1, but it segfaults if I set this variable to >1. The simplest code exhibiting this problem uses only mkl_cspblas_dcsrgemv from mkl.

System: Red Hat Enterprise Linux release 4
MKL: 10.0.011

link sequence:
-lmkl_lapack -lmkl -lmkl_gnu_thread -liomp5 -lpthread -lgfortran -lmkl -lmkl_gnu_thread -liomp5 -lpthread -lgfortran


backtrace in gdb (thread which segfaults):
#0 0x0000002a9b54e9b9 in L_mkl_spblas_lp64_mkl_cspblas_dcsrgemv_128__par_loop0 ()
from /opt/intel/mkl/10.0.011/lib/em64t//libmkl_intel_thread.so
#1 0x0000002a9bdbb903 in __kmp_invoke_microtask () from /opt/intel/mkl/10.0.011/lib/em64t/libiomp5.so
#2 0x0000002a9bda93e0 in __kmpc_invoke_task_func () from /opt/intel/mkl/10.0.011/lib/em64t/libiomp5.so
#3 0x0000002a9bdaa0cc in __kmp_launch_thread () from /opt/intel/mkl/10.0.011/lib/em64t/libiomp5.so
#4 0x0000002a9bdbda43 in __kmp_create_monitor () from /opt/intel/mkl/10.0.011/lib/em64t/libiomp5.so
#5 0x0000003a626060aa in start_thread () from /lib64/tls/libpthread.so.0
#6 0x0000003a61bc53d3 in clone () from /lib64/tls/libc.so.6
#7 0x0000000000000000 in ?? ()

threads (OMP_NUM_THREADS=2):
3 Thread 1077973344 (LWP 28977) 0x0000003a6260ae2d in __lll_mutex_unlock_wake () from /lib64/tls/libpthread.so.0
2 Thread 1073809760 (LWP 28976) 0x0000003a62608acf in pthread_cond_timedwait@@GLIBC_2.3.2 ()
from /lib64/tls/libpthread.so.0
* 1 Thread 182899530176 (LWP 28927) 0x0000002a9b54e9b9 in L_mkl_spblas_lp64_mkl_cspblas_dcsrgemv_128__par_loop0 ()
from /opt/intel/mkl/10.0.011/lib/em64t//libmkl_intel_thread.so



ldd on executable:
libboost_python-gcc-1_32.so.1.32.0 => /local/opt/lib/libboost_python-gcc-1_32.so.1.32.0 (0x0000002a95819000)
/local/libmesh-0.6.1/lib/x86_64-unknown-linux-gnu_opt/libmesh.so (0x0000002a95968000)
/local/libmesh-0.6.1/contrib/lib/x86_64-unknown-linux-gnu_opt/liblaspack.so (0x0000002a95ede000)
/local/libmesh-0.6.1/contrib/lib/x86_64-unknown-linux-gnu_opt/libparmetis.so (0x0000002a95ff9000)
/local/libmesh-0.6.1/contrib/lib/x86_64-unknown-linux-gnu_opt/libmetis.so (0x0000002a96152000)
/local/libmesh-0.6.1/contrib/lib/x86_64-unknown-linux-gnu_opt/libsfcurves.so (0x0000002a962b5000)
/local/libmesh-0.6.1/contrib/lib/x86_64-unknown-linux-gnu_opt/libgzstream.so (0x0000002a963b7000)
libz.so.1 => /usr/lib/gcc/x86_64-redhat-linux/3.4.3/../../../../lib64/libz.so.1 (0x0000002a964bb000)
/local/libmesh-0.6.1/contrib/lib/x86_64-unknown-linux-gnu_opt/libgmv.so (0x0000002a965ce000)
libvtkIO.so.5.0 => /usr/local/lib/libvtkIO.so.5.0 (0x0000002a966e4000)
libvtkCommon.so.5.0 => /usr/local/lib/libvtkCommon.so.5.0 (0x0000002a96b9f000)
& nbsp; /local/libmesh-0.6.1/contrib/lib/x86_64-unknown-linux-gnu_opt/libtetgen.so (0x0000002a96ecb000)
/local/libmesh-0.6.1/contrib/lib/x86_64-unknown-linux-gnu_opt/libtriangle.so (0x0000002a97080000)
libpetscsnes.so => /local/petsc-2.3.3-p8/lib/linux-mkl-dbg5/libpetscsnes.so (0x0000002a971a6000)
libpetscksp.so => /local/petsc-2.3.3-p8/lib/linux-mkl-dbg5/libpetscksp.so (0x0000002a972e2000)
libpetscdm.so => /local/petsc-2.3.3-p8/lib/linux-mkl-dbg5/libpetscdm.so (0x0000002a9761f000)
libpetscmat.so => /local/petsc-2.3.3-p8/lib/linux-mkl-dbg5/libpetscmat.so (0x0000002a97797000)
libpetscvec.so => /local/petsc-2.3.3-p8/lib/linux-mkl-dbg5/libpetscvec.so (0x0000002a97b7d000)
libpetsc.so => /local/petsc-2.3.3-p8/lib/linux-mkl-dbg5/libpetsc.so (0x0000002a97d1a000)
libmkl_lapack.so => /opt/intel/mkl/10.0.011/lib/em64t/libmkl_lapack.so (0x0000002a97ef0000)
/opt/intel/mkl/10.0.011/lib/em64t//libmkl_intel_lp64.so (0x0000002a9862f000)
/opt/intel/mkl/10.0.011/lib/em64t//libmkl_intel_thread.so (0x0000002a98935000)
/opt/intel/mkl/10.0.011/lib/em64t//libmkl_core.so (0x0000002a98d76000)
libmkl_gnu_thread.so => /opt/intel/mkl/10.0.011/lib/em64t/libmkl_gnu_thread.so (0x0000002a98f47000)
libiomp5.so => /opt/intel/mkl/10.0.011/lib/em64t/libiomp5.so (0x0000002a9929f000)
libpthread.so.0 => /lib/../lib64/tls/libpthread.so.0 (0x0000002a9940b000)
libgfortran.so.1 => /usr/lib/gcc/x86_64-redhat-linux/3.4.3/../../../../lib64/libgfortran.so.1 (0x0000002a99520000)
libdl.so.2 => /lib/../lib64/libdl.so.2 (0x0000002a996b5000)
libgcc_s.so.1 => /lib/../lib64/libgcc_s.so.1 (0x0000002a997b9000)
libg2c.so.0 => /usr/lib/gcc/x86_64-redhat-linux/3.4.3/../../../../lib64/libg2c.so.0 (0x0000002a998c4000)
libm.so.6 => /lib/../lib64/tls/libm.so.6 (0x0000002a999e5000)
libX11.so.6 => /usr/X11R6/lib64/libX11.so.6 (0x0000002a99b91000)
libmkl_mc.so => /opt/intel/mkl/10.0.011/lib/em64t/libmkl_mc.so (0x0000002a99d72000)
libmkl_def.so => /opt/intel/mkl/10.0.011/lib/em64t/libmkl_def.so (0x0000002a9a57f000)
libc.so.6 => /lib/../lib64/tls/libc.so.6 (0x0000002a9ab4d000)
libstdc++.so.6 => /usr/lib/gcc/x86_64-redhat-linux/3.4.3/../../../../lib64/libstdc++.so.6 (0x0000002a9ad80000)
libvtkFiltering.so.5.0 => /usr/local/lib/libvtkFiltering.so.5.0 (0x0000002a9af71000)
l ibvtkDICOMParser.so.5.0 => /usr/local/lib/libvtkDICOMParser.so.5.0 (0x0000002a9b3c1000)
libvtkpng.so.5.0 => /usr/local/lib/libvtkpng.so.5.0 (0x0000002a9b503000)
libvtkzlib.so.5.0 => /usr/local/lib/libvtkzlib.so.5.0 (0x0000002a9b633000)
libvtkjpeg.so.5.0 => /usr/local/lib/libvtkjpeg.so.5.0 (0x0000002a9b747000)
libvtktiff.so.5.0 => /usr/local/lib/libvtktiff.so.5.0 (0x0000002a9b872000)
libvtkexpat.so.5.0 => /usr/local/lib/libvtkexpat.so.5.0 (0x0000002a9b9d0000)
libvtkMPEG2Encode.so.5.0 => /usr/local/lib/libvtkMPEG2Encode.so.5.0 (0x0000002a9bafe000)
libvtksys.so.5.0 => /usr/local/lib/libvtksys.so.5.0 (0x0000002a9bc17000)
libmpich.so => /local/petsc-2.3.3-p8/externalpackages/mpich2-1.0.5p4/linux-mkl-dbg5/lib/libmpich.so (0x0000002a9bd52000)
librt.so.1 => /lib/../lib64/tls/librt.so.1 (0x0000002a9bfff000)
/lib64/ld-linux-x86-64.so.2 (0x000000552aaaa000)

[Todd_R_Intel]

The file libmkl.so is a linker script that links libmkl_intel_lp64.so, libmkl_intel_thread.so and libmkl_core.so. It's meant to help users in the transition of library names that was made with version 10.0. Having '-lmkl' on the link line together with '-lmkl_gnu_thread' may be causing some problems since libmkl_gnu_thread.so and libmkl_intel_thread.so are both threading layer libraries and the one that is actually linked in may be unpredictable. Could you give the following link sequence a try?

link sequence:
-Wl,--start-group -lmkl_gf_lp64 -lmkl_gnu_thread -lmkl_core -Wl,--end-group -liomp5 -lpthread -lgfortran

Further linking information can be found in userguide which is available in the doc directory or online.

Todd