3 programs (fortran and c++) started with mpirun + intel mkl => simulation hang

[Guillaume_De_Nayer]

Hi,

I start 3 programs with mpirun:
- prog1: fortran program
- prog2: C++ program
- prog3: C++ program which is using mkl of intel

prog3 uses these options:

-Wl,--start-group $(MKLROOT)/lib/intel64/libmkl_intel_lp64.a $(MKLROOT)/lib/intel64/libmkl_intel_thread.a $(MKLROOT)/lib/intel64/libmkl_core.a -Wl,--end-group -openmp -lpthread

When I start the 3 programs with mpirun of intel (intel cluster studio 2011) it hangs...then I have to kill the 3 programs.

When I'm disable the use of MKL in the prog3, the mpirun and the 3 programs are running without problem.

When I enable the use of MKL in the prog3 without -mt_mpi option the program alone "hang".
When I enable the use of MKL in the prog3 with -mt_mpi option the program alone runs.
When I enable the use of MKL in the prog3 with -mt_mpi option and starts the 3 programs (prog1, prog2, prog3) with mpirun, it "hang"

The use of the -mt_mpi option in the other programs doesn't help. The 2 others programs do not use openmp.

I need help, I don't have idea anymore ;)

Best regards,
Guillaume

[Guillaume_De_Nayer]

additional infos:

when I'm using in prog3:
-Wl,--start-group $(MKLROOT)/lib/intel64/libmkl_intel_lp64.a $(MKLROOT)/lib/intel64/libmkl_sequential.a $(MKLROOT)/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread

without -mt_mpi option it works with the mpirun and the 2 others programs. It seems to be related to openmp in an hybrid environement.

[TimP]

In case this is what you are asking about, since you appear to be linking against the libiomp5.so, you must take care that this library is present on LD_LIBRARY_PATH at run time on each node.
You could check by individdual ssh to each node, opening the same environment as in your MPI run, see what ldd yourapp shows, and check if you can run a single process copy of your application.

[Guillaume_De_Nayer]

Is there knows compatibility problems with MKL + openmp + a mix of different programming languages (here Fortran and C) ?

For example on a national supercomputer I have to link my 3 programs with extra lib:
Fortran: -lffio -lsma -lpthread
C++: -lmpi++abi1002 -lsma -lpthread
in order that they can work together.

As you have linked the Intel OpenMP dynamic library, you must take care that no other OpenMP libraries are linked, and that the compilers in use all use compatible run-time libraries. Intel Fortran treats both icc and gcc as "companion processors" for the purpose of C interoperability, and libiomp5 is designed to work as a replacement for libgomp. However, other Fortran compilers, such as gfortran, will have run-time library incompatibility with ifort, so any libraries compiled with gnu fortran or some other fortran would be troublesome. Some of our colleagues have worked with libffio, and that is not easy to get working correctly.

You might note that MKL can work with gfortran, but this requires use of the gnu_thread library in place of the ifort compatible intel_thread library; either of those works with libiomp5, but they don't work together.