Sergei, thank you for your efforts describing your concerns.
I'll try to make it a bit more clear for you.
- Do I need -lpthread in link line?
- What is the disastrous effect on scalapack tests of -openmp compiler flag? Can I use it in general?
- What is the effect of wrong iomp5?
If you link your application with icc and give it -openmp flag - the compiler goes by it self and adds -lpthread to the link line - you can see it by compiling with -v icc flag. But it is better to save your self from unexpected behaviour (i.e. while switching to another compiler) explicitly stating -lpthread.
Regarding iomp5 - this is icc compiler library and MKL has it's own version distributed to make it possible to link against it with other compilers such as gcc. So if you state -openmp - icc will link against it's version of iomp5.
All your link lines heavily depend upon the working environment. If I have icc and intel mpi and trying to compile hybrid application (threads + mpi) I have the following link line according to the http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor/ which is the same as you wrote (and at least seems fine to me):
-L$MKLPATH -lmkl_scalapack_lp64 $MKLPATH/libmkl_solver_lp64.a -Wl,--start-group -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_intelmpi_lp64 -Wl,--end-group -openmp -lpthread
I don't see the sequential library that you have mentioned. But indeed hybrid application is not default case for MPI applications - usually people use pure MPI apps and this would be sequential library since we don't have threads there, aren't we? :)
Can't predict the behaviour of application that is linked with the wrong version of liomp5 - you have a compiler's version and it is better to stick to this version until you are positive about what you are doing.
- Do I nee -lmkl_lapack in link line?
Usually it is not needed since you have mkl_core there that contains some parts of lapack that ScaLAPACK depends upon. But if it does exists in the link line anyway it will not affect resulting binary.
Answering "double lmkl_lapack" question here: double occurence can be there because of the cyclic dependencies across libraries. Usually you can avoid this by using --start-group / --end-group that forces linker to iterate through libs until dependencies are satisfied.
- Do I need mt_mpi with lmkl_intel_thread? Why GNU is so strict?
Usually you do need to state mt_mpi if you are threading inside MPI applications since it requires MPI to behave differently. Intel compiler can guess for you what you really want while gcc can't afford to know everything about all MPI vendors.
- There is no net effect of using -mkl 11.1 compiler flag. By the way compiler manual also suggests to use sequential version with scalapack. Why?
Again, threading is not the default behaviour behavior for most of MPI applications. Regarding -mkl compiler flag - it is trying to help you to pull the link line for you. You can see what it's actually doing with icc -v flag. In fact it's doing fine for me - though I don't use it and would restrain my self from using it - it's potentially unexpected behaviour between different compiler's version. So my advise would be not to use it until you are positive about compatibility of the concrete versions of MKL and ICC.
I hope that I've cleared it at least a bit for you.
Thank you very much for the comments. It is really helpful. My questions are certainly answered.
The only one dim spot still exists. Do not you think it will be VERY useful to supply MKL Scalapack tests with more complete makefile and test apps allowing users to check the MKL/MPI installation in real conditions? The most important, I guess, is threading at different levels - app, mpi, mkl. The user then will be able to see link line that works for sure and the reason for asking questions will almost dissapear.
I knew the hope is dying the last.