Beginners guide to scalapack link line

[savinovsv]

Hi all!
I am very sorry for bothering the respectable community, but I suppose I need a couple of coments from the first hands.
I honestly digged Intel manuals and googled (surprisingly the fastest way to find something on intel forums!!??) quite a lot, but still do not have perfectly clear understanding. Please, help.

The question is as old as earth.
How to link with scalapack and impi correctly?

Environment
Ifort 11.1.059, mkl 10.2.2.025, impi 3.2.2.006
unix x86_64

Using mpiifort.
PATHs, LD_LIBRARY_PATHs etc are correctly set.

Let us see advices.

1. MKL users guide
Table 5-3, p 5-5

ScaLAPACK libmkl_scalapack_lp64.so
lp64 interface libmkl_lapack.so
libmkl_core.so

Single lapack, no group

2. Scalapack test (/opt/intel/Compiler/mkl/tests/scalapack/make.inc)

ifeq ($(mpi),intelmpi)

BLACSLIB=$(MKL_PATH)/libmkl_blacs_intelmpi$(IFACE_PART).$(EXT_LIB)

SCALAPACKLIB= $(MKL_PATH)/libmkl_scalapack$(IFACE_PART).$(EXT_LIB)

**** Default
MKL_LIBS=-L$(MKL_PATH)
$(MKL_PATH)/libmkl_intel$(ADD_IFACE).so
-Wl,--start-group
$(MKL_PATH)/libmkl_sequential.so
$(MKL_PATH)/libmkl_lapack.so
$(MKL_PATH)/libmkl_core.so
-Wl,--end-group
-lpthread

Only sequential by default (Why?)

****** My variant
MKL_LIBS=-L$(MKL_PATH)
$(MKL_PATH)/libmkl_intel$(ADD_IFACE).so
-Wl,--start-group
$(MKL_PATH)/libmkl_intel_thread.so
$(MKL_PATH)/libmkl_lapack.so
$(MKL_PATH)/libmkl_core.so
-Wl,--end-group
-liomp5
-lpthread

LIBS=$(LIBpath) $(TOOLSLIB) $(SCALAPACKLIB) $(FBLACSLIB) $(MKL_LIBS) $(INSLIB) $(MPILIB)

This also pass tests.
To pass tests I do NOT need lpthread !!?? It is linked by mpiifort. So, do I need it in link line?
If I would add -openmp to compiler flags all tests will crash. Personally I can not follow what is the reason.
Nothing about threaded MPI (opposite to GNU, which insists on mt_mpi immediately when sees mkl_intel_thread (equaly mkl_gnu_thread))

By the way, what does it mean in TESTS?!!!
Finished 36 tests, with the following results:
24 tests completed and passed residual checks.
0 tests completed and failed residual checks.
12 tests skipped because of illegal input values.

3. MKL users guide
p 5-8

Dynamic linking of myprog.f and parallel Intel MKL supporting the LP64 interface:
ifort myprog.f -L$MKLPATH -I$MKLINCLUDE
-lmkl_intel_lp64
-lmkl_intel_thread
-lmkl_core
-liomp5
-lpthread

Nothing about scalapack, but OK try to see below

4. MKL users guide
p 9-5

To link with ScaLAPACK for a cluster of systems based on the IA-64 architecture, use the following libraries:
/opt/intel/mpi/3.0/bin/mpiifort
-L$MKLPATH
-lmkl_scalapack_lp64
-lmkl_blacs_intelmpi_lp64
-lmkl_lapack
-lmkl_intel_lp64
lmkl_intel_thread
-lmkl_lapack
lmkl_core
-liomp5
-lpthread

Two lapack libs !!! Does it really matter?

5. Link advisor http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor

-L$MKLPATH -lmkl_scalapack_lp64
$MKLPATH/libmkl_solver_lp64.a
-Wl,--start-group -
lmkl_intel_lp64
-lmkl_intel_thread
-lmkl_core
-lmkl_blacs_intelmpi_lp64
-Wl,--end-group
-openmp
-lpthread

NO lapack and group

6. My link line (to use scalapack with impi)

SCALAPACK_LIBS=
-L$(MKLPATH)
-lmkl_scalapack_lp64
-Wl,--start-group
-lmkl_lapack
-lmkl_intel_lp64
-lmkl_intel_thread or -lmkl_sequential
-lmkl_core
-lmkl_blacs_intelmpi_lp64
-Wl,--end-group
-L/opt/intel/Compiler/11.1/059/lib/intel64 -liomp5 or nothing
-lpthread

MKL users guide
p 5-6.

If you use the Intel compiler, link in the libiomp/libguide version that comes with the compiler, that is, use the -openmp option.

Is it really truth? What will be the effect of MKL's (instead of compiler's) iomp5?

Finally there are a few questions.

- Do I need -lpthread in link line?
- Do I nee -lmkl_lapack in link line?
- What is the disastrous effect on scalapack tests of -openmp compiler flag? Can I use it in general?
- What is the effect of wrong iomp5?
- Do I need mt_mpi with lmkl_intel_thread? Why GNU is so strict?
- There is no net effect of using -mkl 11.1 compiler flag. By the way compiler manual also suggests to use sequential version with scalapack. Why?
- After all is my link line correct?

Thank in advance.
Sergei

[Gennady_F_Intel]

Sergey,
please see the artilce to help you find out the list libraries that need to link your application
--Gennady

[savinovsv]

Quoting - Gennady Fedorov (Intel)

Sergey,
please see the artilce to help you find out the list libraries that need to link your application
--Gennady

Hi, Gennady

Just have a closer look. One of mentioned link advices is namely "link adviser'.

What is bothering me is the discrepancy. I want straight answer like "do this and this". As you can see from my message it is not a case.

Please, read the questonS. As you can easily see I do not need rather general directions.

Sergei

[TimP]

Rather than everyone asking you what you intend to do, it's more productive for you to use the resources provided, then ask questions.

[savinovsv]

Quoting - tim18

Rather than everyone asking you what you intend to do, it's more productive for you to use the resources provided, then ask questions.

Hello Tim18!

May be I am not smart enough, but I could NOT find the information necessary to answer my questions in Intel's documentation.

I also can not see the reason why this simple and purely technological knowledge should be a kind of magic. Why should I even ask these question having manuals in hands. The answers MUST be inside. They are NOT.

Instead of answers I got advices to read more. Nice. At the same time I have the impression that the message was not even troughtly read. Otherwise you should notice that questions (exept for the last one, which was placed mostly to draw attention) have the right to exist.

And so I can only repeat my questions in the hope that somebody will spend his time explaining to me, for example, what will be the difference in programs behaviour if I will link iomp5 from MKL comparing to linking iomp5 from compiler (see the first post). Or why GNU behaiour is different. I guess this is important both for me and for you.

The hope is dying the last.

And to be honest I did expect the reaction like I have got. I simply pointed some discrepancies and uncertainties. All I want is reasonable comments, which are based on the assumption that I know what I am asking.

Sergei

P.S.

The situation can be really complicated with mixture if libs say on big machine (everything depends on everythig, the order is important and so on). But this lib is the bunch of tools from single source and there should exists simple receipt (preferably with explanations "what if" kind) how to use it.

P.P.S.
Last minute note.

MKL users guide
p3-13

libmkl_solver_lp64.a Deprecated.
Empty library for backward compatibility

see http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor

[Alex_Kosenkov]

Sergei, thank you for your efforts describing your concerns.

I'll try to make it a bit more clear for you.

- Do I need -lpthread in link line?

- What is the disastrous effect on scalapack tests of -openmp compiler flag? Can I use it in general?

- What is the effect of wrong iomp5?

If you link your application with icc and give it -openmp flag - the compiler goes by it self and adds -lpthread to the link line - you can see it by compiling with -v icc flag. But it is better to save your self from unexpected behaviour (i.e. while switching to another compiler) explicitly stating -lpthread.

Regarding iomp5 - this is icc compiler library and MKL has it's own version distributed to make it possible to link against it with other compilers such as gcc. So if you state -openmp - icc will link against it's version of iomp5.

All your link lines heavily depend upon the working environment. If I have icc and intel mpi and trying to compile hybrid application (threads + mpi) I have the following link line according to the http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor/ which is the same as you wrote (and at least seems fine to me):

-L$MKLPATH -lmkl_scalapack_lp64 $MKLPATH/libmkl_solver_lp64.a -Wl,--start-group -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_intelmpi_lp64 -Wl,--end-group -openmp -lpthread

I don't see the sequential library that you have mentioned. But indeed hybrid application is not default case for MPI applications - usually people use pure MPI apps and this would be sequential library since we don't have threads there, aren't we? :)

Can't predict the behaviour of application that is linked with the wrong version of liomp5 - you have a compiler's version and it is better to stick to this version until you are positive about what you are doing.

- Do I nee -lmkl_lapack in link line?

Usually it is not needed since you have mkl_core there that contains some parts of lapack that ScaLAPACK depends upon. But if it does exists in the link line anyway it will not affect resulting binary.

Answering "double lmkl_lapack" question here: double occurence can be there because of the cyclic dependencies across libraries. Usually you can avoid this by using --start-group / --end-group that forces linker to iterate through libs until dependencies are satisfied.

- Do I need mt_mpi with lmkl_intel_thread? Why GNU is so strict?

Usually you do need to state mt_mpi if you are threading inside MPI applications since it requires MPI to behave differently. Intel compiler can guess for you what you really want while gcc can't afford to know everything about all MPI vendors.

- There is no net effect of using -mkl 11.1 compiler flag. By the way compiler manual also suggests to use sequential version with scalapack. Why?

Again, threading is not the default behaviour behavior for most of MPI applications. Regarding -mkl compiler flag - it is trying to help you to pull the link line for you. You can see what it's actually doing with icc -v flag. In fact it's doing fine for me - though I don't use it and would restrain my self from using it - it's potentially unexpected behaviour between different compiler's version. So my advise would be not to use it until you are positive about compatibility of the concrete versions of MKL and ICC.

I hope that I've cleared it at least a bit for you.

Cheers

-- Alexander

[savinovsv]

Hello Alexander!

Thank you very much for the comments. It is really helpful. My questions are certainly answered.

The only one dim spot still exists. Do not you think it will be VERY useful to supply MKL Scalapack tests with more complete makefile and test apps allowing users to check the MKL/MPI installation in real conditions? The most important, I guess, is threading at different levels - app, mpi, mkl. The user then will be able to see link line that works for sure and the reason for asking questions will almost dissapear.

Best wishes,

Sergei

P.S.

I knew the hope is dying the last.