How to link MKL with MPI (openmpi)?

[Bogdan_S_]

Hi all,

I've been trying to link my openMPI program with MKL library for 24 hours now. All the options that I tried have failed and I'm just bedazzled with the amount of possible compiler swithces needed to compile and link these two things togeteher. Basically I have some matrix calculation that I do in parallel in openMPI (which works fine) and I want to check final results with 1 or 2 MKL functions. I have to mention that I have installed Intel composer_xe_2013. Although it has 2 subfolders: composer_xe_2013_sp1  and composer_xe_2013_sp1.1.106 ...

I've also tried MKL Linking advisor (if it is called that way), but even with it I cannot seem to find a way to connect these two things. Logically I was thinking that first thing to find out was which swithces are needed for GCC compiler to comile MKL program and then apply the same switches to MPICC (I believe there is 100% analogy). I've followed tutorial on MKL installation (http://software.intel.com/en-us/articles/intel-mkl-103-getting-started.), all the install scripts provided there (var scripts, paths etc.) but nothing helped. Compiling with line:

$gcc  mkl_lab_solution.c -lmkl_intel -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm

Gives me error: /usr/bin/ld: cannot find -lmkl_intel

The good thing is that I was able to compile plain MKL program (no MPI) with ICC compiler. However to be able to that I had to source two sh files, and it seems that I have to do that every time I restart system. Is there another way to enable icc other then repeatedly sourcing these two files every time I restart computer? It is not such a big problem, but  if that is the only way then I will have to do automatic sourcing in my project running sh file.

Thanks in advance for any help or insight provided,

Bogdan

[TimP]

A single sourceing of compilervars.sh would set up both icc and MKL, for the MKL installed with icc.  This would also make the MKL shared objects available to a build made with gcc, or mpicc built against that gcc, but you would probably include a -L entry pointing to the mkl/lib directory in your link step (the same directory which compilervars adds to you LD_LIBRARY_PATH),  You are correct that it's possible to support all gcc OpenMP calls by using the libraries you quoted, same as for icc.

If you so choose, you can include the sourceing of compilervars in your .profile (if using e.g. bash) so that the icc/mkl environment appears whenever you open a new shell.

Assuming that your openmpi is installed off the default paths (as is preferable), you would also need to set the path for mpicc. You might choose to build a copy of mpicc which uses icc and install it in an alternate path so you can select which mpicc you use, e.g.

configure and install openmpi with for example --prefix=/opt/ompi1.6/intel/ for icc and /opt/ompi1.6/gcc4.9/ if you want both choices.

icc and gcc are compatible enough for objects to link together with the same mpicc but you might need to deal with differences in library and include paths which mpicc normally is meant to avoid.

[Bogdan_S_]

Hi TimP,

I have understood most of what you where saying in your message. I also realized that I was making mistake by mixing 32 and 64 bit version of MKL libraries so I stick now to 32 bit version (I have 64 bit Ubuntu though - can that be a problem?) since that is the only version that has full tutorial on web: http://software.intel.com/en-us/articles/intel-mkl-103-getting-started

By following this tutorial strictly and sourcing 2 necessary files I was able to set my $LD_LIBRARY_PATH and $LIBRARY_PATH veriables to correct values (32-bit versions of MKL libraries). After thet I strictly applied the following line on your MKL example code:

gcc -m32 mkl_lab_solution.c -lmkl_intel -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm

However then I get this error:

/home/sataric/intel/composer_xe_2013_sp1.1.106/mkl/lib/ia32/libmkl_intel_thread.so: undefined reference to `__kmpc_ok_to_fork'
/home/sataric/intel/composer_xe_2013_sp1.1.106/mkl/lib/ia32/libmkl_intel_thread.so: undefined reference to `__kmpc_end_single'

etc...

However if I try to compile only with just adding -c to previous line compiling goes fine and I get .o file, which I unfortunatelly cannot turn into .out file after that. I have read that these kind of errors containing '_kmpc_ok_to_fork'' are due to not putting -fopenmp or -openmp compiler flag. I tried also that but that didn't help much or I don't know where exactly to put this flag (I don't know ordering of flags well enough). 

I feel like I'm close to solving this issue, as it already took me too much time.

Thanks in advance for help provided.

[Bogdan_S_]

Thanks for the answer TimP,

I have been able to change my settings and now I'm literally using the same settings as in your tutorial (I had problems mixing 32 and 64 bit settings). Now I'm able to compile program but I cannot link it. My environment variables are set up according to 2 .sh files that you provided. I am able to compile program with the gcc line provided in my previous message, but I'm unable to link it into .out file. I get error related to lots of (apparently) OpenMP functions and the first one is:

 undefined reference to `__kmpc_ok_to_fork'

I know this is related to OpenMP libraries. I tried putting -fopenmp, -openmp and other combinations into my compiling line but I cannot force it to compile anyways. Maybe it is a minor or stupid mistake but I just don't know what to do anymore... 

[Gennady_F_Intel]

-lomp5 may ( which is intel treading library - libiomp5 ) help you.

[Bogdan_S_]

Hi Gennady,

I used following line for compiling:

gcc -m32 mkl-lab-solution.c -lmkl_intel -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm

As you can see it has liopm5 library but that doesn\'t help. I also tried  your suggestion: lomp5 and libiomp5  but compiler cannot find those two libraires. I think those two libraries you mentioned are same/similar to liomp5 which I did put in, but still the kmpc_ok_to_fork problem persists.

[TimP]

Your command line should be OK,  with the addition of -L specification pointing to the (ia32 32-bit) directories where those mkl shared objects and the libiomp5.so reside.  I would expect MKL to provide libiomp5.so in the same folders as its own .so for use with gcc.  When you have an icc installation, libiomp5.so will be provided in the compiler's own shared object folder.

[Bogdan_S_]

My task is actually to link mkl with mpicc so I now try the mpicc compiler instead of gcc (same issue as with gcc).

TimP I tried your suggestion, and my compiling line now looks like this:

mpicc mkl-lab-solution.c -Ihome/sataric/intel/composer_xe_2013_sp1.1.106/compiler/include -Lhome/sataric/intel/composer_xe_2013_sp1.1.106/compiler/lib/intel64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm

The -I and -L directories are correct ones. i searched for libiomp5.so file and it resides in exact intel64 folder, also in ia32 and mic folders but I chose intel64 as I have 64 bit Ubuntu. However the same issue persist!?! I don't know weather it has to do with relative paths to the compiling folder? My mkl-lab-solution.c file is placed in my /home/sataric/Downloads folder. I'm really getting frustrated here...

[Bogdan_S_]

I found a way to do this. I changed mpicc wrapper so it can use icc instead of gcc compiler (change gcc into icc). How to do this manually I have no idea. Thankfully there is a wrapper that solves this issue.

[Ying_H_Intel]

Hi Bogdan,

if with manually, the link code should be like :

gcc mkl-lab-solution.c -I/home/sataric/intel/composer_xe_2013_sp1.1.106/mkl/include 

-L/home/sataric/intel/composer_xe_2013_sp1.1.106/mkl/lib/intel64

-L/home/sataric/intel/composer_xe_2013_sp1.1.106/compiler/lib/intel64

-lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm

or  run scriipt

>source /home/sataric/intel/composer_xe_2013_sp1.1.106/mkl/bin/mklvars.sh intel64

>gcc main.c -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm

[yhu5@snb04 ReproducerForMerry]$ source /opt/intel/composer_xe_2013_sp1.1.106/mkl/bin/mklvars.sh intel64
[yhu5@snb04 ReproducerForMerry]$ gcc main.c -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm
[yhu5@snb04 ReproducerForMerry]$ ls
a.out  compile.sh  main.c  repro  Runner.sh
[yhu5@snb04 ReproducerForMerry]$ ./a.out
Time for size 22: 2.27809e-06
Time for size 24: 2.49358e-06
GFlops for size 22: 9.3482
GFlops for size 24: 11.0877

Best Regards,

Ying

[Ying_H_Intel]

Hi Bogdan,

One more tiny problem.  I suppose your "home" dir should be under root directory, right?   Then the path should be -I/home/,,,  and  -L/home/... ,   please add "/" before the home.

 not the path -Lhome/.../..

Best Regards,

Ying

Bogdan S. wrote:

My task is actually to link mkl with mpicc so I now try the mpicc compiler instead of gcc (same issue as with gcc).

TimP I tried your suggestion, and my compiling line now looks like this:

mpicc mkl-lab-solution.c -Ihome/sataric/intel/composer_xe_2013_sp1.1.106/compiler/include -Lhome/sataric/intel/composer_xe_2013_sp1.1.106/compiler/lib/intel64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm

The -I and -L directories are correct ones. i searched for libiomp5.so file and it resides in exact intel64 folder, also in ia32 and mic folders but I chose intel64 as I have 64 bit Ubuntu. However the same issue persist!?! I don't know weather it has to do with relative paths to the compiling folder? My mkl-lab-solution.c file is placed in my /home/sataric/Downloads folder. I'm really getting frustrated here...

[Bogdan_S_]

I was able to compile program with MPICC compiler now. What I did is I changed MPICC wrapper to use ICC comipler instead of GCC. Any other solution didn't work. Now if I source everything needed before I use MPICC on code having both MPI and MKLeverything compiles fine. However now I'm unable to run MPI process. The mpirun throws following error:

/home/sataric/intel/composer_xe_2013_sp1.1.106/mpirt/bin/intel64/mpirun: 96: .: Can't open /home/sataric/intel/composer_xe_2013_sp1.1.106/mpirt/bin/intel64/mpivars.sh

I looked directly into mentioned folder for mpivars.sh file and there is no such file in that folder (there is mpirun though). Also I did systemwide search and I couldn't find mpivars.sh file... I tried to find solution on several websites but solutions look kind of obscure (they mention fortran for example). What should I do now?

[Bogdan_S_]

I was able to compile program now. Not directly (even your solution Ying didn't help), but thtough MPICC wrapper. I changed GCC to ICC and now MPICC compiles. However now MPIRUN is the problem. When I start MPIRUN I get this error: 

Can't open /home/sataric/intel/composer_xe_2013_sp1.1.106/mpirt/bin/intel64/mpivars.sh

I searched for this file everywhere and there is no such file on entire system. How to solve this issue now when there is no such file?

[Bogdan_S_]

I found solution finally. I overrided Intel's mpirun with original mpirun (direct path \usr\bin\mpirun in script) and now after a week long struggle everything works...

[Ying_H_Intel]

Hi Bogdan,

Glad to know everything works.

Acording to your discription,  i guess, you build the code with mpicc (change the gcc to icc), which is from your OpenMPI, right?

Then you run with /usr/bin/mpirun, is it from OpenMPI too?

Could you please tell us the result if you run the below command separately?

/home/sataric/intel/composer_xe_2013_sp1.1.106/mpirt/bin/intel64/mpirun -V

and

/usr/bin/mpirun -V

Regards,

Ying

[Bogdan_S_]

Hi Ying,

Yes I built my code by switching mpicc wrapper to use icc instead of gcc. mpicc is from my OpenMPI. mpirun is also from my OpenMPI. 

If I run /home/sataric/intel/composer_xe_2013_sp1.1.106/mpirt/bin/intel64/mpirun -V I get error of missing mpivars.sh file. I tried to find that file on my disk but there is none...That was the issue I battled for past day or two and most solutions I googled said that I should ovveride intel's mpirun (which has to do with some kind of fortran coarrays), and insted use openmpi mpirun. I did that and it worked. How and why it has to be done that way I don't understand, and honestly the only important thing is that it works now. Running /usr/bin/mpirun -V returns: mpirun (Open MPI) 1.6.4.

[Ying_H_Intel]

Hi Bogdan,

Thanks for letting us know.  I guess the mpirun under directory Composer_xe is supposed to look for Intel MPI or MPICH (they has the mpivars.sh). So for OpenMPI, It is correct to either override it or ignore it  (to use OpenMPI's mpirun).  whatever,  you may go ahead for your project with MKL.

Best Regards,
Ying .

[TimP]

I think the question of setting the MPI path after the Intel Fortran path is already an FAQ for OpenMPI.  Intel Fortran provides an mpirun for support of coarrays but it must be kept from appearing on PATH before your MPI.  Current versions of Intel Fortran compilervars.sh (including yours) have checks to avoid setting the coarray support path when Intel MPI is already on path; if you are ambitious, you might try to modify that so it also checks for OpenMPI.  Otherwise, you must simply set your MPI paths after setting the ifort paths.