Is it possible to compile Intel MKL application to MPI executable?

[alevseev]

Is it possible to compile Intel MKL application to MPI executable?
For example, I have example program that uses ParDiSo library from MKL. Is it possible to compile it into mpi executable, so I can run it by mpi scheduler: mpirun?

[Sergey_P_Intel2]

Yes, it is possible. You can find an example of using MKL + MPI in MP LINPACK part of MKL.

- Sergey

[alevseev]

Quoting - Sergey Pudov (Intel)

Yes, it is possible. You can find an example of using MKL + MPI in MP LINPACK part of MKL.

- Sergey

Could you help with compilation options for simple example (pardiso_sym_f.f from mkl library examples)? Target architecture is Intel64.

[Sergey_P_Intel2]

Let me clarify my previous answer because I think that some misunderstanding still exists. MKL contains different applications; some of them were developed for clusters and used MPI. So you CAN compile Intel MKL application to MPI executable.

But current PARDISO library doesn't have MPI support, so PARDISO examples can't be build to MPI executable without modifications.

For example you can use ScaLAPACK tests as basic for your MPI application and insert PARDISO calls into it, if necessary. Below you can see example of compiler options for ScaLAPACK tests:



mpiifort -nocompchk -c -w -u -O0 psludriver.f

mpiifort -nocompchk -c -w -u -O0 psluinfo.f

mpiifort -nocompchk -c -w -u -O0 psgetrrv.f

mpiifort -nocompchk -c -w -u -O0 psmatgen.f

mpiifort -nocompchk -c -w -u -O0 pmatgeninc.f

mpiifort -nocompchk -c -w -u -O0 pslaschk.f

mpiifort -nocompchk -c -w -u -O0 pslafchk.f

mpiifort -nocompchk -o ../xslu_libem64t_intelmpi_intel_noopt_lp64 psludriver.o psluinfo.o psgetrrv.o psmatgen.o pmatgeninc.o pslaschk.o pslafchk.o -L/__release_lnx/tests/scalapack/../../lib/em64t /__release_lnx/tests/scalapack/../../lib/em64t/libmkl_scalapack_lp64.a /__release_lnx/tests/scalapack/../../lib/em64t/libmkl_blacs_intelmpi_lp64.a -L/__release_lnx/tests/scalapack/../../lib/em64t /__release_lnx/tests/scalapack/../../lib/em64t/libmkl_intel_lp64.a -Wl,--start-group /__release_lnx/tests/scalapack/../../lib/em64t/libmkl_sequential.a /__release_lnx/tests/scalapack/../../lib/em64t/libmkl_core.a -Wl,--end-group -lpthread

[alevseev]

I tried to compile this example but get many compilation errors. What did I do wrong?:

-sh-3.1$ mpif77 -nocompchk -c -w -u -O0 psludriver.f
-sh-3.1$ mpif77 -nocompchk -c -w -u -O0 psluinfo.f
-sh-3.1$ mpif77 -nocompchk -c -w -u -O0 psgetrrv.f
-sh-3.1$ mpif77 -nocompchk -c -w -u -O0 psmatgen.f
-sh-3.1$ mpif77 -nocompchk -c -w -u -O0 pmatgeninc.f
-sh-3.1$ mpif77 -nocompchk -c -w -u -O0 pslaschk.f
-sh-3.1$ mpif77 -nocompchk -c -w -u -O0 pslafchk.f
-sh-3.1$ mpif77 -nocompchk -o ../xslu_libem64t_intelmpi_intel_noopt_lp64 psludriver.o psluinfo.o psgetrrv.o psmatgen.o pmatgeninc.o pslaschk.o pslafchk.o -L/__release_lnx/tests/scalapack/../../lib/em64t /__release_lnx/tests/scalapack/../../lib/em64t/libmkl_scalapack_lp64.a /__release_lnx/tests/scalapack/../../lib/em64t/libmkl_blacs_intelmpi_lp64.a -L/__release_lnx/tests/scalapack/../../lib/em64t /__release_lnx/tests/scalapack/../../lib/em64t/libmkl_intel_lp64.a -Wl,--start-group /__release_lnx/tests/scalapack/../../lib/em64t/libmkl_sequential.a /__release_lnx/tests/scalapack/../../lib/em64t/libmkl_core.a -Wl,--end-group -lpthread

ifort: command line warning #10006: ignoring unknown option '-nocompchk'
/opt/intel/fce/10.1.021/lib/libimf.so: warning: warning: feupdateenv is not implemented and will always fail
psludriver.o: In function `MAIN__':
psludriver.f:(.text+0x31): undefined reference to `blacs_pinfo_'
psludriver.f:(.text+0x61d): undefined reference to `blacs_get_'
psludriver.f:(.text+0x649): undefined reference to `blacs_gridinit_'
psludriver.f:(.text+0x681): undefined reference to `blacs_gridinfo_'
psludriver.f:(.text+0x993): undefined reference to `igsum2d_'
psludriver.f:(.text+0xbf9): undefined reference to `igsum2d_'
psludriver.f:(.text+0xcc6): undefined reference to `numroc_'
psludriver.f:(.text+0xd19): undefined reference to `numroc_'
psludriver.f:(.text+0xd6c): undefined reference to `numroc_'
psludriver.f:(.text+0xeaa): undefined reference to `descinit_'
psludriver.f:(.text+0xf43): undefined reference to `igsum2d_'
psludriver.f:(.text+0x10a2): undefined reference to `iceil_'
psludriver.f:(.text+0x135d): undefined reference to `igsum2d_'
psludriver.f:(.text+0x15c1): undefined reference to `psfillpad_'
psludriver.f:(.text+0x164b): undefined reference to `psfillpad_'
psludriver.f:(.text+0x1704): undefined reference to `psfillpad_'
psludriver.f:(.text+0x17a8): undefined reference to `pslange_'
psludriver.f:(.text+0x1864): undefined reference to `pslange_'
psludriver.f:(.text+0x1926): undefined reference to `pschekpad_'
psludriver.f:(.text+0x19f4): undefined reference to `pschekpad_'
psludriver.f:(.text+0x1bef): undefined reference to `psfillpad_'
psludriver.f:(.text+0x1bfa): undefined reference to `slboot_'
psludriver.f:(.text+0x1c17): undefined reference to `blacs_barrier_'
psludriver.f:(.text+0x1c28): undefined reference to `sltimer_'
psludriver.f:(.text+0x1cbf): undefined reference to `psgetrf_'
psludriver.f:(.text+0x1cd4): undefined reference to `sltimer_'
psludriver.f:(.text+0x1e37): undefined reference to `pschekpad_'
psludriver.f:(.text+0x1ed6): undefined reference to `pschekpad_'
psludriver.f:(.text+0x212f): undefined reference to `pschekpad_'
psludriver.f:(.text+0x21ce): undefined reference to `pschekpad_'
psludriver.f:(.text+0x229c): undefined reference to `pschekpad_'
psludriver.f:(.text+0x245b): undefined reference to `slcombine_'
psludriver.f:(.text+0x24e3): undefined reference to `slcombine_'
psludriver.f:(.text+0x2cbf): undefined reference to `iceil_'
psludriver.f:(.text+0x2ce9): undefined reference to `iceil_'
psludriver.f:(.text+0x2e1f): undefined reference to `iceil_'
psludriver.f:(.text+0x2faa): undefined reference to `igsum2d_'
psludriver.f:(.text+0x30c7): undefined reference to `psfillpad_'
psludriver.f:(.text+0x3180): undefined reference to `psfillpad_'
psludriver.f:(.text+0x326f): undefined reference to `psgecon_'
psludriver.f:(.text+0x3324): undefined reference to `pschekpad_'
psludriver.f:(.text+0x33c3): undefined reference to `pschekpad_'
psludriver.f:(.text+0x3491): undefined reference to `pschekpad_'
psludriver.f:(.text+0x35b7): undefined reference to `descinit_'
psludriver.f:(.text+0x3650): undefined reference to `igsum2d_'
psludriver.f:(.text+0x3729): undefined reference to `numroc_'
psludriver.f:(.text+0x382b): undefined reference to `ilcm_'
psludriver.f:(.text+0x389f): undefined reference to `numroc_'
psludriver.f:(.text+0x38f8): undefined reference to `numroc_'
psludriver.f:(.text+0x3b4f): undefined reference to `igsum2d_'
psludriver.f:(.text+0x3db3): undefined reference to `psfillpad_'
psludriver.f:(.text+0x3f9a): undefined reference to `psfillpad_'
psludriver.f:(.text+0x402b): undefined reference to `psfillpad_'
psludriver.f:(.text+0x40bc): undefined reference to `psfillpad_'
psludriver.f:(.text+0x40dd): undefined reference to `blacs_barrier_'
psludriver.f:(.text+0x40ee): undefined reference to `sltimer_'
psludriver.f:(.text+0x41d9): undefined reference to `psgetrs_'
psludriver.f:(.text+0x41ee): undefined reference to `sltimer_'
psludriver.f:(.text+0x429f): undefined reference to `pschekpad_'
psludriver.f:(.text+0x433e): undefined reference to `pschekpad_'
psludriver.f:(.text+0x43e8): undefined reference to `pschekpad_'
psludriver.f:(.text+0x44a1): undefined reference to `psfillpad_'
psludriver.f:(.text+0x46c5): undefined reference to `pschekpad_'
psludriver.f:(.text+0x4793): undefined reference to `pschekpad_'
psludriver.f:(.text+0x4911): undefined reference to `iceil_'
psludriver.f:(.text+0x4a9c): undefined reference to `igsum2d_'
psludriver.f:(.text+0x4bb9): undefined reference to `psfillpad_'
psludriver.f:(.text+0x4c72): undefined reference to `psfillpad_'
psludriver.f:(.text+0x4e6a): undefined reference to `psgerfs_'
psludriver.f:(.text+0x4f22): undefined reference to `pschekpad_'
psludriver.f:(.text+0x4fcc): undefined reference to `pschekpad_'
psludriver.f:(.text+0x506b): undefined reference to `pschekpad_'
psludriver.f:(.text+0x5115): undefined reference to `pschekpad_'
psludriver.f:(.text+0x51bf): undefined reference to `pschekpad_'
psludriver.o:psludriver.f:(.text+0x5265): more undefined references to `pschekpad_' follow
psludriver.o: In function `MAIN__':
psludriver.f:(.text+0x5531): undefined reference to `psfillpad_'
psludriver.f:(.text+0x5755): undefined reference to `pschekpad_'
psludriver.f:(.text+0x5823): undefined reference to `pschekpad_'
psludriver.f:(.text+0x58ab): undefined reference to `slcombine_'
psludriver.f:(.text+0x5933): undefined reference to `slcombine_'
psludriver.f:(.text+0x62b8): undefined reference to `pschekpad_'
psludriver.f:(.text+0x6357): undefined reference to `pschekpad_'
psludriver.f:(.text+0x6425): undefined reference to `pschekpad_'
psludriver.f:(.text+0x64d6): undefined reference to `blacs_gridexit_'
psludriver.f:(.text+0x67f3): undefined reference to `blacs_exit_'
psluinfo.o: In function `psluinfo_':
psluinfo.f:(.text+0x115b): undefined reference to `blacs_setup_'
psluinfo.f:(.text+0x117d): undefined reference to `blacs_get_'
psluinfo.f:(.text+0x11af): undefined reference to `blacs_gridinit_'
psluinfo.f:(.text+0x11cc): undefined reference to `pslamch_'
psluinfo.f:(.text+0x125e): undefined reference to `sgebs2d_'
psluinfo.f:(.text+0x1359): undefined reference to `igebs2d_'
psluinfo.f:(.text+0x13b0): undefined reference to `icopy_'
psluinfo.f:(.text+0x1408): undefined reference to `icopy_'
psluinfo.f:(.text+0x1460): undefined reference to `icopy_'
psluinfo.f:(.text+0x14b8): undefined reference to `icopy_'
psluinfo.f:(.text+0x1510): undefined reference to `icopy_'
psluinfo.o:psluinfo.f:(.text+0x1568): more undefined references to `icopy_' follow
psluinfo.o: In function `psluinfo_':
psluinfo.f:(.text+0x1643): undefined reference to `igebs2d_'
psluinfo.f:(.text+0x3402): undefined reference to `blacs_setup_'
psluinfo.f:(.text+0x3424): undefined reference to `blacs_get_'
psluinfo.f:(.text+0x3456): undefined reference to `blacs_gridinit_'
psluinfo.f:(.text+0x3473): undefined reference to `pslamch_'
psluinfo.f:(.text+0x3517): undefined reference to `sgebr2d_'
psluinfo.f:(.text+0x35b0): undefined reference to `igebr2d_'
psluinfo.f:(.text+0x36e0): undefined reference to `igebr2d_'
psluinfo.f:(.text+0x3737): undefined reference to `icopy_'
psluinfo.f:(.text+0x3789): undefined reference to `icopy_'
psluinfo.f:(.text+0x37db): undefined reference to `icopy_'
psluinfo.f:(.text+0x382d): undefined reference to `icopy_'
psluinfo.f:(.text+0x387f): undefined reference to `icopy_'
psluinfo.o:psluinfo.f:(.text+0x38d1): more undefined references to `icopy_' follow
psluinfo.o: In function `psluinfo_':
psluinfo.f:(.text+0x3931): undefined reference to `blacs_gridexit_'
psluinfo.f:(.text+0x3a14): undefined reference to `blacs_abort_'
psgetrrv.o: In function `psgetrrv_':
psgetrrv.f:(.text+0x8f): undefined reference to `blacs_gridinfo_'
psgetrrv.f:(.text+0x106): undefined reference to `indxg2p_'
psgetrrv.f:(.text+0x16f): undefined reference to `numroc_'
psgetrrv.f:(.text+0x1da): undefined reference to `pb_topget_'
psgetrrv.f:(.text+0x225): undefined reference to `pb_topget_'
psgetrrv.f:(.text+0x270): undefined reference to `pb_topset_'
psgetrrv.f:(.text+0x2bb): undefined reference to `pb_topset_'
psgetrrv.f:(.text+0x44e): undefined reference to `iceil_'
psgetrrv.f:(.text+0x4ee): undefined reference to `indxg2p_'
psgetrrv.f:(.text+0x549): undefined reference to `indxg2p_'
psgetrrv.f:(.text+0x629): undefined reference to `descset_'
psgetrrv.f:(.text+0x6e5): undefined reference to `descset_'
psgetrrv.f:(.text+0x73c): undefined reference to `numroc_'
psgetrrv.f:(.text+0x801): undefined reference to `descset_'
psgetrrv.f:(.text+0x9a8): undefined reference to `pslacpy_'
psgetrrv.f:(.text+0xa63): undefined reference to `pslaset_'
psgetrrv.f:(.text+0xb39): undefined reference to `pslacpy_'
psgetrrv.f:(.text+0xbf9): undefined reference to `pslaset_'
psgetrrv.f:(.text+0xcb3): undefined reference to `pslaset_'
psgetrrv.f:(.text+0xd62): undefined reference to `pslaset_'
psgetrrv.f:(.text+0xec0): undefined reference to `psgemm_'
psgetrrv.f:(.text+0x10cd): undefined reference to `pslacpy_'
psgetrrv.f:(.text+0x1168): undefined reference to `pslaset_'
psgetrrv.f:(.text+0x121c): undefined reference to `pslacpy_'
psgetrrv.f:(.text+0x12c0): undefined reference to `pslaset_'
psgetrrv.f:(.text+0x136b): undefined reference to `pslaset_'
psgetrrv.f:(.text+0x13fe): undefined reference to `pslaset_'
psgetrrv.f:(.text+0x1522): undefined reference to `psgemm_'
psgetrrv.f:(.text+0x1634): undefined reference to `pslapiv_'
psgetrrv.f:(.text+0x167f): undefined reference to `pb_topset_'
psgetrrv.f:(.text+0x16ca): undefined reference to `pb_topset_'
psmatgen.o: In function `psmatgen_':
psmatgen.f:(.text+0x64): undefined reference to `numroc_'
psmatgen.f:(.text+0x9f): undefined reference to `numroc_'
psmatgen.f:(.text+0xca): undefined reference to `lsame_'
psmatgen.f:(.text+0xf5): undefined reference to `lsame_'
psmatgen.f:(.text+0x120): undefined reference to `lsame_'
psmatgen.f:(.text+0x152): undefined reference to `lsame_'
psmatgen.f:(.text+0x177): undefined reference to `lsame_'
psmatgen.f:(.text+0x398): undefined reference to `pxerbla_'
psmatgen.f:(.text+0x480): undefined reference to `iceil_'
psmatgen.f:(.text+0x4ad): undefined reference to `iceil_'
psmatgen.f:(.text+0x19bf): undefined reference to `lsame_'
psmatgen.f:(.text+0x2b25): undefined reference to `lsame_'
pslaschk.o: In function `pslaschk_':
pslaschk.f:(.text+0x86): undefined reference to `blacs_gridinfo_'
pslaschk.f:(.text+0xa3): undefined reference to `pslamch_'
pslaschk.f:(.text+0x178): undefined reference to `infog2l_'
pslaschk.f:(.text+0x208): undefined reference to `infog2l_'
pslaschk.f:(.text+0x2a1): undefined reference to `numroc_'
pslaschk.f:(.text+0x2fe): undefined reference to `numroc_'
pslaschk.f:(.text+0x54c): undefined reference to `pbstran_'
pslaschk.f:(.text+0x9a6): undefined reference to `sgemm_'
pslaschk.f:(.text+0xa5f): undefined reference to `slaset_'
pslaschk.f:(.text+0xaf7): undefined reference to `sgsum2d_'
pslaschk.f:(.text+0xb81): undefined reference to `isamax_'
pslaschk.f:(.text+0xc0b): undefined reference to `isamax_'
pslaschk.f:(.text+0xd6b): undefined reference to `sgamx2d_'
pslaschk.f:(.text+0xe80): undefined reference to `sgesd2d_'
pslaschk.f:(.text+0xf0d): undefined reference to `sgerv2d_'
pslaschk.f:(.text+0x101a): undefined reference to `sgebs2d_'
pslaschk.f:(.text+0x10b5): undefined reference to `sgebr2d_'
pslafchk.o: In function `pslafchk_':
pslafchk.f:(.text+0x86): undefined reference to `blacs_gridinfo_'
pslafchk.f:(.text+0xa3): undefined reference to `pslamch_'
pslafchk.f:(.text+0x13e): undefined reference to `infog2l_'
pslafchk.f:(.text+0x1da): undefined reference to `numroc_'
pslafchk.f:(.text+0x234): undefined reference to `numroc_'
pslafchk.f:(.text+0x286): undefined reference to `iceil_'
pslafchk.f:(.text+0x4d7): undefined reference to `smatadd_'
pslafchk.f:(.text+0x74a): undefined reference to `smatadd_'
pslafchk.f:(.text+0x7e9): undefined reference to `pslange_'

[Sergey_P_Intel2]

Please try to use mpiifort instead of mpif77.

- Sergey

[alevseev]

I don't know why, but there is no mpiifort available on our cluster.

[TimP]

Quoting - alevseev

I don't know why, but there is no mpiifort available on our cluster.

mpiifort is the wrapper which uses ifort in Intel MPI. Naturally, we would advocate you add Intel MPI to your cluster.
If you use an open source MPI, you should configure and rebuild it to use ifort as the f77 and f90 compiler. The new mpif77 you make this way should fix your linking problems. If you aren't using ifort, gfortran is more likely to be satisfactory than g77, as the latter has not been supported for several years, has different default linking conventions, and current MKL has libraries intended for gfortran.

[rdees]

Quoting - Sergey Pudov (Intel)

Let me clarify my previous answer because I think that some misunderstanding still exists. MKL contains different applications; some of them were developed for clusters and used MPI. So you CAN compile Intel MKL application to MPI executable.

But current PARDISO library doesn't have MPI support, so PARDISO examples can't be build to MPI executable without modifications.

For example you can use ScaLAPACK tests as basic for your MPI application and insert PARDISO calls into it, if necessary. Below you can see example of compiler options for ScaLAPACK tests:



mpiifort -nocompchk -c -w -u -O0 psludriver.f

mpiifort -nocompchk -c -w -u -O0 psluinfo.f

mpiifort -nocompchk -c -w -u -O0 psgetrrv.f

mpiifort -nocompchk -c -w -u -O0 psmatgen.f

mpiifort -nocompchk -c -w -u -O0 pmatgeninc.f

mpiifort -nocompchk -c -w -u -O0 pslaschk.f

mpiifort -nocompchk -c -w -u -O0 pslafchk.f

mpiifort -nocompchk -o ../xslu_libem64t_intelmpi_intel_noopt_lp64 psludriver.o psluinfo.o psgetrrv.o psmatgen.o pmatgeninc.o pslaschk.o pslafchk.o -L/__release_lnx/tests/scalapack/../../lib/em64t /__release_lnx/tests/scalapack/../../lib/em64t/libmkl_scalapack_lp64.a /__release_lnx/tests/scalapack/../../lib/em64t/libmkl_blacs_intelmpi_lp64.a -L/__release_lnx/tests/scalapack/../../lib/em64t /__release_lnx/tests/scalapack/../../lib/em64t/libmkl_intel_lp64.a -Wl,--start-group /__release_lnx/tests/scalapack/../../lib/em64t/libmkl_sequential.a /__release_lnx/tests/scalapack/../../lib/em64t/libmkl_core.a -Wl,--end-group -lpthread

Sergey,

I'm new to this forum, so, please bear with me. I've been searching all around, but, can't seem to find an answer.

I need to run PARDISO on myAMD Opteron x86_64 cluster, using Intel FORTRAN compiler,using Intel MPI. I've just successfully compiled and linked the PARDISO FORTRAN sample problem. But ... the executable is only for a single processor and a single thread. I need to use MPI in order to gain access to the aggregate memory of the cluster.

In the thread shown above you said two things of interest.

1) MPI support was not available for PARDISO. Is MPI support now available?
2) PARDISO MPI executable could only be built with modifications. What modifications would I use?

Oh, by the way, when I issue the command "cpuinfo", here is what I get:

Architecture : x86_64
Hyperthreading: not supported
Packages : 4
Cores : 8
Processors : 8
===== Processor identification =====
Processor Thread Core Package
0 0 0 0
1 0 1 0
2 0 0 1
3 0 1 1
4 0 0 2
5 0 1 2
6 0 0 3
7 0 1 3
===== Processor placement =====
Package Cores Processors
0 0,1 0,1
1 0,1 2,3
2 0,1 4,5
3 0,1 6,7
===== Cache sharing =====
Cache Size Processors
L1 0 B no sharing
L2 0 B no sharing
L3 0 B no sharing
=======================

Any suggestions?

Thank you very much,
Bob

[Gennady_F_Intel]

Is MPI support now available?
- not in the current version. but we are working on this problem and probably this functionality will available the next release.
--Gennady