parallel mpi program linked with mkl libraries

[combe]

Dear all,

I am trying to compile the code SIESTA(a scientific code : www.uam.es/siesta) on a SGI altix 3200 clusters with the intel fortran compiler, INTEL MKL (10.0.2.018) and SGI MPI libraries.

The SIESTA code uses BLAS, LAPACK (or scalapack) libraries.
I succeed in compiling the sequential version of the code, but I would like now to compile a parallel version.


However, I would like to use the SGI MPI libraries instead of the INTEL MPI libraries (because the SGI MPI is precisely adapted to our system) but also the MKL libraries (because they are fast!)


So I would like to use the scalapack and blacs MKL intel libraries with the SGI MPI libraries.
I have tried to use the following libraries :
-L/usr/lib64 -lmpi -lpthread (SGI MPI libraries)
-L/opt/intel/mkl/10.0.2.018/lib/em64t -lmkl -lmkl_scalapack_lp64 -lmkl_blacs_lp64

but I miss something since I get error messages like :
/opt/intel/mkl/10.0.2.018/lib/em64t/libmkl_intel_thread.so: undefined reference to `omp_in_parallel'
/opt/intel/mkl/10.0.2.018/lib/em64t/libmkl_intel_thread.so: undefined reference to `__kmpc_end_critical'

So my questions are :
1) Can I use scalapack and blacs MKL with the SGI MPI libraries ?
2) which library is missing for my compilation ?
3) How can I know the dependances between libraries (for instance here libmkl_intel_thread.so depends on another intel libraries, how can I know which one ?) , how can I have access to the list of functions contains in one library ?


Thank you very much for your help

Nicolas

[TimP]

MKL scalapack has built-in dependency on Intel MPI, which cannot be mixed with SGI MPT, as far as I know. There are cases, such as mixing Intel MPI with MPICH2, where it may be possible to substitute one MPI run-time for the other.
If you avoid Intel MPI calls, you should have no trouble using MKL with MPT. The messages you quote don't show an MPI dependence; they appear to indicate that you didn't link in an Intel OpenMP library. I thought it was fairly well documented that libmkl_intel_thread depends on libguide or libiomp5 and lpthread (which you already linked).
It looks as if you haven't decided whether you want a sort of hybrid OpenMP/MPI created by using libmkl_intel_thread inside MPI, or whether you wanted mkl sequential, which would remove the dependency on libguide or libiomp5. The hybrid raises the problem of using the affinity tools to allocate a suitable group of cores to each MPI process. MPT may be "precisely adapted" to your system, but it has no way of guessing what you are doing behind its back.
Comparison of Intel MPI to MPT performance could be a research project in itself. If your OS installation is up to date, Intel MPI and MKL scalapack should run well.

[TimP]

I understand thatboth Intel MPI and MPT are supported by BLACS libraries, while scalapack has no MPI dependencies. As noted above, the reported errors are due to specifying threaded support, but not including the OpenMP library. libmkl_blacs_sgimpt_lp64.a issupplied to support MPT.

[TimP]

reminder: any libraries for Itanium would be found in /64/ directories

[combe]

Thank very much Tim,
you avoid me severals weeks of works !

There is still something I don't understand (even reading carefully the MKL user guide).

There are /lib/64/ librairies for Itanium architecture, some /lib/em64t/ librairies for intel 64 architecture.
We actually have intel Xeon processor in our cluster.
However, inside the /lib/em64t/ directory I found some libmkl_xx_lp64 for intel 64 architecture but also some libmkl_xx_ilp64 for iitanium architecture.. and the same in the /lib/64 directory.
The MKL user guide refers to the concept of "interface" I have not understand.

For my precise problem, I would need the libmkl_blacs_sgimpt_lp64.a librairy I can found in /lib/64.
How should I understand it : the lib is in the itanium directory, but it 's name refer to intel 64 archecture.
I have not get the aim of all these denominations.
Can I use /lib/64/libmkl_blacs_sgimpt_lp64.a with other of /lib/em64t/ ?
Or should I now use all the libmlk_xxx_lp64 of /lib/64 ?
I am confused !

Thank you very much in advance.
Nicolas

[combe]

Well, I have ridden again the MKL user guide... more carefully,much more carefully than the first time!
I now understand the difference between libmkl_xxx_lp64 and libmlk_xxx_ilp64 (nothing to do with the architecture !...). Thus I conclude my last message was stupid! I apologize.

So would it be possible to have a libmkl_blacs_sgimpt_lp64.a for em64t architecture ?

Thank a lot

N.

[TimP]

You're correct, if your machine is Xeon, the /64/ Itanium libraries will not work, you must use /em64t/. All the references I could find indicated that Altix 3200 meant Itanium, like Altix 3000. Guess I didn't find reliable references. Can't ever believe marketing.

The distinction between ilp64 and lp64, as you may have figured out, is in the data type of the integer arguments; ilp64 means 64-bit integers, lp64 means 32-bit integers (omission of the i meaning default sized integers, definitely somewhat obscure).

If you want an MPT library which is not provided for Xeon/em64t, a request against the product on your premier.intel.com account would be appropriate. I was chastised roundly for suggesting there might be a certainlibrary available for Intel MPI but not MPT.

[Todd_R_Intel]

Nicolas,

A version of the BLACS compiled against SGI MPT on systems with Intel EM64T has already been requested and is targeted for a release coming this summer.

Todd

[magdian]

Quoting - combe

Dear all,

I am trying to compile the code SIESTA(a scientific code : www.uam.es/siesta) on a SGI altix 3200 clusters with the intel fortran compiler, INTEL MKL (10.0.2.018) and SGI MPI libraries.

The SIESTA code uses BLAS, LAPACK (or scalapack) libraries.
I succeed in compiling the sequential version of the code, but I would like now to compile a parallel version.


However, I would like to use the SGI MPI libraries instead of the INTEL MPI libraries (because the SGI MPI is precisely adapted to our system) but also the MKL libraries (because they are fast!)


So I would like to use the scalapack and blacs MKL intel libraries with the SGI MPI libraries.
I have tried to use the following libraries :
-L/usr/lib64 -lmpi -lpthread (SGI MPI libraries)
-L/opt/intel/mkl/10.0.2.018/lib/em64t -lmkl -lmkl_scalapack_lp64 -lmkl_blacs_lp64

but I miss something since I get error messages like :
/opt/intel/mkl/10.0.2.018/lib/em64t/libmkl_intel_thread.so: undefined reference to `omp_in_parallel'
/opt/intel/mkl/10.0.2.018/lib/em64t/libmkl_intel_thread.so: undefined reference to `__kmpc_end_critical'

So my questions are :
1) Can I use scalapack and blacs MKL with the SGI MPI libraries ?
2) which library is missing for my compilation ?
3) How can I know the dependances between libraries (for instance here libmkl_intel_thread.so depends on another intel libraries, how can I know which one ?) , how can I have access to the list of functions contains in one library ?


Thank you very much for your help

Nicolas

Hi combe

I'm trying to install parallel SIESTA in a Intel Xeon with openmpi-1.2.7 and mkl 10.0.2.018, and I'm still have problems with mkl libraries, I get the following errors

/packages/intel/mkl/10.0.2.018/lib/em64t/libmkl_intel_thread.a(zunmqr_omp.o): In function `L_mkl_lapack_zunmqr_314__par_region0':

/packages/intel/mkl/10.0.2.018/lib/em64t/libmkl_scalapack_ilp64.a(pbdvecadd.o): In function `pbdvecadd_':

I wonder if can tell me what is wrong with my arch.make


#
SIESTA_ARCH=intel9-mkl8
#

#
FC=mpif90
#
FFLAGS= -w -O2 -mp
FFLAGS_DEBUG= -g
LDFLAGS=
COMP_LIBS=
RANLIB=echo
#
NETCDF_LIBS=
NETCDF_INTERFACE=
DEFS_CDF=
#
MPI_INTERFACE=libmpi_f90.a
MPI_INCLUDE=/packages/intel-openmpi-1.2.7/include
DEFS_MPI=-DMPI
#
COMP=-L/packages/intel/mkl/10.0.2.018/lib/em64t
LAPACK=/packages/intel/mkl/10.0.2.018/lib/em64t/libmkl_lapack.a
BLAS=/packages/intel/mkl/10.0.2.018/lib/em64t/libmkl_em64t.a
BLACS=/packages/intel/mkl/10.0.2.018/lib/em64t/libmkl_blacs_openmpi_ilp64.a
SCALAPACK=/packages/intel/mkl/10.0.2.018/lib/em64t/libmkl_scalapack_ilp64.a
THREAD=/packages/intel/mkl/10.0.2.018/lib/em64t/libmkl_intel_thread.a
VML=/packages/intel/mkl/10.0.2.018/em64t/libmkl_vml_p4n.so
LIBS=$(LAPACK) $(COMP) $(BLAS) $(BLACS) $(THREAD) $(VML)
SYS=nag
DEFS= $(DEFS_CDF) $(DEFS_MPI)
#
.F.o:
$(FC) -c $(FFLAGS) $(INCFLAGS) $(DEFS) $<
.f.o:
$(FC) -c $(FFLAGS) $(INCFLAGS) $<
.F90.o:
$(FC) -c $(FFLAGS) $(INCFLAGS) $(DEFS) $<
.f90.o:
$(FC) -c $(FFLAGS) $(INCFLAGS) $<
#