Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Roberta_G_
Beginner
326 Views

MKL with MUMPS?

I'm trying to install MUMPS using MKL on my Linux cluster... I'm using MVAPICH-0.9.9

In the makefile for MUMPS,I need to specify the various BLACS and SCALAPACK libraries... which should I be using for MVAPICH-0.9.9? I only see options for Intel MPI, OpenMPI and SGI's MPT.

Thanks in advance for any help you can provide.
0 Kudos
12 Replies
Tabrez_Ali
Beginner
326 Views

Quoting - rgigon@slb.com
I'm trying to install MUMPS using MKL on my Linux cluster... I'm using MVAPICH-0.9.9

In the makefile for MUMPS,I need to specify the various BLACS and SCALAPACK libraries... which should I be using for MVAPICH-0.9.9?

Thanks in advance for any help you can provide.
What do you mean by "I only see options for Intel MPI, OpenMPI and SGI's MPT."

In the mumps makefile just put the path for scalapack, blacs, lapack, blas, metis etc. that are installed on your cluster and thats pretty much it.

You can test what works by using a trivial test program given at scalapacks home page. If that works with your mpi then it should work with mumps too.
Roberta_G_
Beginner
326 Views

This is the Makefile.inc file we ended up using to get MUMPS compiled with Intel Compilers, Intel MPI and MKL. Hope this helps!

#
# This file is part of MUMPS 4.9.2, built on Thu Nov 5 07:05:08 UTC 2009
#
#Begin orderings

# NOTE that PORD is distributed within MUMPS by default. If you would like to
# use other orderings, you need to obtain the corresponding package and modify
# the variables below accordingly.
# For example, to have Metis available within MUMPS:
# 1/ download Metis and compile it
# 2/ uncomment (suppress # in first column) lines
# starting with LMETISDIR, LMETIS
# 3/ add -Dmetis in line ORDERINGSF
# ORDERINGSF = -Dpord -Dmetis
# 4/ Compile and install MUMPS
# make clean; make (to clean up previous installation)
#
# Metis/ParMetis and SCOTCH/PT-SCOTCH (ver 5.1 and later) orderings are now available for MUMPS.
#

#SCOTCHDIR = ${HOME}/scotch_5.1_esmumps
#ISCOTCH = -I$(SCOTCHDIR)/include
# You have to choose one among the following two lines depending on
# the type of analysis you want to perform. If you want to perform only
# sequential analysis choose the first (remember to add -Dscotch in the ORDERINGSF
# variable below); for both parallel and sequential analysis choose the second
# line (remember to add -Dptscotch in the ORDERINGSF variable below)

#LSCOTCH = -L$(SCOTCHDIR)/lib -lesmumps -lscotch -lscotcherr
#LSCOTCH = -L$(SCOTCHDIR)/lib -lptesmumps -lptscotch -lptscotcherr


LPORDDIR = $(topdir)/PORD/lib/
IPORD = -I$(topdir)/PORD/include/
LPORD = -L$(LPORDDIR) -lpord

#LMETISDIR = /local/metis/
#IMETIS = # Metis doesn't need include files (Fortran interface avail.)

# You have to choose one among the following two lines depending on
# the type of analysis you want to perform. If you want to perform only
# sequential analysis choose the first (remember to add -Dmetis in the ORDERINGSF
# variable below); for both parallel and sequential analysis choose the second
# line (remember to add -Dparmetis in the ORDERINGSF variable below)

#LMETIS = -L$(LMETISDIR) -lmetis
#LMETIS = -L$(LMETISDIR) -lparmetis -lmetis

# The following variables will be used in the compilation process.
# Please note that -Dptscotch and -Dparmetis imply -Dscotch and -Dmetis respectively.
#ORDERINGSF = -Dscotch -Dmetis -Dpord -Dptscotch -Dparmetis
ORDERINGSF = -Dpord
ORDERINGSC = $(ORDERINGSF)

LORDERINGS = $(LMETIS) $(LPORD) $(LSCOTCH)
IORDERINGSF = $(ISCOTCH)
IORDERINGSC = $(IMETIS) $(IPORD) $(ISCOTCH)

#End orderings
########################################################################
################################################################################
MKLPATH = /opt/intel/Compiler/11.0/083/mkl/lib/em64t
PLAT =
RM = /bin/rm -f
CC = mpiicc
FC = ifort
FL = mpiifort
AR = ar vr
#RANLIB = ranlib
RANLIB = echo
SCALAP = -L$(MKLPATH) -lmkl_scalapack_lp64 $(MKLPATH)/libmkl_solver_lp64.a -Wl,--start-group -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_intelmpi_lp64 -Wl,--end-group -openmp -lpthread

INCPAR = -I/home/opt/intel/impi/4.0.0.025/include64

LIBPAR = $(SCALAP) -L/home/opt/intel/impi/4.0.0.025/lib64 -lmpi

INCSEQ = -I$(topdir)/libseq

LIBSEQ = -L$(topdir)/libseq -lmpiseq

LIBBLAS = -L$(MKLPATH) $(MKLPATH)/libmkl_solver_lp64.a -Wl,--start-group -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_intelmpi_lp64 -Wl,--end-group -openmp -lpthread

LIBOTHERS = -lpthread

#Preprocessor defs for calling Fortran from C (-DAdd_ or -DAdd__ or -DUPPER)
CDEFS = -DAdd_

#Begin Optimized options
OPTF = -O -Dintel_ -DALLOW_NON_INIT -nofor_main
OPTL = -O -nofor_main
OPTC = -O
#End Optimized options
INC = $(INCPAR)
LIB = $(LIBPAR)
LIBSEQNEEDED =

Best regards,
Roberta

JM9999
Beginner
326 Views

Thanks! It is very helpful. I am trying to install mumps with the MKL library. The "LIBBLAS" is not correct, as shown at the end.

ifort: command line warning #10006: ignoring unknown option '-/global/software/intel2012/composer_xe_2011_sp1.11.339/mkl/lib/intel64/libmkl_solver_lp64.a'

Any idea to fix it? The cluster has Xeon X5650 2.66GHz.

TimP
Black Belt
326 Views

Apparently, you didn't succeed in setting MKLPATH so that the Makefile could use it.

  Did you source the mklvars script first?

JM9999
Beginner
326 Views

Thank you, Tim. There is a typo in the path. It is running now!

JM9999
Beginner
326 Views

Problem on another cluster with Xeon X5650 2.66GHz processors with more processors. I tried different IntelMPI and MKL using module load. It seems that only MKL 10.2.6 works with Intel 11.1. But this combination does not finally fit my main program (don't know why). So I tried to use MKL and compilers from the same package of Intel14/composer_xe_2013_sp1.2.144 with IntelMPI 4.1.1.036/intel64. Makefile and output files are attached.

Anybody can help again? I had similar mistake for quite a while.

mpif90 -O  -DINTEL_ -DALLOW_NON_INIT -nofor_main -I/global/software/intel14/impi/4.1.1.036/intel64/include -I. -I/home/MyName/parallel/MUMPS/include -c dsimpletest.F -o dsimpletest.o
mpif90 -o dsimpletest -O -nofor_main dsimpletest.o  /home/MyName/parallel/MUMPS/lib/libdmumps.a /home/MyName/parallel/MUMPS/lib/libmumps_common.a -L/home/MyName/parallel/parmetis -lparmetis -lmetis -L/home/MyName/parallel/MUMPS/PORD/lib/ -lpord  -L/global/software/intel14/mkl/lib/intel64   /global/software/intel14/mkl/lib/intel64/libmkl_blas95_lp64.a /global/software/intel14/mkl/lib/intel64/libmkl_lapack95_lp64.a /global/software/intel14/mkl/lib/intel64/libmkl_scalapack_lp64.a -Wl,--start-group /global/software/intel14/mkl/lib/intel64/libmkl_cdft_core.a /global/software/intel14/mkl/lib/intel64/libmkl_intel_lp64.a /global/software/intel14/mkl/lib/intel64/libmkl_core.a /global/software/intel14/mkl/lib/intel64/libmkl_intel_thread.a -Wl,--end-group /global/software/intel14/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.a -lpthread -lm -openmp -I/global/software/intel14/mkl/include/intel64/lp64 -I/global/software/intel14/mkl/include -L/global/software/intel14/impi/4.1.1.036/intel64/lib -lmpi  -L/global/software/intel14/mkl/lib/intel64  /global/software/intel14/mkl/lib/intel64/libmkl_blas95_lp64.a /global/software/intel14/mkl/lib/intel64/libmkl_lapack95_lp64.a -Wl,--start-group /global/software/intel14/mkl/lib/intel64/libmkl_intel_lp64.a /global/software/intel14/mkl/lib/intel64/libmkl_core.a /global/software/intel14/mkl/lib/intel64/libmkl_intel_thread.a -Wl,--end-group -lpthread -lm -openmp -I/global/software/intel14/mkl/include/intel64/lp64 -I/global/software/intel14/mkl/include -lpthread
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
make[1]: *** [dsimpletest] Error 1
make[1]: Leaving directory `/g01/home/MyName/parallel/MUMPS/examples'
make: *** [dexamples] Error 2

TimP
Black Belt
326 Views

On the face of it, your collection of .o files doesn't include a successful compilation of a C main() (which your nofor_main option tells the linker script to expect).

Your gcc libraries are rather old to be using with up to date MKL.  I don't know whether that's a problem. I suppose Red Hat 5.5 should still be fully supported on Westmere CPUs.

JM9999
Beginner
326 Views

Thank you, Tim.

Does Intel compiler rely on default gcc? It does not seem that I can change gcc after I load Intel compiler and MKL. The Red Hat should have no problem, as it is being used on several clusters here. And it works with an older MKL (with Intel 11). 

I attempted to load gcc-4.4.4 and set MPI /MKl path to Intel's directory. It does not work at all. 

TimP
Black Belt
326 Views

Intel tools which support Red Hat 6.1 should have no difficulty with gcc 4.4 being on PATH and LD_LIBRARY_PATH.  

I don't know whether the configuration of mvapich or your application might be affected by so large a change in gcc version.

JM9999
Beginner
326 Views

Thank you very much, Tim.

I don't know how to set LD_LIBRARY_PATH for gcc 4.4 after I load Intel14 and MKL. It takes me lots of time to make small progress, as I am doing that in trial-and-error. Just set something in Makefile like: 

LD_LIBRARY_PATH=usr/lib/gcc/x86_64-redhat-linux6E/4.4.6 ?

I know this one does not work. I have to try something else. Would you please give more instruction?

 

TimP
Black Belt
326 Views

Setting up an Intel compiler by sourcing the compilervars script adds required entries to LD_LIBRARY_PATH.  In order to change your gcc paths afterwards, you would add the new gcc paths to the beginning of PATH and LD_LIBRARY_PATH.

I find it a little difficult to believe that anyone would run a 32-bit cluster OS as I haven't seen one for 8 years.  I don't know the installation path of your gcc 4.4, so I'll give as an example supposing that gcc 64-bit were installed with prefix set as /usr/local/gcc4.4:

export PATH=/usr/local/gcc4.4/bin:$PATH

export LD_LIBRARY_PATH=/usr/local/gcc4.4/lib64:$LD_LIBRARY_PATH

I'd reiterate that the main problem is that your link line expects your application to use a C main().  If there is a Fortran main program (normally, one headed up by PROGRAM .....) you would remove the nofor_main options.  If it is actually a C main(), you would check that this compiled successfully to a .o file included in your list.

 

JM9999
Beginner
326 Views

Maybe I asked too much!

 

Reply