Can't get pardiso to multithread (MKL linking issue?)

[Viktor_J_]

Hello, I'm currently trying to get pardiso to work with multi threading and I'm wondering if it is a linking issue or something else. I have tried some "easy" fixes that didn't work, then I tried the link advisor and get an error when linking.

Question: How do I get pardiso to work with multiple cores?

Background:

When calling pardiso I use the following iparm

  iparm= 0
  iparm(1) = 1! !0=solver default
  iparm(2) = 2 !  !2
  iparm(3) = 0 !reserved, set to zero
  iparm(4) = 0 ! Preconditioned CGS/CG.
  iparm(5) = 0 !
  iparm(6) = 0 !
  iparm(7) = 0 !
  iparm(8) = 9 ! Iterative refinement step.
  iparm(9) = 0 ! reserved, set to zero
  iparm(10) = 13
  iparm(11) = 1
  iparm(12) = 0
  iparm(13) = 0
  iparm(14) = 0
  iparm(15) = 0
  iparm(16) = 0
  iparm(17) = 0
  iparm(18) = -1
  iparm(19) = -1
  iparm(20) = 0
 

The problem is "large" with

             #equations:                                     71574
             #non-zeros in A:                                3815460
             non-zeros in A (%):                            0.074479

The simplest fix

  call mkl_set_dynamic(0)  ! disable adjustment of the number of threads
  call mkl_set_num_threads(4)
  call omp_set_num_threads(4)

This is done right before I call pardiso and has no effect on the number of cores used, the next step was to check the MKL linking, I then tried to use the link advisor, https://software.intel.com/en-us/articles/intel-mkl-link-line-advisor/ but I get and error.

With the advisor,I choose the "Composer XE 2011" in the link advisor since that is where my MKL roots are in the makefile(please correct me if this is a wrong assumption),

MKLroot=/software/intel/parallel_studio/composer_xe_2011_sp1.7.256/mkl/lib/intel64

My OS is then Linux, the compiler is iFort/intel fortran, linking is dynamic(unsure of this, trying static yields same result/error), the interface is LP64.

For threading layer Im uncertain of which option to select, sequential or OpenMP. In the makefile there is "sequential" in LLIBS, but there are sections of the code that are already multithreading with OpenMP, only do-loops and no explicit MKL routines.

LLIBS =  -L/software/matlab/R2011a/bin/glnxa64 -leng -lmat -lmx -lut -Wl,--start-group ${MKLroot}/libmkl_intel_lp64.a ${MKLroot}/libmkl_sequential.a ${MKLroot}/libmkl_core.a -Wl,--end-group -lpthread 

Lastly I'm advised to do the following linking and add to my compiler

 -lpthread -lm
 -openmp -mkl=parallel 

Adding the compiler options work but when linking I get the following message/error

$ ifort -lpthread -lmifort: warning #10315: specifying -lm before files may supercede the Intel(R) math library and affect performance
/software/intel/parallel_studio/composer_xe_2011_sp1.7.256/compiler/lib/intel64/for_main.o: In function `main':
/export/users/nbtester/efi2linux_nightly/branch-12_1/20111012_000000/libdev/frtl/src/libfor/for_main.c:(.text+0x38): undefined reference to `MAIN__'

I assume this means that the linking is incomplete, how can I complete it? And hopefully get pardiso to work on more then 1 core

--

Sorry if this post is messy, this is my first time attempting to link anything. If any clarification is needed please ask, any help is much appreciated

/Viktor

[Ying_H_Intel]

Hi Viktor, 

It seems you have complex environment including Matlab and but there are sections of the code that are already multithreading with OpenMP, only do-loops. 

Let's simplify the problem first. 

Do you have the input data ready? I may suggest you to modify the MKL pardiso sample code pardiso_sym_f.f or  pardiso_unsym_f.f  , which in

/software/intel/parallel_studio/composer_xe_2011_sp1.7.256/

/mkl/examples/solver/source

1) copy the solver folder to one folder you can write

2) modify pardiso_unsym_f.f (if you are working on fortran ) 

3) then build it , ite

make libem64 function=pardiso_unsym_f 

You will see the exact link option and exe. 

4) run the exe to see if the multiple core was used.

5) Then reconsider your real makefile, also please use the same order, for example,  -lphread, -lm  is supposed at the end of link line etc. 

Best Regards,

Ying 

[Viktor_J_]

Hello and thank you for your answer,

I don't have ready input data but I could copy the sample code and make the example work on multiple cores.
I then looked at the sample makefile and copied the links and flags the parts that were missing from my own makefile,

CF90=ifort
F77FLAGS= -O3    
F90FLAGS = ${F77FLAGS} -I${SOLIDroot} -openmp -m64 -w -fpp -I$(MKLROOT)/../.. $(OMP_LIB) -lpthread -lm -L"/software/intel/parallel_studio/composer_xe_2011_sp1.7.256/mkl/../compiler/lib/intel64" -liomp5 -lmkl_core 
SOLIDroot=../Modules

OMP_LIB=-L"/software/intel/parallel_studio/composer_xe_2011_sp1.7.256/mkl/../compiler/lib/intel64" -liomp5 
MKLroot=/software/intel/parallel_studio/composer_xe_2011_sp1.7.256/mkl/lib/intel64
LLIBS =  -L/software/matlab/R2011a/bin/glnxa64 -leng -lmat -lmx -lut -Wl,--start-group ${MKLroot}/libmkl_intel_lp64.a ${MKLroot}/libmkl_sequential.a ${MKLroot}/libmkl_core.a -Wl,--end-group -lpthread  -Wl,--start-group ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a ${MKLROOT}/lib/intel64/libmkl_core.a ${MKLROOT}/lib/intel64/libmkl_intel_thread.a -Wl,--end-group -lpthread -lm   
LD_LIBRARY_PATH=/software/matlab/R2011a/bin/glnxa64

I can compile the code but I get the following error:

./run_tl_J2iso.x: symbol lookup error: /software/intel/parallel_studio/composer_xe_2011_sp1.7.256/mkl/lib/intel64/libmkl_avx.so: undefined symbol: mkl_serv_check_ptr

I never had this error before, so it must be from the additional linking, tried to eliminate different parts of the linking and
 found that the problem is caused by:

-L"/software/intel/parallel_studio/composer_xe_2011_sp1.7.256/mkl/../compiler/lib/intel64" -liomp5 -lmkl_core

Is the linking done correctly? I quick search showed that this error can occur when different versions of openmp are used by fortran and matlab(and I use matlab in my program) Previously the compiler looked like this

CF90=ifort
F77FLAGS= -O3  
F90FLAGS = ${F77FLAGS} -I${SOLIDroot} -openmp  #linking done here
SOLIDroot=../Modules


#extra path here
MKLroot=/software/intel/parallel_studio/composer_xe_2011_sp1.7.256/mkl/lib/intel64
LLIBS =  -L/software/matlab/R2011a/bin/glnxa64 -leng -lmat -lmx -lut -Wl,--start-group ${MKLroot}/libmkl_intel_lp64.a ${MKLroot}/libmkl_sequential.a ${MKLroot}/libmkl_core.a -Wl,--end-group -lpthread 
LD_LIBRARY_PATH=/software/matlab/R2011a/bin/glnxa64

If any clarification is needed please just ask and any help is much appreciated 

/Viktor

[Ying_H_Intel · Accepted solution]

Hi Viktor, 

I can't try your command line, so just comment

It seems the line F90FLAGS is not expected. please use your original one. or 

remove  "$(OMP_LIB) -lpthread -lm -L"/software/intel/parallel_studio/composer_xe_2011_sp1.7.256/mkl/../compiler/lib/intel64" -liomp5 -lmkl_core  "

they are not needed.  

The line 

LLIBS =  -L/software/matlab/R2011a/bin/glnxa64 -leng -lmat -lmx -lut -Wl,--start-group ${MKLroot}/libmkl_intel_lp64.a ${MKLroot}/libmkl_sequential.a ${MKLroot}/libmkl_core.a -Wl,--end-group -lpthread  -Wl,--start-group ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a ${MKLROOT}/lib/intel64/libmkl_core.a ${MKLROOT}/lib/intel64/libmkl_intel_thread.a -Wl,--end-group -lpthread -lm

Please remove the below part. (it used sequential mkl, will cause mkl run in 1 thread. )

-Wl,--start-group ${MKLroot}/libmkl_intel_lp64.a ${MKLroot}/libmkl_sequential.a ${MKLroot}/libmkl_core.a -Wl,--end-group -lpthread. 

the thread MKL should be enough, the correct line is like

-L/software/matlab/R2011a/bin/glnxa64 -leng -lmat -lmx -lut -Wl,--start-group ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a ${MKLROOT}/lib/intel64/libmkl_intel_thread.a ${MKLROOT}/lib/intel64/libmkl_core.a  -Wl,--end-group -liomp5 -lpthread -lm

Best Regards,

Ying

[Viktor_J_]

Hello, removing that line work.

Thank you so much for helping and and giving advice.

With sincere thanks,

Viktor