Pardiso multi processor

eh4 · ‎11-16-2011

Hi all,
I have been reading many posts here about trying to let PARDISO run in more than processor. However, I still cant get a proper idea on how this can be done.

What I have done is:

call omp_set_num_threads(4).

iparm(1) = 1 ! no solver default

iparm(2) = 3 ! fill-in reordering from METIS

iparm(3) = 4 ! numbers of processors, value of MKL_NUM_THREADS

iparm(4) = 0 ! no iterative-direct algorithm

iparm(5) = 0 ! no user fill-in reducing permutation

iparm(6) = 0 ! =0 solution on the first n components of x

iparm(7) = 0 ! not in use

iparm(8) = 20 ! numbers of iterative refinement steps

iparm(9) = 0 ! not in use

iparm(10) = 13 ! perturb the pivot elements with 1E-13

iparm(11) = 1 ! use nonsymmetric permutation and scaling MPS

iparm(12) = 0 ! not in use

iparm(13) = 1 ! not in use

iparm(14) = 0 ! Output: number of perturbed pivots

iparm(15) = 0 ! not in use

iparm(16) = 0 ! not in use

iparm(17) = 0 ! not in use

iparm(18) = -1 ! Output: number of nonzeros in the factor LU

iparm(19) = -1 ! Output: Mflops for LU factorization

iparm(20) = 0 ! Output: Numbers of CG Iterations

iparm(27) = 1 !Matrix checker on

iparm(60) = 1

During the run, it always say that the number of procesors use is 1.

(Direct factorization using #processors) 1

Does this mean that only one processor is running?

The machine that I am using is Intel Xeon W55803.2GHz. There are 4 physical processors. The funny thing is when i check the task manager, all 4 processors are running to 100% during factorization.

Thank you.

EH

Alexander_K_Intel2 · ‎11-16-2011

Hi,

Could you provide size of your matrix?

With best regards,

Alexander Kalinkin

eh4 · ‎11-16-2011

Hi Alex,
There are 113762 equations. The matrix is sparse and the number of nonzeros are 11741784

The PARDISO is running in the in-core mode

Do I need to specify anything for mkl_num_threads and omp_num_threads?

Thank you.

Best wishes,
EH

Alexander_K_Intel2 · ‎11-16-2011

Size of matrix is pretty big so PARDISO could run in multithread mode... Could you provide us linkline of your project?

With best regards,

Alexander Kalinkin

eh4 · ‎11-16-2011

Hi Alex,
Sorry because I am not very familiar with the programming jargons. I have here the entire subroutine where pardiso is called. Would this help? If not, could you tell me what linkline of a project is?

Thank you and sorry for the inconvenience.

!------------------------------------------------------------------------------------!

SUBROUTINE PARDISO_SOLVER(Nc,n,A,ia,ja,Bin,Z)

! SUBROUTINE PARDISO_SOLVER(Nc,n,Ain,iin,jin,Bin,Z)

! THIS SUBROUTINE CALLS THE SOLVER PARADISO TO SOLVE THE SYSTEM...

! OF LINEAR ALGEBRAIC EQUATIONS

!------------------------------------------------------------------------------------!

IMPLICIT NONE

! Internal solver memory pointer for 64-bit architectures

! INTEGER*8 pt(64)

! Internal solver memory pointer for 32-bit architectures

! INTEGER*4 pt(64)

! This is OK in both cases

! DOUBLE PRECISION, ALLOCATABLE :: A(:)

! INTEGER, ALLOCATABLE :: ia(:), ja(:)

INTEGER*4 pt(64)

INTEGER maxfct, mnum, mtype, phase, n, nrhs, error, msglvl, Nc, i, idum

INTEGER iparm(64)

INTEGER ja(Nc), ia(n+1)

DOUBLE PRECISION A(Nc), Bin(n), Z(n)

DOUBLE PRECISION waltime1, waltime2, ddum

INTEGER mkl_get_max_threads

INTEGER MKL_NUM_THREADS,OMP_NUM_THREADS

EXTERNAL mkl_get_max_threads

EXTERNAL omp_num_threads

MKL_NUM_THREADS=2

call OMP_SET_NUM_THREADS(4)

! Set up PARDISO control parameter

DO i=1,64

iparm(i) = 0

END DO

iparm(1) = 1 ! no solver default

iparm(2) = 3 ! fill-in reordering from METIS

iparm(3) = 4 ! numbers of processors, value of MKL_NUM_THREADS

iparm(4) = 0 ! no iterative-direct algorithm

iparm(5) = 0 ! no user fill-in reducing permutation

iparm(6) = 0 ! =0 solution on the first n components of x

iparm(7) = 0 ! not in use

iparm(8) = 20 ! numbers of iterative refinement steps

iparm(9) = 0 ! not in use

iparm(10) = 13 ! perturb the pivot elements with 1E-13

iparm(11) = 1 ! use nonsymmetric permutation and scaling MPS

iparm(12) = 0 ! not in use

iparm(13) = 1 ! not in use

iparm(14) = 0 ! Output: number of perturbed pivots

iparm(15) = 0 ! not in use

iparm(16) = 0 ! not in use

iparm(17) = 0 ! not in use

iparm(18) = -1 ! Output: number of nonzeros in the factor LU

iparm(19) = -1 ! Output: Mflops for LU factorization

iparm(20) = 0 ! Output: Numbers of CG Iterations

iparm(27) = 1 !Matrix checker on

iparm(60) = 1

error = 0 ! initialize error flag

msglvl = 1 ! print statistical information

mtype = 11 ! real unsymmetric

maxfct=1

mnum=1

nrhs=1

! Initialize the internal solver memory pointer. This is only...

! necessary for the FIRST call of the PARDISO solver.

DO i=1,64

pt(i) = 0

END DO

! Reordering and Symbolic Factorization, This step also allocates...

! all memory that is necessary for the factorization

phase = 11 ! only reordering and symbolic factorization

CALL pardiso(pt,maxfct,mnum,mtype,phase,n,A,ia,ja,idum,nrhs,iparm,msglvl,ddum,ddum,error)

WRITE(*,*) 'Reordering completed ... '

IF (error .NE. 0) THEN

WRITE(*,*) 'The following ERROR was detected: ', error

PAUSE

STOP

END IF

WRITE(*,*) 'Number of nonzeros in factors = ',iparm(18)

WRITE(*,*) 'Number of factorization MFLOPS = ',iparm(19)

! Factorization.

phase = 22 ! only factorization

CALL pardiso(pt,maxfct,mnum,mtype,phase,n,A,ia,ja,idum,nrhs,iparm,msglvl,ddum,ddum,error)

WRITE(*,*) 'Factorization completed ... '

IF (error .NE. 0) THEN

WRITE(*,*) 'The following ERROR was detected: ', error

PAUSE

STOP

ENDIF

! Back substitution and iterative refinement

iparm(8) = 20 ! max numbers of iterative refinement steps

phase = 33 ! only factorization

CALL pardiso(pt,maxfct,mnum,mtype,phase,n,A,ia,ja,idum,nrhs,iparm,msglvl,Bin,Z,error)

WRITE(*,*) 'Solve completed ... '

! Termination and release of memory

phase = -1 ! release internal memory

CALL pardiso(pt,maxfct,mnum,mtype,phase,n,ddum,idum,idum,idum,nrhs,iparm,msglvl,ddum,ddum,error)

! DEALLOCATE(A,ia,ja)

RETURN

END

Thank you.

Best wishes,
EH

Alexander_K_Intel2 · ‎11-16-2011

Ok. If you run your program from visual studio you put several mkl library names in field names "additional libraries" or something like it... If you run you example from command line you call some command to compile and run it. Could you provide names of mkl library in first case or command in second case?

Also, could you print result ofmkl_get_max_threads() before first call of PARDISO?

With best regards,

Alexander Kalinkin

eh4 · ‎11-16-2011

Hi,
I did not put in any mkl library names. Instead, under Properties -> Fortran -> Libraries:Include Math Kernel Library option, I chose the Parallel (/Qmkl:parallel) options. Is this not the correct way of doing it?

Regarding the mkl_get_max_threads(), the printed value was 4. However the Parallel Direct Factorization with #provessors still say 1.

Thanks.

EH

Alexander_K_Intel2 · ‎11-16-2011

Hi,

One additional question, I hope the last one: which version of stufio do you use?

With best regards,

Alexander Kalinkin

eh4 · ‎11-16-2011

I am using Microsoft Visual Studio 2008, Version 9.0.21022.8 RTM. The installed product is Intel Visual Fortran Compiler Integration.

Konstantin_A_Intel · ‎11-16-2011

Hi, could you please go to

Project -> Your Project Properties... -> Configuration Properties -> Linker -> Command line

and send us the content of this text field?

Regards,
Konstantin

Gennady_F_Intel · ‎11-16-2011

actually, if customer set the option(/Qmkl:parallel ), then it would be enough to link all threading runtime.

but, for eliminations all concerns, You can explicitly toset all requiredlibraries for your application and check how it would works on your side.

as an additional info, it might be useful for you to see this KB article shows how configure Intel MKL in Microsoft* Visual Studio.

eh4 · ‎11-16-2011

Hi,
The content in the Command line reads:

/OUT:"x64\Debug\Test1.exe" /INCREMENTAL:NO /NOLOGO /LIBPATH:"C:\Program Files\Intel\Compiler\11.1\048\mkl\ia64\lib" /MANIFEST /MANIFESTFILE:"D:\USERS\EHOOI\Fortran\3D\Natural_Convection\Test3-steady\x64\Debug\Test1.exe.intermediate.manifest" /DEBUG /PDB:"D:\USERS\EHOOI\Fortran\3D\Natural_Convection\Test3-steady\x64\Debug\Test1.pdb" /SUBSYSTEM:CONSOLE /IMPLIB:"D:\USERS\EHOOI\Fortran\3D\Natural_Convection\Test3-steady\x64\Debug\Test1.lib"

Thank you.

EH

eh4 · ‎11-16-2011

Hi Konst
Thank you. I will check it out. By the way, if all four CPU in the Task Manager show 100% computation during the running of PARDISO (only fortran is running), then is this an indicator that it is using 4 processors in parallel although in the Command Window it says 1 processor in the Direct Factorization?

Thank you.

EH

Konstantin_A_Intel · ‎11-16-2011

Hi,

Hmm, it's strange that you useIntel Xeon W5580, but it seems like you are linking with IA-64 architecture (Itanium-2):
/OUT:"x64\Debug\Test1.exe" /INCREMENTAL:NO /NOLOGO /LIBPATH:"C:\Program Files\Intel\Compiler\11.1\048\mkl\ia64\lib"

Also you can easily understand is your code parallelized or not running your program in 2 modes: with mkl_set_num_threads(1); and mkl_set_num_threads(4); set prior to calling PARDISO andcompare the timings reported.

Regards,
Konstantin

eh4 · ‎11-17-2011

Hi,
When I let mkl_set_num_threads(4), during PARDISO all 4 CPUs in Task manager were running at 100%. However, when I change it to mkl_set_num_threads(1), only 1CPU is running. I guess it is running in parallel, although I dont know why the CommandWindow says only 1 processor is running for the Direct Factorization.

Is it wrong to link with the IA-64 architecture?

Thanks.

BW,
EH