Question about parameter setting of PARDISO - Page 2

Zhanghong_T_ · ‎09-01-2014

Dear all,

I use the MKL under WIndows 7 64bit + VS 2012 + MKL 11.0 or 11.2.

I need to solve very large FEM matrix by PARDISO. Currently I set parameters as follows:

iparm(1) = 1 ! no solver default
iparm(2) = 3 ! parallel (OpenMP) version of the nested dissection algorithm
iparm(4) = 0 ! no iterative-direct algorithm
iparm(5) = 0 ! no user fill-in reducing permutation
iparm(6) = 0 ! =0 solution on the first n compoments of x
iparm(8) = 9 ! numbers of iterative refinement steps
iparm(10) = 13 ! perturbe the pivot elements with 1E-13
iparm(11) = 1 ! use nonsymmetric permutation and scaling MPS
iparm(13) = 0 ! maximum weighted matching algorithm is switched-off (default for symmetric). Try iparm(13) = 1 in case of inappropriate accuracy
iparm(18) = -1 ! Output: number of nonzeros in the factor LU
iparm(19) = -1 ! Output: Mflops for LU factorization
iparm(20) = 0 ! Output: Numbers of CG Iterations
iparm(21) = 1 ! Apply 1x1 and 2x2 Bunch and Kaufman pivoting during the factorization process
iparm(24) = 1 ! PARDISO uses new two-level factorization algorithm

The peak memory usage would reach to 20 GB (windows task manager->commit size). Is there any better parameter settings to decrease memory usage but don't increase the solution time significant?

Thanks,

Zhanghong Tang

Alexander_K_Intel2 · ‎10-02-2014

Just for testing - could you send iparm(60) to 1? I try to check that you correctly read config file. Also, could you set some file name but not only the folder: 'c:\temp\'? Thanks, Alex P.S. Am I correct that you have several process that going to work with one ooc file? It is better to choose different file name for different processes.

Zhanghong_T_ · ‎10-02-2014

Dear Alex,

Thank you very much for your kindly reply. I have set file name not just the folder, the file name is

'c:\temp\'//trim(ss2)

where 'ss2' is a character from 'myid'.

In fact, just now I tested by the following command:

mpiexec -wdir "Z:\test" -mapall -hosts 1 n01 2 Z:\fem

where n01 is just the local machine.

and the PARDISO work successfully.

Thanks,

Zhanghong Tang

Alexander_K_Intel2 · ‎10-02-2014

Good! And what the name of ooc files stored in correspondent machine in such case?

Zhanghong_T_ · ‎10-02-2014

Hi Alex,

Thank you very much for your kindly reply.

The names of these files are:

Directory of C:\temp

2014/10/02 23:25    <DIR>          .
2014/10/02 23:25    <DIR>          ..
2014/10/02 23:22       196,902,156 0_.ind
2014/10/02 23:22       295,044,736 0_.jal
2014/10/02 23:23     2,147,483,608 0_.lnz
2014/10/02 23:22       147,522,368 0_.lup
2014/10/02 23:22        34,423,440 0_.sin
2014/10/02 23:22        34,423,440 0_.sle
2014/10/02 23:22       202,666,344 0_2.lnz
2014/10/02 23:22                 0 0_3.lnz
2014/10/02 23:22                 0 0_4.lnz
2014/10/02 23:23       195,247,668 1_.ind
2014/10/02 23:22       291,282,696 1_.jal
2014/10/02 23:23       305,741,768 1_.lnz
2014/10/02 23:22       145,641,348 1_.lup
2014/10/02 23:22        34,093,524 1_.sin
2014/10/02 23:22        34,093,524 1_.sle
2014/10/02 23:23                 0 1_2.lnz
2014/10/02 23:23                 0 1_3.lnz
2014/10/02 23:23                 0 1_4.lnz
2014/10/02 23:25                 0 out.txt
              19 File(s) 4,064,566,620 bytes
               2 Dir(s) 54,129,131,520 bytes free

But the problem is that the PARDISO returns error -11 when I run by the following command:

mpiexec -wdir "Z:\test" -mapall -hosts 10 n01 2 n02 2 n03 4 n04 4 n05 4 n06 4 n07 4 n08 4 n09 4 n10 4 Z:\fem

Alexander_K_Intel2 · ‎10-02-2014

Small correction, the initial error is -10, the -11 returned by solving step because factorization have not been competed....My feeling is that something incorrect with sile name - they became the same, but i far from my machine to check it. I understand that i am boring, but it is only way to find the issue - could you call pardiso_getenv routine and pring filename that have been returned by pardiso? To make a proof that name a different? Thanks, Alex

Zhanghong_T_ · ‎10-02-2014

Dear Alex,

Thank you very much for your kindly reply. I did as you suggested, the screen output is attached.

Thanks,

Zhanghong Tang

Alexander_K_Intel2 · ‎10-02-2014

Ok, everything looks correct. Let me some time to play with such configuration (several processes per one node with OOC PARDISO on each). Also, just to reproduce it more correctly, could you print here parameters of iparm(15), iparm(16), iparm(17) and iparm(63) after reordering step?

Thanks,

Alex

Zhanghong_T_ · ‎10-02-2014

Dear Alex,

Thank you very much for your kindly reply. I did as you suggested, the screen output is attached.

Thanks,

Zhanghong Tang

Zhanghong_T_ · ‎10-03-2014

Dear Alex,

Is there any progress the investigate this problem?

BTW: I have set MKL_PARDISO_OOC_KEEP_FILE = 0 but I noticed that the OOC files are not deleted after the solver finished (phase = -1). Is there anything missed?

Finally I wish the PARDISO work like this: if evaluated peak memory is larger than the limit set before (for example, 16 GB), the OOC is enabled and the peak memory should not larger than 16 GB, if evaluated peak memory is smaller than the limit, the IC is enabled. For choosing OOC or IC, I think it can be solved by setting Iparm(60) = 1, the key issue is that after enabled OOC, the peak memory should be limitted to 16 GB, is there any additional settings?

Thanks,

Zhanghong Tang

Zhanghong_T_ · ‎10-23-2014

Dear Alex,

Is there any progress to investigate this problem?

Thanks,

Zhanghong Tang