- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

Hello everyone!

I'm using PARDISO to solve Navier Stoks and Temperature equations in one program.

I use phase 11 only at program start to tell solver what my matrixes look like. I need to refill both matrixes, make factorization and find solution at every time layer. To do so I use *maxfct *parameter equal 2 and change *mnum *from 1 to 2 for different equations. I also use 2 different *pt *arrays for different equations. I need to mention that I use phase 0 after each time layer in order to free memory.

I want to accelerate computations by using several cluster nodes. I found Cluster version of PARDISO and it was a surprise for me that *maxfct *and *mnum* parameters are ignored. I also didnt find phase 0.

My first question is: do I really need those parameters to solve my problem on a cluster? I dont want to use phase 11 at every time layer because it will be too slow.

Secondly I'm having a problem using PARDISO on a single cluster node when the number of equations is ~ 13 million. I recieve error -2. Output below I get after phase 11 for both matrixes and phase 23 for Navier Stoks equation.

=== PARDISO: solving a real nonsymmetric system === 0-based array is turned ON PARDISO double precision computation is turned ON Parallel METIS algorithm at reorder step is turned ON Scaling is turned ON Matching is turned ON Summary: ( reordering phase ) ================ Times: ====== Time spent in calculations of symmetric matrix portrait (fulladj): 1.680254 s Time spent in reordering of the initial matrix (reorder) : 15.632488 s Time spent in symbolic factorization (symbfct) : 128.883790 s Time spent in data preparations for factorization (parlist) : 2.423303 s Time spent in allocation of internal data structures (malloc) : 168.633973 s Time spent in additional calculations : 13.837179 s Total time spent : 331.090987 s Statistics: =========== Parallel Direct Factorization is running on 28 OpenMP < Linear system Ax = b > number of equations: 13198336 number of non-zeros in A: 107134147 number of non-zeros in A (%): 0.000062 number of right-hand sides: 1 < Factors L and U > number of columns for each panel: 72 number of independent subgraphs: 0 number of supernodes: 6792104 size of largest supernode: 56189 number of non-zeros in L: 52781978834 number of non-zeros in U: 52521656342 number of non-zeros in L+U: 105303635176 === PARDISO: solving a real nonsymmetric system === 0-based array is turned ON PARDISO double precision computation is turned ON Parallel METIS algorithm at reorder step is turned ON Scaling is turned ON Matching is turned ON Summary: ( reordering phase ) ================ Times: ====== Time spent in calculations of symmetric matrix portrait (fulladj): 0.309069 s Time spent in reordering of the initial matrix (reorder) : 2.541601 s Time spent in symbolic factorization (symbfct) : 2.516607 s Time spent in data preparations for factorization (parlist) : 0.269381 s Time spent in allocation of internal data structures (malloc) : 1.306915 s Time spent in additional calculations : 1.014548 s Total time spent : 7.958121 s Statistics: =========== Parallel Direct Factorization is running on 28 OpenMP < Linear system Ax = b > number of equations: 3368499 number of non-zeros in A: 22736533 number of non-zeros in A (%): 0.000200 number of right-hand sides: 1 < Factors L and U > number of columns for each panel: 72 number of independent subgraphs: 0 number of supernodes: 2244305 size of largest supernode: 13571 number of non-zeros in L: 2906625075 number of non-zeros in U: 2863523632 number of non-zeros in L+U: 5770148707 === PARDISO is running in In-Core mode, because iparam(60)=0 === *** Error in PARDISO ( insufficient_memory) error_num= 8 *** Error in PARDISO memory allocation: FACTORIZE_SOLVING_LU_DATA, allocation of 412359210 bytes failed total memory wanted here: 435961139 kbyte === PARDISO: solving a real nonsymmetric system === Summary: ( starting phase is factorization, ending phase is solution ) ================ Times: ====== Time spent in additional calculations : 0.000074 s Total time spent : 0.000074 s Statistics: =========== Parallel Direct Factorization is running on 28 OpenMP < Linear system Ax = b > number of equations: 13198336 number of non-zeros in A: 107134147 number of non-zeros in A (%): 0.000062 number of right-hand sides: 1 < Factors L and U > number of columns for each panel: 72 number of independent subgraphs: 0 number of supernodes: 6792104 size of largest supernode: 56189 number of non-zeros in L: 52781978834 number of non-zeros in U: 52521656342 number of non-zeros in L+U: 105303635176 gflop for the numerical factorization: 3841175.475158 PARDISO ERROR = -2

Do I really need so much memory to solve system or it is some kind of bug? If it is not, can this problem be solved by using cluster PARDISO?

My iparm setup is below.

IPARM(1) = 1 ! NO SOLVER DEFAULT IPARM(2) = 3 ! FILL-IN REORDERING FROM METIS IPARM(3) = 0 ! NUMBERS OF PROCESSORS IPARM(4) = 0 ! NO ITERATIVE-DIRECT ALGORITHM IPARM(5) = 0 ! NO USER FILL-IN REDUCING PERMUTATION IPARM(6) = 0 ! =0 SOLUTION ON THE FIRST N COMPONENTS OF X IPARM(7) = 0 ! NOT IN USE IPARM(8) = 5 ! NUMBERS OF ITERATIVE REFINEMENT STEPS IPARM(9) = 0 ! NOT IN USE IPARM(10) = 16 ! PERTURB THE PIVOT ELEMENTS WITH 1E-13 IPARM(11) = 1 ! USE NONSYMMETRIC PERMUTATION AND SCALING MPS IPARM(12) = 0 ! NOT IN USE IPARM(13) = 1 ! MAXIMUM WEIGHTED MATCHING ALGORITHM IS SWITCHED-ON (DEFAULT FOR NON-SYMMETRIC) IPARM(14) = 0 ! OUTPUT: NUMBER OF PERTURBED PIVOTS IPARM(15) = 0 ! NOT IN USE IPARM(16) = 0 ! NOT IN USE IPARM(17) = 0 ! NOT IN USE IPARM(18) = -1 ! OUTPUT: NUMBER OF NONZEROS IN THE FACTOR LU IPARM(19) = -1 ! OUTPUT: MFLOPS FOR LU FACTORIZATION IPARM(20) = 0 ! OUTPUT: NUMBERS OF CG ITERATIONS IPARM(24) = 0 IPARM(34) = 0 IPARM(27) = 0 IPARM(35) = 1 ! ZERO BASE INDEXING IPARM(39) = 0

Im using Parallel Studio XE 2017.4.196 on a single cluster node. Each node has 2 х Intel Xeon E5-2690 v4 and 256GB of RAM and it runs on CentOS 7.3

Thanks in advance for your help.

Link Copied

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

>> Do I really need so much memory to solve system or it is some kind of bug? If it is not, can this problem be solved by using cluster PARDISO?

yes, this is not a bug. Solver reports you the number of nnz aftere factorization is nnz ~ 57*10^9, therefore the size of memory needed would be sizeof(double) * nnz ~ 420 Gb.

you may solve this case by using iparm(60) = 1 or 2 ( hybrid or OOC modes). Please refer to the Developer Reference manual for details.

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

Thank you for response. Will it be enough memory if I use 5 nodes? 10? How do Cluster Pardiso allocate memory?

What about my first question?

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

Hello,

I have a problem causing the same error message, but using iparm (60) = 1, so OOC mode is enabled. Here is the log:

ooc_max_core_size got from config file=200000 === PARDISO is running in Out-Of-Core mode, because iparam(60)=1 and there is not enough RAM for In-Core === *** Error in PARDISO ( insufficient_memory) error_num= 8 *** Error in PARDISO memory allocation: FACTORIZE_SOLVING_LU_DATA, allocation of 136172344 bytes failed total memory wanted here: 195700928 kbyte === PARDISO: solving a symmetric indefinite system === 1-based array indexing is turned ON PARDISO double precision computation is turned ON Parallel METIS algorithm at reorder step is turned ON Scaling is turned ON Matching is turned ON Summary: ( starting phase is reordering, ending phase is solution ) ================ Times: ====== Time spent in calculations of symmetric matrix portrait (fulladj): 37.209805 s Time spent in reordering of the initial matrix (reorder) : 0.174363 s Time spent in symbolic factorization (symbfct) : 244.759732 s Time spent in data preparations for factorization (parlist) : 7.739215 s Time spent in allocation of internal data structures (malloc) : 1371.208639 s Time spent in additional calculations : 230.071368 s Total time spent : 1891.163122 s ============================================================== ----------- Out of core time (in percent (%)) -------------- Factorization step (100 (%)): write to files : 0 % read from files: 0 % factorization - write&read: 100 % Solution step (100 (%)): read from files: 0 % solve - write&read: 100 % Total time (100 (%)): read from files: 0 % total - write&read: 100 % ----------- Out of core Mb -------------- Factorization step: write to files : 0.000 Mb read from files: 0.000 Mb Solution step: read from files: 0.000 Mb Total size of data transferred: write&read : 0.000 Mb ============================================================== Statistics: =========== Parallel Direct Factorization is running on 16 OpenMP < Linear system Ax = b > number of equations: 16651601 number of non-zeros in A: 1070209671 number of non-zeros in A (%): 0.000386 number of right-hand sides: 1 < Factors L and U > number of columns for each panel: 192 number of independent subgraphs: 0 number of supernodes: 3858946 size of largest supernode: 76371 number of non-zeros in L: 24985066246 number of non-zeros in U: 1 number of non-zeros in L+U: 24985066247 gflop for the numerical factorization: 700621.967324

I think this is strange because these large arrays should be stored out of core.

Also you write

Solver reports you the number of nnz aftere factorization is nnz ~ 57*10^9

Where do you find that number? I find in Oleg's error message

number of non-zeros in L+U: 105303635176

which would give a necessary memory of about 840 GB which does not correspond to the 420 GB the solver says it needs.

In the case of my problem,

number of non-zeros in L+U: 24985066247

which would require about 192 GB of memory. That corresponds to the

total memory wanted here: 195700928 kbyte

which is about 195 GB. So in my case, those numbers match. Also I do not understand why 195 GB is too much when

ooc_max_core_size

is set to 200 GB.

Thanks in advance for your help.

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page