- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

Hello,

I am working with a sparse matrix A and trying to solve <Ax =b>. b is an array of 1 million and it is in double precision. The matrix is sufficiently sparse (number of non-zeros in A (%): 0.000694). I am using mkl_pardiso.f90 in 4 nodes. The factorization is taking 10 min to complete but I was expecting that the solution phase shall not take longer but it is more than one hour and it is still in that phase. Is this normal ? I provide the output till now (before solution phase). Can anybody please share any ideas in order to improve this situation ? Any help will be much appreciated.

=== PARDISO: solving a real nonsymmetric system === 1-based array indexing is turned ON PARDISO double precision computation is turned ON Parallel METIS algorithm at reorder step is turned ON Scaling is turned ON Matching is turned ON Summary: ( reordering phase ) ================ Times: ====== Time spent in calculations of symmetric matrix portrait (fulladj): 0.081618 s Time spent in reordering of the initial matrix (reorder) : 1.912543 s Time spent in symbolic factorization (symbfct) : 1.559847 s Time spent in data preparations for factorization (parlist) : 0.075198 s Time spent in allocation of internal data structures (malloc) : 0.340503 s Time spent in additional calculations : 0.220093 s Total time spent : 4.189802 s Statistics: =========== Parallel Direct Factorization is running on 4 OpenMP < Linear system Ax = b > number of equations: 1000000 number of non-zeros in A: 6940000 number of non-zeros in A (%): 0.000694 number of right-hand sides: 1 < Factors L and U > number of columns for each panel: 96 number of independent subgraphs: 0 number of supernodes: 654600 size of largest supernode: 11181 number of non-zeros in L: 782333465 number of non-zeros in U: 766664027 number of non-zeros in L+U: 1548997492 Reordering completed ... Number of nonzeros in factors = 1548997492 Number of factorization MFLOPS = 10596105 === PARDISO is running in In-Core mode, because iparam(60)=0 === Percentage of computed non-zeros for LL^T factorization 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 === PARDISO: solving a real nonsymmetric system === Single-level factorization algorithm is turned ON Summary: ( factorization phase ) ================ Times: ====== Time spent in copying matrix to internal data structure (A to LU): 0.000000 s Time spent in factorization step (numfct) : 953.925513 s Time spent in allocation of internal data structures (malloc) : 0.170726 s Time spent in additional calculations : 0.089973 s Total time spent : 954.186212 s Statistics: =========== Parallel Direct Factorization is running on 4 OpenMP < Linear system Ax = b > number of equations: 1000000 number of non-zeros in A: 6940000 number of non-zeros in A (%): 0.000694 number of right-hand sides: 1 < Factors L and U > number of columns for each panel: 96 number of independent subgraphs: 0 number of supernodes: 654600 size of largest supernode: 11181 number of non-zeros in L: 782333465 number of non-zeros in U: 766664027 number of non-zeros in L+U: 1548997492 gflop for the numerical factorization: 10596.105218 gflop/s for the numerical factorization: 11.107896 Factorization completed ...

Thanks,

Dhiraj

Link Copied

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

The ratio of the number of non-zeroes in the L and U factors is over 200 times the number of non-zeroes in the original matrix. That is a lot of fill-in, and is probably responsible for the solution phase taking unexpectedly large execution time.

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

Hello,

Thanks a lot for your explanation. Actually my problem is that I have to do the factorization only once and then solution phase has to be repeated. So is there a way to do the fill in stuff once and for all so that if keeps the matrices in memory (I guess it automatically does unless phase=-1 is called). Specifically speaking I want something between phase 22 and 33, i.e. the fill-in part to be done at initialization stage, since only array b is going to change in my program. Or even during the fill-in it needs b ?

Thanks again for your help,

Dhiraj

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

You can make one set of calls (or single call) to complete phases 1 and 2 at the beginning of your run. Then, you can write a loop in which you assign new values to the r.h.s. vector b ( as in A.x = b), and obtain the solution with PHASE=33, do what you want with the solution x, and go on to the next r.h.s.

However, given the large fill-in that occurs during factorization of your atypical matrix, it can very well happen that the solution phase consumes CPU time that is not negligible compared to the factorization phase, especially if multiple solutions are asked for.

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

Hello,

Thank you for your answer. I am already calling the 11 and 22 phase at the initialization stage as my matrix A is not going to change. I am just calling 33 in a loop where matrix b gets changed and I get different x. And the solution phase is taking significantly larger time, seems that 10 times more than the factorization phase. Now I have 1 query. I shall be thankful if you can help in this matter.

The solution step is anyway going to take time as you said. I guess the forward and backward substitution for these elements takes time. In this case, can the cluster sparse solve help? I am yet to know the detailed structure of the cluster sparse solve but it seems that the more node I use, the faster the program will run. Am I right? or the result phase is expected to take nearly equal amount of time like this ?

Thanks again,

Dhiraj

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

It may be worthwhile for you to attempt to renumber the variables/reorder the equations so as to reduce the fill-in. If that fails, you can use the somewhat brute force approach of using a cluster solver, but I do not have any experience with clusters. Pardiso gives you some choice as to the algorithms available for reordering.

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

Hi Dhiraj,

Could you provide full log with time of solving step? it is not expected that time of solving step was bigger than factorization time only if number of rhs is huge.

Thanks,

Alex

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

Hi Alexander,

Actually last time I stopped the program after 1 hour. Today I ran it again and found that at nearly one hour the program stopped (automatically killed). It seems that the desktop ram (8GB) exhausted during the solution phase. I started the run again on our cluster and found that the program runs completely fine. The output is provided below. The cluster has 32 GB ram. Hence it seems that it is completely a ram problem. But I still do not understand when in cluster it took only 0.85 seconds in the solution phase why the desktop took so long before it got killed. In a smaller size problem (where size of b is 400000) computation time in the cluster and in my desktop was similar. Anyway my main problem here is solved. Thanks for all your input

Cheers,

Dhiraj

=== PARDISO: solving a real nonsymmetric system === The local (internal) PARDISO version is : 103911000 1-based array indexing is turned ON PARDISO double precision computation is turned ON Parallel METIS algorithm at reorder step is turned ON Scaling is turned ON Matching is turned ON Summary: ( reordering phase ) ================ Times: ====== Time spent in calculations of symmetric matrix portrait (fulladj): 0.101121 s Time spent in reordering of the initial matrix (reorder) : 1.829722 s Time spent in symbolic factorization (symbfct) : 3.486545 s Time spent in data preparations for factorization (parlist) : 0.096258 s Time spent in allocation of internal data structures (malloc) : 0.447319 s Time spent in additional calculations : 0.271942 s Total time spent : 6.232907 s Statistics: =========== < Parallel Direct Factorization with number of processors: > 20 < Numerical Factorization with BLAS3 and O(n) synchronization > < Linear system Ax = b > number of equations: 1000000 number of non-zeros in A: 6940000 number of non-zeros in A (%): 0.000694 number of right-hand sides: 1 < Factors L and U > number of columns for each panel: 72 number of independent subgraphs: 0 number of supernodes: 655255 size of largest supernode: 11186 number of non-zeros in L: 781479562 number of non-zeros in U: 768317946 number of non-zeros in L+U: 1549797508 Reordering completed ... Number of nonzeros in factors = 1549797508 Number of factorization MFLOPS = 10707689 === PARDISO is running in In-Core mode, because iparam(60)=0 === Percentage of computed non-zeros for LL^T factorization 0 % 1 % 2 % 3 % 4 % 5 % 6 % 7 % 8 % 9 % 10 % 11 % 12 % 13 % 14 % 15 % 16 % 17 % 18 % 19 % 20 % 21 % 22 % 23 % 24 % 25 % 26 % 27 % 28 % 29 % 30 % 31 % 32 % 33 % 35 % 36 % 37 % 39 % 40 % 41 % 42 % 43 % 44 % 45 % 47 % 48 % 49 % 51 % 52 % 54 % 55 % 57 % 59 % 60 % 62 % 63 % 65 % 67 % 69 % 71 % 73 % 75 % 77 % 79 % 82 % 84 % 86 % 89 % 92 % 93 % 95 % 96 % 98 % 99 % 100 % === PARDISO: solving a real nonsymmetric system === Single-level factorization algorithm is turned ON Summary: ( factorization phase ) ================ Times: ====== Time spent in copying matrix to internal data structure (A to LU): 0.000001 s Time spent in factorization step (numfct) : 140.436790 s Time spent in allocation of internal data structures (malloc) : 0.001189 s Time spent in additional calculations : 0.000002 s Total time spent : 140.437982 s Statistics: =========== < Parallel Direct Factorization with number of processors: > 20 < Numerical Factorization with BLAS3 and O(n) synchronization > < Linear system Ax = b > number of equations: 1000000 number of non-zeros in A: 6940000 number of non-zeros in A (%): 0.000694 number of right-hand sides: 1 < Factors L and U > number of columns for each panel: 72 number of independent subgraphs: 0 number of supernodes: 655255 size of largest supernode: 11186 number of non-zeros in L: 781479562 number of non-zeros in U: 768317946 number of non-zeros in L+U: 1549797508 gflop for the numerical factorization: 10707.689303 gflop/s for the numerical factorization: 76.245614 Factorization completed ... === PARDISO: solving a real nonsymmetric system === Summary: ( solution phase ) ================ Times: ====== Time spent in direct solver at solve step (solve) : 0.850383 s Time spent in additional calculations : 1.698694 s Total time spent : 2.549077 s Statistics: =========== < Parallel Direct Factorization with number of processors: > 20 < Numerical Factorization with BLAS3 and O(n) synchronization > < Linear system Ax = b > number of equations: 1000000 number of non-zeros in A: 6940000 number of non-zeros in A (%): 0.000694 number of right-hand sides: 1 < Factors L and U > number of columns for each panel: 72 number of independent subgraphs: 0 number of supernodes: 655255 size of largest supernode: 11186 number of non-zeros in L: 781479562 number of non-zeros in U: 768317946 number of non-zeros in L+U: 1549797508 gflop for the numerical factorization: 10707.689303 gflop/s for the numerical factorization: 76.245614 Solve completed ...

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page