topic Re:PARDISO is running in In-Core mode, because iparam(60)=0 in Intel® oneAPI Math Kernel Library

PARDISO is running in In-Core mode, because iparam(60)=0

Hassan-Ebrahimi — Wed, 13 Sep 2023 11:55:11 GMT

I am trying to make use of OOC PARIDSO but it seems that setting iparam(60)=1 is neglected.

For example with a small matrix I get the following messages:

*************************************

Statistics:
===========
Parallel Direct Factorization is running on 1 OpenMP

< Linear system Ax = b >
number of equations: 500000
number of non-zeros in A: 50494950
number of non-zeros in A (%): 0.020198

number of right-hand sides: 1

< Factors L and U >
number of columns for each panel: 128
number of independent subgraphs: 0
< Preprocessing with state of the art partitioning metis>
number of supernodes: 10238
size of largest supernode: 242
number of non-zeros in L: 157921408
number of non-zeros in U: 1
number of non-zeros in L+U: 157921409
Reordering Time [s]: 15.5904239001684
=== PARDISO is running in In-Core mode, because iparam(60)=0 ===
**********************

Is there other setting needed to see the effect of setting iparam(60)=1?

The wrapper of pardiso solver is attached.

MKL version: Intel(R) oneAPI Math Kernel Library Version 2022.1-Product Build 20220311 for Intel(R) 64 architecture applications

Thank you in advance.

Re:PARDISO is running in In-Core mode, because iparam(60)=0

AryanK_Intel — Thu, 14 Sep 2023 10:39:21 GMT

Hi,

Thanks for posting in Intel Communities.

To make use of OOC(Out Of Core) PARDISO, please assign the value of iparm[59] to 2. If iparm[59] is set to 2, then Intel® MKL PARDISO OOC mode is used.

The default value of iparm[59] is 0.

If iparm[59] is set to 0, then the standard in-core Intel® MKL PARDISO is used.

If iparm[59] is set to 1, then the standard in-core Intel® MKL PARDISO is used.

If iparm[59] is set to 2, then Intel® MKL PARDISO OOC mode is used.

Please refer to the below link for more details regarding the significance of iparm[59].

https://www.intel.com/content/www/us/en/developer/articles/training/how-to-use-ooc-pardiso.html

Best regards,

Sri Raj Aryan.K

Re: Re:PARDISO is running in In-Core mode, because iparam(60)=0

Hassan-Ebrahimi — Fri, 15 Sep 2023 01:12:28 GMT

Hi Sri Raj Aryan.K,

I am using the Fortran interface of PARDISO. Am I not supposed to set iparm(60)?

When with iparm(60) =2 the message is still "PARDISO is running in In-Core mode, because iparam(60)=0" for a matrix of size 1 GB.

For a matrix of size 6.73 GB I get the following error in reordering phase:

****************************************************************************************

************************** mkl_test starts ...
reordering
*** Error in PARDISO ( insufficient_memory) error_num= 1
*** Error in PARDISO memory allocation: MATCHING_REORDERING_DATA, allocation of 1 bytes failed
total memory wanted here: 4906038 kbyte

=== PARDISO: solving a symmetric indefinite system ===
1-based array indexing is turned ON
PARDISO double precision computation is turned ON
METIS algorithm at reorder step is turned ON

Summary: ( reordering phase )
================

Times:
======
Time spent in calculations of symmetric matrix portrait (fulladj): 2.843310 s
Time spent in reordering of the initial matrix (reorder) : 0.000000 s
Time spent in symbolic factorization (symbfct) : 0.000000 s
Time spent in allocation of internal data structures (malloc) : 0.007406 s
Time spent in additional calculations : 5.792175 s
Total time spent : 8.642892 s

Statistics:
===========
Parallel Direct Factorization is running on 1 OpenMP

< Linear system Ax = b >
number of equations: 2000000
number of non-zeros in A: 601954850
number of non-zeros in A (%): 0.015049

number of right-hand sides: 1

< Factors L and U >
number of columns for each panel: 128
number of independent subgraphs: 0
< Preprocessing with state of the art partitioning metis>
number of supernodes: 0
size of largest supernode: 0
number of non-zeros in L: 0
number of non-zeros in U: 0
number of non-zeros in L+U: 0
iparm(15) 0
iparm(16) 0
iparm(18) -1
Reordering completed ...
The following ERROR was detected: -2
1
****************************************************************************************

The Fortran file is attached

Re: Re:PARDISO is running in In-Core mode, because iparam(60)=0

Hassan-Ebrahimi — Thu, 21 Sep 2023 06:41:46 GMT

Hi Sri Raj Aryan.K,

The problem is not solved yet.

Would you please look into the matter?

Hassan

Re:PARDISO is running in In-Core mode, because iparam(60)=0

AryanK_Intel — Thu, 21 Sep 2023 10:08:46 GMT

Hi Hassan-Ebrahimi,

Thanks for sharing the Fortran source. We have tried running the shared source code at our end and we could see that the code exiting abruptly with error code -1 which means that the input is inconsistent as per the below Pardiso reference.

https://www.intel.com/content/www/us/en/docs/onemkl/developer-reference-fortran/2023-2/pardiso.html

Regrets for earlier reply, as iparm[60]=2 is the expected value for the OOC mode.

Could you please share with us a sample working reproducer, so that we could work on this accordingly.

Best Regards,

Sri Raj Aryan.K

Re: Re:PARDISO is running in In-Core mode, because iparam(60)=0

Hassan-Ebrahimi — Thu, 21 Sep 2023 10:18:45 GMT

Hi Sri Raj Aryan.K,

Thanks for your reply.

The crash may be due to the size of the matrix.

Let's make it work for a smaller matrix:

In line 40 of pardiso_sym_iparam60=2.f, we have w=300. Please reduce it w=30.

With w=30, the factorization completes in less than 30 seconds on my side.

Best regards,

Hassan

Re:PARDISO is running in In-Core mode, because iparam(60)=0

AryanK_Intel — Fri, 22 Sep 2023 11:58:38 GMT

Hi Hassan-Ebrahimi,

Thanks for the suggestion. We have tried running the code by changing the value of w=300 to w=30. However, the issue persists for both iparm(60)=0 and iparm(60)=2.

Best Regards,

Sri Raj Aryan.K

Re:PARDISO is running in In-Core mode, because iparam(60)=0

AryanK_Intel — Fri, 29 Sep 2023 08:41:41 GMT

Hi Hassan-Ebrahimi,

We have not heard back from you. Could you please provide an update on this thread? If possible, please share a sample working reproducer, so that we could work on this accordingly.

Thanks & Regards,

Sri Raj Aryan.K

Re: Re:PARDISO is running in In-Core mode, because iparam(60)=0

Hassan-Ebrahimi — Mon, 02 Oct 2023 07:44:01 GMT

Hi Sri Raj Aryan.K

Thanks for your follow-up.

After days of struggling, I found the cause of the original problem.

In my code IPARM(1) = 0. This seems to reset IPARM(60).

By setting IPARM(1) = 1 and IPARM(60)=2, PARSDISP runs in OOC mode.

Hassan

Re: PARDISO is running in In-Core mode, because iparam(60)=0

AryanK_Intel — Tue, 03 Oct 2023 12:34:52 GMT

Hi Hassan-Ebrahimi,

Thanks for sharing the solution with us.

It’s great to know that the issue has been resolved, in case you run into any other issues please feel free to create a new thread.

Have a Good Day!

Thanks & Regards,

Sri Raj Aryan.K