MKL PARDISO iparm[23] behaviour

[morskaya_svinka_1]

I use MKL version 2024.2. Please mention in the MKL documentation in iparm[23] description that phase=11 with iparm[23]=0 is incompatible with phase=23 with iparm[23]=1, phase=11 with iparm[23]=1 is incompatible with phase=23 with iparm[23]=0. Please mention about all such incompatibilities or provide a link to the place its mentioned (I didnt find it after searching through all iparm[23] occurencies). Also just in case some user comes by and finds this information helpful iparm[23]=1 requires more RAM than iparm[23]=0 especially in OOC mode. Questions:
1. When iparm[23] is set to 1 the program prints very little statistics to the screen compared to iparm[23]=0, is it ok? 
2. iparm[16] remains the same after phase=11 and phase=23 for iparm[23]=1 but something is written there on phase=23 for iparm[23]=0. What is written there?
3. Does MKL_PARDISO_OOC_MAX_CORE_SIZE control the amount of memory can be given to OOC mode for all solver process or just additional memory for factorization and solution on top of analysis memory? I ask that because in the last test I provided 10'000 MB restriction but in the beginning of phase=23 it printed 14'500 MB allocated. I also wonder why iparm[23]=0 provides lower memory bound on OOC but its PeakWorkingSet (11 GB) is bigger than iparm[23]=1 PeakWorkingSet (8 GB) and also iparm[23]=0 PeakWorkingSet is same in IC and OOC in this example. Can you give some advice when I should use iparm[23]=0 or iparm[23]=1 on a shared memory machine?

In-core results iparm[23]=0
minimum memory requirement IC (MB) = 10731
minimum memory requirement OOC (MB) = 3206
iparm[14] (MB) = 3206
iparm[15] (MB) = 2976
iparm[16] (MB) = 7754
iparm[15] + iparm[62] (MB) = 2976
phase=11
PeakWorkingSet64 = 2875588608
phase=23
time in seconds = 32.8
minimum memory requirement IC (MB) = 10697
minimum memory requirement OOC (MB) = 3206
iparm[14] (MB) = 3206
iparm[15] (MB) = 2976
iparm[16] (MB) = 7720
iparm[15] + iparm[62] (MB) = 2976
PeakWorkingSet64 = 11316170752 bytes

In-core results iparm[23]=1
phase=11
Memory allocated on phase 11 4166.4923 MB
PeakWorkingSet64 = 4440424448
minimum memory requirement IC (MB) = 11281
minimum memory requirement OOC (MB) = 6728
iparm[14] (MB) = 4192
iparm[15] (MB) = 4166
iparm[16] (MB) = 7115
iparm[15] + iparm[62] (MB) = 6728
phase=23
Memory allocated on phase 22 11281.5165 MB
time in seconds = 34.8
minimum memory requirement IC (MB) = 11281
minimum memory requirement OOC (MB) = 6728
iparm[14] (MB) = 4192
iparm[15] (MB) = 4166
iparm[16] (MB) = 7115
iparm[15] + iparm[62] (MB) = 6728

PeakWorkingSet64 = 11855372288 bytes

OOC results iparm[23]=0 MKL_PARDISO_OOC_MAX_CORE_SIZE=10000

phase=11
minimum memory requirement IC (MB) = 10880
minimum memory requirement OOC (MB) = 3172
iparm[14] (MB) = 3172
iparm[15] (MB) = 2957
iparm[16] (MB) = 7922
iparm[15] + iparm[62] (MB) = 2957
phase=23
time in seconds = 56.2
minimum memory requirement IC (MB) = 9295
minimum memory requirement OOC (MB) = 3172
iparm[14] (MB) = 3172
iparm[15] (MB) = 2957
iparm[16] (MB) = 6337
iparm[15] + iparm[62] (MB) = 2957

PeakWorkingSet64 = 11338207232 bytes

OOC results iparm[23]=1 MKL_PARDISO_OOC_MAX_CORE_SIZE=10000

phase=11
Memory allocated on phase 11 4166.4923 MB (printed by PARDISO)
PeakWorkingSet64 = 4308307968
minimum memory requirement IC (MB) = 11281
minimum memory requirement OOC (MB) = 6728
iparm[14] (MB) = 4192
iparm[15] (MB) = 4166
iparm[16] (MB) = 7115
iparm[15] + iparm[62] (MB) = 6728
phase=23
Memory allocated on phase 22 14588.5565 MB (printed by PARDISO)
time in seconds = 44.4
minimum memory requirement IC (MB) = 11281
minimum memory requirement OOC (MB) = 6728
iparm[14] (MB) = 4192
iparm[15] (MB) = 4166
iparm[16] (MB) = 7115
iparm[15] + iparm[62] (MB) = 6728

PeakWorkingSet64 = 7979446272 bytes

[morskaya_svinka_1]

It seems like iparm[23]=1 does not work at all with OOC mode and 64-bit integer (ILP64 interface). I build the executable with MSBuild in Visual Studio 2022. When I set iparm[23]=0 it works correctly, but iparm[23]=1 in the same code fails. In the debug mode the following message is returned: "Unhandled exception at [some address] (mkl_core.2.dll) in [my_executable.exe]: An invalid parameter was passed to a function that considers invalid parameters fatal".  I have not found anything on this in the documentation. It happens on some percentage of factorization (phase=22) with both Parallel and Sequential versions used. For a big matrix it fails on say 21%, but on a small one it fails after 100% of factorization but before leaving phase=22 pardiso subroutine. By the way, percentage printing is broken (some percents are not printed) in multithreaded mode for several combinations of parameters, including iparm[23]=1 with OOC.
P.S. I tried iparm[1]=2,3 and 0, does not help. All iparms are consistent through all the phases.

[Ruqiu_C_Intel]

Thank you for posting the issue. We are investigating it and will update here once there is progress.

[Ruqiu_C_Intel]

Hi Evgeny,

Thank you for your patient.

Here are our updates:

When iparm[23] is set to 1 the program prints very less statistics to the screen compared to iparm[23]=0.

iparm[16] is documented as the extra memory required for factorization in in-core mode after phase 1. The value after phase 2 and 3 can be ignored, it is undocumented and just for internal calculations. We might remove it in future and make the value constant throughout.

MKL_PARDISO_OOC_MAX_CORE_SIZE is not a parameter to be tuned. It should be close to the size of RAM. Actually, MKL_PARDISO_OOC_MAX_CORE_SIZE will be compared with the memory required by the whole solver process to determine if PARDISO has sufficient memory to perform the solve. This comparison, however, can only be done after phase 1 since only then the structure of LU factors can be determined. If your test case allocated more than 10 GB, it looks like a bug, can you share us a simple reproducer to investigate? Also if you met an unexpected crash with iparm[23]=1, we need the reproducer as well.

Regards,

Ruqiu

[morskaya_svinka_1]

I found out what is the problem with iparm[23]=1 OOC mode. Both int32 and int64 versions crash when I specify MKL_PARDISO_OOC_PATH environment variable like this in C++ (tried MSVC and Intel-cpp-compiler): _putenv("MKL_PARDISO_OOC_PATH=C:\\somepath\\pardiso_ooc_tmpdir\\ooc_tmp_file"). OOC files are not written in that directory, and I think the program crashes at the moment it tries to write. iparm[23]=0 works fine in that case. Reproducer attached.

I also attached the reproducer of the issue when MKL_PARDISO_OOC_MAX_CORE_SIZE=10000 is set and PARDISO prints that 14'000 MB has been allocated. I cannot attach the matrix here though as it weights 500 MB and the restriction here is 23 MB. Show me the way I can upload  it if you want it please.

Why should MKL_PARDISO_OOC_MAX_CORE_SIZE not be tuned? I think the tuning is helpful if the user wants to be sure that he will have say 10GB available RAM left while the OOC calculation is running. I saw that restricting MKL_PARDISO_OOC_MAX_CORE_SIZE can significantly slow down the calculation, but I gave about 200% lower bound for OOC mode and it worked fine for all the matrices I have tested. Moreover, if you keep giving more than 150% or 200% of lower bound, PeakWorkingSet grows a bit, but speed does not improve.
There is another drawback of not tuning MKL_PARDISO_OOC_MAX_CORE_SIZE. If you want to use OOC mode explicitly (iparm[59]=2), you can get about 1.5x slowdown in case you cross minimum IC requirement for shared memory machines, as discussed here: https://community.intel.com/t5/Intel-oneAPI-Math-Kernel-Library/OneAPI-PARDISO-iparm-62-0/m-p/1636246#M36518. Though I have not tested this effect on clusters and matrices with factorization larger than 200GB.

[morskaya_svinka_1]

Was anyone able to reproduce the unexpected crash? I could not reproduce it with MKL 2024.0, but I cannot also find a download page for MKL 2024.2. Is there a way to download this version?

[morskaya_svinka_1]

I obtained the following behaviour when I specify _putenv("MKL_PARDISO_OOC_PATH=C:\\tmp\\tmptmp").  iparm[62]=2 with iparm[23]=1 creates files named "mkl_pardiso_lnz..." in the directory C:\\tmp\\tmptmp, and crashes if such directory does not exist - that is the crash I got. But iparm[62]=2 with iparm[23]=0 creates files named "tmptmp..." in the directory C:\\tmp independently of C:\\tmp\\tmptmp existing or not. It also appears that _putenv("MKL_PARDISO_OOC_KEEP_FILE=0") does not affect iparm[23]=1 OOC deleting temporary files, it deletes them anyways (with MKL_PARDISO_OOC_KEEP_FILE=1 as well). Is this fixed in the current MKL version (2025.0.1)?

[Ruqiu_C_Intel]

Thank you for your patient.

We reproduced the issue with oneMKL 2025.0, we are fixing it now.