Pardiso Crash in MKL 2017

[schulzey]

After upgrading to MKL 2017 (as part of the 2017 Intel Fortran compiler for Windows) I get a Pardiso crash when calling it to solve a simple matrix. The crash doesn't happen in MKL 11.3.4 or in any other earlier versions.

I have created a test example that demonstrates the crash (see attachment). The data that gets passed to Pardiso (which is normally generated by my application) is read from a file in my test example which is also attached. The test example runs Ok in MKL 11.3.4 and earlier versions but crashes in MKL 2017.

I have been using Pardiso successfully for many years and don't think I am doing anything wrong.

Peter

[Gennady_F_Intel]

Do you mean this is some sort of regression problem and this case passed well with previous updates? 

[Gennady_F_Intel]

the light testing your example, shows no problems.  Win, 64 bit, lp64 mode. 

the only changes I did:  made the program instead of subroutine, added mkl_get_version_string and prints for output array.

here is the results:

_Forums\u696757>1.exe
 **********************************************
ntel(R) Math Kernel Library Version 2017.0.0 Product Build 20160801 for Intel(R) 64 architecture applications

 **********************************************
.... before Pardiso call .......
 after Pardiso call, phase == 13, ERROR ==            0
 **********************************************
 **********************************************
 after Pardiso call, Phase ==-1, ERROR ==            0
 ... X  ...==  0.000000000000000E+000 -1.130282459811037E-003
 3.526077288086858E-003 -3.592627831828867E-005  0.000000000000000E+000
 0.000000000000000E+000  0.000000000000000E+000 -5.602969108474065E-004
 2.777262958946455E-003 -6.643974549978017E-005  0.000000000000000E+000
 0.000000000000000E+000  1.149659736043504E-003  2.549216482510058E-003
-4.677457951176992E-003  0.000000000000000E+000  0.000000000000000E+000
 0.000000000000000E+000  4.259138634320509E-003  2.111724010591131E-003
-1.320197447645458E-002  0.000000000000000E+000  0.000000000000000E+000
 0.000000000000000E+000  5.987196295190753E-003  1.128306758578033E-003
-3.235162985183892E-002  0.000000000000000E+000  0.000000000000000E+000
 6.276281397749442E-003  1.021550220088535E-010 -5.476862158031406E-002
 0.000000000000000E+000  0.000000000000000E+000  8.459754026750180E-014
-1.128306554232226E-003 -3.235162985252373E-002  0.000000000000000E+000
 0.000000000000000E+000 -6.276281397759122E-003 -2.111723799051891E-003
-1.320197305531472E-002  0.000000000000000E+000  0.000000000000000E+000
 0.000000000000000E+000 -5.987196127432409E-003 -2.549216311189985E-003
-4.677461520114669E-003  0.000000000000000E+000  0.000000000000000E+000
 0.000000000000000E+000 -4.259139853882195E-003 -2.777262844894757E-003
-6.643974549908814E-005  0.000000000000000E+000  0.000000000000000E+000
-1.149659751248887E-003 -3.526077231058885E-003 -3.592627831794265E-005
 0.000000000000000E+000  0.000000000000000E+000  0.000000000000000E+000
 5.602968956403245E-004  0.000000000000000E+000  1.130282444603389E-003

[schulzey]

I'm not quite sure in what context you are using the term regression, but yes my test example works well in all previous versions of Pardiso. The problem only started in the 2017 version.

My test case is a small 65 x 65 symmetrical banded matrix with a half bandwidth of 12. I'm using CSR storage and calling Pardiso with mtype=2 (real and symmetric positive definite matrix).and phase=13. If I call Pardiso with phase=12 then it still crashes.
 

[schulzey]

The results you got are identical to mine if I use an earlier version of Pardiso. Do you have any idea of why my version would be crashing? It is very consistent.

[Gennady_F_Intel]

- regression means if 11.3.4 fails but the previous update 11.3.3 works as expected...

- I still don't see the problem on my side. here are the conditions which I use for validation your case:

window 8.1, MKL version 2017; command line build and execution; 

linking with /Qmkl or "mkl_core.lib mkl_intel_lp64.lib mkl_intel_thread.lib libiomp5md.lib"

CPU - AVX2 based system.

- Could you give the environment you use when the problem with 2017 happens?

[schulzey]

My environment is Windows 10-64 with Visual Studio 2015. The VS command line is:

ifort /nologo /O2 /fpp /warn:noalignments /align:rec1byte /fp:strict /module:&quotx64\Release\\&quot /object:&quotx64\Release\\&quot /Fd&quotx64\Release\vc140.pdb&quot /libs:dll /threads /c /Qlocation,link,&quotC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\\bin\amd64&quot /Qm64 &quotC:\work\PardisoTest.FOR&quot

Could you send me your modified source code and command lines so that I can try your method?

[mecej4]

No crashes with IPS2017 on W10-X64, run after compiling 32- or 64-bit EXEs.

[schulzey]

C'mon guys, I know what regression means. I was asking about the context in case Gennady was asking if my application uses Pardiso to do some sort of regression analysis.

[Gennady_F_Intel]

the attached is your example I used.

compile and link lines:

ifort /Qmkl PardisoTest_v1.for 

or

ifort /I "..\windows\mkl\include" PardisoTest_v1.for mkl_intel_lp64.lib mkl_intel_thread.lib  mkl_core.lib \libiomp5md.lib 

[schulzey]

After further testing I have been able to establish that the Pardiso crash only happens in the 64-bit version if I use /Qmkl:parallel. If I use /Qmkl:sequential then it works Ok. I have also established that the problem occurs in MKL 11.3.4 and MKL 2017, whereas earlier versions seem to be Ok.

The problem for me is that running Pardiso in sequential mode is too slow for my calculation intensive application.

​Can you see if you can replicate this?

Peter

[schulzey]

After a bit more testing I have found that the Pardiso crash only happens if I compile and link in Visual Studio but not if I do it with a command line prompt (eg. ifort /Qmkl:parallel PardisoTest.for). The problem therefore seems to be related to something in the Visual Studio 2015 implementation. I have used default VS settings everywhere except for turning on MKL with /Qmkl:parallel. I have attached my VS2015 solution which demonstrates the crash.

Peter

[Ferris_H_]

I wonder if your issue is related to the one I just posted about. Try changing the OMP_NUM_THREADS to see if only certain values cause it to crash like with me.

Edit: Also try changing iparm[0]= 1 or 0 to see if one works.

[schulzey]

Interesting... setting OMP_NUM_THREADS ​= 1 makes my example work with /Qmkl:parallel but anything greater than 1 (or leaving it out altogether) makes it crash. My system is an i7 3537U with 4 cores.

I think setting iparm[0]=1 might be a red herring. It just means that you have to set all the other iparm values yourself rather than using the default values. Are you calling PardisoInit before Pardiso and, if not, are you initializing pt to zeroes first?

I just noticed that someone else reported a similar problem back in November 2015 (see link below):

http://stackoverflow.com/questions/33539521/intel-mkl-pardiso-wont-run-parallel-in-fortran

[Ferris_H_]

I am not using Pardisoinit, but I am initializing pt to 0's:

  for(i=0;i<64;i++){pt=0;}

For some reason setting iparm[0]=1 allows me to use any value for OMP_NUM_THREADS ( I do not set any other iparm values ). If I keep it at zero, then I get the crash for OMP_NUM_THREADS=2 or 3.

[Gennady_F_Intel]

Peter, I reproduced the problem with the example you provided at #12. 

The exception happens when the code links with threaded versions of mkl as you said. In the case if of sequential libs or MKL_NUM_THREADS = 1, Pardiso works as expected. 

when you link with threaded versions of mkl's libs, as a temporarily workaround, we may recommend to set      call mkl_set_num_threads( 1 ) before the first Pardiso call and then restore the #of threads.

wbr, Gennady

[schulzey]

I can't really use sequential or threads=1 because that will slow down my application too much.

Do you know why if I compile and link with a command prompt and still use /Qmkl:parallel it seems to work? Is there a particular VS2015 compiler setting that is causing the problem that I could possibly change to avoid the problem and get a result similar to the command prompt method (other than changing to sequential or one thread)?

[Alexander_K_Intel2]

Hi,

looks like your forgot to add include file in your example. After some slight change of file (in attach) to remove warning, example passed correctly on all mkl version that I have

Thanks,

Alex 

schulzey wrote:

I can't really use sequential or threads=1 because that will slow down my application too much.

Do you know why if I compile and link with a command prompt and still use /Qmkl:parallel it seems to work? Is there a particular VS2015 compiler setting that is causing the problem that I could possibly change to avoid the problem and get a result similar to the command prompt method (other than changing to sequential or one thread)?

[schulzey]

Hi Alex, it still crashes in my Visual Studio application if I have the extra include line in there or not. You seem to be compiling it with a command line but have you tried it with my VS project (see post #12 above). Compiling it with a command line works for me too, but it is the VS implementation that is the problem.

[Alexander_K_Intel2]

Schulzey,

Can I ask you to send me project with link mkl which differ from default one? I see similar behaviour in case of linking non-default MKL without removing check mark in "Use Intel Math Kernel Library" field.

Thanks,

Alex

[schulzey]

Alex,

I always have "Use Intel Math Kernel Library" set to "Parallel". I am able to change to an older version of the MKL library by changing the Include and Library paths in the Visual Studio "Tools" => "Options" settings for the Intel Visual Fortran compiler. When running the test program I also make sure I am using the MKL runtime DLLs to match the version of MKL I have compiled with. See attached images.

​If I use MKL 11.3.4 or MKL 2017 my test program crashes, but if I go back to any older version of MKL then it works Ok. I am using the Intel Fortran compiler 17.0.0.109.

​Changing the VS Tools => Options settings doesn't change the project and so there is no point uploading it again.

Regards,
Peter