topic mecej4 - if you are willing in Intel® oneAPI Math Kernel Library

trouble going from XE2013 to XE2020

Brian_Murphy — Sun, 01 Mar 2020 22:26:55 GMT

I have an intel fortran console program which uses pardiso and works great when compiled with XE 2013 & visual studio 2010. I am trying to get it to run with XE 2020 and visual studio 2019 and the very first call to pardiso (phase=11) completes successfully but produces a different values in the iparm vector, and later on the calculation is failing and produces garbage (i.e. NaN). Any suggestions?

Hello Brian Murphy,

Ruqiu_C_Intel — Mon, 02 Mar 2020 01:56:55 GMT

Hello Brian Murphy,

Thanks for bringing the problem. Could you help provide reproducer for us to investigate it?

Best Regards,

Ruqiu

Thanks for the reply. My

Brian_Murphy — Mon, 02 Mar 2020 02:09:56 GMT

Thanks for the reply. My program with the problem is a very big program. It will me a lot of time to make something simple that reproduces the problem.

Are there any known issues when going from XE 2013 to XE 2020?

msglvl=1 is printing messages that say OpenMP is being used, but I am using /Qmkl:sequential

Brian, we have fixed pretty

Gennady_F_Intel — Mon, 02 Mar 2020 03:23:44 GMT

Brian, we have fixed pretty many issues since the version of XE 2013 regarding PARDISO solver, especially regarding the factorization step. You could try to set iparm(2) == 0 and check the problem once again.

When you linked with a sequential version of MKL and msglvl==1, you the expected to see something like this:

Statistics:
===========
Parallel Direct Factorization is running on 1 OpenMP

What do you have on your side?

I see Parallel Direct

Brian_Murphy — Mon, 02 Mar 2020 03:44:00 GMT

I see Parallel Direct Factorization is running on 1 OpenMP being printed to the console window by XE 2020.

XE 2013 prints the following, which does not mention OpenMP, which is why

Statistics:
===========
< Parallel Direct Factorization with number of processors: > 1
< Numerical Factorization with Level-3 BLAS performance >

I now think my problems are happening before calling pardiso and instead are with LAPACK. A LAPACK routine named dlarnv is returning garbage and I don't know why. It worked perfectly with XE 2013 but not XE 2020. I think XE 2020 is getting mixed up between MKL's lapack and a version of lapack compiled with the project. I'll bet there is a visual studio setting which worked for XE 2013 that isn't working for XE 2020.

Ok. 1 OpenMP thread ==

Gennady_F_Intel — Mon, 02 Mar 2020 04:22:30 GMT

Ok. 1 OpenMP thread == sequential mode.

regarding dlarnv() routine -- this is an unknown issue. could you give us the details to check the problem on our side?

Brian Murphy:

mecej4 — Mon, 02 Mar 2020 11:59:43 GMT

Brian Murphy:

I responded in your other thread, https://software.intel.com/en-us/forums/intel-fortran-compiler/topic/849486#comment-1953826 , regarding DLARNV.

If the random numbers returned by DLARNV are being used in a subsequent call to Pardiso, you should definitely ensure that the DLARNV call is functioning correctly before you suspect errors in Pardiso.

I see no problem with this

Gennady_F_Intel — Tue, 03 Mar 2020 03:24:57 GMT

I see no problem with this function. Could you check how it works on your side? my test is attached.

Here is the log I see on my part. MKL_VERBOSE is on. the very first elements are not a NAN as you reported.
MKL_VERBOSE Intel(R) MKL 2020.0 Product build 20191125 for Intel(R) 64 architecture Intel(R) Advanced Vector Extensions 2 (Intel(R) AVX2) enabled processors, Win 2.60GHz cdecl intel_thread
MKL_VERBOSE DLARNV(1,00000013B337FB60,140,000001E5E1A0E540) 3.02us CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:2
.. 0 , 0.131856
.. 1 , 0.233811

I am using fortran. I doubt

Brian_Murphy — Tue, 03 Mar 2020 03:40:23 GMT

I am using fortran. I doubt my problem is actually with dlarnv because a simple test program like your works fine. I've spent all day on this and I'm no closer to finding the root problem than when I started.

I'm getting that gesvx linear

Brian_Murphy — Tue, 03 Mar 2020 04:56:29 GMT

I'm getting that gesvx linear equation solver produces slightly different results in XE 2020 versus XE 2013. Slightly means a double precision calculation with a 4x4 matrix is different in the last digit. Has something changed in the MKL that could explain this, or should the results be identical if I don't have a mistake somewhere? The input matrices to gesvx are identical, but the output matrix is not.

I doubt this the root of my problem, but I'm running out of places to look.

Regarding consistency of FP

mecej4 — Tue, 03 Mar 2020 11:32:25 GMT

Regarding consistency of FP results, you can consider using compiler options that emphasize consistency at the expense of reduced performance, and consider the MKL features described in https://software.intel.com/en-us/articles/introduction-to-the-conditional-numerical-reproducibility-cnr .

I suspect that your large program has some of the common errors such as array overruns, uninitialized variables, data input errors and incorrect subprogram arguments. In the presence of such errors, consistency is neither desirable nor is perceived inconsistency something to put under the microscope.

Those NaNs that you reported did not come out of nowhere.

When you wrote "The input matrices to gesvx are identical", how did you determine that they were so?

If possible/permissible, post your full sources or, preferably, a cut-down version of those sources.

Thanks for the reply, mecej4.

Brian_Murphy — Tue, 03 Mar 2020 14:32:00 GMT

Thanks for the reply, mecej4. The entire code is far too big to post. I have observed that XE2020 initializes variables differently than XE2013 even when using the same compiler options. XE2020 seems to be initializing real's and integers to 0, whereas XE2013 does not. Since XE2013 is less forgiving about this, it sort of forces you to explicitly initialize variables to avoid problems.

This project started pre-Powerstation, and has gone through many migrations of compilers (by persons other than myself). Currently, in visual studio, all options at Properties/Fortran/Compatibility/ are set the same for XE2013 and XE2020. Only two are set to Yes; Treat Backslash as Normal Character and Use PowerStation Portability Library. Oddly, the corresponding directives /assume:bscc and /fpscomp:libs are not in the compiler command line or linker command line, which suggests to me that these are defaults.

I'm with you that the NaN from dlarnv did not come out of nowhere. I've been working my way backward trying to find where intermediate numerical results in XE2013 and XE2020 first become different. That led me to results returned by gesvx. If all 15 digits displayed in the debugger for all elements of an array, I call them equivalent.

Here is the compiler command line formatted for easier reading. Can you suggest any others that will help me find the problem? I wonder why there is a forward slash at the end of the first /I item.

/nologo 
/debug:full 
/Od 
/I"C:\Users\Brian\Documents\Visual Studio 2019\Projects\TRC-2013\Lateral Torsion Coupling\DLLUser\Debug/" 
/I"C:\Users\Brian\Documents\Visual Studio 2019\Projects\TRC-2013\Lateral Torsion Coupling\ARPACK\UTIL\Debug" 
/I"C:\Users\Brian\Documents\Visual Studio 2019\Projects\TRC-2013\Lateral Torsion Coupling\ARPACK\SRC\Debug" 
/DXLTRC2_INTEL_VF 
/warn:interfaces 
/module:"Debug\\" 
/object:"Debug\\" 
/Fd"Debug\vc160.pdb" 
/traceback 
/check:bounds 
/check:stack 
/libs:static 
/threads 
/dbglibs 
/Qmkl:sequential 
/c

If is is permissible for you

mecej4 — Tue, 03 Mar 2020 15:36:39 GMT

If is is permissible for you to post the code, rather than attaching it to a post here, zip up the whole source along with include files, data files and project files, upload the zip file (less than a couple of 100 megabytes is OK) to a cloud service (Google drive, Onedrive, Dropbox, etc.), enable public access to the cloud file, and post the link or shortcut in your response.

We have used this procedure a few times in the past in the Fortran forum, and that has worked out OK. One objection to posting large source codes is that their size may discourage some forum members from trying to help because of the correspondingly large time and effort needed.

I can't post the code because

Brian_Murphy — Tue, 03 Mar 2020 17:14:50 GMT

I can't post the code because it is the property of Texas A&M and is distributed to paying members of a turbomachinery research consortium.

My goal is to get the code running and validated against a collection of published test cases. I accomplished that with XE 2013, but I'm failing miserably with XE 2020. Tracing through calculation and comparing 2020 to 2013 is not working because tiny numerical differences make it impossible to see where things first go wrong.

I turned on /check:all but that didn't turn up anything, or change the final calculation result. Lots of warnings 406 were printed about creating temporary arrays for various arguments, but I don't think those are indicative of a problem. I tried to use piping to capture that output to a text file, but that didn't work on my windows 7 system.

If I don't come up with any new ideas, I will try XE 2013 with visual studio 2019.

mecej4 - if you are willing

Brian_Murphy — Tue, 03 Mar 2020 17:18:39 GMT

mecej4 - if you are willing to take a shot at this privately, I can get you paid to do so. Let me know by using the email link at the bottom of the home page at www.xlrotor.com