Intel® oneAPI Math Kernel Library
Ask questions and share information with other developers who use Intel® Math Kernel Library.
7111 Discussions

Program crash in mkl_avx.dll on Windows with Intel MKL 11.0

Travis_O_
Beginner
2,329 Views

One of our customers has encountered a problem with our library (NumPy) when linked against MKL version 11.0.3 on the Windows platform.    The program dies during an eigenvalue decomposition.    The problem does not show up on a build of NumPy against an older version of the MKL.

Attached are some screenshots which provide information about: 

1) Where the crash occurs (mkl_avx)

2) Which instruction it seems not to like (vandpd)

3) call stack

4&5) Configuration information (Windows Version and Hardware).   

The software is Anaconda 1.5 available here:  www.continuum.io  but the same problem was replicated with other versions of NumPy downloaded elsewhere.   Several machines with this kind of hardware seem to have the same problem. 

0 Kudos
13 Replies
mecej4
Honored Contributor III
2,329 Views

I think versions of Windows older than W7SP1/Windows Server 2008 R2 did not support AVX instructions. Does this pertain to your customer's installation?

0 Kudos
SergeyKostrov
Valued Contributor II
2,329 Views
This is simply to let you know that the 2nd screenshot ( Screen Shot 2013-05-23 at 4.39.54 PM.png ) has Size = 0.
0 Kudos
SergeyKostrov
Valued Contributor II
2,329 Views
>>...I think versions of Windows older than W7SP1/Windows Server 2008 R2 did not support AVX instructions... On a couple of screenshots there is information about a version of Windows and this is: Windows 7 Professional SP1 ( 6.1.7601 ) I also see that Travis's computer has lots of memory and I don't think this is a memory related problem. It looks like a problem with MKL.
0 Kudos
Bernard
Valued Contributor I
2,329 Views

If you are using already windbg please run automated analysis with command !analyze -v or !analyze -hang and post the result.

0 Kudos
Bernard
Valued Contributor I
2,328 Views

Error code points to Access violation exception and it seems that wrong pointer was dereferenced or wrong memory address was calculated.

0 Kudos
SergeyKostrov
Valued Contributor II
2,329 Views
Unfortunately, it is Not clear what MKL function was called... Since the user is Not following up it is possible that the problem is resolved.
0 Kudos
Travis_O_
Beginner
2,329 Views

The problem is not resolved.  I don't have access to the machine directly so am waiting to see if I can get additional output from windbg.   Also, I don't know what this means exactly "run automated analysis with command !analyze -v or !analyze -hang and post the result"  I am not an expert windbg user.     

Our only option is to disable AVX instructions, unfortunately. 

0 Kudos
Travis_O_
Beginner
2,329 Views

The problem is not resolved.  I don't have access to the machine directly so am waiting to see if I can get additional output from windbg.   Also, I don't know what this means exactly "run automated analysis with command !analyze -v or !analyze -hang and post the result"  I am not an expert windbg user.     

Our only option is to disable AVX instructions, unfortunately. 

0 Kudos
Travis_O_
Beginner
2,329 Views

Doesn't this screen-shot show the stack of MKL functions that was called:  http://software.intel.com/sites/default/files/forum/392958/screen-shot-2013-05-23-at-4.40.48-pm.png

0 Kudos
Shane_S_Intel
Employee
2,329 Views

Hi Travis - I'll see if I can find someone on the MKL side to do a deeper analysis.

-Shane

0 Kudos
Alexander_K_Intel3
2,329 Views

Hello,

With very high confidence this is the same issue as described in http://software.intel.com/en-us/articles/svd-multithreading-bug-in-mkl. It goes to the same root cause according to the stack trace provided. A higher-level function (dlange) generates out of bound array reference and uses it to call AVX optimized subfunction, which crashes due to out of memory access. There is nothing specific to AVX. This is 6 cored machine, and this is the number of cores on which the issue is known to appear.

Please recommend the customer to upgrade to MKL 11.0.4 where the issue is already fixed. A temporal workaround is to use 4 threads by setting environment variable MKL_NUM_THREADS=4.

W.B.R.,

Alexander

0 Kudos
SergeyKostrov
Valued Contributor II
2,329 Views
>>...Doesn't this screen-shot show the stack of MKL functions that was called: >>http://software.intel.com/sites/default/files/forum/392958/screen-shot-2. You're right and I missed it even if I've looked at that screenshot several times. Thanks to Alexander for additional technical details!
0 Kudos
Travis_O_
Beginner
2,329 Views

Thank you Alexander.   I think you are exactly right and this is the same bug.   In fact, I can verify that the problem goes away for the user when they install the software compiled with MKL 11.0 Update 4

Thanks..

-Travis

0 Kudos
Reply