Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
46 Views

Intel MKL 2018 update 2 freezes during numpy test

Newly released Intel MKL 2018 Update 2 freezes when running standard numpy (1.14.2) test suite. It stops running

numpy.distutils.tests.test_exec_command.TestExecCommand.test_basic ...

test while using 1 CPU core with 100% loading. With Intel MKL 2018 initial release all tests are passed without problems.

0 Kudos
10 Replies
Highlighted
Black Belt
46 Views

If you narrow down the question to which MKL routine caused the failure and what argument values were passed to that routine, more people would be able to take a look. If you use Intel's Python distribution, there is a separate forum for questions related to it, see https://software.intel.com/en-us/forums/intel-distribution-for-python ;.

0 Kudos
Highlighted
46 Views

It is ordinary C/C++/Fortran MKL installation with manually built numpy with MKL as BLAS on Linux machine.

Well, I tried but I am not able to identify exact routine. But I reproduced behavior on another machine.

0 Kudos
Highlighted
Employee
46 Views

Hi Mvrht,

​I may first to recommend your try Intel distributed python, which have optimized numpy and more by MKL.

Then consider build numpy manually, are you using Intel C/C++/fortran compiler or using GNU gcc/gfortran to build the numpy?
​Here is one article discussed some issues for your reference:
https://software.intel.com/en-us/articles/numpyscipy-with-intel-mkl
​Best Regards,
​Ying

 

 

 

0 Kudos
Highlighted
46 Views

Hi Ying,

Thanks for your advices! I am using GNU gcc/gfortran for building numpy. I tried to add additional compilation flags from referenced article but unfortunately with no luck - testing still freezes.

Considering python distribution - I am using deep learning library which require C/C++ MKL as BLAS backend as well, so only python distribution is not a solution and it is bit heavy (Anaconda+a lot of libraries) considering that I need only numpy.

0 Kudos
Highlighted
Employee
46 Views

Hi Mvrht,
what is exact command line are you using? or do you have details steps to reproduce the problem?

and have you tried

>export MKL_THREADING_LAYER=GNU
​>export MKL_NUM_THREADS=1
>export MKL_VERBOSE=1
​> your test.py
​and what is the output?
Best Regards,
Ying

 

 

 

 

 

 

 

0 Kudos
Highlighted
46 Views

I tried "export MKL_THREADING_LAYER=GNU" and I will try today a bit later other flags

 

Steps:

1. compile Numpy

wget https://github.com/numpy/numpy/archive/v1.14.2.tar.gz
tar -xzf v1.14.2.tar.gz
cd numpy-1.14.2
echo "[mkl]" >> site.cfg
echo "library_dirs = /opt/intel/mkl/lib/intel64" >> site.cfg
echo "include_dirs = /opt/intel/mkl/include" >> site.cfg
echo "mkl_libs = mkl_rt" >> site.cfg
echo "lapack_libs = " >> site.cfg
python3 setup.py build
sudo python3 setup.py install

2. run standard Numpy test

python3
>>import numpy as np
>>np.test(verbose=3)

 

0 Kudos
Highlighted
46 Views

Hi, Ying

well with defined all 3 flags

>export MKL_THREADING_LAYER=GNU
​>export MKL_NUM_THREADS=1
>export MKL_VERBOSE=1

NumPy tests passed.

But I guess MKL_NUM_THREADS=1 flag kills most of BLAS benefits? Somehow MKL 2018 initial release do not require defining this flags to pass the tests. BTW I tried MKL 2018 update 1 the problem is the same as with update 2.

0 Kudos
Highlighted
46 Views

In fact it is not necessary to use GNU threads. I attached outputs of running test with 1 and 2 threads with MKL_VERBOSE=1.

It stops during

numpy.distutils.tests.test_exec_command.TestExecCommand.test_basic ...

test. But I guess it is not the real problem - it does not call BLAS routines and if I add return statement in the beginning of this test it will anyway freeze with one core 100% usage.

0 Kudos
Highlighted
46 Views

Hi,

The issue you are describing is similar to what our Intel Python Distribution team already faced a few times.

They suggested a workaround: set environment variable KMP_INIT_AT_FORK=FALSE.

Can you please check and let us know is it helpful or not?

Thank you,

Konstantin

0 Kudos
Highlighted
46 Views

Hi, Konstantin

Yes, with KMP_INIT_AT_FORK=FALSE NumPy tests passed successfully! Thank you for helping!

I have some more questions: how does this flag affect performance (somehow https://software.intel.com/en-us/cpp-compiler-18.0-developer-guide-and-reference-supported-environme... even does not list such env variable)? Or maybe it just NumPy test corner case and generally it is not required anywhere else (in my application MKL will be called not only by NumPy, but by mix of NumPy code + deep learning framework C++ code)?

0 Kudos