Intel® oneAPI Math Kernel Library
Ask questions and share information with other developers who use Intel® Math Kernel Library.
6956 Discussions

Intel MKL 2018 update 2 freezes during numpy test

mvrht__u3425923
Beginner
1,081 Views

Newly released Intel MKL 2018 Update 2 freezes when running standard numpy (1.14.2) test suite. It stops running

numpy.distutils.tests.test_exec_command.TestExecCommand.test_basic ...

test while using 1 CPU core with 100% loading. With Intel MKL 2018 initial release all tests are passed without problems.

0 Kudos
10 Replies
mecej4
Honored Contributor III
1,081 Views

If you narrow down the question to which MKL routine caused the failure and what argument values were passed to that routine, more people would be able to take a look. If you use Intel's Python distribution, there is a separate forum for questions related to it, see https://software.intel.com/en-us/forums/intel-distribution-for-python ;.

0 Kudos
mvrht__u3425923
Beginner
1,081 Views

It is ordinary C/C++/Fortran MKL installation with manually built numpy with MKL as BLAS on Linux machine.

Well, I tried but I am not able to identify exact routine. But I reproduced behavior on another machine.

0 Kudos
Ying_H_Intel
Employee
1,081 Views

Hi Mvrht,

​I may first to recommend your try Intel distributed python, which have optimized numpy and more by MKL.

Then consider build numpy manually, are you using Intel C/C++/fortran compiler or using GNU gcc/gfortran to build the numpy?
​Here is one article discussed some issues for your reference:
https://software.intel.com/en-us/articles/numpyscipy-with-intel-mkl
​Best Regards,
​Ying

 

 

 

0 Kudos
mvrht__u3425923
Beginner
1,082 Views

Hi Ying,

Thanks for your advices! I am using GNU gcc/gfortran for building numpy. I tried to add additional compilation flags from referenced article but unfortunately with no luck - testing still freezes.

Considering python distribution - I am using deep learning library which require C/C++ MKL as BLAS backend as well, so only python distribution is not a solution and it is bit heavy (Anaconda+a lot of libraries) considering that I need only numpy.

0 Kudos
Ying_H_Intel
Employee
1,082 Views

Hi Mvrht,
what is exact command line are you using? or do you have details steps to reproduce the problem?

and have you tried

>export MKL_THREADING_LAYER=GNU
​>export MKL_NUM_THREADS=1
>export MKL_VERBOSE=1
​> your test.py
​and what is the output?
Best Regards,
Ying

 

 

 

 

 

 

 

0 Kudos
mvrht__u3425923
Beginner
1,082 Views

I tried "export MKL_THREADING_LAYER=GNU" and I will try today a bit later other flags

 

Steps:

1. compile Numpy

wget https://github.com/numpy/numpy/archive/v1.14.2.tar.gz
tar -xzf v1.14.2.tar.gz
cd numpy-1.14.2
echo "[mkl]" >> site.cfg
echo "library_dirs = /opt/intel/mkl/lib/intel64" >> site.cfg
echo "include_dirs = /opt/intel/mkl/include" >> site.cfg
echo "mkl_libs = mkl_rt" >> site.cfg
echo "lapack_libs = " >> site.cfg
python3 setup.py build
sudo python3 setup.py install

2. run standard Numpy test

python3
>>import numpy as np
>>np.test(verbose=3)

 

0 Kudos
mvrht__u3425923
Beginner
1,082 Views

Hi, Ying

well with defined all 3 flags

>export MKL_THREADING_LAYER=GNU
​>export MKL_NUM_THREADS=1
>export MKL_VERBOSE=1

NumPy tests passed.

But I guess MKL_NUM_THREADS=1 flag kills most of BLAS benefits? Somehow MKL 2018 initial release do not require defining this flags to pass the tests. BTW I tried MKL 2018 update 1 the problem is the same as with update 2.

0 Kudos
mvrht__u3425923
Beginner
1,082 Views

In fact it is not necessary to use GNU threads. I attached outputs of running test with 1 and 2 threads with MKL_VERBOSE=1.

It stops during

numpy.distutils.tests.test_exec_command.TestExecCommand.test_basic ...

test. But I guess it is not the real problem - it does not call BLAS routines and if I add return statement in the beginning of this test it will anyway freeze with one core 100% usage.

0 Kudos
Konstantin_A_Intel
1,082 Views

Hi,

The issue you are describing is similar to what our Intel Python Distribution team already faced a few times.

They suggested a workaround: set environment variable KMP_INIT_AT_FORK=FALSE.

Can you please check and let us know is it helpful or not?

Thank you,

Konstantin

0 Kudos
mvrht__u3425923
Beginner
1,082 Views

Hi, Konstantin

Yes, with KMP_INIT_AT_FORK=FALSE NumPy tests passed successfully! Thank you for helping!

I have some more questions: how does this flag affect performance (somehow https://software.intel.com/en-us/cpp-compiler-18.0-developer-guide-and-reference-supported-environment-variables even does not list such env variable)? Or maybe it just NumPy test corner case and generally it is not required anywhere else (in my application MKL will be called not only by NumPy, but by mix of NumPy code + deep learning framework C++ code)?

0 Kudos
Reply