Intel MKL 2018 update 2 freezes during numpy test

[mvrht__u3425923]

Newly released Intel MKL 2018 Update 2 freezes when running standard numpy (1.14.2) test suite. It stops running

numpy.distutils.tests.test_exec_command.TestExecCommand.test_basic ...

test while using 1 CPU core with 100% loading. With Intel MKL 2018 initial release all tests are passed without problems.

[mecej4]

If you narrow down the question to which MKL routine caused the failure and what argument values were passed to that routine, more people would be able to take a look. If you use Intel's Python distribution, there is a separate forum for questions related to it, see https://software.intel.com/en-us/forums/intel-distribution-for-python ;.

[mvrht__u3425923]

It is ordinary C/C++/Fortran MKL installation with manually built numpy with MKL as BLAS on Linux machine.

Well, I tried but I am not able to identify exact routine. But I reproduced behavior on another machine.

[Ying_H_Intel]

Hi Mvrht,

​I may first to recommend your try Intel distributed python, which have optimized numpy and more by MKL.

Then consider build numpy manually, are you using Intel C/C++/fortran compiler or using GNU gcc/gfortran to build the numpy?
​Here is one article discussed some issues for your reference:
https://software.intel.com/en-us/articles/numpyscipy-with-intel-mkl
​Best Regards,
​Ying

[mvrht__u3425923]

Hi Ying,

Thanks for your advices! I am using GNU gcc/gfortran for building numpy. I tried to add additional compilation flags from referenced article but unfortunately with no luck - testing still freezes.

Considering python distribution - I am using deep learning library which require C/C++ MKL as BLAS backend as well, so only python distribution is not a solution and it is bit heavy (Anaconda+a lot of libraries) considering that I need only numpy.

[Ying_H_Intel]

Hi Mvrht,
what is exact command line are you using? or do you have details steps to reproduce the problem?

and have you tried

>export MKL_THREADING_LAYER=GNU
​>export MKL_NUM_THREADS=1
>export MKL_VERBOSE=1
​> your test.py
​and what is the output?
Best Regards,
Ying

[mvrht__u3425923]

I tried "export MKL_THREADING_LAYER=GNU" and I will try today a bit later other flags

Steps:

1. compile Numpy

wget https://github.com/numpy/numpy/archive/v1.14.2.tar.gz
tar -xzf v1.14.2.tar.gz
cd numpy-1.14.2
echo "[mkl]" >> site.cfg
echo "library_dirs = /opt/intel/mkl/lib/intel64" >> site.cfg
echo "include_dirs = /opt/intel/mkl/include" >> site.cfg
echo "mkl_libs = mkl_rt" >> site.cfg
echo "lapack_libs = " >> site.cfg
python3 setup.py build
sudo python3 setup.py install

2. run standard Numpy test

python3

>>import numpy as np
>>np.test(verbose=3)

[mvrht__u3425923]

Hi, Ying

well with defined all 3 flags

>export MKL_THREADING_LAYER=GNU
​>export MKL_NUM_THREADS=1
>export MKL_VERBOSE=1

NumPy tests passed.

But I guess MKL_NUM_THREADS=1 flag kills most of BLAS benefits? Somehow MKL 2018 initial release do not require defining this flags to pass the tests. BTW I tried MKL 2018 update 1 the problem is the same as with update 2.

[mvrht__u3425923]

In fact it is not necessary to use GNU threads. I attached outputs of running test with 1 and 2 threads with MKL_VERBOSE=1.

It stops during

numpy.distutils.tests.test_exec_command.TestExecCommand.test_basic ...

test. But I guess it is not the real problem - it does not call BLAS routines and if I add return statement in the beginning of this test it will anyway freeze with one core 100% usage.

[Konstantin_A_Intel]

Hi,

The issue you are describing is similar to what our Intel Python Distribution team already faced a few times.

They suggested a workaround: set environment variable KMP_INIT_AT_FORK=FALSE.

Can you please check and let us know is it helpful or not?

Thank you,

Konstantin

[mvrht__u3425923]

Hi, Konstantin

Yes, with KMP_INIT_AT_FORK=FALSE NumPy tests passed successfully! Thank you for helping!

I have some more questions: how does this flag affect performance (somehow https://software.intel.com/en-us/cpp-compiler-18.0-developer-guide-and-reference-supported-environment-variables even does not list such env variable)? Or maybe it just NumPy test corner case and generally it is not required anywhere else (in my application MKL will be called not only by NumPy, but by mix of NumPy code + deep learning framework C++ code)?