- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Newly released Intel MKL 2018 Update 2 freezes when running standard numpy (1.14.2) test suite. It stops running
numpy.distutils.tests.test_exec_command.TestExecCommand.test_basic ...
test while using 1 CPU core with 100% loading. With Intel MKL 2018 initial release all tests are passed without problems.
Link Copied
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
If you narrow down the question to which MKL routine caused the failure and what argument values were passed to that routine, more people would be able to take a look. If you use Intel's Python distribution, there is a separate forum for questions related to it, see https://software.intel.com/en-us/forums/intel-distribution-for-python ;.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
It is ordinary C/C++/Fortran MKL installation with manually built numpy with MKL as BLAS on Linux machine.
Well, I tried but I am not able to identify exact routine. But I reproduced behavior on another machine.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Mvrht,
I may first to recommend your try Intel distributed python, which have optimized numpy and more by MKL.
Then consider build numpy manually, are you using Intel C/C++/fortran compiler or using GNU gcc/gfortran to build the numpy?
Here is one article discussed some issues for your reference:
https://software.intel.com/en-us/articles/numpyscipy-with-intel-mkl
Best Regards,
Ying
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Ying,
Thanks for your advices! I am using GNU gcc/gfortran for building numpy. I tried to add additional compilation flags from referenced article but unfortunately with no luck - testing still freezes.
Considering python distribution - I am using deep learning library which require C/C++ MKL as BLAS backend as well, so only python distribution is not a solution and it is bit heavy (Anaconda+a lot of libraries) considering that I need only numpy.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Mvrht,
what is exact command line are you using? or do you have details steps to reproduce the problem?
and have you tried
>export MKL_THREADING_LAYER=GNU
>export MKL_NUM_THREADS=1
>export MKL_VERBOSE=1
> your test.py
and what is the output?
Best Regards,
Ying
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I tried "export MKL_THREADING_LAYER=GNU" and I will try today a bit later other flags
Steps:
1. compile Numpy
wget https://github.com/numpy/numpy/archive/v1.14.2.tar.gz tar -xzf v1.14.2.tar.gz cd numpy-1.14.2 echo "[mkl]" >> site.cfg echo "library_dirs = /opt/intel/mkl/lib/intel64" >> site.cfg echo "include_dirs = /opt/intel/mkl/include" >> site.cfg echo "mkl_libs = mkl_rt" >> site.cfg echo "lapack_libs = " >> site.cfg python3 setup.py build sudo python3 setup.py install
2. run standard Numpy test
python3
>>import numpy as np >>np.test(verbose=3)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi, Ying
well with defined all 3 flags
>export MKL_THREADING_LAYER=GNU
>export MKL_NUM_THREADS=1
>export MKL_VERBOSE=1
NumPy tests passed.
But I guess MKL_NUM_THREADS=1 flag kills most of BLAS benefits? Somehow MKL 2018 initial release do not require defining this flags to pass the tests. BTW I tried MKL 2018 update 1 the problem is the same as with update 2.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
In fact it is not necessary to use GNU threads. I attached outputs of running test with 1 and 2 threads with MKL_VERBOSE=1.
It stops during
numpy.distutils.tests.test_exec_command.TestExecCommand.test_basic ...
test. But I guess it is not the real problem - it does not call BLAS routines and if I add return statement in the beginning of this test it will anyway freeze with one core 100% usage.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
The issue you are describing is similar to what our Intel Python Distribution team already faced a few times.
They suggested a workaround: set environment variable KMP_INIT_AT_FORK=FALSE.
Can you please check and let us know is it helpful or not?
Thank you,
Konstantin
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi, Konstantin
Yes, with KMP_INIT_AT_FORK=FALSE NumPy tests passed successfully! Thank you for helping!
I have some more questions: how does this flag affect performance (somehow https://software.intel.com/en-us/cpp-compiler-18.0-developer-guide-and-reference-supported-environment-variables even does not list such env variable)? Or maybe it just NumPy test corner case and generally it is not required anywhere else (in my application MKL will be called not only by NumPy, but by mix of NumPy code + deep learning framework C++ code)?
![](/skins/images/D2683F18326913BBA0436CB7114DD569/responsive_peak/images/icon_anonymous_message.png)
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page