Community
cancel
Showing results for 
Search instead for 
Did you mean: 
mvrht__u3425923
Beginner
277 Views

SciPy built with Intel MKL Update 2 fails test

Hi,

SciPy v1.0.0 built with newly released Intel MKL 2018 Update 2 using GNU gcc/gfortran on Ubuntu 16.04 x64 PC fails standard tests. According to test output (attached), as I understand, some modules not included into final shared library despite that building process ended without errors (log also attached). It even do not start testing process detecting that module is absent:

AttributeError: module 'scipy.spatial.qhull' has no attribute 'ConvexHull'

where qhull.py only loads and imports symbols from dedicated shared library. Problem appeared since MKL 2018 update 1 release, SciPy built with MKL 2018 initial release successfully passes tests.

Steps to reproduce:

1. build scipy:

wget https://github.com/scipy/scipy/archive/v1.0.0.tar.gz
tar -xzf v1.0.0.tar.gz
cd scipy-1.0.0
echo "[mkl]" >> site.cfg
echo "library_dirs = /opt/intel/mkl/lib/intel64" >> site.cfg
echo "include_dirs = /opt/intel/mkl/include" >> site.cfg
echo "mkl_libs = mkl_rt" >> site.cfg
echo "lapack_libs = " >> site.cfg
python3 setup.py build
sudo python3 setup.py install

2. Run standard SciPy test:

python3
>> import scipy as sp
>> sp.test(verbose=3)

I am trying to build custom Docker image. As Intel python distribution contains a lot of packages and has completely non-customizable install it is not a suitable solution, it is too heavy (5Gb), and in fact I still need original MKL for C++ code. I am able to reproduces this error on multiple PC.

 

0 Kudos
9 Replies
Gennady_F_Intel
Moderator
277 Views

thanks for the issue. We will ask Python developers to investigate this case.

Ying_H_Intel
Employee
277 Views

Hi Mvrht,

Do set KMP_INIT_AT_FORK=FALSE work around the problem?

if it is same reason from
https://software.intel.com/en-us/forums/intel-math-kernel-library/topic/760830, ;

We will fix in latest next release.

Thanks
​Ying

 

 

 

mvrht__u3425923
Beginner
277 Views

Hi, Ying,

no, it does not help. I think it is more compilation time problem than runtime, though I am not 100% sure

Rohit_J_Intel
Employee
277 Views

 
Hello,
 
Since your primary concern is having Intel optimized Scipy, and not the complete installation of Intel Distribution for Python, functional in a container, I'd suggest the following:
1. Install miniconda in container (https://conda.io/miniconda.html)
2. <miniconda_installation>/bin/conda create -n intel_core_pkgs -c intel intelpython3_core python=3.6 -y
3. source <miniconda_installation>/bin/activate intel_core_pkgs
When you activate the "intel_core_pkgs" conda-environment (Step.3), you should have Scipy and all the required packages installed in that environment. I'd advise you to take this route since we build Numpy and Scipy using Intel compilers and have made various optimizations to the sources prior to building these modules.
 
Hope this helps!
 
Thanks,
Rohit
mvrht__u3425923
Beginner
277 Views

Hello, Rohit,

thank you for advice, but I guess in such case I will have to install separate MKL for C/C++ code?

Rohit_J_Intel
Employee
277 Views

 
Hi,
 
You won't have to; since we build Scipy and Numpy while dynamically linking against MKL, we ship MKL runtimes with the distribution. In your conda-environment, you would have all the MKL runtimes at `python3-config --exec-prefix`/lib directory. Also, if you require MKL's header-files, you can get them by running: `conda install -c intel mkl-include`, from your conda-environment. The header-files would be placed in `python3-config --exec-prefix`/include directory. 
 
Thanks,
Rohit
mvrht__u3425923
Beginner
277 Views

Sounds good! I will give a try, thank you!

277 Views

Hi, the error of compiling qhull.pyx is due to use of Cython 0.28, which broke build of SciPy. See https://github.com/scipy/scipy/pull/8581

The fix from the aforementioned pull request has been incorporated in 1.0.1 sources. Please use those, or use Cython 0.27.3 to build scipy.

mvrht__u3425923
Beginner
277 Views

Hi,

haven't noticed that SciPy update. Thank you!

Reply