SciPy built with Intel MKL Update 2 fails test

[mvrht__u3425923]

Hi,

SciPy v1.0.0 built with newly released Intel MKL 2018 Update 2 using GNU gcc/gfortran on Ubuntu 16.04 x64 PC fails standard tests. According to test output (attached), as I understand, some modules not included into final shared library despite that building process ended without errors (log also attached). It even do not start testing process detecting that module is absent:

AttributeError: module 'scipy.spatial.qhull' has no attribute 'ConvexHull'

where qhull.py only loads and imports symbols from dedicated shared library. Problem appeared since MKL 2018 update 1 release, SciPy built with MKL 2018 initial release successfully passes tests.

Steps to reproduce:

1. build scipy:

wget https://github.com/scipy/scipy/archive/v1.0.0.tar.gz
tar -xzf v1.0.0.tar.gz
cd scipy-1.0.0
echo "[mkl]" >> site.cfg
echo "library_dirs = /opt/intel/mkl/lib/intel64" >> site.cfg
echo "include_dirs = /opt/intel/mkl/include" >> site.cfg
echo "mkl_libs = mkl_rt" >> site.cfg
echo "lapack_libs = " >> site.cfg
python3 setup.py build
sudo python3 setup.py install

2. Run standard SciPy test:

python3
>> import scipy as sp
>> sp.test(verbose=3)

I am trying to build custom Docker image. As Intel python distribution contains a lot of packages and has completely non-customizable install it is not a suitable solution, it is too heavy (5Gb), and in fact I still need original MKL for C++ code. I am able to reproduces this error on multiple PC.

[Gennady_F_Intel]

thanks for the issue. We will ask Python developers to investigate this case.

[Ying_H_Intel]

Hi Mvrht,

Do set KMP_INIT_AT_FORK=FALSE work around the problem?

if it is same reason from
https://software.intel.com/en-us/forums/intel-math-kernel-library/topic/760830, ;

We will fix in latest next release.

Thanks
​Ying

[mvrht__u3425923]

Hi, Ying,

no, it does not help. I think it is more compilation time problem than runtime, though I am not 100% sure

[Rohit_J_Intel]

Hello,

Since your primary concern is having Intel optimized Scipy, and not the complete installation of Intel Distribution for Python, functional in a container, I'd suggest the following:

1. Install miniconda in container (https://conda.io/miniconda.html)
2. <miniconda_installation>/bin/conda create -n intel_core_pkgs -c intel intelpython3_core python=3.6 -y
3. source <miniconda_installation>/bin/activate intel_core_pkgs

When you activate the "intel_core_pkgs" conda-environment (Step.3), you should have Scipy and all the required packages installed in that environment. I'd advise you to take this route since we build Numpy and Scipy using Intel compilers and have made various optimizations to the sources prior to building these modules.

Hope this helps!

Thanks,

Rohit

[mvrht__u3425923]

Hello, Rohit,

thank you for advice, but I guess in such case I will have to install separate MKL for C/C++ code?

[Rohit_J_Intel]

Hi,

You won't have to; since we build Scipy and Numpy while dynamically linking against MKL, we ship MKL runtimes with the distribution. In your conda-environment, you would have all the MKL runtimes at `python3-config --exec-prefix`/lib directory. Also, if you require MKL's header-files, you can get them by running: `conda install -c intel mkl-include`, from your conda-environment. The header-files would be placed in `python3-config --exec-prefix`/include directory. 

Thanks,

Rohit

[mvrht__u3425923]

Sounds good! I will give a try, thank you!

[Oleksandr_P_Intel]

Hi, the error of compiling qhull.pyx is due to use of Cython 0.28, which broke build of SciPy. See https://github.com/scipy/scipy/pull/8581

The fix from the aforementioned pull request has been incorporated in 1.0.1 sources. Please use those, or use Cython 0.27.3 to build scipy.

[mvrht__u3425923]

Hi,

haven't noticed that SciPy update. Thank you!