Problem building 64 bit numpy using MKL and vc11 (Windows)

[Jay_R_]

Hi, I’ve built numpy (1.9) 64 bit using vc11, the Intel Fortran compiler and the MKL ‘mkl_rt’ library.

*why? (see end of message for the reason, if interested)

Any advice or assistance would be greatly appreciated.  If I can offer additional information, I will happily do so.

The build appears to go just fine (no errors noted), and numpy loads into python just fine as well.

(I note a warning:  ### Warning:  python_xerbla.c is disabled ### -- however, it doesn’t appear to be problematic?)

I have also confirmed that numpy sees the mkl blas and lapack libs. 

>>> numpy.__config__.show()

lapack_opt_info:

    libraries = ['mkl_lapack', 'mkl_rt']

    library_dirs = ['C:\\Program Files (x86)\\Intel\\Composer XE 2015\\mkl\\lib\\intel64']

    define_macros = [('SCIPY_MKL_H', None)]

    include_dirs = ['C:\\Program Files (x86)\\Intel\\Composer XE 2015\\mkl\\include']

blas_opt_info:

    libraries = ['mkl_rt']

    library_dirs = ['C:\\Program Files (x86)\\Intel\\Composer XE 2015\\mkl\\lib\\intel64']

    define_macros = [('SCIPY_MKL_H', None)]

    include_dirs = ['C:\\Program Files (x86)\\Intel\\Composer XE 2015\\mkl\\include']

openblas_lapack_info:

  NOT AVAILABLE

lapack_mkl_info:

    libraries = ['mkl_lapack', 'mkl_rt']

    library_dirs = ['C:\\Program Files (x86)\\Intel\\Composer XE 2015\\mkl\\lib\\intel64']

    define_macros = [('SCIPY_MKL_H', None)]

    include_dirs = ['C:\\Program Files (x86)\\Intel\\Composer XE 2015\\mkl\\include']

blas_mkl_info:

    libraries = ['mkl_rt']

    library_dirs = ['C:\\Program Files (x86)\\Intel\\Composer XE 2015\\mkl\\lib\\intel64']

    define_macros = [('SCIPY_MKL_H', None)]

    include_dirs = ['C:\\Program Files (x86)\\Intel\\Composer XE 2015\\mkl\\include']

mkl_info:

    libraries = ['mkl_rt']

    library_dirs = ['C:\\Program Files (x86)\\Intel\\Composer XE 2015\\mkl\\lib\\intel64']

    define_macros = [('SCIPY_MKL_H', None)]

    include_dirs = ['C:\\Program Files (x86)\\Intel\\Composer XE 2015\\mkl\\include']

Everything *looks* to be in order upon casual inspection (*I think*, please correct me if I’m wrong!)

However, there is no performance boost when running a few different tests in numpy (singular value decomposition, for example), and only a single thread appears to be in play.

Running numpy.test(‘full’) reveals 21 errors.

For instance,

LINK : fatal error LNK1104: cannot open file 'ifconsol.lib'

And, the other being a recurring error with f2py,

ERROR: test_size.TestSizeSumExample.test_transpose

----------------------------------------------------------------------

Traceback (most recent call last):

  File "C:\Program Files\Side Effects Software\Houdini 13.0.509\python27\lib\site-packages\nose\case.py", line 371, in setUp

    try_run(self.inst, ('setup', 'setUp'))

  File "C:\Program Files\Side Effects Software\Houdini 13.0.509\python27\lib\site-packages\nose\util.py", line 478, in try_run

    return func()

  File "C:\Program Files\Side Effects Software\Houdini 13.0.509\python27\lib\site-packages\numpy\f2py\tests\util.py", line 353, in setUp

    module_name=self.module_name)

  File "C:\Program Files\Side Effects Software\Houdini 13.0.509\python27\lib\site-packages\numpy\f2py\tests\util.py", line 80, in wrapper

   raise ret

RuntimeError: Running f2py failed: ['-m', '_test_ext_module_5403', 'c:\\users\\jareyn~1\\appdata\\local\\temp\\tmpvykewl\\foo.f90']

Reading .f2py_f2cmap ...

        Mapping "real(kind=rk)" to "double"

Succesfully applied user defined changes from .f2py_f2cmap

Everything that requires configuration appears to be in agreement with this Intel Application Note, minus use of the Intel C++ compiler:

https://software.intel.com/en-us/articles/numpyscipy-with-intel-mkl

I have also referenced the Windows build docs on scipy.org:

http://www.scipy.org/scipylib/building/windows.html#building-scipy

Some info about my configuration:

site.cfg:

include_dirs = C:\Program Files (x86)\Intel\Composer XE 2015\mkl\include

library_dirs = C:\Program Files (x86)\Intel\Composer XE 2015\mkl\lib\intel64

mkl_libs = mkl_rt

PATH = (paths separated by line for easy reading)

C:\Program Files\Side Effects Software\Houdini 13.0.509\python27;

C:\Program Files\Side Effects Software\Houdini 13.0.509\python27\Scripts;

C:\Program Files (x86)\Microsoft Visual Studio 11.0\VC\bin\x86_amd64;

C:\Program Files (x86)\Intel\Composer XE 2015\bin\intel64

LD_LIBRARY_PATH =

C:\Program Files (x86)\Microsoft Visual Studio 11.0\VC\bin\x86_amd64;

C:\Program Files (x86)\Intel\Composer XE 2015\bin\intel64;

C:\Program Files (x86)\Intel\Composer XE 2015\mkl\lib\intel64;

C:\Program Files (x86)\Intel\Composer XE 2015\compiler\lib\intel64

Thank you in advance for your time,  

-Jay

PS - Please note that I've cross-posted this message to the numpy-discussion mailing list as well.  I will post any useful information I receive here.

=====

*why am I doing this? 

The reason I’m doing this is because I need numpy with MKL to run with the version of python that comes packaged with Houdini (Python 2.7.5 (default, Oct 24 2013, 17:49:49) [MSC v.1700 64 bit (AMD64)] on win32).

So, downloading a prebuilt 64 bit numpy isn’t an option due to the unavailability of a compatible compiler version.

[Zhang_Z_Intel]

Jay,

Thanks for sharing your experience. Building numpy/scipy using Visual C++ is no easy task. So far I haven't seen a good and smooth solution. If anyone on this forum has a success story, please share with us.

The easiest way of using MKL with numpy/scipy on Windows is grabbing the Anaconda python distribution (free) with the MKL optimization add-on ($29). See more here: http://continuum.io/anaconda-addons#mkl

If you insist on building it from scratch yourself then it's probably easier to install a Unix-like environment (e.g. cygwin) on Windows and do it from there.

If you don't see multiple threads in your testing, check your matrix size. You'll need a reasonable size (for example, a few hundreds elements on both dimensions) to see the benefit of MKL threading. Also check your environment settings, in particular MKL_NUM_THREADS (or, OMP_NUM_THREADS), and MKL_THREADING_LAYER (the value should be INTEL).

[TimP]

In cygwin, it seems you would still have the problem that the numpy which comes with it would expect you to use the cygwin gcc compilers, not easily interfaced with MKL.

[Ying_H_Intel]

Dear all, 

Just for your reference, we investigate the way to building them on windows and published in the article. 

https://software.intel.com/en-us/articles/building-numpyscipy-with-intel-mkl-and-intel-fortran-on-windows

Best Regards,
Ying