Problem building 64 bit numpy using MKL and vc11 (Windows)

Jay_R_ — Fri, 03 Oct 2014 00:42:21 GMT

Hi, I’ve built numpy (1.9) 64 bit using vc11, the Intel Fortran compiler and the MKL ‘mkl_rt’ library.

*why? (see end of message for the reason, if interested)

Any advice or assistance would be greatly appreciated. If I can offer additional information, I will happily do so.

The build appears to go just fine (no errors noted), and numpy loads into python just fine as well.

(I note a warning: ### Warning: python_xerbla.c is disabled ### -- however, it doesn’t appear to be problematic?)

I have also confirmed that numpy sees the mkl blas and lapack libs.

>>> numpy.__config__.show()

lapack_opt_info:

libraries = ['mkl_lapack', 'mkl_rt']

library_dirs = ['C:\\Program Files (x86)\\Intel\\Composer XE 2015\\mkl\\lib\\intel64']

define_macros = [('SCIPY_MKL_H', None)]

include_dirs = ['C:\\Program Files (x86)\\Intel\\Composer XE 2015\\mkl\\include']

blas_opt_info:

libraries = ['mkl_rt']

library_dirs = ['C:\\Program Files (x86)\\Intel\\Composer XE 2015\\mkl\\lib\\intel64']

define_macros = [('SCIPY_MKL_H', None)]

include_dirs = ['C:\\Program Files (x86)\\Intel\\Composer XE 2015\\mkl\\include']

openblas_lapack_info:

NOT AVAILABLE

lapack_mkl_info:

libraries = ['mkl_lapack', 'mkl_rt']

library_dirs = ['C:\\Program Files (x86)\\Intel\\Composer XE 2015\\mkl\\lib\\intel64']

define_macros = [('SCIPY_MKL_H', None)]

include_dirs = ['C:\\Program Files (x86)\\Intel\\Composer XE 2015\\mkl\\include']

blas_mkl_info:

libraries = ['mkl_rt']

library_dirs = ['C:\\Program Files (x86)\\Intel\\Composer XE 2015\\mkl\\lib\\intel64']

define_macros = [('SCIPY_MKL_H', None)]

include_dirs = ['C:\\Program Files (x86)\\Intel\\Composer XE 2015\\mkl\\include']

mkl_info:

libraries = ['mkl_rt']

library_dirs = ['C:\\Program Files (x86)\\Intel\\Composer XE 2015\\mkl\\lib\\intel64']

define_macros = [('SCIPY_MKL_H', None)]

include_dirs = ['C:\\Program Files (x86)\\Intel\\Composer XE 2015\\mkl\\include']

Everything *looks* to be in order upon casual inspection (*I think*, please correct me if I’m wrong!)

However, there is no performance boost when running a few different tests in numpy (singular value decomposition, for example), and only a single thread appears to be in play.

Running numpy.test(‘full’) reveals 21 errors.

For instance,

LINK : fatal error LNK1104: cannot open file 'ifconsol.lib'

And, the other being a recurring error with f2py,

ERROR: test_size.TestSizeSumExample.test_transpose

----------------------------------------------------------------------

Traceback (most recent call last):

File "C:\Program Files\Side Effects Software\Houdini 13.0.509\python27\lib\site-packages\nose\case.py", line 371, in setUp

try_run(self.inst, ('setup', 'setUp'))

File "C:\Program Files\Side Effects Software\Houdini 13.0.509\python27\lib\site-packages\nose\util.py", line 478, in try_run

return func()

File "C:\Program Files\Side Effects Software\Houdini 13.0.509\python27\lib\site-packages\numpy\f2py\tests\util.py", line 353, in setUp

module_name=self.module_name)

File "C:\Program Files\Side Effects Software\Houdini 13.0.509\python27\lib\site-packages\numpy\f2py\tests\util.py", line 80, in wrapper

raise ret

RuntimeError: Running f2py failed: ['-m', '_test_ext_module_5403', 'c:\\users\\jareyn~1\\appdata\\local\\temp\\tmpvykewl\\foo.f90']

Reading .f2py_f2cmap ...

Mapping "real(kind=rk)" to "double"

Succesfully applied user defined changes from .f2py_f2cmap

Everything that requires configuration appears to be in agreement with this Intel Application Note, minus use of the Intel C++ compiler:

https://software.intel.com/en-us/articles/numpyscipy-with-intel-mkl

I have also referenced the Windows build docs on scipy.org:

http://www.scipy.org/scipylib/building/windows.html#building-scipy

Some info about my configuration:

site.cfg:

include_dirs = C:\Program Files (x86)\Intel\Composer XE 2015\mkl\include

library_dirs = C:\Program Files (x86)\Intel\Composer XE 2015\mkl\lib\intel64

mkl_libs = mkl_rt

PATH = (paths separated by line for easy reading)

C:\Program Files\Side Effects Software\Houdini 13.0.509\python27;

C:\Program Files\Side Effects Software\Houdini 13.0.509\python27\Scripts;

C:\Program Files (x86)\Microsoft Visual Studio 11.0\VC\bin\x86_amd64;

C:\Program Files (x86)\Intel\Composer XE 2015\bin\intel64

LD_LIBRARY_PATH =

C:\Program Files (x86)\Microsoft Visual Studio 11.0\VC\bin\x86_amd64;

C:\Program Files (x86)\Intel\Composer XE 2015\bin\intel64;

C:\Program Files (x86)\Intel\Composer XE 2015\mkl\lib\intel64;

C:\Program Files (x86)\Intel\Composer XE 2015\compiler\lib\intel64

Thank you in advance for your time,

-Jay

PS - Please note that I've cross-posted this message to the numpy-discussion mailing list as well. I will post any useful information I receive here.

=====

*why am I doing this?

The reason I’m doing this is because I need numpy with MKL to run with the version of python that comes packaged with Houdini (Python 2.7.5 (default, Oct 24 2013, 17:49:49) [MSC v.1700 64 bit (AMD64)] on win32).

So, downloading a prebuilt 64 bit numpy isn’t an option due to the unavailability of a compatible compiler version.

Jay,

Zhang_Z_Intel — Fri, 03 Oct 2014 16:52:00 GMT

Jay,

Thanks for sharing your experience. Building numpy/scipy using Visual C++ is no easy task. So far I haven't seen a good and smooth solution. If anyone on this forum has a success story, please share with us.

The easiest way of using MKL with numpy/scipy on Windows is grabbing the Anaconda python distribution (free) with the MKL optimization add-on ($29). See more here: http://continuum.io/anaconda-addons#mkl

If you insist on building it from scratch yourself then it's probably easier to install a Unix-like environment (e.g. cygwin) on Windows and do it from there.

If you don't see multiple threads in your testing, check your matrix size. You'll need a reasonable size (for example, a few hundreds elements on both dimensions) to see the benefit of MKL threading. Also check your environment settings, in particular MKL_NUM_THREADS (or, OMP_NUM_THREADS), and MKL_THREADING_LAYER (the value should be INTEL).

In cygwin, it seems you would

TimP — Fri, 03 Oct 2014 17:01:11 GMT

In cygwin, it seems you would still have the problem that the numpy which comes with it would expect you to use the cygwin gcc compilers, not easily interfaced with MKL.

Dear all,

Ying_H_Intel — Tue, 28 Jul 2015 02:52:02 GMT

Dear all,

Just for your reference, we investigate the way to building them on windows and published in the article.

https://software.intel.com/en-us/articles/building-numpyscipy-with-intel-mkl-and-intel-fortran-on-windows

Best Regards,
Ying

topic Problem building 64 bit numpy using MKL and vc11 (Windows) in Intel® oneAPI Math Kernel Library

Problem building 64 bit numpy using MKL and vc11 (Windows)

Jay,

In cygwin, it seems you would

Dear all,