topic Re: Cannot build numpy from source with intel c++ and fortran compilers in Intel® Distribution for Python*

Cannot build numpy from source with intel c++ and fortran compilers

MattHouston — Sun, 10 Apr 2022 19:53:28 GMT

I followed the section "Windows platforms" from https://www.intel.com/content/www/us/en/developer/articles/technical/build-numpy-with-mkl-and-icc.html

It didn't work. I tried to inspire myself from https://gehrcke.de/2014/02/building-numpy-and-scipy-with-intel-compilers-and-intel-mkl-on-a-64-bit-machine/ and https://christoph-conrads.name/building-numpy-and-scipy-with-intel-compilers-and-intel-mkl/ without success.

Concretly I have I am using Python 3.8.10 64bits under windows 11 64bits last update with intel One Api Base and HPC kit 2021.3.0.3221.

I have the following site.cfg file :

[mkl]
include_dirs = C:\Program Files (x86)\Intel\oneAPI\mkl\2021.3.0\include
library_dirs = C:\Program Files (x86)\Intel\oneAPI\mkl\2021.3.0\lib\intel64
libraries = mkl_rt

and I modified the intelcompiler.py as follows :

and the intel.py file (for the fortran compiler) as follows :

I compiled with the following command :

python setup.py config --compiler=intelemw --fcompiler=intelvem build_clib --compiler=intelemw --fcompiler=intelvem build_ext --compiler=intelemw --fcompiler=intelvem

I have then the following errors (see log attached as well) :

C:\CODING\OTHERS\NUMPY\numpy>python setup.py config --compiler=intelemw --fcompiler=intelvem build_clib --compiler=intelemw --fcompiler=intelvem build_ext --compiler=intelemw --fcompiler=intelvem
Running from numpy source directory.
setup.py:86: DeprecationWarning:

`numpy.distutils` is deprecated since NumPy 1.23.0, as a result
of the deprecation of `distutils` itself. It will be removed for
Python >= 3.12. For older Python versions it will remain present.
It is recommended to use `setuptools < 60.0` for those Python versions.
For more details, see:
https://numpy.org/devdocs/reference/distutils_status_migration.html

import numpy.distutils.command.sdist
Cythonizing sources
numpy/random\_bounded_integers.pxd.in has not changed
numpy/random\bit_generator.pyx has not changed
numpy/random\mtrand.pyx has not changed
numpy/random\_bounded_integers.pyx has not changed
numpy/random\_bounded_integers.pyx.in has not changed
numpy/random\_common.pyx has not changed
numpy/random\_generator.pyx has not changed
numpy/random\_mt19937.pyx has not changed
numpy/random\_pcg64.pyx has not changed
numpy/random\_philox.pyx has not changed
numpy/random\_sfc64.pyx has not changed
C:\CODING\OTHERS\NUMPY\numpy\numpy\core\setup.py:484: MismatchCAPIWarning: API mismatch detected, the C API version numbers have to be updated. Current C api version is 15, with checksum 04a7bf1e65350926a0e528798da263c0, but recorded checksum for C API version 15 in core/codegen_dir/cversions.txt is b8783365b873681cd204be50cdfb448d. If functions were added in the C API, you have to update C_API_VERSION in C:\CODING\OTHERS\NUMPY\numpy\numpy\core\setup_common.py.
check_api_version(C_API_VERSION, codegen_dir)
INFO: blas_opt_info:
INFO: blas_armpl_info:
INFO: No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils
INFO: customize MSVCCompiler
INFO: libraries armpl_lp64_mp not found in ['C:\\PYTHON\\3\\3.8.10\\lib', 'C:\\', 'C:\\PYTHON\\3\\3.8.10\\libs']
INFO: NOT AVAILABLE
INFO:
INFO: blas_mkl_info:
INFO: FOUND:
INFO: libraries = ['mkl_rt', 'mkl_rt']
INFO: library_dirs = ['C:\\Program Files (x86)\\Intel\\oneAPI\\mkl\\2021.3.0\\lib\\intel64']
INFO: define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
INFO: include_dirs = ['C:\\Program Files (x86)\\Intel\\oneAPI\\mkl\\2021.3.0\\include']
INFO:
INFO: FOUND:
INFO: libraries = ['mkl_rt', 'mkl_rt']
INFO: library_dirs = ['C:\\Program Files (x86)\\Intel\\oneAPI\\mkl\\2021.3.0\\lib\\intel64']
INFO: define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
INFO: include_dirs = ['C:\\Program Files (x86)\\Intel\\oneAPI\\mkl\\2021.3.0\\include']
INFO:
INFO: lapack_opt_info:
INFO: lapack_armpl_info:
INFO: libraries armpl_lp64_mp not found in ['C:\\PYTHON\\3\\3.8.10\\lib', 'C:\\', 'C:\\PYTHON\\3\\3.8.10\\libs']
INFO: NOT AVAILABLE
INFO:
INFO: lapack_mkl_info:
INFO: FOUND:
INFO: libraries = ['mkl_rt', 'mkl_rt']
INFO: library_dirs = ['C:\\Program Files (x86)\\Intel\\oneAPI\\mkl\\2021.3.0\\lib\\intel64']
INFO: define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
INFO: include_dirs = ['C:\\Program Files (x86)\\Intel\\oneAPI\\mkl\\2021.3.0\\include']
INFO:
INFO: FOUND:
INFO: libraries = ['mkl_rt', 'mkl_rt']
INFO: library_dirs = ['C:\\Program Files (x86)\\Intel\\oneAPI\\mkl\\2021.3.0\\lib\\intel64']
INFO: define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
INFO: include_dirs = ['C:\\Program Files (x86)\\Intel\\oneAPI\\mkl\\2021.3.0\\include']
INFO:
Warning: attempted relative import with no known parent package
C:\PYTHON\3\3.8.10\lib\distutils\dist.py:274: UserWarning: Unknown distribution option: 'define_macros'
warnings.warn(msg)
running config
running build_clib
running build_src
INFO: build_src
INFO: building py_modules sources
INFO: building library "npymath" sources
WARN: Could not locate executable icc
WARN: Could not locate executable ecc
WARN: Could not locate executable C:\Program
MSVCRT.lib(loadcfg.obj) : error LNK2001: unresolved external symbol __enclave_config
MSVCRT.lib(loadcfg.obj) : error LNK2001: unresolved external symbol __volatile_metadata
_configtest.exe : fatal error LNK1120: 2 unresolved externals
Traceback (most recent call last):
File "setup.py", line 492, in <module>
setup_package()
File "setup.py", line 484, in setup_package
setup(**metadata)
File "C:\CODING\OTHERS\NUMPY\numpy\numpy\distutils\core.py", line 169, in setup
return old_setup(**new_attr)
File "C:\PYTHON\3\3.8.10\lib\site-packages\setuptools\__init__.py", line 153, in setup
return distutils.core.setup(**attrs)
File "C:\PYTHON\3\3.8.10\lib\distutils\core.py", line 148, in setup
dist.run_commands()
File "C:\PYTHON\3\3.8.10\lib\distutils\dist.py", line 966, in run_commands
self.run_command(cmd)
File "C:\PYTHON\3\3.8.10\lib\distutils\dist.py", line 985, in run_command
cmd_obj.run()
File "C:\CODING\OTHERS\NUMPY\numpy\numpy\distutils\command\build_clib.py", line 95, in run
self.run_command('build_src')
File "C:\PYTHON\3\3.8.10\lib\distutils\cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "C:\PYTHON\3\3.8.10\lib\distutils\dist.py", line 985, in run_command
cmd_obj.run()
File "C:\CODING\OTHERS\NUMPY\numpy\numpy\distutils\command\build_src.py", line 144, in run
self.build_sources()
File "C:\CODING\OTHERS\NUMPY\numpy\numpy\distutils\command\build_src.py", line 155, in build_sources
self.build_library_sources(*libname_info)
File "C:\CODING\OTHERS\NUMPY\numpy\numpy\distutils\command\build_src.py", line 288, in build_library_sources
sources = self.generate_sources(sources, (lib_name, build_info))
File "C:\CODING\OTHERS\NUMPY\numpy\numpy\distutils\command\build_src.py", line 378, in generate_sources
source = func(extension, build_dir)
File "C:\CODING\OTHERS\NUMPY\numpy\numpy\core\setup.py", line 757, in get_mathlib_info
raise RuntimeError(
RuntimeError: Broken toolchain: cannot link a simple C program.

Can you please help ?

I must precise that I am aware of https://community.intel.com/t5/Intel-Distribution-for-Python/Compiling-numpy-amp-scipy-with-Intel-MKL-in-windows-10/m-p/1318469 and that I don't want to use prebuilt packages. I really want to build it mysefl on my machine.

Re: Cannot build numpy from source with intel c++ and fortran compilers

Gopika_Intel — Wed, 13 Apr 2022 14:29:17 GMT

Hi,

Thank you for posting in Intel Communities.

We were unable to reproduce the issue from our end. We did not face any error with and without modifying the intelccompiler.py and intel.py.

Details of our machine

NOTE: Ensure the versions are the same

· OS: Windows 10

· Intel® MKL version used: 2020 update 3

NOTE: You need a license serial number to install any older version of Intel® oneAPI Toolkit. You must purchase a license if you do not have one.

Access the Registration Center and follow the instructions below.

Select the product

Select the OS

Specify the required component, version and click download.

· Visual studio version :2017(Visit https://visualstudio.microsoft.com/vs/older-downloads/ to download older versions)

(If you build on Windows platforms, Visual Studio 2017 and Visual C++ Build Tools 2015 are also required. When install VS2017, just select the “VC++ 2015.3 v14.00 (v140) toolset for desktop” at the bottom of the “Desktop development with C++” section)

Details needed

Could you please share with us the following details?

1. The numpy version you are using

2. The MKL version you are using

3. Can you confirm whether you are using the components installed from Base Toolkit and HPC Toolkit (2021.3.0.3221 version)

Please let us know if you still face issues even after using the same versions mentioned above

Regards

Gopika

Re: Cannot build numpy from source with intel c++ and fortran compilers

MattHouston — Wed, 13 Apr 2022 21:20:05 GMT

I am under windows 10 or 11 (I tried both) pro last updates + visual studio 2019 enterprise full install (all win 10 and 11 SDKs installed). I have intel the following version of intel parallel studio and intel one api :

IPSXE2020u4
intel one api w_BaseKit_p_2021.3.0.3221 + w_HPCKit_p_2021.3.0.3227

(Both are full installs, hence : all components of Base Toolkit and HPC Toolkit (2021.3.0.3221 version) are installed.)

(This also gives you the exact versions of my intel c/c++ and fortran compilers as well as the MKL version.)

I have official python 3.8.10 64 bits.

I don't know what do you mean with "the numpy version I am using". I indeed have the pip package numpy version 1.21.5 installed, but I am trying to build numpy from source myself, so that I just git cloned numpy repo the day I asked the question here, and it is this numpy version I am trying to build.

I must precise that in my four configurations (win 10 or 11, ipsxe2020u4 or one api 2021.3.0.3221) I have the very same error log, I just attached. The crucial bit seems to be :

MSVCRT.lib(loadcfg.obj) : error LNK2001: unresolved external symbol __enclave_config
MSVCRT.lib(loadcfg.obj) : error LNK2001: unresolved external symbol __volatile_metadata
_configtest.exe : fatal error LNK1120: 2 unresolved externals
Traceback (most recent call last):
File "setup.py", line 492, in <module>
setup_package()
File "setup.py", line 484, in setup_package
setup(**metadata)
File "C:\CODING\OTHERS\NUMPY\numpy\numpy\distutils\core.py", line 169, in setup
return old_setup(**new_attr)
File "C:\PYTHON\3\3.8.10\lib\site-packages\setuptools\__init__.py", line 153, in setup
return distutils.core.setup(**attrs)
File "C:\PYTHON\3\3.8.10\lib\distutils\core.py", line 148, in setup
dist.run_commands()
File "C:\PYTHON\3\3.8.10\lib\distutils\dist.py", line 966, in run_commands
self.run_command(cmd)
File "C:\PYTHON\3\3.8.10\lib\distutils\dist.py", line 985, in run_command
cmd_obj.run()
File "C:\CODING\OTHERS\NUMPY\numpy\numpy\distutils\command\build_clib.py", line 95, in run
self.run_command('build_src')
File "C:\PYTHON\3\3.8.10\lib\distutils\cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "C:\PYTHON\3\3.8.10\lib\distutils\dist.py", line 985, in run_command
cmd_obj.run()
File "C:\CODING\OTHERS\NUMPY\numpy\numpy\distutils\command\build_src.py", line 144, in run
self.build_sources()
File "C:\CODING\OTHERS\NUMPY\numpy\numpy\distutils\command\build_src.py", line 155, in build_sources
self.build_library_sources(*libname_info)
File "C:\CODING\OTHERS\NUMPY\numpy\numpy\distutils\command\build_src.py", line 288, in build_library_sources
sources = self.generate_sources(sources, (lib_name, build_info))
File "C:\CODING\OTHERS\NUMPY\numpy\numpy\distutils\command\build_src.py", line 378, in generate_sources
source = func(extension, build_dir)
File "C:\CODING\OTHERS\NUMPY\numpy\numpy\core\setup.py", line 757, in get_mathlib_info
raise RuntimeError(
RuntimeError: Broken toolchain: cannot link a simple C program.

(I attached the new log because I corrected the site.cfg file ; indeed, although the windows plateform section from https://www.intel.com/content/www/us/en/developer/articles/technical/build-numpy-with-mkl-and-icc.html mentions paths of the form :

library_dirs = C:\Program Files (x86)\IntelSWTools\compilers_and_libraries \windows\mkl\lib\intel64; 
include_dirs = C:\Program Files (x86)\IntelSWTools\compilers_and_libraries \windows\mkl\include

it appears that the fact they are containing spaces causes issues, hence I replaced the "C:\Program Files (x86)" bit with "C:\PROGRA~2")