- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Short version:
How to build, from source, numpy so I can use Intel VTune to profile
_multiarray_umath.cp36-win_amd64.pyd
Long version:
I am using Intel Parallel Studio XE 2019 Update 5, with the parallel installed Intel Python 3.6
We have a massively parallel calculation system using Numpy all using the Intel Python. Intel VTune shows lots of time is spent in the DLL
C:\Program Files (x86)\IntelSWTools\intelpython3\Lib\site-packages\numpy\core\_multiarray_umath.cp36-win_amd64.pyd
I believe this .pyd/dll is here. https://github.com/numpy/numpy/tree/master/numpy/core/src/multiarray
Release: https://github.com/numpy/numpy/releases/tag/v1.17.0
1) Doesnt Intel have a Symbol server (.pdf files) I can point VTune at to show me the symbols? The .pdb files for NymPy/MKD are NOT installed as part of the Intel Parallel Studio, I searched. Strange that Intel doesnt put the Intel Symbol server instructions here.
https://software.intel.com/en-us/vtune-amplifier-help-debug-information-for-windows-system-libraries
2) If there is no .pdb sybmols, how do I compile NumPy using the Intel Parallel Studio so I get local debugging symbols (.pyd files)
Erik Anderson
Bloomberg
Link Copied
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
It should not be very complicated.
1. Activate PSXE environment which should activate Intel Python within PSXE.
2. Create a conda environment for building NumPy by running "conda create -n build_numpy --offline cython setuptools mkl-devel python=3.6". The intent is to reuse the packages included in the PSXE.
3. Activate that environment: "conda activate build_numpy"
3. Clone sources for NumPy in IDP: "git clone https://github.com/IntelPython/numpy.git ; numpy_src -b intel/1.17.0"
4. Change into that folder: "cd numpy_src"
5. Create site.cfg file which looks like
[mkl] include_dirs = @PREFIX@\Library\include library_dirs = @PREFIX@\Library\lib lapack_libs = mkl_rt mkl_libs = mkl_rt
where the token "@PREFIX@" is replaced with the value of environment variable "%CONDA_PREFIX%"
6. Execute
set CC=icl set LD=xilink rem here add the flag to generate PDB files, such as "/Zi" set CFLAGS=/Qstd=c99 -DNDEBUG -I%CONDA_PREFIX%\Library\include /Zi %CFLAGS% python setup.py config_cc -compiler=intelemw build_ext --inplace
7. Once complete, the PDB file can be revealed with "dir *.pdb".
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Oleksandr
Oleksandr P. (Intel) wrote:
1. Activate PSXE environment which should activate Intel Python within PSXE.
I assume that is
C:\Program Files (x86)\IntelSWTools\parallel_studio_xe_2019.5.068\bin>psxevars.bat intel64
Oleksandr P. (Intel) wrote:
2. Create a conda environment for building NumPy by running "conda create -n build_numpy --offline cython setuptools mkl-devel python=3.6". The intent is to reuse the packages included in the PSXE.
This gives an error.
C:\projects\numpy_build>conda create -n build_numpy --offline cython setuptools mkl-devel python=3.6
Fetching package metadata .................
PackageNotFoundError: Packages missing in current channels:
- cython
We have searched for the packages in the following channels:
- file:///C:/Program%20Files%20%28x86%29/IntelSWTools/conda_channel/win-64
- file:///C:/Program%20Files%20%28x86%29/IntelSWTools/conda_channel/noarch
- https://conda.anaconda.org/intel/win-64
- https://conda.anaconda.org/intel/noarch
- https://repo.continuum.io/pkgs/main/win-64
- https://repo.continuum.io/pkgs/main/noarch
- https://repo.continuum.io/pkgs/free/win-64
- https://repo.continuum.io/pkgs/free/noarch
- https://repo.continuum.io/pkgs/r/win-64
- https://repo.continuum.io/pkgs/r/noarch
- https://repo.continuum.io/pkgs/pro/win-64
- https://repo.continuum.io/pkgs/pro/noarch
- https://repo.continuum.io/pkgs/msys2/win-64
- https://repo.continuum.io/pkgs/msys2/noarch
*****
If I do a
"conda install cython" that also gives an error.
...
The following NEW packages will be INSTALLED:
tqdm: 4.36.1-py_0
...
tqdm-4.36.1-py 100% |###############################| Time: 0:00:00 3.21 MB/s
ERROR conda.core.link:_execute_actions(339): An error occurred while installing package 'defaults::tqdm-4.36.1-py_0'.
CondaError: Cannot link a source that does not exist. C:\Program Files (x86)\IntelSWTools\intelpython3\Scripts\conda.exe
Running `conda clean --packages` may resolve your problem.
Attempting to roll back.
CondaError: Cannot link a source that does not exist. C:\Program Files (x86)\IntelSWTools\intelpython3\Scripts\conda.exe
Running `conda clean --packages` may resolve your problem.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Please confirm that the "cython" conda -ball is indeed in the conda_channel by running
dir "%PROGRAMFILES%\IntelSWTools\conda_channel\win-64\cython*"
It should be.
The reason why you running into an issue when executing "conda install cython" has to do with conda trying to update the version of conda while it is using it. The workaround is to find out which version of conda is installed in your environment:
conda list conda
In the output, pay locate the version, and the build number, and then try "conda install cython conda=conda_version=conda_build" indicating conda that you do not want to upgrade the conda itself.
You can try creating the build_numpy environment from online channel:
conda create -b build_numpy -c intel --override-channels cython setuptools mkl-devel=2019.5 python=3.6
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Few issues that I had to address but it compiled. There are NO PDB files though.
I see the
icl.exe ..... /Zi for the debug compiling but when I look at the linker line I dont see any /PDB blah blah.
C:\Program Files (x86)\IntelSWTools\parallel_studio_xe_2019.5.068\compilers_and_libraries_2019\windows\bin\intel64\xilink.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:C:\Users\AndersonErik\AppData\Local\conda\conda\envs\build_numpy\Library\lib /LIBPATH:C:\Users\AndersonErik\AppData\Local\conda\conda\envs\build_numpy\libs /LIBPATH:C:\Users\AndersonErik\AppData\Local\conda\conda\envs\build_numpy\PCbuild\amd64 /LIBPATH:build\temp.win-amd64-3.6 loops.lib npymath.lib npysort.lib mkl_cp.lib mkl_rt.lib npymath.lib python36.lib /EXPORT:PyInit__multiarray_umath
.....obj(s)
/OUT:numpy\core\_multiarray_umath.cp36-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.6\Release\numpy\core\src\multiarray\_multiarray_umath.cp36-win_amd64.lib /MANIFEST /MANIFESTFILE:build\temp.win-amd64-3.6\Release\numpy\core\src\multiarray\_multiarray_umath.cp36-win_amd64.pyd.manifest
building 'numpy.core._umath_tests' extension
Usually the linker respects the compiler but in this case the xilink.exe isnt doing the right thing.
Are we missing a
set LDFLAGS=blah
I attached the compile and link log to this post.
Erik Anderson
Bloomberg
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Likely we are missing
set LDFLAGS=/DEBUG
per https://docs.microsoft.com/en-us/cpp/build/reference/debug-generate-debug-info?view=vs-2019
At the end of the compilation you should find "vc140.pdb" in the numpy_src/ folder.

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page