How to build and profile numpy with Intel Tool Chain for performance profiling

[anderson__erik]

Short version:
How to build, from source, numpy so I can use Intel VTune to profile 
_multiarray_umath.cp36-win_amd64.pyd

Long version:
I am using Intel Parallel Studio XE 2019 Update 5, with the parallel installed Intel Python 3.6

We have a massively parallel calculation system using Numpy all using the Intel Python. Intel VTune shows lots of time is spent in the DLL 

C:\Program Files (x86)\IntelSWTools\intelpython3\Lib\site-packages\numpy\core\_multiarray_umath.cp36-win_amd64.pyd

I believe this .pyd/dll is here. https://github.com/numpy/numpy/tree/master/numpy/core/src/multiarray
  Release: https://github.com/numpy/numpy/releases/tag/v1.17.0

1) Doesnt Intel have a Symbol server (.pdf files) I can point VTune at to show me the symbols? The .pdb files for NymPy/MKD are NOT installed as part of the Intel Parallel Studio, I searched. Strange that Intel doesnt put the Intel Symbol server instructions here.
   https://software.intel.com/en-us/vtune-amplifier-help-debug-information-for-windows-system-libraries
2) If there is no .pdb sybmols, how do I compile NumPy using the Intel Parallel Studio so I get local debugging symbols (.pyd files)

Erik Anderson
Bloomberg

[Oleksandr_P_Intel]

It should not be very complicated.

1. Activate PSXE environment which should activate Intel Python within PSXE.
2. Create a conda environment for building NumPy by running "conda create -n build_numpy --offline cython setuptools mkl-devel python=3.6". The intent is to reuse the packages included in the PSXE.
3. Activate that environment: "conda activate build_numpy"
3. Clone sources for NumPy in IDP: "git clone https://github.com/IntelPython/numpy.git ; numpy_src -b intel/1.17.0"
4. Change into that folder: "cd numpy_src"
5. Create site.cfg file which looks like 
 

[mkl]
include_dirs = @PREFIX@\Library\include
library_dirs = @PREFIX@\Library\lib
lapack_libs = mkl_rt
mkl_libs = mkl_rt

where the token "@PREFIX@" is replaced with the value of environment variable "%CONDA_PREFIX%"

6. Execute
 

set CC=icl
set LD=xilink

rem here add the flag to generate PDB files, such as "/Zi" 
set CFLAGS=/Qstd=c99 -DNDEBUG -I%CONDA_PREFIX%\Library\include /Zi %CFLAGS%

python setup.py config_cc -compiler=intelemw build_ext --inplace

7. Once complete, the PDB file can be revealed with "dir *.pdb".

[anderson__erik]

Oleksandr

Oleksandr P. (Intel) wrote:

1. Activate PSXE environment which should activate Intel Python within PSXE.

I assume that is
C:\Program Files (x86)\IntelSWTools\parallel_studio_xe_2019.5.068\bin>psxevars.bat intel64

Oleksandr P. (Intel) wrote:

2. Create a conda environment for building NumPy by running "conda create -n build_numpy --offline cython setuptools mkl-devel python=3.6". The intent is to reuse the packages included in the PSXE.

This gives an error.

C:\projects\numpy_build>conda create -n build_numpy --offline cython setuptools mkl-devel python=3.6
Fetching package metadata .................

PackageNotFoundError: Packages missing in current channels:

  - cython

We have searched for the packages in the following channels:

  - file:///C:/Program%20Files%20%28x86%29/IntelSWTools/conda_channel/win-64
  - file:///C:/Program%20Files%20%28x86%29/IntelSWTools/conda_channel/noarch
  - https://conda.anaconda.org/intel/win-64
  - https://conda.anaconda.org/intel/noarch
  - https://repo.continuum.io/pkgs/main/win-64
  - https://repo.continuum.io/pkgs/main/noarch
  - https://repo.continuum.io/pkgs/free/win-64
  - https://repo.continuum.io/pkgs/free/noarch
  - https://repo.continuum.io/pkgs/r/win-64
  - https://repo.continuum.io/pkgs/r/noarch
  - https://repo.continuum.io/pkgs/pro/win-64
  - https://repo.continuum.io/pkgs/pro/noarch
  - https://repo.continuum.io/pkgs/msys2/win-64
  - https://repo.continuum.io/pkgs/msys2/noarch

*****

If I do a
"conda install cython" that also gives an error.

...

The following NEW packages will be INSTALLED:
    tqdm:                   4.36.1-py_0

...

tqdm-4.36.1-py 100% |###############################| Time: 0:00:00   3.21 MB/s
ERROR conda.core.link:_execute_actions(339): An error occurred while installing package 'defaults::tqdm-4.36.1-py_0'.
CondaError: Cannot link a source that does not exist. C:\Program Files (x86)\IntelSWTools\intelpython3\Scripts\conda.exe
Running `conda clean --packages` may resolve your problem.
Attempting to roll back.

CondaError: Cannot link a source that does not exist. C:\Program Files (x86)\IntelSWTools\intelpython3\Scripts\conda.exe
Running `conda clean --packages` may resolve your problem.

[Oleksandr_P_Intel]

Please confirm that the "cython" conda -ball is indeed in the conda_channel by running

dir "%PROGRAMFILES%\IntelSWTools\conda_channel\win-64\cython*"

It should be.

The reason why you running into an issue when executing "conda install cython" has to do with conda trying to update the version of conda while it is using it. The workaround is to find out which version of conda is installed in your environment:

conda list conda

In the output, pay locate the version, and the build number, and then try "conda install cython conda=conda_version=conda_build" indicating conda that you do not want to upgrade the conda itself. 

You can try creating the build_numpy environment from online channel:

conda create -b build_numpy -c intel --override-channels cython setuptools mkl-devel=2019.5 python=3.6

[anderson__erik]

Few issues that I had to address but it compiled. There are NO PDB files though.

I see the 
icl.exe ..... /Zi   for the debug compiling but when I look at the linker line I dont see any /PDB blah blah.

C:\Program Files (x86)\IntelSWTools\parallel_studio_xe_2019.5.068\compilers_and_libraries_2019\windows\bin\intel64\xilink.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:C:\Users\AndersonErik\AppData\Local\conda\conda\envs\build_numpy\Library\lib /LIBPATH:C:\Users\AndersonErik\AppData\Local\conda\conda\envs\build_numpy\libs /LIBPATH:C:\Users\AndersonErik\AppData\Local\conda\conda\envs\build_numpy\PCbuild\amd64 /LIBPATH:build\temp.win-amd64-3.6 loops.lib npymath.lib npysort.lib mkl_cp.lib mkl_rt.lib npymath.lib python36.lib /EXPORT:PyInit__multiarray_umath 
.....obj(s)
 /OUT:numpy\core\_multiarray_umath.cp36-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.6\Release\numpy\core\src\multiarray\_multiarray_umath.cp36-win_amd64.lib /MANIFEST /MANIFESTFILE:build\temp.win-amd64-3.6\Release\numpy\core\src\multiarray\_multiarray_umath.cp36-win_amd64.pyd.manifest
building 'numpy.core._umath_tests' extension

Usually the linker respects the compiler but in this case the xilink.exe isnt doing the right thing.

Are we missing a

set LDFLAGS=blah

I attached the compile and link log to this post.

Erik Anderson
Bloomberg

[Oleksandr_P_Intel]

Likely we are missing

set LDFLAGS=/DEBUG

per https://docs.microsoft.com/en-us/cpp/build/reference/debug-generate-debug-info?view=vs-2019

At the end of the compilation you should find "vc140.pdb" in the numpy_src/ folder.