numpy.dot not multi-threading

Geary__Robert · ‎09-05-2018

I installed Intel's Python distribution on my i9 7980XE running Windows 10 because I was curious to see how it performed compared to Python 3.7 with pip-installed numpy, particularly with dot products.

I first uninstalled Python 3.7 and then installed Intel's Python. I then ran the script

import numpy as np
A = np.random.rand(30000,30000)
B = np.dot(A, A)

and found that it runs far slower than under Python 3.7. This seems to be the result of Intel's Python not multi-threading the dot product because ASUS CAM shows my CPU only reaching about 6% utilization, whereas when run under Python 3.7, my CPU runs at 100% during the entire dot product calculation.

How do I get Intel's Python to multi-thread the dot product?

Thanks

Oleksandr_P_Intel · ‎09-06-2018

Hi,

Could you please provide the output of conda list --explicit to make sure that the Intel Python environment has not been inadvertently altered.

Also, please inspect np.__config__.show() to make sure that NumPy has been configured to use MKL.

If so, please set MKL_VERBOSE=1 environment variable, execute your script, and provide the output. The output will show MKL version, as well as the number of threads used, function executed, and dimensions.

Thank you,
Oleksandr

Geary__Robert · ‎09-06-2018

Geary__Robert · ‎09-06-2018

>>> np.__config__.show()
mkl_info:
libraries = ['mkl_rt']
library_dirs = ['.\\Library\\lib']
define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
include_dirs = ['.\\Library\\include']
blas_mkl_info:
libraries = ['mkl_rt']
library_dirs = ['.\\Library\\lib']
define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
include_dirs = ['.\\Library\\include']
blas_opt_info:
libraries = ['mkl_rt']
library_dirs = ['.\\Library\\lib']
define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
include_dirs = ['.\\Library\\include']
lapack_mkl_info:
libraries = ['mkl_rt']
library_dirs = ['.\\Library\\lib']
define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
include_dirs = ['.\\Library\\include']
lapack_opt_info:
libraries = ['mkl_rt']
library_dirs = ['.\\Library\\lib']
define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
include_dirs = ['.\\Library\\include']

Geary__Robert · ‎09-06-2018

I inserted

import os
os.environ["MKL_VERBOSE"] = "1"

at the beginning of my script but don't get any feedback when I run it. So it seems MKL is not being used.

Oleksandr_P_Intel · ‎09-06-2018

Ok, thank you for these response. Here is what I get:

(idp3) C:\Users\me\devel>ipython
Python 3.6.2 |Intel Corporation| (default, Aug 15 2017, 11:34:02) [MSC v.1900 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 6.1.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import os

In [2]: os.environ['MKL_VERBOSE']="1"

In [3]: import numpy as np

In [4]: x = np.random.randn(1000)

In [5]: np.dot(x,x)
MKL_VERBOSE Intel(R) MKL 2018.0 Product build 20170720 for Intel(R) 64 architecture Intel(R) Advanced Vector Extensions 2 (Intel(R) AVX2) enabled processors, Win 2.60GHz cdecl intel_thread NMICDev:0
MKL_VERBOSE DDOT(1000,000001D258374D20,1,000001D258374D20,1) 1.93ms CNR:OFF Dyn:1 FastMM:1 TID:0  NThr:2 WDiv:HOST:+0.000
Out[5]: 928.24284740935025

In [6]:

Could you please try the following, to verify that MKL does not fail to load:

# coding: utf-8
import ctypes
mkl = ctypes.cdll.LoadLibrary('mkl_rt.dll')

# See https://software.intel.com/en-us/mkl-developer-reference-c-mkl-get-version-string
fn = mkl.mkl_get_version_string
fn.argtypes = (ctypes.POINTER(ctypes.c_char), ctypes.c_int)
p = ctypes.create_string_buffer(198)
fn(p, ctypes.sizeof(p))

# Output version string
print(p.value.decode('ascii'))

Geary__Robert · ‎09-06-2018

That code prints

Intel(R) Math Kernel Library Version 2018.0.3 Product Build 20180406 for Intel(R) 64 architecture applications

Oleksandr_P_Intel · ‎09-06-2018

Thank you for trying that and sorry for the trouble. The execution seem to confirm that MKL library can be loaded just fine, but for some reason fails to be called from numpy.dot.

I concocted a small script that calls DDOT manually, and then through numpy.dot

import ctypes, os
os.environ['MKL_VERBOSE']='1'
mkl = ctypes.cdll.LoadLibrary('mkl_rt.dll')
import numpy as np
x = np.random.randn(100)
ddot = mkl.cblas_ddot
ddot.argtypes = (ctypes.c_int, ctypes.c_void_p, ctypes.c_int, ctypes.c_void_p, ctypes.c_int)
ddot.restype = ctypes.c_double
print("DDOT result = {}".format(ddot(x.shape[0], x.ctypes.data_as(ctypes.POINTER(ctypes.c_double)), 1, x.ctypes.data_as(ctypes.POINTER(ctypes.c_double)), 1)))
print("np.dot result = {}".format(np.dot(x,x)))

When I execute it on my dated Win-64 installation, I see the following output:

MKL_VERBOSE Intel(R) MKL 2018.0 Product build 20170720 for Intel(R) 64 architecture Intel(R) Advanced Vector Extensions 2 (Intel(R) AVX2) enabled processors, Win 2.60GHz lp64 intel_thread NMICDev:0
MKL_VERBOSE DDOT(100,00000175A0048180,1,00000175A0048180,1) 3.32ms CNR:OFF Dyn:1 FastMM:1 TID:0  NThr:2 WDiv:HOST:+0.000
DDOT result = 90.116370539625
MKL_VERBOSE DDOT(100,00000175A0048180,1,00000175A0048180,1) 3.76us CNR:OFF Dyn:1 FastMM:1 TID:0  NThr:2 WDiv:HOST:+0.000
np.dot result = 90.116370539625

The exact value of the result can vary from run to run as I neglected to fix the seed.

The output of this experiment should confirm that numpy.dot fails to use BLAS and falls back on generic sequential loops if the second MKL_VERBOSE is not echoed to the terminal.

Please run that script.

I would suggest for you to create an alternative environment with minimal IDP installation as follows:

conda create -n idp_core -c intel  intelpython3_core ipython

Then activate that environment and check whether problems persists.

Thank you,
Oleksandr

Geary__Robert · ‎09-06-2018

I get this output:

MKL_VERBOSE Intel(R) MKL 2018.0 Update 3 Product build 20180406 for Intel(R) 64 architecture Intel(R) Advanced Vector Extensions 512 (Intel(R) AVX-512) enabled processors, Win 2.60GHz lp64 intel_thread
MKL_VERBOSE DDOT(100,000001524BB5B7A0,1,000001524BB5B7A0,1) 710.75us CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:18
DDOT result = 96.64738197234854
MKL_VERBOSE DDOT(100,000001524BB5B7A0,1,000001524BB5B7A0,1) 686ns CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:18
np.dot result = 96.64738197234854

Geary__Robert · ‎09-06-2018

I've set up the core environment as you suggested, but I still only get single-threaded performance for my original matrix multiplication.

Oleksandr_P_Intel · ‎09-06-2018

We established that numpy.dot actualy does call into MKL BLAS function.

Could you then please append your benchmark to the end of the script:

import ctypes, os
os.environ['MKL_VERBOSE']='1'
mkl = ctypes.cdll.LoadLibrary('mkl_rt.dll')
import numpy as np
x = np.random.randn(100)
ddot = mkl.cblas_ddot
ddot.argtypes = (ctypes.c_int, ctypes.c_void_p, ctypes.c_int, ctypes.c_void_p, ctypes.c_int)
ddot.restype = ctypes.c_double
print("DDOT result = {}".format(ddot(x.shape[0], x.ctypes.data_as(ctypes.POINTER(ctypes.c_double)), 1, x.ctypes.data_as(ctypes.POINTER(ctypes.c_double)), 1)))
print("np.dot result = {}".format(np.dot(x,x)))

n = 30*1000
A = np.random.randn(n, n)

import timeit
t0 = timeit.default_timer() 
B = np.dot(A, A)
t1 = timeit.default_timer()
print("Time {}".format(t1-t0))

You should be able to see all threads being used.

Sampling of 9*10**8 Gaussian variates may take longish time with numpy.random. You can try replacing np.random.randn sampling with numpy.random_intel.randn (present in IDP), in which case faster MKL sampling functions will be used.

Geary__Robert · ‎09-07-2018

I have a new problem now: As soon as the matrix A is done populating, my PC crashes. No BSOD, just a chirp from my motherboard and then power off.

My CPU is overclocked (all cores at 4.2GHz), but it's been stable for months. That doesn't mean it will be stable on everything, but I've never seen an unstable overclock not produce a BSOD in Windows 10 with Skylake.

Geary__Robert · ‎09-07-2018

I've ruled out an unstable overclock: With all BIOS settings restored to defaults, I get exactly the same black screen of death when the script begins the dot product.

Running out of RAM is not a problem: I have 128GB.

Oleksandr_P_Intel · ‎09-07-2018

Thank for trying it out and reporting the subsequent issue.

I think it is worthwhile to check if setting os.environ['MKL_ENABLE_ARCHITECTURE'] = 'AVX2' atop the script helps with the crash. This will instruct MKL to use AVX2 instruction set instead of AVX512.

It's also useful to try setting os.environ['MKL_THREADING_LAYER'] = 'TBB' , and see it that helps with the crash. This will replace use of Intel OpenMP with use of Intel TBB.

If it doesn't, I'd suggest to start with a smaller matrix size, let's say, 1k by 1k, and gradually increase the size to identify the size triggering the crash, so that this issue could not taken to the MKL engineering team.

Thank you,
Oleksandr

Geary__Robert · ‎09-07-2018

I tried all three of your suggestions at the same time (os.environ['MKL_ENABLE_ARCHITECTURE'] = 'AVX2' , os.environ['MKL_THREADING_LAYER'] = 'TBB' , and a 1000x1000 matrix) and it still crashes my PC.

Oleksandr_P_Intel · ‎09-07-2018

It is my bad, I should have checked better. My suggestion should have been

# https://software.intel.com/en-us/mkl-linux-developer-guide-instruction-set-specific-dispatching-on-intel-architectures
os.environ['MKL_ENABLE_INSTRUCTIONS'] = 'AVX2'

Regardless of whether this helps, please try lowering matrix size by binary search until the machine stops crashing. It would help MKL developers identify the problem behind the issues you are experiencing.

Thank you,
Oleksandr

Geary__Robert · ‎09-07-2018

Sorry, I've already uninstalled Intel's Python Distribution. I can't afford to play Russian roulette with my hard drives any more.

I can tell you that my system has not had any problems with AVX512 instructions in other software.

Oleksandr_P_Intel · ‎09-07-2018

Thank you for taking the time. I will try to reproduce the issue in house,

Oleksandr