MKL > 2018.3 doesn't work with NVIDIA nvblas

[kkamm1]

Platform is Ubuntu 16.04.5. GPU NVIDIA1070 cuda-9.2 python3.5

I use python with nvblas support by compiling numpy against intel MKL. It works with 2018.1 and 2018.2 but GPU is not used for MKL 2018.3 and 2019.0

Here is the experiment (have tried with different versions of numpy same issue)

LD_PRELOAD=/usr/local/cuda/targets/x86_64-linux/lib/libnvblas.so python3

NVBLAS_CONFIG_FILE environment variable is set to '/home/bernard/.config/nvblas.conf'

Python 3.5.2 (default, Nov 23 2017, 16:37:01)

[GCC 5.4.0 20160609] on linux

Type "help", "copyright", "credits" or "license" for more information.

>>> import numpy as np

>>> np.__version__ '1.15.2'

>>> np.show_config()

lapack_opt_info: define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]

library_dirs = ['/opt/intel/mkl/lib/intel64/']

include_dirs = ['/opt/intel/mkl/include']

libraries = ['mkl_rt', 'pthread'] blas_mkl_info: define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]

library_dirs = ['/opt/intel/mkl/lib/intel64/']

include_dirs = ['/opt/intel/mkl/include']

libraries = ['mkl_rt', 'pthread']

lapack_mkl_info: define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]

library_dirs = ['/opt/intel/mkl/lib/intel64/']

include_dirs = ['/opt/intel/mkl/include']

libraries = ['mkl_rt', 'pthread'] blas_opt_info: define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]

library_dirs = ['/opt/intel/mkl/lib/intel64/']

include_dirs = ['/opt/intel/mkl/include'] libraries = ['mkl_rt', 'pthread']

>>> a = np.random.rand(10000, 10000)

>>> b = np.random.rand(10000, 10000)

>>> a@b

now check with the command: nvidia-smi

For MKL <= 2018.3 you can see the gpu is being used 100%, with MKL > 2018.3 it is 0% so the gpu is not used,

[Gennady_F_Intel]

here is the link to the MKL 2018 System Requirements - https://software.intel.com/en-us/articles/intel-math-kernel-library-intel-mkl-2018-system-requirements

[kkamm1]

Gennady F. (Intel) wrote:

here is the link to the MKL 2018 System Requirements - https://software.intel.com/en-us/articles/intel-math-kernel-library-inte...

Not sure what this has to do with my post.The system requirement is certainly met and it works. The only thing that doesn't work is Nvidia's nvblas with numpy compiled against MKL (only for MKL version >= 2018.3, it works for 2018.2 and before)

[AndrewC]

I would go back and experiment with a simple C program that makes BLAS calls (say one from the MKL examples ) and tries to do what you are attempting. I assume that would be linking against MKL, but then setting LD_PRELOAD to try and force nvblas to be used.

It's also possible to turn on LD debugging at runtime to see where BLAS symbols are being resolved.

[kkamm1]

@vasci

I would go back and experiment with a simple C program that makes BLAS calls (say one from the MKL examples ) and tries to do what you are attempting. I assume that would be linking against MKL, but then setting LD_PRELOAD to try and force nvblas to be used.

I am not a C programmer. My use case is very simple. I use INTEL MKL as an alternative  cpu blas for R and python. That works so the linking is not an issue. But to use nvblas with python requires numpy be compiled against MKL (done) and it is used as a fall back cpu blas as nvblas only works for so called level 3 operations (matrix to matrix). See http://scelementary.com/2015/04/09/nvidia-nvblas-in-numpy.html
Since this is a very specific issue I don't think running some generic C examples would produce any insight (and as I said I am no C programmer)

It's also possible to turn on LD debugging at runtime to see where BLAS symbols are being resolved.

How do you do that?

FWIW I have also created an issue on Nvidia's forums. Here it is.