Scipy test fails when compiled against MKL (oneAPI version 2023.1)

[AgilentGCMS]

Hello,

I've been trying to build numpy and scipy with intel MKL from oneAPI version 2023.1. The build of scipy proceeds OK, but the test fails at scipy.linalg.logm. If I set MKL_VERBOSE=1, I get a lot of these lines:

MKL_VERBOSE DDOT(7,0x3482328,2,0x3482320,2) 654ns CNR:OFF Dyn:1 FastMM:1 TID:0  NThr:1
MKL_VERBOSE ZGEMM(N,N,2,7,7,0x14e586c88238,0x35cc960,2,0x347b5e0,7,0x14e586c88228,0x35b10e0,2) 3.22us CNR:OFF Dyn:1 FastMM:1 TID:0  NThr:1
MKL_VERBOSE ZGEMM(N,N,2,7,7,0x14e586c88238,0x35cc960,2,0x347b5e0,7,0x14e586c88228,0x35b10e0,2) 1.20us CNR:OFF Dyn:1 FastMM:1 TID:0  NThr:1
MKL_VERBOSE ZGEMM(N,N,2,7,7,0x14e586c88238,0x2b8c170,2,0x347b5e0,7,0x14e586c88228,0x35cc960,2) 940ns CNR:OFF Dyn:1 FastMM:1 TID:0  NThr:1
MKL_VERBOSE ZGEMM(N,T,2,7,7,0x14e586c88238,0x2b8c170,2,0x347b5e0,7,0x14e586c88228,0x35b10e0,2) 2.01us CNR:OFF Dyn:1 FastMM:1 TID:0  NThr:1
MKL_VERBOSE ZGEMM(N,T,2,7,7,0x14e586c88238,0x35cc960,2,0x347b5e0,7,0x14e586c88228,0x35b10e0,2) 1.10us CNR:OFF Dyn:1 FastMM:1 TID:0  NThr:1
MKL_VERBOSE ZGEMM(N,T,2,7,7,0x14e586c88238,0x35a60b0,2,0x347b5e0,7,0x14e586c88228,0x35cc960,2) 968ns CNR:OFF Dyn:1 FastMM:1 TID:0  NThr:1
MKL_VERBOSE ZGEMM(N,N,2,7,7,0x14e586c88238,0x35b10e0,2,0x347b5e0,7,0x14e586c88228,0x35a60b0,2) 2.13us CNR:OFF Dyn:1 FastMM:1 TID:0  NThr:1
MKL_VERBOSE ZGEMM(N,N,2,7,7,0x14e586c88238,0x35b10e0,2,0x347b5e0,7,0x14e586c88228,0x35a60b0,2) 1.10us CNR:OFF Dyn:1 FastMM:1 TID:0  NThr:1
MKL_VERBOSE ZGEMM(N,N,2,7,7,0x14e586c88238,0x35cc960,2,0x347b5e0,7,0x14e586c88228,0x35b10e0,2) 1.05us CNR:OFF Dyn:1 FastMM:1 TID:0  NThr:1

and the call to 'logm' gets stuck forever. Can anyone tell me what these mean and how to fix this? 

[Ruqiu_C_Intel]

Hello,

Thank you for posting.

Have you tried oneMKL latest verion(2024.0)? At the same time, instead of build Numpy/Scipy with Intel® oneAPI Math Kernel Library (oneMKL) manually, we strongly recommend developer to use  Intel®  Distribution  for  Python* , which has prebuild Numpy/Scipy based on oneMKL. Here is one article for your reference:

https://www.intel.com/content/www/us/en/developer/articles/technical/numpyscipy-with-intel-mkl.html

[AgilentGCMS]

I understand that Intel would like everyone to use the Intel Distribution for Python, but I would still like to be able to compile scipy and numpy locally in order to take advantage of processor-specific optimizations. Nonetheless, I downloaded and installed Intel Distribution for Python, and scipy.test() fails with the following:

In [1]: import scipy

In [2]: scipy.test()
===================================================================== test session starts ======================================================================
platform linux -- Python 3.9.18, pytest-8.0.0, pluggy-1.3.0
rootdir: /home/sbasu
collected 0 items / 1 error

============================================================================ ERRORS ============================================================================
________________________________________________________________ ERROR collecting test session _________________________________________________________________
/work2/noaa/co2/sbasu/packages/python/intelpython3/lib/python3.9/site-packages/pluggy/_hooks.py:493: in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
        firstresult = True
        kwargs     = {'parent': <Dir work2>, 'path': PosixPath('/work2/hpc-computing')}
        self       = <HookCaller 'pytest_collect_directory'>
/work2/noaa/co2/sbasu/packages/python/intelpython3/lib/python3.9/site-packages/pluggy/_manager.py:115: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
        firstresult = True
        hook_name  = 'pytest_collect_directory'
        kwargs     = {'parent': <Dir work2>, 'path': PosixPath('/work2/hpc-computing')}
        methods    = [<HookImpl plugin_name='main', plugin=<module '_pytest.main' from '/work2/noaa/co2/sbasu/packages/python/intelpython3/...test.python' from '/work2/noaa/co2/sbasu/packages/python/intelpython3/lib/python3.9/site-packages/_pytest/python.py'>>]
        self       = <_pytest.config.PytestPluginManager object at 0x1516fc117cd0>
/work2/noaa/co2/sbasu/packages/python/intelpython3/lib/python3.9/site-packages/_pytest/python.py:210: in pytest_collect_directory
    if pkginit.is_file():
        parent     = <Dir work2>
        path       = PosixPath('/work2/hpc-computing')
        pkginit    = PosixPath('/work2/hpc-computing/__init__.py')
/work2/noaa/co2/sbasu/packages/python/intelpython3/lib/python3.9/pathlib.py:1456: in is_file
    return S_ISREG(self.stat().st_mode)
        self       = PosixPath('/work2/hpc-computing/__init__.py')
/work2/noaa/co2/sbasu/packages/python/intelpython3/lib/python3.9/pathlib.py:1232: in stat
    return self._accessor.stat(self)
E   PermissionError: [Errno 13] Permission denied: '/work2/hpc-computing/__init__.py'
        self       = PosixPath('/work2/hpc-computing/__init__.py')
=================================================================== short test summary info ====================================================================
ERROR ::work2 - PermissionError: [Errno 13] Permission denied: '/work2/hpc-computing/__init__.py'
======================================================================= 1 error in 0.17s =======================================================================
ERROR: found no collectors for /work2/noaa/co2/sbasu/packages/python/intelpython3/lib/python3.9/site-packages/scipy

Out[2]: False

I installed the distribution inside /work2/noaa/co2/sbasu/packages/python/intelpython3, absolutely no idea where it's getting /work2/hpc-computing. In any case, /work2/hpc-computing is not even readable by users on this system, and I don't see why python needs access to it.

[PengHuang]

Hi,

do you install Intel Python following the guide https://www.intel.com/content/www/us/en/developer/articles/technical/get-started-with-intel-distribution-for-python.html ?

you can also install different packages under https://www.intel.com/content/www/us/en/developer/articles/tool/whats-included-distribution-for-python.html#packageEnvironmentManagers.

Intel® Distribution for Python* is a cluster of packages includes Numba*, NumPy* and more. All these packages are optimized via Intel® oneAPI Math Kernel Library (oneMKL) and Intel® oneAPI Data Analytics Library (oneDAL) to make Python* application more efficient, looks like you don't need to build those by yourself.

could you also provides us your scipy testing env (including hardware/OS/tools version) and steps in this post?

do you run the scipy under conda env? we tried Intel Python in our lab but can't reproduce your above issue.

thanks.

[AgilentGCMS]

Yes, I did follow the guide at https://www.intel.com/content/www/us/en/developer/articles/technical/get-started-with-intel-distribution-for-python.html. Specifically, 

bash ~/intelpython3.sh 

And then I initiated the environment with

source activate base

And then from an interactive python shell, I typed

>>> import scipy
>>> scipy.test()

Which resulted in the error

=============================================================================== test session starts ================================================================================
platform linux -- Python 3.9.18, pytest-8.0.0, pluggy-1.3.0
rootdir: /home/sbasu
collected 0 items / 1 error

====================================================================================== ERRORS ======================================================================================
__________________________________________________________________________ ERROR collecting test session ___________________________________________________________________________
/work2/noaa/co2/sbasu/packages/python/intelpython3/lib/python3.9/site-packages/pluggy/_hooks.py:493: in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
        firstresult = True
        kwargs     = {'parent': <Dir work2>, 'path': PosixPath('/work2/hpc-computing')}
        self       = <HookCaller 'pytest_collect_directory'>
/work2/noaa/co2/sbasu/packages/python/intelpython3/lib/python3.9/site-packages/pluggy/_manager.py:115: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
        firstresult = True
        hook_name  = 'pytest_collect_directory'
        kwargs     = {'parent': <Dir work2>, 'path': PosixPath('/work2/hpc-computing')}
        methods    = [<HookImpl plugin_name='main', plugin=<module '_pytest.main' from '/work2/noaa/co2/sbasu/packages/python/intelpython3/...test.python' from '/work2/noaa/co2/sbasu/packages/python/intelpython3/lib/python3.9/site-packages/_pytest/python.py'>>]
        self       = <_pytest.config.PytestPluginManager object at 0x14a7f9d5b550>
/work2/noaa/co2/sbasu/packages/python/intelpython3/lib/python3.9/site-packages/_pytest/python.py:210: in pytest_collect_directory
    if pkginit.is_file():
        parent     = <Dir work2>
        path       = PosixPath('/work2/hpc-computing')
        pkginit    = PosixPath('/work2/hpc-computing/__init__.py')
/work2/noaa/co2/sbasu/packages/python/intelpython3/lib/python3.9/pathlib.py:1456: in is_file
    return S_ISREG(self.stat().st_mode)
        self       = PosixPath('/work2/hpc-computing/__init__.py')
/work2/noaa/co2/sbasu/packages/python/intelpython3/lib/python3.9/pathlib.py:1232: in stat
    return self._accessor.stat(self)
E   PermissionError: [Errno 13] Permission denied: '/work2/hpc-computing/__init__.py'
        self       = PosixPath('/work2/hpc-computing/__init__.py')
============================================================================= short test summary info ==============================================================================
ERROR ::work2 - PermissionError: [Errno 13] Permission denied: '/work2/hpc-computing/__init__.py'
================================================================================= 1 error in 0.66s =================================================================================
ERROR: found no collectors for /work2/noaa/co2/sbasu/packages/python/intelpython3/lib/python3.9/site-packages/scipy

[PengHuang]

Hi,

from your error logs, looks like it was caused by your permission, but not Intel Python's issue, maybe you can change to root user and run again, I also find some suggestions for your error in google as below,

The PermissionError: [errno 13] permission denied error occurs when you try to access a file from Python without having the necessary permissions. To fix this error, use the chmod or chown command to change the permissions of the file so that the right user and/or group can access the file.

[AgilentGCMS]

I am installing on a high performance compute cluster where I cannot and will never have root access. Are you saying that Intel Python cannot be installed without root access? Then please mention that clearly in the install guide. I can imagine I'm not the only one who will otherwise try to install Intel Python as a regular user.

If this is indeed true, i.e., Intel Python cannot be installed in a user's space and requires root access, it breaks precedent with almost every piece of Linux software I know of. The general principle of Linux packages is not to ask for more permissions than needed, as a result of which only software that needs root access, such as NFS servers and mailer daemons, ask for root access. There is (AFAIK) no need for a python interpreter to require root access for installation, and I've compiled and installed python as a regular user on many different Linux compute clusters. Unless there's a good reason for this demand for elevated privilege, Intel Python's demand for root access is a result of sloppy programming/packaging, which is unfortunate for a product that's touted as an improvement on regular python for numerical programming.

[PengHuang]

No, I'm not saying that Intel Python must be installed and used with root access, just want to confirm whether your issue is caused by permission problem, because from your error logs it's caused by permission denied to access '/work2/hpc-computing/__init__.py'.

if you can't get root access, maybe could try to use the chmod or chown command to change the permissions of the file so that the right user and/or group can access the file.