- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello,
I've been trying to build numpy and scipy with intel MKL from oneAPI version 2023.1. The build of scipy proceeds OK, but the test fails at scipy.linalg.logm. If I set MKL_VERBOSE=1, I get a lot of these lines:
MKL_VERBOSE DDOT(7,0x3482328,2,0x3482320,2) 654ns CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1 MKL_VERBOSE ZGEMM(N,N,2,7,7,0x14e586c88238,0x35cc960,2,0x347b5e0,7,0x14e586c88228,0x35b10e0,2) 3.22us CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1 MKL_VERBOSE ZGEMM(N,N,2,7,7,0x14e586c88238,0x35cc960,2,0x347b5e0,7,0x14e586c88228,0x35b10e0,2) 1.20us CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1 MKL_VERBOSE ZGEMM(N,N,2,7,7,0x14e586c88238,0x2b8c170,2,0x347b5e0,7,0x14e586c88228,0x35cc960,2) 940ns CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1 MKL_VERBOSE ZGEMM(N,T,2,7,7,0x14e586c88238,0x2b8c170,2,0x347b5e0,7,0x14e586c88228,0x35b10e0,2) 2.01us CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1 MKL_VERBOSE ZGEMM(N,T,2,7,7,0x14e586c88238,0x35cc960,2,0x347b5e0,7,0x14e586c88228,0x35b10e0,2) 1.10us CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1 MKL_VERBOSE ZGEMM(N,T,2,7,7,0x14e586c88238,0x35a60b0,2,0x347b5e0,7,0x14e586c88228,0x35cc960,2) 968ns CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1 MKL_VERBOSE ZGEMM(N,N,2,7,7,0x14e586c88238,0x35b10e0,2,0x347b5e0,7,0x14e586c88228,0x35a60b0,2) 2.13us CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1 MKL_VERBOSE ZGEMM(N,N,2,7,7,0x14e586c88238,0x35b10e0,2,0x347b5e0,7,0x14e586c88228,0x35a60b0,2) 1.10us CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1 MKL_VERBOSE ZGEMM(N,N,2,7,7,0x14e586c88238,0x35cc960,2,0x347b5e0,7,0x14e586c88228,0x35b10e0,2) 1.05us CNR:OFF Dyn:1 FastMM:1 TID:0 NThr:1
and the call to 'logm' gets stuck forever. Can anyone tell me what these mean and how to fix this?
Link Copied
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello,
Thank you for posting.
Have you tried oneMKL latest verion(2024.0)? At the same time, instead of build Numpy/Scipy with Intel® oneAPI Math Kernel Library (oneMKL) manually, we strongly recommend developer to use Intel® Distribution for Python* , which has prebuild Numpy/Scipy based on oneMKL. Here is one article for your reference:
https://www.intel.com/content/www/us/en/developer/articles/technical/numpyscipy-with-intel-mkl.html
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I understand that Intel would like everyone to use the Intel Distribution for Python, but I would still like to be able to compile scipy and numpy locally in order to take advantage of processor-specific optimizations. Nonetheless, I downloaded and installed Intel Distribution for Python, and scipy.test() fails with the following:
In [1]: import scipy
In [2]: scipy.test()
===================================================================== test session starts ======================================================================
platform linux -- Python 3.9.18, pytest-8.0.0, pluggy-1.3.0
rootdir: /home/sbasu
collected 0 items / 1 error
============================================================================ ERRORS ============================================================================
________________________________________________________________ ERROR collecting test session _________________________________________________________________
/work2/noaa/co2/sbasu/packages/python/intelpython3/lib/python3.9/site-packages/pluggy/_hooks.py:493: in __call__
return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
firstresult = True
kwargs = {'parent': <Dir work2>, 'path': PosixPath('/work2/hpc-computing')}
self = <HookCaller 'pytest_collect_directory'>
/work2/noaa/co2/sbasu/packages/python/intelpython3/lib/python3.9/site-packages/pluggy/_manager.py:115: in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
firstresult = True
hook_name = 'pytest_collect_directory'
kwargs = {'parent': <Dir work2>, 'path': PosixPath('/work2/hpc-computing')}
methods = [<HookImpl plugin_name='main', plugin=<module '_pytest.main' from '/work2/noaa/co2/sbasu/packages/python/intelpython3/...test.python' from '/work2/noaa/co2/sbasu/packages/python/intelpython3/lib/python3.9/site-packages/_pytest/python.py'>>]
self = <_pytest.config.PytestPluginManager object at 0x1516fc117cd0>
/work2/noaa/co2/sbasu/packages/python/intelpython3/lib/python3.9/site-packages/_pytest/python.py:210: in pytest_collect_directory
if pkginit.is_file():
parent = <Dir work2>
path = PosixPath('/work2/hpc-computing')
pkginit = PosixPath('/work2/hpc-computing/__init__.py')
/work2/noaa/co2/sbasu/packages/python/intelpython3/lib/python3.9/pathlib.py:1456: in is_file
return S_ISREG(self.stat().st_mode)
self = PosixPath('/work2/hpc-computing/__init__.py')
/work2/noaa/co2/sbasu/packages/python/intelpython3/lib/python3.9/pathlib.py:1232: in stat
return self._accessor.stat(self)
E PermissionError: [Errno 13] Permission denied: '/work2/hpc-computing/__init__.py'
self = PosixPath('/work2/hpc-computing/__init__.py')
=================================================================== short test summary info ====================================================================
ERROR ::work2 - PermissionError: [Errno 13] Permission denied: '/work2/hpc-computing/__init__.py'
======================================================================= 1 error in 0.17s =======================================================================
ERROR: found no collectors for /work2/noaa/co2/sbasu/packages/python/intelpython3/lib/python3.9/site-packages/scipy
Out[2]: False
I installed the distribution inside /work2/noaa/co2/sbasu/packages/python/intelpython3, absolutely no idea where it's getting /work2/hpc-computing. In any case, /work2/hpc-computing is not even readable by users on this system, and I don't see why python needs access to it.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
do you install Intel Python following the guide https://www.intel.com/content/www/us/en/developer/articles/technical/get-started-with-intel-distribution-for-python.html ?
you can also install different packages under https://www.intel.com/content/www/us/en/developer/articles/tool/whats-included-distribution-for-python.html#packageEnvironmentManagers.
Intel® Distribution for Python* is a cluster of packages includes Numba*, NumPy* and more. All these packages are optimized via Intel® oneAPI Math Kernel Library (oneMKL) and Intel® oneAPI Data Analytics Library (oneDAL) to make Python* application more efficient, looks like you don't need to build those by yourself.
could you also provides us your scipy testing env (including hardware/OS/tools version) and steps in this post?
do you run the scipy under conda env? we tried Intel Python in our lab but can't reproduce your above issue.
thanks.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Yes, I did follow the guide at https://www.intel.com/content/www/us/en/developer/articles/technical/get-started-with-intel-distribution-for-python.html. Specifically,
bash ~/intelpython3.sh
And then I initiated the environment with
source activate base
And then from an interactive python shell, I typed
>>> import scipy
>>> scipy.test()
Which resulted in the error
=============================================================================== test session starts ================================================================================
platform linux -- Python 3.9.18, pytest-8.0.0, pluggy-1.3.0
rootdir: /home/sbasu
collected 0 items / 1 error
====================================================================================== ERRORS ======================================================================================
__________________________________________________________________________ ERROR collecting test session ___________________________________________________________________________
/work2/noaa/co2/sbasu/packages/python/intelpython3/lib/python3.9/site-packages/pluggy/_hooks.py:493: in __call__
return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
firstresult = True
kwargs = {'parent': <Dir work2>, 'path': PosixPath('/work2/hpc-computing')}
self = <HookCaller 'pytest_collect_directory'>
/work2/noaa/co2/sbasu/packages/python/intelpython3/lib/python3.9/site-packages/pluggy/_manager.py:115: in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
firstresult = True
hook_name = 'pytest_collect_directory'
kwargs = {'parent': <Dir work2>, 'path': PosixPath('/work2/hpc-computing')}
methods = [<HookImpl plugin_name='main', plugin=<module '_pytest.main' from '/work2/noaa/co2/sbasu/packages/python/intelpython3/...test.python' from '/work2/noaa/co2/sbasu/packages/python/intelpython3/lib/python3.9/site-packages/_pytest/python.py'>>]
self = <_pytest.config.PytestPluginManager object at 0x14a7f9d5b550>
/work2/noaa/co2/sbasu/packages/python/intelpython3/lib/python3.9/site-packages/_pytest/python.py:210: in pytest_collect_directory
if pkginit.is_file():
parent = <Dir work2>
path = PosixPath('/work2/hpc-computing')
pkginit = PosixPath('/work2/hpc-computing/__init__.py')
/work2/noaa/co2/sbasu/packages/python/intelpython3/lib/python3.9/pathlib.py:1456: in is_file
return S_ISREG(self.stat().st_mode)
self = PosixPath('/work2/hpc-computing/__init__.py')
/work2/noaa/co2/sbasu/packages/python/intelpython3/lib/python3.9/pathlib.py:1232: in stat
return self._accessor.stat(self)
E PermissionError: [Errno 13] Permission denied: '/work2/hpc-computing/__init__.py'
self = PosixPath('/work2/hpc-computing/__init__.py')
============================================================================= short test summary info ==============================================================================
ERROR ::work2 - PermissionError: [Errno 13] Permission denied: '/work2/hpc-computing/__init__.py'
================================================================================= 1 error in 0.66s =================================================================================
ERROR: found no collectors for /work2/noaa/co2/sbasu/packages/python/intelpython3/lib/python3.9/site-packages/scipy
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
from your error logs, looks like it was caused by your permission, but not Intel Python's issue, maybe you can change to root user and run again, I also find some suggestions for your error in google as below,
The PermissionError: [errno 13] permission denied error occurs when you try to access a file from Python without having the necessary permissions. To fix this error, use the chmod or chown command to change the permissions of the file so that the right user and/or group can access the file.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I am installing on a high performance compute cluster where I cannot and will never have root access. Are you saying that Intel Python cannot be installed without root access? Then please mention that clearly in the install guide. I can imagine I'm not the only one who will otherwise try to install Intel Python as a regular user.
If this is indeed true, i.e., Intel Python cannot be installed in a user's space and requires root access, it breaks precedent with almost every piece of Linux software I know of. The general principle of Linux packages is not to ask for more permissions than needed, as a result of which only software that needs root access, such as NFS servers and mailer daemons, ask for root access. There is (AFAIK) no need for a python interpreter to require root access for installation, and I've compiled and installed python as a regular user on many different Linux compute clusters. Unless there's a good reason for this demand for elevated privilege, Intel Python's demand for root access is a result of sloppy programming/packaging, which is unfortunate for a product that's touted as an improvement on regular python for numerical programming.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
No, I'm not saying that Intel Python must be installed and used with root access, just want to confirm whether your issue is caused by permission problem, because from your error logs it's caused by permission denied to access '/work2/hpc-computing/__init__.py'.
if you can't get root access, maybe could try to use the chmod or chown command to change the permissions of the file so that the right user and/or group can access the file.
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page