topic Re: MKL library scans available cores, disregards existing cpu affinity in Intel® oneAPI Math Kernel Library

MKL library scans available cores, disregards existing cpu affinity

EddyF — Fri, 22 Oct 2021 19:54:10 GMT

Hi, this is about the same issue as this earlier thread: https://community.intel.com/t5/Intel-oneAPI-Math-Kernel-Library/MKL-library-scans-available-cores-touching-nodes-it-absolutely/m-p/1284598

Since the earlier thread never had any resolution, I'm reopening a new thread (as suggested in that thread).

Here is what I'm running to reproduce the problem:

conda create -n mkl -c intel mkl-service conda activate mkl taskset -c 2,4,5 strace -e trace=sched_setaffinity python -c 'import mkl; mkl.get_num_stripes()'

Output:

sched_setaffinity(0, 8, [2]) = 0 sched_setaffinity(0, 8, [4]) = 0 sched_setaffinity(0, 8, [5]) = 0 sched_setaffinity(0, 8, [2, 4, 5]) = 0 sched_setaffinity(0, 8, [2, 4, 5]) = 0 sched_setaffinity(0, 8, [2, 4, 5]) = 0 sched_setaffinity(0, 8, [0]) = 0 sched_setaffinity(0, 8, [1]) = 0 sched_setaffinity(0, 8, [2]) = 0 sched_setaffinity(0, 8, [2, 4, 5]) = 0 +++ exited with 0 +++

As we can see there are two phases happening here:

First it scans through cpus 2, 4, and 5
Later, it scans through cpus 0, 1, and 2

It seems that if the initial affinity set contains N cpus, then the second phase above will always scan through cpus 0 through N-1, regardless of which cpus were actually in the affinity set. This seems like a very strange and patently buggy behavior?

Using gdb, I was able to figure out that all of these sched_setaffinity calls are happening inside of a function called mkl_serv_get_num_stripes. Furthermore, the "first phase" (where we scan through the correct cpus) is happening inside of a sub-call to omp_get_num_procs ; the "second phase" (which is buggy) happens inside of mkl_serv_get_num_stripes itself.

What can be done to fix this?

Re:MKL library scans available cores, disregards existing cpu affinity

RahulV_intel — Mon, 25 Oct 2021 11:32:12 GMT

Hi,

Thanks for posting on the MKL forum. I could reproduce your output with oneAPI 2021.4, but I would further need to check with the team internally whether MKL is causing this behavior.

Regards,

Rahul

Re:MKL library scans available cores, disregards existing cpu affinity

Ruqiu_C_Intel — Thu, 28 Oct 2021 08:21:26 GMT

Hi ,

As the early thread mentioned that the issue might be python itself or openMP rather than MKL.

You can test without MKL involved, the issue still exist. For more details, please refer to the early thread: https://community.intel.com/t5/Intel-oneAPI-Math-Kernel-Library/MKL-library-scans-available-cores-to...

Thanks,

Ruqiu

Re: MKL library scans available cores, disregards existing cpu affinity

EddyF — Thu, 28 Oct 2021 12:16:15 GMT

Hi, thanks for the response! Unfortunately I'm quite sure that the issue is coming from inside of MKL. I have a simpler reproducible example than the earlier thread, which makes it more clear that MKL is the problem.

This is how I create my python environment and install MKL:

conda create -n mkl -c intel mkl-service conda activate mkl

As you can see, I'm only installing the mkl-service package (along with its dependencies), from the Intel conda channel to make sure it's the latest official version.

If I run the following then the issue appears:

Input:

taskset -c 2,4,5 strace -e trace=sched_setaffinity python -c 'import mkl; mkl.get_num_stripes()'

Output:

However, if I skip calling mkl.get_num_stripes(), then the issue does not appear:

Input:

taskset -c 2,4,5 strace -e trace=sched_setaffinity python -c 'import mkl'

Output:

+++ exited with 0 +++

This shows that the sched_setaffinity calls are happening inside of mkl.get_num_stripes().

As I was saying at the start of this thread, I have already used gdb to investigate further, and I figured out exactly where sched_setaffinity was being called from:

The first few calls (scanning through 2, 4, 5) happen inside of on OpenMP function called omp_get_num_procs which is called from an MKL function called mkl_serv_get_num_stripes
The remaining calls (scanning through 0, 1, 2, which is erroneous) happens directly inside of mkl_serv_get_num_stripes

I have also spent some time stepping through the execution of mkl_serv_get_num_stripes in gdb (one assembly instruction at a time) and I could see exactly where it was triggering the erroneous sched_setaffinity syscalls. Are you familiar with this function? I guess it is an internal MKL function and its source code is not publicly available. It would be great if someone within Intel who has access to the source code and knows how it is built could have a closer look and confirm if these observations make sense.

Re:MKL library scans available cores, disregards existing cpu affinity

Ruqiu_C_Intel — Tue, 02 Nov 2021 04:41:24 GMT

Hi Eddy,

Thanks for the information! We are investigating it internally and will let you know once there is any update.

Re:MKL library scans available cores, disregards existing cpu affinity

Ruqiu_C_Intel — Tue, 23 Nov 2021 01:24:10 GMT

Hi,

Have you tried to set MKL_DYNAMIC=false? Its default value is Ture, oneMKL can adjust number of threads to get the best performance. Switching off MKL_DYNAMIC will let user set whatever he wants. For more details, please check the MKL document here:

https://www.intel.com/content/www/us/en/develop/documentation/onemkl-windows-developer-guide/top/managing-performance-and-memory/improving-performance-with-threading/using-additional-threading-control/mkl-dynamic.html

Regards,

Ruqiu

Re: Re:MKL library scans available cores, disregards existing cpu affinity

EddyF — Tue, 23 Nov 2021 03:18:48 GMT

I just tried it but it doesn't seem to affect anything and the end result is the same. Using the same minimal reproducible example from my earlier posts:

Input:

MKL_DYNAMIC=false taskset -c 2,4,5 strace -e trace=sched_setaffinity python -c 'import mkl; mkl.get_num_stripes()'

Output:

I also tried MKL_DYNAMIC=FALSE and the result is exactly the same. @Ruqiu_C_Intel is it unexpected that this didn't work? Does it work for you? Perhaps there are some additional environment variables that I need to set? I have tried many combinations and have not yet been able to find anything that works for skipping the faulty sched_setaffinity logic.

Re:MKL library scans available cores, disregards existing cpu affinity

Ruqiu_C_Intel — Wed, 22 Dec 2021 03:35:24 GMT

Hi Eddy,

Thanks for your patience. We are still investigating the issue.

Why skip calling mkl.get_num_stripes(), then the issue does not appear, The reason is that no MKL functions gets called. In fact the observed issues is only triggered when a global thread-control function is called.

Regards,

Ruqiu

Re: MKL library scans available cores, disregards existing cpu affinity

EddyF — Tue, 04 Jan 2022 17:58:27 GMT

Thanks for the update! Yes, I agree, the issue only gets triggered when a certain global thread-control function is called (and a lot of MKL functions cause that thread-control function to be called, so the issue ends up happening with almost any MKL function). I'll be on the lookout for further updates here.

Re:MKL library scans available cores, disregards existing cpu affinity

Ruqiu_C_Intel — Tue, 01 Mar 2022 10:20:33 GMT

The fixed will be available in our next version. Thank you for your patience.

Re: MKL library scans available cores, disregards existing cpu affinity

EddyF — Fri, 11 Mar 2022 15:41:38 GMT

Great! Thank you!