oneMKL debug library mkl_tbb_threadd

[jeremybouny]

Hello,

It has been few years since I have been using mkl/oneMKL. Until now, we made some shortcuts to use it with oneTBB: we were consuming the Release-only version of oneTBB, even in Debug mode.
While reworking our build process, I am trying to remove this hack and benefit from the oneMKL (2024.2) debug oneTBB threading layer library to allow using the Debug oneTBB library when building our project in Debug.

I am experiencing the exact same issue described in an older post which got no answers: https://community.intel.com/t5/Intel-oneAPI-Math-Kernel-Library/Using-debug-version-of-TBB-with-MKL-on-Windows/m-p/1397723/highlight/true#M33358

Here are some details about what I am doing:

• linking against oneMKL 2024.2 (dynamic mode)
• shipping only the required oneMKL libraries
• using the official CMake config file (which should be this version https://github.com/uxlfoundation/oneMath/blob/09b4123a126318e5cff0f6c32b5388ae8e8624cb/cmake/mkl/MKLConfig.cmake)
  • it is globally matching what is described on the link advisor
  • I am here using the MKL::MKL target
• configuring oneMKL to use lp64 interface / tbb threading layer
• building oneTBB 2021.13.0 on our side

The main difference between the link advisor and the CMake config file is the reference to the mkl_tbb_threadd library in CMake, the Debug version of the oneTBB threading layer used in Windows/Debug mode.
This library appeared a few years ago, but its use was never documented I think.
In theory, it seems it should finally allow us to use oneTBB in Debug, when working in Debug.

I think the only available documentation of this library is just the confirmation of its existence here: https://www.intel.com/content/www/us/en/docs/onemkl/developer-guide-windows/2024-0/contents-of-the-bin-directory.html

After dumping the dependencies I can confirm that mkl_tbb_threadd.2.dll expects tbb12_debug.dll, which is what I want.
However, when linking against mkl_tbb_threadd_dll.lib, it is at runtime failing with the error:

INTEL oneMKL ERROR: The specified module could not be found. mkl_sequential.2.dll.
Intel oneMKL FATAL ERROR: Cannot load <mkl-thread-layer>.

I thought maybe it was still searching for the Release oneTBB layer by mistake and tried to manually copy mkl_tbb_thread.2.dll to the build folder, but same issue.
I then tried to copy mkl_sequential.2.dll, which is working.

So,
It seems that when linking against the Release library mkl_tbb_thread, things are working properly.
However, when linking against the Debug library mkl_tbb_threadd, I also need to add the mkl_sequential.2.dll file in the application directory to make it work.

I am now doubting if oneMKL is using oneTBB in debug, or just using the sequential version. If it is really the behavior, I still don't see any reason to depend on the sequential version as we are asking for the oneTBB layer.
I checked the symbols out of curiosity, to see if the debug threading layer was exposing just a subset, but no, all the exposed symbols are similar on these three libraries.

In the end, I don't know if this Debug version of the threading layer should work or not. On the other side, it is partially handled on the official CMake file.
I would like to confirm what the status is and how to use oneMKL in Debug.
Forcing the use of oneTBB Release even in Debug is something I would prefer to avoid as it complexifies a lot our build process with no real reason. I am not sure neither of the Linux behavior, if there is something similar.

I may have missed some documentation or info. Moreover, I tried to deduce things, but I may also be completely wrong. Any official information would be helpful!

Additional note 1:

The official MKLConfig.cmake file has only a partial support for this debug library. It is not working when using a multi configuration generator as it is based on the CMAKE_BUILD_TYPE value.
Moreover, the latest version of this file is forcing oneTBB as release only, which makes me think something is not stable with the debug libraries: https://github.com/uxlfoundation/oneMath/blob/cf9c3cb51e72c033e7dcfdbc3d5c9227ef28ba2b/cmake/mkl/MKLConfig.cmake#L865.

Additional note 2:

mkl_tbb_threadd library can only be found in the installer. It seems it is not present in the pip nor conda packages while they both contain the MKLConfig.cmake file referencing it, which is weird.

[Fengrui]

Hi,

I tried to test on a simple code with tbb debug threading layer, but didn't reproduce the issue. The case is attached.

Did you see cmake look for mkl_sequential.2.dll during configuration? In my case, the following command is used for configuration, 

"cmake . -G Ninja -DCMAKE_BUILD_TYPE=Debug -DMKL_THREADING=tbb_thread",

and the corresponding output for tbb threading library is correct,

...

-- Found C:/Program Files (x86)/Intel/oneAPI/mkl/latest/lib/mkl_tbb_threadd_dll.lib
-- Found DLL: C:/Program Files (x86)/Intel/oneAPI/mkl/latest/bin/mkl_tbb_threadd.2.dll

...

Could you check the output of configuration and share related lines?

Thanks,

Fengrui

[jeremybouny]

Hi,

Thank you for the answer.

I confirm that in my case mkl_sequential.2.dll is never mentioned/searched at configuration time. I dumped all the CMake targets to check the behavior.

It is really a runtime behavior once the application is compiled: When calling oneMKL functions, it is trying to dlopen mkl_sequential.2.dll, which is not found and failing, at least with the tested oneMKL 2024.2.
My main point is to understand if we should make the mkl_sequential.2.dll library available at runtime in debug. If yes, is it an expected behavior, a misconfiguration from our side, or an issue on oneMKL side.

We will soon try to switch to oneMKL 2025.x so I would prefer to be sure of the behavior to help this new migration.

[Fengrui]

Thank you for the clarification! I can reproduce it now in debug mode if I remove mkl_sequential.2.dll. It is unexpected behavior to me. Let me report it internally as a bug. 

[jeremybouny]

Great thank you for the confirmation!

If it appears it is not an expected behavior and if it is possible, I would still be curious to know if internally in the end it is using the oneTBB or the sequential threading layer to better understand the performances of our algorithms (even in debug).