topic Re: oneMKL debug library mkl_tbb_threadd in Intel® oneAPI Math Kernel Library

oneMKL debug library mkl_tbb_threadd

jeremybouny — Thu, 31 Jul 2025 08:42:55 GMT

Hello,

It has been few years since I have been using mkl/oneMKL. Until now, we made some shortcuts to use it with oneTBB: we were consuming the Release-only version of oneTBB, even in Debug mode.
While reworking our build process, I am trying to remove this hack and benefit from the oneMKL (2024.2) debug oneTBB threading layer library to allow using the Debug oneTBB library when building our project in Debug.

I am experiencing the exact same issue described in an older post which got no answers: https://community.intel.com/t5/Intel-oneAPI-Math-Kernel-Library/Using-debug-version-of-TBB-with-MKL-on-Windows/m-p/1397723/highlight/true#M33358

Here are some details about what I am doing:

linking against oneMKL 2024.2 (dynamic mode)
shipping only the required oneMKL libraries
using the official CMake config file (which should be this version https://github.com/uxlfoundation/oneMath/blob/09b4123a126318e5cff0f6c32b5388ae8e8624cb/cmake/mkl/MKLConfig.cmake)
- it is globally matching what is described on the link advisor
- I am here using the MKL::MKL target
configuring oneMKL to use lp64 interface / tbb threading layer
building oneTBB 2021.13.0 on our side

The main difference between the link advisor and the CMake config file is the reference to the mkl_tbb_threadd library in CMake, the Debug version of the oneTBB threading layer used in Windows/Debug mode.
This library appeared a few years ago, but its use was never documented I think.
In theory, it seems it should finally allow us to use oneTBB in Debug, when working in Debug.

I think the only available documentation of this library is just the confirmation of its existence here: https://www.intel.com/content/www/us/en/docs/onemkl/developer-guide-windows/2024-0/contents-of-the-bin-directory.html

After dumping the dependencies I can confirm that mkl_tbb_threadd.2.dll expects tbb12_debug.dll, which is what I want.
However, when linking against mkl_tbb_threadd_dll.lib, it is at runtime failing with the error:

INTEL oneMKL ERROR: The specified module could not be found. mkl_sequential.2.dll. Intel oneMKL FATAL ERROR: Cannot load <mkl-thread-layer>.

I thought maybe it was still searching for the Release oneTBB layer by mistake and tried to manually copy mkl_tbb_thread.2.dll to the build folder, but same issue.
I then tried to copy mkl_sequential.2.dll, which is working.

So,
It seems that when linking against the Release library mkl_tbb_thread, things are working properly.
However, when linking against the Debug library mkl_tbb_threadd, I also need to add the mkl_sequential.2.dll file in the application directory to make it work.

I am now doubting if oneMKL is using oneTBB in debug, or just using the sequential version. If it is really the behavior, I still don't see any reason to depend on the sequential version as we are asking for the oneTBB layer.
I checked the symbols out of curiosity, to see if the debug threading layer was exposing just a subset, but no, all the exposed symbols are similar on these three libraries.

In the end, I don't know if this Debug version of the threading layer should work or not. On the other side, it is partially handled on the official CMake file.
I would like to confirm what the status is and how to use oneMKL in Debug.
Forcing the use of oneTBB Release even in Debug is something I would prefer to avoid as it complexifies a lot our build process with no real reason. I am not sure neither of the Linux behavior, if there is something similar.

I may have missed some documentation or info. Moreover, I tried to deduce things, but I may also be completely wrong. Any official information would be helpful!

Additional note 1:

The official MKLConfig.cmake file has only a partial support for this debug library. It is not working when using a multi configuration generator as it is based on the CMAKE_BUILD_TYPE value.
Moreover, the latest version of this file is forcing oneTBB as release only, which makes me think something is not stable with the debug libraries: https://github.com/uxlfoundation/oneMath/blob/cf9c3cb51e72c033e7dcfdbc3d5c9227ef28ba2b/cmake/mkl/MKLConfig.cmake#L865.

Additional note 2:

mkl_tbb_threadd library can only be found in the installer. It seems it is not present in the pip nor conda packages while they both contain the MKLConfig.cmake file referencing it, which is weird.

Re: oneMKL debug library mkl_tbb_threadd

Fengrui — Wed, 20 Aug 2025 00:14:16 GMT

Hi,

I tried to test on a simple code with tbb debug threading layer, but didn't reproduce the issue. The case is attached.

Did you see cmake look for mkl_sequential.2.dll during configuration? In my case, the following command is used for configuration,

"cmake . -G Ninja -DCMAKE_BUILD_TYPE=Debug -DMKL_THREADING=tbb_thread",

and the corresponding output for tbb threading library is correct,

...

-- Found C:/Program Files (x86)/Intel/oneAPI/mkl/latest/lib/mkl_tbb_threadd_dll.lib
-- Found DLL: C:/Program Files (x86)/Intel/oneAPI/mkl/latest/bin/mkl_tbb_threadd.2.dll

...

Could you check the output of configuration and share related lines?

Thanks,

Fengrui

Re: oneMKL debug library mkl_tbb_threadd

jeremybouny — Tue, 26 Aug 2025 08:40:53 GMT

Hi,

Thank you for the answer.

I confirm that in my case mkl_sequential.2.dll is never mentioned/searched at configuration time. I dumped all the CMake targets to check the behavior.

It is really a runtime behavior once the application is compiled: When calling oneMKL functions, it is trying to dlopen mkl_sequential.2.dll, which is not found and failing, at least with the tested oneMKL 2024.2.
My main point is to understand if we should make the mkl_sequential.2.dll library available at runtime in debug. If yes, is it an expected behavior, a misconfiguration from our side, or an issue on oneMKL side.

We will soon try to switch to oneMKL 2025.x so I would prefer to be sure of the behavior to help this new migration.

Re: oneMKL debug library mkl_tbb_threadd

Fengrui — Tue, 26 Aug 2025 17:48:10 GMT

Thank you for the clarification! I can reproduce it now in debug mode if I remove mkl_sequential.2.dll. It is unexpected behavior to me. Let me report it internally as a bug.

Re: oneMKL debug library mkl_tbb_threadd

jeremybouny — Tue, 26 Aug 2025 19:35:24 GMT

Great thank you for the confirmation!

If it appears it is not an expected behavior and if it is possible, I would still be curious to know if internally in the end it is using the oneTBB or the sequential threading layer to better understand the performances of our algorithms (even in debug).

Re: oneMKL debug library mkl_tbb_threadd

Fengrui — Thu, 18 Sep 2025 19:55:27 GMT

The fix of this issue will be included in oneMKL 2025.3 release.

Although the sequential library is loaded in debug mode in 2025.2, internally oneTBB is used for threading.

Re: oneMKL debug library mkl_tbb_threadd

jeremybouny — Fri, 19 Sep 2025 08:10:22 GMT

Thank you for confirming the behavior, that's helpful! And great news for the fix also.

Re: oneMKL debug library mkl_tbb_threadd

Fengrui — Mon, 03 Nov 2025 21:36:46 GMT

The oneMKL 2025.3 release is available now. Did you get the chance to test it?

Re: oneMKL debug library mkl_tbb_threadd

jeremybouny — Wed, 12 Nov 2025 21:33:38 GMT

Hello,
I am starting some tests with this version to see how it behaves.
However, I am a bit worried of the mentioned known issue in the release note "oneMKL LAPACK direct call routines on Windows with TBB threading may crash". Is there any additional information about it? Like, is which routines and under which conditions? Just wrote this way it looks like a blocker to me as I would like to keep oneTBB as the threading layer.

Re: oneMKL debug library mkl_tbb_threadd

Fengrui — Thu, 13 Nov 2025 00:10:33 GMT

It happens sporadically and we are still working on root-causing this issue. So I'm afraid it is hard to provide more information as of now. But it only affects the direct call usage for small-sized problems. Are direct calls of LAPACK functions used in your application?

Re: oneMKL debug library mkl_tbb_threadd

jeremybouny — Thu, 13 Nov 2025 08:05:23 GMT

That is really interesting information thank you. We are not (to my knowledge) using any direct call after checking it.
To be honest, I am just discovering this feature by the way, it is good to know.
There are however various LAPACK calls either from our projects or from external libraries.
It feels I still have to pay attention to all our Eigen usage where MKL_DIRECT_CALL might be used depending on the configuration.

So, if it appears we are really not using the direct call feature at all, do you confirm there should be no issues?

Re: oneMKL debug library mkl_tbb_threadd

Fengrui — Wed, 19 Nov 2025 00:01:11 GMT

There should be no issues if the direct call feature is not used.