topic Re: Setting MPI for MKL in Intel® oneAPI Math Kernel Library

Setting MPI for MKL

Iakov — Sat, 16 Jul 2022 05:00:28 GMT

Hi all,

I'm trying to dynamically load MKL library during execution and fall into problem during parallel execution. When calling blacs_pinfo function I got this error:

Intel MKL FATAL ERROR: Cannot load symbol MKLMPI_Get_wrappers.

I presume that problem is in the fact that MPI library is not set for MKL, which should be done by calling MKL_Set_mpi function, but calling this function always ends up in getting error code -3 as a return value, which means according to manual that `The MPI library cannot be set at this point.`

Moving MKL_Set_mpi call into different stages of execution didn't help, error code -3 is returned always.

Does anybody have experience with dynamic load of MKL?

Thank you for any advises in advance,

Iakov

Re: Setting MPI for MKL

VidyalathaB_Intel — Mon, 18 Jul 2022 10:14:26 GMT

Hi,

Thanks for reaching out to us.

Could you please confirm if you are using the suggested options by Link Line Advisor and still getting the error?

If yes, please provide us the sample reproducer and steps you are using to compile and run the code so that we can check it from our end as well.

If you are not aware of the Link Line advisor, we suggest you make use of the tool which will help you with the recommended compile and linking options according to your use case.

Additionally, here is a thread with a similar issue that might help in resolving the issue.

https://community.intel.com/t5/Intel-oneAPI-Math-Kernel-Library/Intel-MKL-MPI-2020-4-Cluster-FFT-Fatal-Errors-when-creating/m-p/1340411

Please get back to us with the above-requested details, if you have any issues in doing so.

Regards,

Vidya.

Re: Setting MPI for MKL

Iakov — Mon, 18 Jul 2022 14:24:41 GMT

Hi Vidya,

Thanks for your answer. But my problem is not about linking. It's about dynamic calls of MKL library functions. Here is the workflow:

1. Compile code and link without MKL.

2. Execute binary.

3. During execution:

libHandle = dlopen("libmkl_blacs_intelmpi_lp64.so", RTLD_LAZY);

funcHandle = dlsym(libHandle, "blacs_pinfo");

funcHandle(procID, numProcs);

and this calls gives me an error

Intel MKL FATAL ERROR: Cannot load symbol MKLMPI_Get_wrappers.

Re:Setting MPI for MKL

VidyalathaB_Intel — Tue, 19 Jul 2022 05:23:35 GMT

Hi,

Could you please let us know which version of MKL you are using and your OS environment details?

If possible, could you please provide us with a sample reproducer code and the command you have used to compile and execute so that we will try to reproduce the same from our end?

Regards,

Vidya.

Re: Setting MPI for MKL

Iakov — Tue, 19 Jul 2022 16:14:56 GMT

Hi,

I reproduced this problem with following MKL versions on Linux (CentOS 7.7.1908 (Core)):

Intel(R) MKL version 2018.0.4 (build 20180808) on Intel(R) Advanced Vector Extensions 2 (Intel(R) AVX2) enabled processors, Intel(R) 64 architecture (1 thread)
Intel(R) MKL version 2022.0.0 (build 20211112) on Intel(R) Advanced Vector Extensions 2 (Intel(R) AVX2) enabled processors, Intel(R) 64 architecture (1 thread)

Compilation and link commands (for 2022.0.0 MKL):

icc -I../thirdparty/intel-mpi/v2021.2/linux64_x86_64/include -I../thirdparty/intel-mkl/v2022.0.x/linux64_x86_64/include -c main.cpp

icc main.o -L../thirdparty/intel-mpi/v2021.2/linux64_x86_64/lib -L../thirdparty/intel-mpi/v2021.2/linux64_x86_64/lib/release_mt -lmpi -ldl -lrt -lpthread -ldl -o test

Output for execution with 2 processes:

Start 1 2
Start 0 2
ret_set_mpi = -3
ret_set_mpi = -3
Intel MKL FATAL ERROR: Cannot load symbol MKLMPI_Get_wrappers.
Intel MKL FATAL ERROR: Cannot load symbol MKLMPI_Get_wrappers.

Re: Setting MPI for MKL

VidyalathaB_Intel — Wed, 20 Jul 2022 06:15:53 GMT

Hi,

Thanks for sharing the code and steps to reproduce the issue.

Yes, the issue is reproducible.

Regarding the runtime error that you are getting, I tried following the suggestions from the link line advisor (as I mentioned in my earlier post) and don't see those errors. Here is the screenshot.

linking command:

icc main.o -L${MKLROOT}/lib/intel64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_intelmpi_lp64 -liomp5 -lpthread -lm -ldl -lmpi

But the return value of the function MKL_Set_mpi is still set to -3 as you mentioned in your first post. When searched regarding the same in the MKL manual, it is already mentioned like, the function returns an error with other configurations and maybe it is the reason we are getting that as -3.

mkl_set_mpi

"Call this function to set the MPI implementation to be used by Intel® oneAPI Math Kernel Library on Windows* OS when dynamic Intel® oneAPI Math Kernel Library libraries are used. For all other configurations, the function returns an error indicating that you cannot set the MPI implementation. You can specify your own prebuilt dynamic BLACS library for a custom MPI by setting vendor to MKL_BLACS_CUSTOM"

Please try the suggestions from the Link Line advisor which helps in resolving the issue with runtime error and do let me know if you have any concerns.

Regards,

Vidya.

Re: Setting MPI for MKL

Iakov — Wed, 20 Jul 2022 14:05:25 GMT

Hi Vidya,

Unfortunately I cannot accept linking solution, because the main purpose is to build a binary without any Lapack/Blacs/ScaLapack symbols and during execution depending on some logic load proper library and MKL could be one of the possible libraries.

I did some more tests:

1. On Windows everything is working out of the box without any additional calls/setting environment (I suspect that mpiexec for different MPI libraries set some environment and MKL makes a decision depending on this values). I checked for both IMPI and MSMPI.

2. When I explicitly called MKL_Set_mpi for Windows implementation I again got an error for both IMPI and MSMPI:
Intel MKL FATAL ERROR: Error on loading function mkl_serv_set_mpi.

So the main question is still the same: How to make this logic working on Linux?

The minor question is about not working function MKL_Set_mpi on Windows, but it's not important for me as long as for Windows everything is working without any additional steps.

PS. Reproducible for windows attached. The same MKL version where tested as previously: 2018.0.4 and 2022.0.0.

Re:Setting MPI for MKL

VidyalathaB_Intel — Wed, 27 Jul 2022 07:35:26 GMT

Hi,

We are working on your issue. we will get back to you soon.

Regards,

Vidya.

Re:Setting MPI for MKL

Khang_N_Intel — Wed, 27 Jul 2022 19:23:24 GMT

Hi Lakov,

I will definitely discuss with the team and will get back to you with the next step.

Best,

Khang

Re:Setting MPI for MKL

Khang_N_Intel — Fri, 05 Aug 2022 00:56:47 GMT

Hi Lakov,

The mkl team is looking into this issue.

I will let you know once I receive any news.

Best regards,

Khang

Re:Setting MPI for MKL

Khang_N_Intel — Fri, 05 Aug 2022 01:55:21 GMT

Hi Lakov,

oneMKL libraries cannot resolve internal dependencies on Linux if they were loaded with RTLD_LOCAL (default option for dlopen).

So, for the work-around, you will need to add the option RTLD-GLOBAL in the dlopen calls by replacing RTLD_LAZY with (RTLD_LAZY | RTLD_GLOBAL).

void* rt_lib_ptr = dlopen("libmkl_rt.so", RTLD_LAZY | RTLD_GLOBAL);

void* intel_lib_ptr = dlopen("libmkl_intel_lp64.so", RTLD_LAZY | RTLD_GLOBAL);

void* blacs_lib_ptr = dlopen("libmkl_blacs_intelmpi_lp64.so", RTLD_LAZY | RTLD_GLOBAL);

void* scalapack_lib_ptr = dlopen("libmkl_scalapack_lp64.so", RTLD_LAZY | RTLD_GLOBAL);

The fix will be in the the oneMKL 2023.0 release.

Best regards,

Khang

Re: Re:Setting MPI for MKL

Iakov — Mon, 08 Aug 2022 22:53:36 GMT

Hi Khang,

Thanks for solution! I checked it for our project and everything now works fine.

Sincerely,

Iakov