Intel® oneAPI Math Kernel Library
Ask questions and share information with other developers who use Intel® Math Kernel Library.

Setting MPI for MKL

Iakov
Beginner
2,286 Views

Hi all,

I'm trying to dynamically load MKL library during execution and fall into problem during parallel execution. When calling blacs_pinfo function I got this error:

Intel MKL FATAL ERROR: Cannot load symbol MKLMPI_Get_wrappers.

I presume that problem is in the fact that MPI library is not set for MKL, which should be done by calling MKL_Set_mpi function, but calling this function always ends up in getting error code -3 as a return value, which means according to manual that `The MPI library cannot be set at this point.` 

Moving MKL_Set_mpi  call into different stages of execution didn't help, error code -3 is returned always. 

Does anybody have experience with dynamic load of MKL?

Thank you for any advises in advance,

Iakov

Labels (1)
0 Kudos
1 Solution
Khang_N_Intel
Employee
1,981 Views

Hi Lakov,


oneMKL libraries cannot resolve internal dependencies on Linux if they were loaded with RTLD_LOCAL (default option for dlopen).


So, for the work-around, you will need to add the option RTLD-GLOBAL in the dlopen calls by replacing RTLD_LAZY with (RTLD_LAZY | RTLD_GLOBAL).


void* rt_lib_ptr = dlopen("libmkl_rt.so", RTLD_LAZY | RTLD_GLOBAL);

void* intel_lib_ptr = dlopen("libmkl_intel_lp64.so", RTLD_LAZY | RTLD_GLOBAL);

void* blacs_lib_ptr = dlopen("libmkl_blacs_intelmpi_lp64.so", RTLD_LAZY | RTLD_GLOBAL);

void* scalapack_lib_ptr = dlopen("libmkl_scalapack_lp64.so", RTLD_LAZY | RTLD_GLOBAL);


The fix will be in the the oneMKL 2023.0 release.


Best regards,

Khang




View solution in original post

0 Kudos
11 Replies
VidyalathaB_Intel
Moderator
2,248 Views

Hi,

 

Thanks for reaching out to us.

 

Could you please confirm if you are using the suggested options by Link Line Advisor and still getting the error?

If yes, please provide us the sample reproducer and steps you are using to compile and run the code so that we can check it from our end as well.

 

If you are not aware of the Link Line advisor, we suggest you make use of the tool which will help you with the recommended compile and linking options according to your use case.

 

Additionally, here is a thread with a similar issue that might help in resolving the issue.

https://community.intel.com/t5/Intel-oneAPI-Math-Kernel-Library/Intel-MKL-MPI-2020-4-Cluster-FFT-Fatal-Errors-when-creating/m-p/1340411

 

Please get back to us with the above-requested details, if you have any issues in doing so.

 

Regards,

Vidya.

          

 

0 Kudos
Iakov
Beginner
2,226 Views

Hi Vidya,

 

Thanks for your answer. But my problem is not about linking. It's about dynamic calls of MKL library functions. Here is the workflow:

1. Compile code and link without MKL.

2. Execute binary.

3. During execution:

 

libHandle = dlopen("libmkl_blacs_intelmpi_lp64.so", RTLD_LAZY);

funcHandle = dlsym(libHandle, "blacs_pinfo");

funcHandle(procID, numProcs);

and this calls gives me an error

Intel MKL FATAL ERROR: Cannot load symbol MKLMPI_Get_wrappers.

 

0 Kudos
VidyalathaB_Intel
Moderator
2,200 Views

Hi,


Could you please let us know which version of MKL you are using and your OS environment details?

If possible, could you please provide us with a sample reproducer code and the command you have used to compile and execute so that we will try to reproduce the same from our end?


Regards,

Vidya.


0 Kudos
Iakov
Beginner
2,188 Views

Hi,

 

I reproduced this problem with following MKL versions on Linux (CentOS 7.7.1908 (Core)):

Intel(R) MKL version 2018.0.4 (build 20180808) on Intel(R) Advanced Vector Extensions 2 (Intel(R) AVX2) enabled processors, Intel(R) 64 architecture (1 thread)
Intel(R) MKL version 2022.0.0 (build 20211112) on Intel(R) Advanced Vector Extensions 2 (Intel(R) AVX2) enabled processors, Intel(R) 64 architecture (1 thread)

 

Compilation and link commands (for 2022.0.0 MKL):

icc -I../thirdparty/intel-mpi/v2021.2/linux64_x86_64/include -I../thirdparty/intel-mkl/v2022.0.x/linux64_x86_64/include -c main.cpp

icc main.o -L../thirdparty/intel-mpi/v2021.2/linux64_x86_64/lib -L../thirdparty/intel-mpi/v2021.2/linux64_x86_64/lib/release_mt -lmpi -ldl -lrt -lpthread -ldl -o test

 

Output for execution with 2 processes:

Start 1 2
Start 0 2
ret_set_mpi = -3
ret_set_mpi = -3
Intel MKL FATAL ERROR: Cannot load symbol MKLMPI_Get_wrappers.
Intel MKL FATAL ERROR: Cannot load symbol MKLMPI_Get_wrappers.

0 Kudos
VidyalathaB_Intel
Moderator
2,171 Views

Hi,

 

Thanks for sharing the code and steps to reproduce the issue.

 

Yes, the issue is reproducible.

Regarding the runtime error that you are getting, I tried following the suggestions from the link line advisor (as I mentioned in my earlier post) and don't see those errors. Here is the screenshot.

VidyalathaB_Intel_0-1658297686894.png

 

linking command:

 

icc main.o -L${MKLROOT}/lib/intel64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_intelmpi_lp64 -liomp5 -lpthread -lm -ldl -lmpi

 

But the return value of the function MKL_Set_mpi is still set to -3 as you mentioned in your first post. When searched regarding the same in the MKL manual, it is already mentioned like, the function returns an error with other configurations and maybe it is the reason we are getting that as -3.

mkl_set_mpi

"Call this function to set the MPI implementation to be used by Intel® oneAPI Math Kernel Library on Windows* OS when dynamic Intel® oneAPI Math Kernel Library libraries are used. For all other configurations, the function returns an error indicating that you cannot set the MPI implementation. You can specify your own prebuilt dynamic BLACS library for a custom MPI by setting vendor to MKL_BLACS_CUSTOM

 

Please try the suggestions from the Link Line advisor which helps in resolving the issue with runtime error and do let me know if you have any concerns.

 

Regards,

Vidya.

 

0 Kudos
Iakov
Beginner
2,154 Views

Hi Vidya,

 

Unfortunately I cannot accept linking solution, because the main purpose is to build a binary without any Lapack/Blacs/ScaLapack symbols and during execution depending on some logic load proper library and MKL could be one of the possible libraries.

 

I did some more tests:

1. On Windows everything is working out of the box without any additional calls/setting environment (I suspect that mpiexec for different MPI libraries set some environment and MKL makes a decision depending on this values). I checked for both IMPI and MSMPI.

2. When I explicitly called MKL_Set_mpi for Windows implementation I again got an error for both IMPI and MSMPI:
Intel MKL FATAL ERROR: Error on loading function mkl_serv_set_mpi.

So the main question is still the same: How to make this logic working on Linux?

The minor question is about not working function MKL_Set_mpi on Windows, but it's not important for me as long as for Windows everything is working without any additional steps.

 

PS. Reproducible for windows attached. The same MKL version where tested as previously:  2018.0.4 and 2022.0.0.

0 Kudos
VidyalathaB_Intel
Moderator
2,098 Views

Hi,


We are working on your issue. we will get back to you soon.


Regards,

Vidya.


0 Kudos
Khang_N_Intel
Employee
2,086 Views

Hi Lakov,


I will definitely discuss with the team and will get back to you with the next step.


Best,

Khang


0 Kudos
Khang_N_Intel
Employee
1,987 Views

Hi Lakov,


The mkl team is looking into this issue.

I will let you know once I receive any news.


Best regards,

Khang


0 Kudos
Khang_N_Intel
Employee
1,982 Views

Hi Lakov,


oneMKL libraries cannot resolve internal dependencies on Linux if they were loaded with RTLD_LOCAL (default option for dlopen).


So, for the work-around, you will need to add the option RTLD-GLOBAL in the dlopen calls by replacing RTLD_LAZY with (RTLD_LAZY | RTLD_GLOBAL).


void* rt_lib_ptr = dlopen("libmkl_rt.so", RTLD_LAZY | RTLD_GLOBAL);

void* intel_lib_ptr = dlopen("libmkl_intel_lp64.so", RTLD_LAZY | RTLD_GLOBAL);

void* blacs_lib_ptr = dlopen("libmkl_blacs_intelmpi_lp64.so", RTLD_LAZY | RTLD_GLOBAL);

void* scalapack_lib_ptr = dlopen("libmkl_scalapack_lp64.so", RTLD_LAZY | RTLD_GLOBAL);


The fix will be in the the oneMKL 2023.0 release.


Best regards,

Khang




0 Kudos
Iakov
Beginner
1,921 Views

Hi Khang,

 

Thanks for solution! I checked it for our project and everything now works fine.

 

Sincerely,

Iakov

0 Kudos
Reply