Intel® C++ Compiler
Community support and assistance for creating C++ code that runs on platforms based on Intel® processors.
7942 Discussions

Allow to statically link libiomp5.a to build a shared object

Riccardo_V_
Beginner
542 Views

Hi,

I know this problem is known and i know that apparently the libiomp5.a library is designed intentionally not to work in shared objects. The point is that i would need this behaviour and i would appreciate intel allowing this possibility.

I explain my problem: i am building a python extension in c and i am trying to use openmp and mkl_lapacke. They both rely on libiomp5.so and so my c-program initialize such lib at its first parallel region.

On the other hand at a higher level i use python and in particular i use the numpy module for several calculations. Now it happens that numpy is built using mkl and thus it also initialize libiomp5.so

This creates a wonderful runtime error of the type:

OMP: Error #15: Initializing libiomp5.so, but found libiomp5.so already initialized.
OMP: Hint: This may cause performance degradation and correctness issues. Set environment variable KMP_DUPLICATE_LIB_OK=TRUE to ignore this problem and force the program to continue anyway. Please note that the use of KMP_DUPLICATE_LIB_OK is unsupported and using it may cause undefined behavior. For more information, please see http://www.intel.com/software/products/support/

It took me a while to figure out this was the problem but now it is clear. Setting KMP_DUPLICATE_LIB_OK is not an option i am doing calculations and i have to be reasonably sure about the results. I could make my program work compiling with gcc that links to the gomp library and for the linear system solver i used openblas. In this way my program is not calling at all libiomp5, but only numpy is doing that. No double initialization occurs.

Now the point is that i would like to use mkl_lapacke and intel compiler for my program, instead. To do this i think a solution would be to link statically libiomp5.a inside my program so that numpy and my extension work on different copies of libiomp5 library and they do not try to double initialize the same one.

So, to conclude, why is libiomp5.a library designed to not work in shared objects? And what would be the way to avoid the double initialization otherwise?

As python is interpreted, every imported module loads its own shared libs and there is no way (as far as i know) that two python modules can know they are using the same shared library.

Thanks,

Regards,

Riccardo

0 Kudos
7 Replies
SergeyKostrov
Valued Contributor II
542 Views
>>...t took me a while to figure out this was the problem but now it is clear. Setting KMP_DUPLICATE_LIB_OK >>is not an option i am doing calculations and i have to be reasonably sure about the results. It seems to me that KMP_DUPLICATE_LIB_OK was created to workaround the problem with double-initialization of OpenMP libraries. If it helps to resolve the problem then I wouldn't worry. >>...So, to conclude, why is libiomp5.a library designed to not work in shared objects? I think this is a very old legacy issue and only a person who designed that could explain it.
0 Kudos
Riccardo_V_
Beginner
542 Views

Hi Sergey,

thanks for your answer

Sergey Kostrov wrote:

>>...t took me a while to figure out this was the problem but now it is clear. Setting KMP_DUPLICATE_LIB_OK
>>is not an option i am doing calculations and i have to be reasonably sure about the results.

It seems to me that KMP_DUPLICATE_LIB_OK was created to workaround the problem with double-initialization of OpenMP libraries. If it helps to resolve the problem then I wouldn't worry.

But i mean... they warn that this can cause correctness issues. "Generically", not better specified. The results i obtained with the first trial were correct but i don't know what should i avoid to keep the library problem free.. Furthermore i could see a significant initial slow down of the parallel version compared to the serial one.. so in my case this library become kind of useless..

At this point, i prefer to use gcc and gomp with openblas.

Sergey Kostrov wrote:

>>...So, to conclude, why is libiomp5.a library designed to not work in shared objects?

I think this is a very old legacy issue and only a person who designed that could explain it.

I probably should expose the problem to intel and try to understand better this library but don't know how, i thought this forum would serve this purpose also?

 

0 Kudos
Amanda_S_Intel
Employee
542 Views

FYI - this topic was recently discussed in this thread: https://software.intel.com/en-us/forums/topic/517419

 

0 Kudos
Riccardo_V_
Beginner
542 Views

Hi Amanda,

thanks for the indicated thread. Inside i find (sadly) confirmation of what i wrote in my first post: intel is willing to forbid static linking of libiomp5.a

This means that it is impossible for me to use icc and mkl to compile my python module, because it becomes impossible to avoid double initialization of libiomp5.so, already initialized by numpy in a previous calculation.

How could one solve this problem otherwise?

0 Kudos
TimP
Honored Contributor III
542 Views

The reason for encouraging you to link against a single dynamic copy of OpenMP library is that prevents double initialization of OpenMP library and allows it to manage threads in all parts of your application.

0 Kudos
jimdempseyatthecove
Honored Contributor III
542 Views

>>As python is interpreted, every imported module loads its own shared libs and there is no way (as far as i know) that two python modules can know they are using the same shared library.

Isn't it SOP to test to see if a shared library is already loaded before you load another copy of it? And if it is already loaded, then obtain the desired entry points without calling an initialization function.

Jim Dempsey

0 Kudos
Riccardo_V_
Beginner
542 Views

jimdempseyatthecove wrote:

>>As python is interpreted, every imported module loads its own shared libs and there is no way (as far as i know) that two python modules can know they are using the same shared library.

Isn't it SOP to test to see if a shared library is already loaded before you load another copy of it? And if it is already loaded, then obtain the desired entry points without calling an initialization function.

Jim Dempsey

This seems to center the point. But mmh.. What do you mean by SOP? for example how should i call my lapacke_dgesv routine in my c code (or how should i load numpy module and my module) to avoid double initialization?

0 Kudos
Reply