Cannot call OpenMP functions from libiomp after calling from libomp

[du_boisberranger__je]

Hi,

when a program loads both libiomp and libomp with dlopen, calling an OpenMP function from one then from the other one can cause the program to terminate, depending on the order of the calls.

calling from libiomp first, then from libomp is fine. The issue is when calling from libomp first and then from libiomp.

Here is a reproducible example: test.c

#include <stdlib.h>
#include <dlfcn.h>

int main(int argc, char **argv){
    // load libomp and libiomp
    void *libomp_handle = dlopen("path/to/libomp.so", RTLD_NOW);
    void *libiomp_handle = dlopen("path/to/libiomp5.so", RTLD_NOW);
   
    // get "omp_get_max_threads" from each library
    int (*libiomp_get_max_threads)() = (int (*)()) dlsym(libiomp_handle, "omp_get_num_threads");
    int (*libomp_get_max_threads)() = (int (*)()) dlsym(libomp_handle, "omp_get_max_threads");

    // switch order of calls causes SIGABRT
    libomp_get_max_threads();
    libiomp_get_max_threads();

    return 0;
}

Compile and run:

    icc test.c -o test -ldl && ./test

Error Message:

OMP: Error #13: Assertion failure at z_Linux_util.cpp(2338). OMP: Hint Please submit a bug report with this message, compile and run commands used, and machine configuration info including native compiler and operating system versions. Faster response will be obtained by including all program sources. For information on submitting this issue, please see http://www.intel.com/software/products/support/.

Backtrace:

#0 0x00007ffff783e428 in raise () from /lib/x86_64-linux-gnu/libc.so.6

#1 0x00007ffff784002a in abort () from /lib/x86_64-linux-gnu/libc.so.6

#2 0x00007ffff6f7b2c3 in __kmp_abort_process () at ../../src/kmp_runtime.cpp:481

#3 0x00007ffff6f66237 in __kmp_fatal (message=...) at ../../src/kmp_i18n.cpp:894

#4 0x00007ffff6f3a82c in __kmp_debug_assert (msg=0x76c5 <error: Cannot access memory at address 0x76c5>, file=0x76c5 <error: Cannot access memory at address 0x76c5>, line=6) at ../../src/kmp_debug.cpp:100

#5 0x00007ffff6fb57ee in __kmp_is_address_mapped (addr=0x76c5) at ../../src/z_Linux_util.cpp:2337

#6 0x00007ffff6f78f87 in __kmp_register_library_startup () at ../../src/kmp_runtime.cpp:6684

#7 0x00007ffff6f7a49b in _INTERNAL_25_______src_kmp_runtime_cpp_ddfed41b::__kmp_do_serial_initialize () at ../../src/kmp_runtime.cpp:6834

#8 _INTERNAL_25_______src_kmp_runtime_cpp_ddfed41b::__kmp_do_middle_initialize () at ../../src/kmp_runtime.cpp:7146

#9 __kmp_middle_initialize () at ../../src/kmp_runtime.cpp:7255

#10 0x00007ffff6f5ad75 in __kmp_api_omp_get_max_threads () at ../../src/kmp_ftn_entry.h:366

#11 0x00000000004006ab in main ()

Some remarks:

- Loading libgomp instead of libomp does not raise the error

- Link libraries instead of dlopen does not raise the error

- I know it's bad practice to load different OpenMP runtimes simultaneously. And in fact I have to set KMP_DUPLICATE_LIB_OK=TRUE otherwise it does not work at all. But in the python ecosystem, you can easily install packages from pre-built binaries and we don't know in advance which compiler has been used. For instance if one package uses MKL and another one uses OpenMP and is built with clang.

Thanks for your help

[Viet_H_Intel]

GNU also gave a segmentation fault on your test case. 

$ gcc t.c -o g.out -ldl -O2&&./g.out
Segmentation fault (core dumped)

[du_boisberranger__je]

Sorry I may have not been cleared enough. The issue does not come from the compilation. It comes from the OpenMP library.

The issue is about the fact that intel's OpenMP library (libiomp5) and llvm's OpenMP library (libomp) can't coexist in a same program.

Maybe it was not the right place to ask but I did not find a forum dedicated to OpenMP. Intel compiler seemed the closest to me. If not, could you tell me the right place to ask ?

Thank you

[jimdempseyatthecove]

>>The issue is about the fact that intel's OpenMP library (libiomp5) and llvm's OpenMP library (libomp) can't coexist in a same program.

Why would you expect them to co-exist???

    int (*libiomp_get_max_threads)() = (int (*)()) dlsym(libiomp_handle, "omp_get_num_threads");
    int (*libomp_get_max_threads)() = (int (*)()) dlsym(libomp_handle, "omp_get_max_threads");

first call of libiomp omp_get_num_threads may likely make additional calls (e.g. to initialize the Intel OpenMP runtime library)
fist call of libomp omp_get_max_threads may likely make additional calls (e.g. to initialize the GCC OpenMP runtime library)

*** both libraries likely (most certainly) have conflicting entry point names, and global context buffer names

*** even worse, consider

#pragma omp parallel

Who's thread pool do you intend to be used???

I think you need to resolve your problem differently.

Jim Dempsey

[du_boisberranger__je]

>> e.g. to initialize the GCC OpenMP runtime library

Actually there is no conflict with libgomp. Only with libomp, LLVM OpenMP.

>> Why would you expect them to co-exist???

In the python ecosystem, the main way to install packages is to download pre-built binaries compiled by the package developers themselves. So it's very easy to end up using different packages compiled with different compilers in a same python program.

The way importing packages works in python is through dlopen. So a typical use case would look like

- a dlopen of one shared library liba.so. This lib has a function func_a which makes an OpenMP loop linked to libiomp5.

- a dlopen of another shared library libb.so. This lib has a function func_b which makes an OpenMP loop linked to libomp.

I think we should be able to call func_a and func_b in a same program.

>> #pragma omp parallel

>> Who's thread pool do you intend to be used???

This does not happen in the situation I exposed above.

[jimdempseyatthecove]

me>> Who's thread pool do you intend to be used???

you>>This does not happen in the situation I exposed above.

Actually, by your own statement, it does:

you>>for instance if one package uses MKL and another one uses OpenMP and is built with clang.

Both MKL and the clang portion of the code using OpenMP make OpenMP runtime library calls that reference (in your application loading both OpenMP runtime libraries) name colliding external symbol references accessed via a once-only shared library loader dispatch table. Should one library runtime system use the other library runtime system routines then expect undefined behavior (including crash).

I suggest that you consider having your Python script launch separate processes in this situation.

Jim Dempsey

[du_boisberranger__je]

Sorry my knowledge in shared libraries is quite limited. I understand you may find that my questions or remarks does not make sense :) Thanks for your explanations.

Since it seems to work fine when loading libgomp, I assume that it's not the name clashing of the OpenMP public api that is causing the issue, but maybe the name clashing of libomp and libiomp private apis, which are the same. It's probably a very naive question but wouldn't that be fixed if both had different names for their private apis (something like libomp prepend all its functions by _llvm and libiomp by _intel) ? Although I doubt that this will ever happen :)