dlopen woes

[nflowers]

I'm trying to glue together a scripting language and MKL. The mode of operations is: scripting language dynamically loads my .so (via dlopen -- critically, with RTLD_LOCAL), and exposes my routines to the language. My glue code gets called, transforms scripting structures to MKL arguments, calls MKL, transforms results into scripting structures and returns.

Unfortunately, when the library is is first invoked, it bombs out with

*** libmkl_mc3.so *** failed with error : /usr/local/mkl/10.1.3.027/lib/em64t/libmkl_mc3.so: undefined symbol: mkl_dft_commit_descriptor_s_c2c_md_omp
*** libmkl_def.so *** failed with error : /usr/local/mkl/10.1.3.027/lib/em64t/libmkl_def.so: undefined symbol: mkl_dft_commit_descriptor_s_c2c_md_omp
MKL FATAL ERROR: Cannot load neither libmkl_mc3.so nor libmkl_def.so


After futzing around for several hours, this seems to be the situation:

(for reference, I am running linux 2.6 x86_64 with g++)

I have a small test library libtest.so that is linked as follows

"g++" -L"/usr/local/mkl/10.1.3.027/lib/em64t" -o "bin/gcc-4.3.3/debug/threading-multi/libtest.so" -Wl,-h -Wl,libtest.so -shared -Wl,--start-group "bin/gcc-4.3.3/debug/threading-multi/test.o" -Wl,-Bstatic -Wl,-Bdynamic -lmkl_gf_lp64 -lmkl_gnu_thread -lmkl_lapack -lmkl_core -liomp5 -lrt -Wl,--end-group -g -pthread

I think these are the correct dependencies, though I'm by no means confident.

1. If I write a test binary that directly links against my test function (which calls into lapack), it works just fine.
2. If I write a test binary that at run-time calls dlopen(RTLD_GLOBAL) on libtest.so (not linked in as an execution dependency), it works just fine.
3. If I write a test binary that at run-time calls dlopen(RTLD_LOCAL), I get the error above.

I think what's going on is that MKL is internally calling dlopen to intialize stuff. However, in the third case above, the symbols from libtest.so aren't added to the executables space, so it can't resolve them. I don't have the option of changing the way the scripting executable loads shared objects, even if that were the preferred solution.

For reference, this works:

4. if I invoke my binary #3 with LD_PRELOAD of libtest.so, the symbols get added to the global space, and it works. This also works with the scripting language. But it is a terrible solution for a lot of obvious reasons.

Anyone have any better approaches?

[Gennady_F_Intel]

Quoting - nflowers

I have a small test library libtest.so that is linked as follows

"g++" -L"/usr/local/mkl/10.1.3.027/lib/em64t" -o "bin/gcc-4.3.3/debug/threading-multi/libtest.so" -Wl,-h -Wl,libtest.so -shared -Wl,--start-group "bin/gcc-4.3.3/debug/threading-multi/test.o" -Wl,-Bstatic -Wl,-Bdynamic -lmkl_gf_lp64 -lmkl_gnu_thread -lmkl_lapack -lmkl_core -liomp5 -lrt -Wl,--end-group -g -pthread

If you want to link with intel threading libraries, then please use mkl_intel_thread.a instead of mkl_gnu_thread you used,
i.e -Wl,--start-group $MKLPATH/libmkl_gf_lp64.a $MKLPATH/libmkl_intel_thread.a $MKLPATH/libmkl_core.a -Wl,--end-group -liomp5 -lpthread

Please look at the tool for finding the right Intel Math Kernel Library (Intel MKL) file for you to link with your application "http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor/"

--Gennady

[nflowers]

Quoting - Gennady Fedorov (Intel)

If you want to link with intel threading libraries, then please use mkl_intel_thread.a instead of mkl_gnu_thread you used,
i.e -Wl,--start-group $MKLPATH/libmkl_gf_lp64.a $MKLPATH/libmkl_intel_thread.a $MKLPATH/libmkl_core.a -Wl,--end-group -liomp5 -lpthread

Please look at the tool for finding the right Intel Math Kernel Library (Intel MKL) file for you to link with your application "http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor/"

--Gennady

Thanks Gennady,

I've fixed the link-line as follows

"g++" -L"/usr/local/mkl/10.1.3.027/lib/em64t" -o "bin/gcc-4.3.3/debug/threading-multi/libtest.so" -Wl,-h -Wl,libtest.so -shared -Wl,--start-group "bin/gcc-4.3.3/debug/threading-multi/test.o" -Wl,-Bstatic -Wl,-Bdynamic -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_lapack -lmkl_core -liomp5 -lrt -Wl,--end-group -g -pthread

But still have the problem I originally posted. Any thoughts?

Thanks,
Nick.

[Vladimir_Lunev]

Hi nflowers,

I think the key point of Gennady reply is that you need to use MKL static libraries to build your libtest.so (libmkl_gf_lp64.a instead of -lmkl_gf_lp64 and so on). Note "MKL Custom Builder tool" can help you to build library. You can find it in the directory /tools/builder.

We know the issue you mention and we are trying to resolve it.
The workaround is to use dlopen(RTLD_GLOBAL). But I see this way is not applicable for you. Could I ask - why you can't use RTLD_GLOBAL flag?

Thanks,
-Vladimir

[nflowers]

Vladimir,

Thanks very much.

Unfortunately, the maintainer of the scripting language I'm interfacing with (I don't own the source) considers it bad practice to open third party libraries with RTLD_GLOBAL because they'll clutter the namespace and potentially collide with one another. His case is maybe reasonable, maybe not.

1. Linking statically does work! (Apologies to Gennady for missing the thrust of his response: I read the emphasis on the fact I was linking with gnu_thread instead of intel_thread, which was indeed incorrect.) For the record, I have to link statically with mkl_intel_thread and mkl_lapack. (I can leave mkl_core, mkl_intel_lp64 and iomp5 as dynamic: I certainly don't claim to understand what distinguishes these from the first two.)

I guess the (mild) downside is that code size is increased and I'll have to recompile with new versions of MKL.

2. Here's another solution I found: if inside of my glue, I issue my own dlopen("libtest.so", RTLD_LAZY|RTLD_GLOBAL), ie, reopen myself in global mode, then all the libraries can be brought in dynamically.

I guess the (mild) downside is that the source code needs to know it's own name and use case (ie, the fact that I'm going to be a dynamic library).

Either of these options is far better than LD_PRELOAD.

Thanks for the help,
Nick.

Quoting - Vladimir Lunev (Intel)

Hi nflowers,

I think the key point of Gennady reply is that you need to use MKL static libraries to build your libtest.so (libmkl_gf_lp64.a instead of -lmkl_gf_lp64 and so on). Note "MKL Custom Builder tool" can help you to build library. You can find it in the directory /tools/builder.

We know the issue you mention and we are trying to resolve it.
The workaround is to use dlopen(RTLD_GLOBAL). But I see this way is not applicable for you. Could I ask - why you can't use RTLD_GLOBAL flag?

Thanks,
-Vladimir

[AndrewC]

I am building a python module that uses MKL and I am having the same issue as this poster.
Has this been resolved? I am using MKL 10.2 Update 5

Linking with the static libraries of MKL is not a good option for me as it is not acutally the python module that is using MKL, more other DLLs that the python module depends on ( that ultimately call MKL)

[Artem_V_Intel]

Hello vasci_intel,

Could you please specify what MKLfunction causes this problem?

I attached a simple example of Python module linked against MKL to this post. This module uses dynamic MKL libraries. It contain only one callable function vdMul4.

Invoke the next command to build this module:
$ python setup.py build

Here is an example of usage of this module:
>>> import vmul
>>> vmul.vdMul4((1.0, 2.0, 3.0, 4.0), (3.0, 3.0, 3.0, 3.0))
(3.0, 6.0, 9.0, 12.0)
>>>

This module uses MKL 10.2.5.035 for Linux.

Best regards,
Artem

[AndrewC]

Hi Artem,
My python module is a python module ( SWIG generated) that interfaces to a library that calls most areas of MKL ( BLAS, LAPACK, VML, FFT)

I have tried modifying sys.dlopenflags in python without any luck so far.

So , now forgetting python for the moment, I now have a test program as follows

[cpp]#include "dlfcn.h"
#include "stdio.h"

int main()
{
	void* result=dlopen("./test.so",RTLD_GLOBAL|RTLD_LAZY);
  printf("%p %sn",result, dlerror());
}
[/cpp]

I cannot find any combination of dlopen flags that will "dlopen" the .so without MKL complaining with the error


*** libmkl_mc.so *** failed with error : libmkl_mc.so: undefined symbol: mkl_dft_commit_descriptor_s_c2c_md_omp
*** libmkl_def.so *** failed with error : libmkl_def.so: undefined symbol: mkl_dft_commit_descriptor_s_c2c_md_omp
MKL FATAL ERROR: Cannot load neither libmkl_mc.so nor libmkl_def.so


AS the OP noted also, if I link "directly" to the .so and make a call to a function in it, I have no problems

icpc test2.c test.so -o test2


Andrew

[barragan_villanueva_]

Could you please try linking without -Wl,-Bstatic Wl,-Bdynamic (as Vladimir suggested) but -Bstatic -Bdynamic (and corrected grouping libraries) to create libtest.so:

"g++" -L"/usr/local/mkl/10.1.3.027/lib/em64t"-o "bin/gcc-4.3.3/debug/threading-multi/libtest.so" -Wl,-h -Wl,libtest.so -shared "bin/gcc-4.3.3/debug/threading-multi/test.o" -Bstatic -lmkl_gf_lp64-Wl,--start-group -lmkl_gnu_thread -lmkl_lapack -lmkl_coreWl,--end-group -Bdynamic -liomp5 -lrt --g -pthread

instead of

"g++" -L"/usr/local/mkl/10.1.3.027/lib/em64t" -o "bin/gcc-4.3.3/debug/threading-multi/libtest.so" -Wl,-h -Wl,libtest.so -shared -Wl,--start-group "bin/gcc-4.3.3/debug/threading-multi/test.o" -Wl,-Bstatic -Wl,-Bdynamic -lmkl_gf_lp64 -lmkl_gnu_thread -lmkl_lapack -lmkl_core -liomp5 -lrt -Wl,--end-group -g -pthread

[AndrewC]

Linking against the MKL static libraries is not a solution for me. The "module" I am trying to open with dlopen depends on another DLL that then depends on the MKL libraries and so on. The module itself does not need to even need to link against MKL - it does not make any calls to MKL directly.

I just don't "get" why dlopen behaves so differently to just linking...

[barragan_villanueva_]

Well, you can create needed MKL-custom dll (created from static MKL libraries specifying list of used MKL functions). And after that it will be dlopen'ed at every "module" correctly.

MKL dynamic libraries havedependecy onMKL CPU-specific internal libraries and so direct dlopen is not suitable for such linking you use.

[AndrewC]

Ok, thanks for the advice, it seems suprising to me that there is NO solution to this apart from the custom DLL approach.

It's not like I am getting a long list "missing functions". Just two omp functions that are in mkl_intel_thread

[Artem_V_Intel]

Hi Andrew,

We reproduced this problem. This issue escalated against our engineering team already and this problem will be fixed in the future releases of MKL. I will inform you if any updates will be available. Here is a tracker number for your reference DPD200083570.

For MKL 10.2.5 workaround is available. Additional libraries should be added into the link line of your dynamic library. For instance if your error message looks like
*** libmkl_p4n.so *** failed with error : /opt/intel/Compiler/11.1/046/mkl/lib/em64t/libmkl_p4n.so: undefined symbol: mkl_dft_commit_descriptor_s_c2c_md_omp
*** libmkl_def.so *** failed with error : /opt/intel/Compiler/11.1/046/mkl/lib/em64t/libmkl_def.so: undefined symbol: mkl_dft_commit_descriptor_s_c2c_md_omp
MKL FATAL ERROR: Cannot load neither libmkl_p4n.so nor libmkl_def.so
you should add -lmkl_p4n and -lmkl_def.

Please take a look at the test case I attached. It contains source of dynamic linking library and test program."test" target produces testing program. "all" target produces "broken" dll thatshow the issue. "all_workaround" target produces working dynamic library. Please let me know if this workaround will not work for you.

Best regards,
Artem

[barragan_villanueva_]

Just some remark to the suggested workaround:

instead of directly used there
-lmkl_p4n and -lmkl_def
it needs using all MKL CPU-specific libaryes or at least those which were in error messages

[AndrewC]

Hi Artem, Victor,

That works! If I add -lmkl_def and -lmkl_mc to the link line the problem is fixed. Following on from what Victor said, I should add all the mkl_xxx tp cover all CPU types.

Thanks a lot!

Andrew

[Ying_H_Intel]

To who'm concerns,

The mkl dynamic library issue will be fixed thoroughly in MKL 10.3 and future version. A new librar mkl_rt was introduced. Now to build the custom MKL library based on dynamic MKL, one can easilyuse the library libmkl_rt.so instead of -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core

The new mkl version will be integrated to next intel Compiler version too, which targeted to be release in Nov. The system will send the notification. You are welcomed to try it by that time.

Thanks
Ying