Creating an SO, but how to statically pull in Fortran re-distributable SOs?

[Tony_Garratt]

Sorry if this is an obvious question, but I could not find anything in the documentation about this topic.

We are building a dynamic DLL (Windows) and SO (Linux) of C++ and Fortran code using Intel C++/Fortran 11.1 compilers (wecreate the DLL/SO using the C++ linker by the way).The DLL/SO is loaded by a third party application which itself happens to be built using Fortran 10. We understand that the third party application itself loads the Intel 10 Fortran re-distributabledependent DLLs/SOs itsell at run-time.

The Windows version works fine. Butfor the Linux version, we are seeing unresolved symbols when the application loads our SO. This appears to be because our V11 SO is trying to find symbols that are not in the V10 dependent SOs that were loaded by the application.

Now, if I use the dependency walker on the Windows DLL, I only see a dependancy on libiomp5.dll. If I use ldd on the linux version, I see links toa whole bunch ofre-distributables (libifport etc). My question is this: is there a way we can build an SO on Linux that "binds" these re-distributables into the SO at link time so that they do not need to be loaded at run-time - i.e. the same way as appears to be happening with our Windows dLL? This would avoid our clash with v10.

[mecej4]

The unresolved symbols are probably caused by linking to the newer shared libraries and attempting to run on a system where the older libraries are encountered and do not contain those symbols.

Chapter 5 of the MKL Library User Guide for Linux has a section Building custom shared objects that you may find useful.

[Tony_Garratt]

Thank you for the reference, which I read. If I understand correctly, this allows you to create your own SO that contains MKL functions you are using bound into the SO, so there is no chance of your SO picking up other versions (since other application codes may also use the BLAS) dynamically at run-time.

We are using MKL in our SO, but our problem is that we want to bind to the Intel Fortran (and perhaps C++ too) runtime libraries into our DLL, not just MKL. Using the custom builder would require us to list the entire symbol contents of the re-distributable SOs, right? And that list could change from release to release, which is somewhat of a maintainence headache.

Are my assumptions correct though: this is not an issue on Windows, but is on Linux? I was hoping to find a simple link option that statically pulls in the re-distributables into my SO directly, so that there is no need to pull them in dynamically at run-time. I guess there is no such one and custom SO building is the only way to go?

[mecej4]

I suggested that you look at the MKL reference not because I thought that you used MKL calls in your application, but as it gave examples of building custom shared libraries. With the recent versions of MKL, the linking has become so complex that Intel provided a way to build a custom MKL so that, once built for the specific selected configuration, a single shared library would be all that subsequent applications would be linked to.

I doubt that what you describe is a Windows versus Linux issue. I have seen, on both operating systems, applications running into trouble because the installed DLL/SO and the expected DLL/SO did not match.

For example, there are still some old Linux apps compiled with gcc 2.95 still in use that, when run, complain of missing _errno. Similar problems occur with various versions of the VC DLLs such as msvcrt.dll.

I remember an incident where, in Linux, libc.so.6 was 'updated' and that caused the system to crash because almost every Linux command depends on libc.so.6!

In the part of your code that loads the DLL/SO explicitly, you may have to check the version number after a successful load.

[John4]

Besides the suggestions given by mecej4, the -static-intel switch can be helpful on Linux and Mac OS X. That switch doesn't seem to have a Windows equivalent, though.

[jimdempseyatthecove]

It is unfortunate that there isn't a #pragma or -depreciate... such that you can specify the interface is compatible across a series of versions, or since a specific version, or excepting versions, .... Many API's have been firm since version 1.0 of various libraries. This would make life easire in making redistributable libraries.

Jim

[Tony_Garratt]

I tried the -static-intel idea - according to the documentation is should do exactly what we need. However, I got an error message - see below. Is this telling me that I should re-compile my code with -static-intel or use a different version of libiomp5.a?

%icpc -shared -fPIC -o ourSO.a someofourobjectfiles.o someofourlibs.a /opt/intel/Compiler/11.1/069/mkl/lib/em64t/libmkl_blas95_lp64.a -Wl,--start-group /opt/intel/Compiler/11.1/069/mkl/lib/em64t/libmkl_intel_lp64.a /opt/intel/Compiler/11.1/069/mkl/lib/em64t/libmkl_intel_thread.a /opt/intel/Compiler/11.1/069/mkl/lib/em64t/libmkl_core.a -Wl,--end-group -Lsomeofourlibs -lifport -lifcore -limf -lsvml -lstdc++ -L/opt/intel/Compiler/11.1/069/mkl/lib/em64t -lpthread -openmp -static-intel -L/opt/intel/Compiler/11.1/069/lib -ldl

ld: /opt/intel/Compiler/11.1/069/lib/intel64/libiomp5.a(kmp_csupport.o): relocation R_X86_64_32 against `a local symbol' can not be used when making a shared object; recompile with -fPIC

/opt/intel/Compiler/11.1/069/lib/intel64/libiomp5.a: could not read symbols: Bad value

[mecej4]

On Linux X64, the ICC installation's lib directory contains three pairs (static, shared) of OpenMP libraries: libiomp5, libiompprof5 and libiompstubs5.

Rather than using a compiler switch to link in OpenMP, you may need to use ld with the proper switches to coax it to use the shared library libiomp5.so instead of lthe static library ibiomp.a.

[TimP]

This simply tells you the static objects in libiomp5 aren't relocatable. It's probably intentional, to require objects from libiomp5.a to be linked only once at the end. Various bad things happen when portions of the OpenMP run-time are linked at different stages.

[John4]

The error seems to suggest that the libiomp5.a library was not compiled with the -fPIC switch, so is not compatible with the rest of of your object code.

Your best choice is to try to link against as many static libraries as possible, explicitly, say:

[bash]$ INTEL_PATH=/opt/intel/Compiler/11.1/069
$ icpc -shared -fPIC -o ourSO.so someofourobjectfiles.o someofourlibs.a -L${INTEL_PATH}/lib \
-L${INTEL_PATH}/mkl/lib/em64t -Lsomeofourlibs -Wl,--start-group libmkl_blas95_lp64.a \
libmkl_intel_lp64.a libmkl_intel_thread.a libmkl_core.a libifport.a libifcore.a libimf.a \
libsvml.a -Wl,--end-group -lstdc++ -lpthread -ldl -liomp5[/bash]

You can also try to see if the static versions of -lpthread and -ldl work with the rest of your object code; keep in mind that such option might restrict compatibility with different versions of the C library (I'm assuming -lpthread and -ldl are in /usr/lib; modify accordingly):

[bash]$ icpc -shared -fPIC -o ourSO.so someofourobjectfiles.o someofourlibs.a -L${INTEL_PATH}/lib \
-L${INTEL_PATH}/mkl/lib/em64t -Lsomeofourlibs -Wl,--start-group libmkl_blas95_lp64.a \
libmkl_intel_lp64.a libmkl_intel_thread.a libmkl_core.a libifport.a libifcore.a libimf.a libsvml.a \
libpthread.a libdl.a  -Wl,--end-group -lstdc++ -liomp5
[/bash]

There's also a sequential version of Intel MKL, in case you want to avoid linking against (the shared version of) -liomp5 ---check the Intel MKL documentation for details.

[Tony_Garratt]

Thank you all for your suggestions - I will give them a try.

[Tony_Garratt]

Unfortunately, I still find the .so requires the redestibutables, ever after using -static-intel and including them explicit aslibifport.a etc. The output of ldd shows me

libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x0000002a964e1000)

libpthread.so.0 => /lib64/tls/libpthread.so.0 (0x0000002a966d1000)

libdl.so.2 => /lib64/libdl.so.2 (0x0000002a967e6000)

libm.so.6 => /lib64/tls/libm.so.6 (0x0000002a968ea000)

libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000002a96a70000)

libc.so.6 => /lib64/tls/libc.so.6 (0x0000002a96b7b000)

libimf.so => /opt/intel/Compiler/11.1/069/lib/intel64/libimf.so (0x0000002a96db0000)

libsvml.so => /opt/intel/Compiler/11.1/069/lib/intel64/libsvml.so (0x0000002a97142000)

libiomp5.so => /opt/intel/Compiler/11.1/069/lib/intel64/libiomp5.so (0x0000002a97358000)

libintlc.so.5 => /opt/intel/Compiler/11.1/069/lib/intel64/libintlc.so.5 (0x0000002a9750a000)

/lib64/ld-linux-x86-64.so.2 (0x000000552aaaa000)

[Tony_Garratt]

It just occurred to me that one conclusion from my experiments is that -static-intel does not work! The doc clearly states:

This option causes Intel-provided libraries to be linked in statically. It is the opposite of -shared-intel.

and ldd is reporting that it still is trying to pull in the .so's....

[TimP]

libiomp5 specifically is excluded from -static-intel. It has its own options under ifort and icc for static vs. dynamic and default to profile or not.

[Tony_Garratt]

Hi Tim,
I understand that liomp5 has to be pulled in dynamically, but the fact that the other libs are still being pulled in makes me think that static-intel is not working for those. What do you think?

Tony

[TimP]

As pointed out in another post, the objects in the ifort .a libraries likely aren't built with -fPIC, so would not be suitable for inclusion in your .so. I don't know whether the link script builder is taking that into account automatically. Sorry, I haven 't fully followed what you are trying to do.

[John4]

In my previous post I emphasized on the word "explicitly", to imply that you shouldn't rely on the (probably) obscure behavior of the -static-intel switch. Have you tried compiling without that switch? What's the outcome?

Looking at the compiler's invocation of ld (through the -# switch), it seems that the compiler is doing the right thing (by using the -Bstatic switch right before the libraries provided by Intel), so maybe it's ld who is in fault here.

In Linux, ld is designed to look for lib.so libraries by default, whenever a -l option is provided, so maybe there's an undocumented rule (**) stating that ld can choose to ignore the -Bstatic switch for compatibility reasons ---as Tim suggested, the -fPIC requirement for shared objects limits your options, since GCC's runtime must be linked dynamically.

If instead of -l implied by -static-intel your command provides lib.a explicitly, ld will have to honor the request (probably unsuccessfully) .

(**) According to Google, ld used to have a bug by which "Cascading of shared object defeat[ed] the -Bstatic option"... Maybe the bug was fixed by simply removing that line from the man page.