Creating a shared library

Petros · ‎01-27-2014

Hello everybody,

I've been trying to create a shared library from some fortran code in order to be loaded and used from a python script using ctypes. My code consists of several fortran files. I compile and link them as:

[fortran]ifort -fpp -O2 -mkl=sequential -shared-intel -fPIC -openmp -DBUILD_DLL -module obj/Release/ -c file1.f90 -o obj/Release/file1.o ifort -fpp -O2 -mkl=sequential -shared-intel -fPIC -openmp -DBUILD_DLL -module obj/Release/ -c file2.f90 -o obj/Release/file2.o ... ifort -fpp -O2 -mkl=sequential -shared-intel -fPIC -openmp -DBUILD_DLL -module obj/Release/ -c fileN.f90 -o obj/Release/fileN.o ifort -shared obj/Release/file1.o obj/Release/file2.o ... obj/Release/fileN.o -o lib/Release/mylib_shared.so -shared-intel -openmp -lpthread -lm [/fortran]

Then I try to load it in python as: [python]from ctypes import cdll mylib_shared = cdll.LoadLibrary('/path/to/lib/Release/mylib_shared_shared.so') [/python]

I get back: [python]Traceback (most recent call last): File "shared_lib_test.py", line 14, in libramses = cdll.LoadLibrary('/home/p3tris/Documents/ULg/codeblocks/Projects/libramses/lib/Release/libramses_shared.so') File "/usr/lib/python2.7/ctypes/__init__.py", line 443, in LoadLibrary return self._dlltype(name) File "/usr/lib/python2.7/ctypes/__init__.py", line 365, in __init__ self._handle = _dlopen(self._name, mode) OSError: /opt/intel/composer_xe_2013_sp1.1.106/ipp/../compiler/lib/intel64/libifcoremt.so.5: undefined symbol: _intel_fast_memmove[/python]

I tried adding -lirc -limf to the fortran linking, exactly the same... Any ideas? My system: Debian Gnu Linux 7 Intel(R) Fortran Intel(R) 64 Compiler XE for applications running on Intel(R) 64, Version 14.0.1.106 Build 20131008 Just for the record, the same library I have been using for some time as a static library which is linked to C programs and it works fine. Thanks in advance, Petros

Petros · ‎01-27-2014

For some reason all the code became flat. Rewriting the code:
[fortran]ifort -fpp -O2 -mkl=sequential -shared-intel -fPIC -openmp -DBUILD_DLL -module obj/Release/ -c file1.f90 -o obj/Release/file1.o 
ifort -fpp -O2 -mkl=sequential -shared-intel -fPIC -openmp -DBUILD_DLL -module obj/Release/ -c file2.f90 -o obj/Release/file2.o 
... 
ifort -fpp -O2 -mkl=sequential -shared-intel -fPIC -openmp -DBUILD_DLL -module obj/Release/ -c fileN.f90 -o obj/Release/fileN.o 
ifort -shared obj/Release/file1.o obj/Release/file2.o ... obj/Release/fileN.o -o lib/Release/mylib_shared.so -shared-intel -openmp -lpthread -lm

[/fortran]

The python code:

[python]from ctypes import cdll

mylib_shared = cdll.LoadLibrary('/path/to/lib/Release/mylib_shared_shared.so') [/python]

The error:

[python]Traceback (most recent call last):
File "shared_lib_test.py", line 14, in <module>
libramses = cdll.LoadLibrary('/home/p3tris/Documents/ULg/codeblocks/Projects/libramses/lib/Release/libramses_shared.so')
File "/usr/lib/python2.7/ctypes/__init__.py", line 443, in LoadLibrary
return self._dlltype(name)
File "/usr/lib/python2.7/ctypes/__init__.py", line 365, in __init__
self._handle = _dlopen(self._name, mode)
OSError: /opt/intel/composer_xe_2013_sp1.1.106/ipp/../compiler/lib/intel64/libifcoremt.so.5: undefined symbol: _intel_fast_memmove [/python]

Izaak_Beekman · ‎01-27-2014

I am curious whether you use any modules and/or bind(c) for you procedure interfaces. In general I wouldn't expect a tool designed to interface python with C (ctypes/cdll) to work out of the box with Fortran unless special steps were taken due to name mangling etc. although I'm sure you already know this and have taken the appropriate steps to account for this.

It seems that python is having issues with the intel libraries. Can you try building them with static linking to the intel libraries (-static-intel)?

After reading the documentation on building shared libraries with ifort, that's the only thing that comes to mind to try.

http://software.intel.com/sites/products/documentation/doclib/stdxe/2013/composerxe/compiler/fortran-mac/GUID-DCD8A9F4-2F25-4A03-8676-AC5510750A97.htm

Also, there are projects like fwrap and F2PY to help wrap fortran code for python, but I haven't used them much, and creating c interfaces might be a saner approach.

Petros · ‎01-27-2014

I have tried small examples (like just adding two numbers, etc.) and it works. Example:

The fortran code:

[fortran]integer(c_int) function multiply(a, b) bind(C, name="multiply")
USE, INTRINSIC :: ISO_C_BINDING
integer(c_int), intent(in) :: a, b
multiply = a * b
end function multiply[/fortran]

Compiled with:

[fortran]ifort -fPIC -O2 -module obj/Release/ -c main.f90 -o obj/Release/main.o
ifort -shared obj/Release/main.o -o bin/Release/libshared_lib_for_python_example.so[/fortran]

Python code:

[python]

from ctypes import byref, cdll, c_int

mult = cdll.LoadLibrary('/path/to/bin/Release/libshared_lib_for_python_example.so')

a = c_int(2)

b = c_int(4)

print mult.multiply(byref(a), byref(b))[/python]

Output: 8

You need to bind in C and take care of the interfacing. My main subroutine has this interface:

[fortran]subroutine main_subroutine(ccmdname) bind(C, name="main_subroutine")

use omp_lib

use ISO_FORTRAN_ENV

use, intrinsic :: ISO_C_BINDING

implicit none

character(c_char), dimension(*), optional, intent(in) :: ccmdname

...

end subroutine main_subroutine[/fortran]

I did try linking statically but it doesn't compile as the static intel libraries are not compiled with -fPIC, which is needed for dynamic libraries. Thus, I get:

[fortran]ld: /opt/intel/composer_xe_2013_sp1.1.106/compiler/lib/intel64/libifcoremt.a(for_close.o): relocation R_X86_64_32 against `.rodata.str1.4' can not be used when making a shared object; recompile with -fPIC
/opt/intel/composer_xe_2013_sp1.1.106/compiler/lib/intel64/libifcoremt.a: could not read symbols: Bad value[/fortran]

Any other ideas?

Petros

Izaak_Beekman · ‎01-27-2014

This is starting to get well out of my area of expertise, but I wonder if the optional attribute or the assumed shape if the dummy argument is creating issues with the c interface. (This is pure uneducated speculation at this point.)

Also, your error message (about position independent code and -static-intel) seems to be directly contradictory to what it says in the documentation (albeit the 13.1 documentation). I wonder if this changed in 14.x or if there is a documentation bug or if -fPIC accidentally got turned off when they built the static intel libs.

Petros · ‎01-28-2014

Thanks Izaak. It's not the extra attribute. I tried removing it and I get the same problem. I'm convinced it has something to do with the linking. Either I link wrong libraries or in the wrong order.

I hope someone has an idea...