Fortran C interoperability: Returning string from Fortran to C

[Petros]

Hi, I'm building a C wrapper for my Fortran simulator to be used in Python. I made this function that takes the number of an element and returns its name:

type(c_ptr) function get_elem_name(i) bind(C, name="get_elem_name")
#IFDEF DLL
!DEC$ ATTRIBUTES DLLEXPORT :: get_elem_name
#ENDIF
  use ELEMENTS, only: elemname, nbelem
  implicit none
  integer(c_int), INTENT(IN), value :: i
  character(len=21,kind=c_char) :: returnval
  returnval=''//C_NULL_CHAR
  if(i<=nbelem .and. i>0)returnval=trim(elemname(i))//C_NULL_CHAR
  get_elem_name = C_LOC(returnval)
end function get_elem_name

It is called from Python as:

import ctypes
mylib = ctypes.CDLL("mylib.dll")
mylib.init() # initialize simulator
mylib.get_elem_name(1) # get the name of the first element

It works perfectly for the first 200 calls or something. After that, it randomly returns an empty string (''). I cannot get my head around the problem. Could it be that retval is automatically allocated inside Fortran and then the pointer passed to Python? Could that create an issue?

Thanks for any help.

[Petros]

It seems I fixed the problem. Instead of creating the string inside Fortran, I pass a buffer to fortran, fill the buffer, and get it back. The fortran code is:

integer(c_int) function get_elem_name(i,name) bind(C, name="get_elem_name")
#IFDEF DLL
!DEC$ ATTRIBUTES DLLEXPORT :: get_elem_name
#ENDIF
use ELEMENTS, only: elemname, nbelem
character(c_char), dimension(21), intent(inout) :: name
character(len=21) :: f_name
integer :: n, j
name(1)=C_NULL_CHAR
if(i<=nbelem.and. i>0)then
   f_name=trim(elemname(i))
   n = len_trim(f_name)
   do j = 1, n
      name(j) = f_name(j:j)
   end do
   name(n + 1) = c_null_char
   get_elem_name= 0
   return
endif
get_elem_name= 1
end function get_elem_name

The Python call becomes:

import ctypes
mylib = ctypes.CDLL("mylib.dll")
mylib.init() # initialize simulator
name=ctypes.create_string_buffer(21)
mylib.get_elem_name(1,name) # get the name of the first element
name_str=name.value

[jimdempseyatthecove]

Your first case had two pitfalls. 1) the returned address was to that which was local to the function. Depending on compiler options the buffer may or may not have been on the stack frame of your function and that frame goes out of scope upon return. While you could mitigate this by attributing the buffer with SAVE, this then restricts your code to having only one active returned element name. This may have been why you observed a null element (get element A, get element B-not found thus destroying element A return value). 2) returning a C_PTR, your code may erroneously attempt to perform a free. Though this may not be the case with your Python script as illustrated above, it may affect your script as actually implemented.

I suggest for your second implementation that you pass in the size of the buffer as opposed to assuming (requiring) it be 21 bytes long. This will give you more flexibility and be less error prone.

Jim Dempsey

[mecej4]

Here is what I think was happening with the code in #1, and why the fix of #2 worked.

The variable returnval is a local variable. Therefore, once the function returns to the caller, the variable goes out of scope. It was allocated on the stack and code executed later may overwrite the variable and the memory that had been temporarily allocated on the stack to that variable may be reused as part of the stack of some other routine or in the caller itself.

A variable that went out of scope may retain its value indefinitely. A pointer to the variable (pointer in the C sense) may yield the correct value when dereferenced, but this is not something that one can depend upon.

P.S.: After pressing "SUBMIT", I found that Jim's reply had appeared. His diagnosis agrees with mine as to the main point.

[Petros]

Thanks Jim and mecej4. I should have realized it was an issue with the allocation.

[JVanB]

My take on the problem is that you should go back to the original method but allocate the memory in the function. Also you should allocate using a technique that is consistent with the caller's dynamic memory scheme. Here is a fleshed out example with gfortran and gcc. ifort and cl should be similar.

module elements
   implicit none
   integer, parameter :: nbelem = 109
   character(13) :: elemname(nbelem) = [character(13) :: &
      'Hydrogen', 'Helium', 'Lithium', 'Beryllium', &
      'Boron', 'Carbon', 'Nitrogen', 'Oxygen', &
      'Fluorine', 'Neon', 'Sodium', 'Magnesium', &
      'Aluminum', 'Silicon', 'Phosphorus', 'Sulfur', &
      'Chlorine', 'Argon', 'Potassium', 'Calcium', &
      'Scandium', 'Titanium', 'Vanadium', 'Chromium', &
      'Manganese', 'Iron', 'Cobalt', 'Nickel', &
      'Copper', 'Zinc', 'Gallium', 'Germanium', &
      'Arsenic', 'Selenium', 'Bromine', 'Krypton', &
      'Rubidium', 'Strontium', 'Yttrium', 'Zirconium', &
      'Niobium', 'Molybdenum', 'Technetium', 'Ruthenium', &
      'Rhodium', 'Palladium', 'Silver', 'Cadmium', &
      'Indium', 'Tin', 'Antimony', 'Tellurium', &
      'Iodine', 'Xenon', 'Cesium', 'Barium', &
      'Lanthanum', 'Cerium', 'Praseodymium', 'Neodymium', &
      'Promethium', 'Samarium', 'Europium', 'Gadolinium', &
      'Terbium', 'Dysprosium', 'Holmium', 'Erbium', &
      'Thulium', 'Ytterbium', 'Lutetium', 'Hafnium', &
      'Tantalum', 'Tungsten', 'Rhenium', 'Osmium', &
      'Iridium', 'Platinum', 'Gold', 'Mercury', &
      'Thallium', 'Lead', 'Bismuth', 'Polonium', &
      'Astatine', 'Radon', 'Francium', 'Radium', &
      'Actinium', 'Thorium', 'Protactinium', 'Uranium', &
      'Neptunium', 'Plutonium', 'Americium', 'Curium', &
      'Berkelium', 'Californium', 'Einsteinium', 'Fermium', &
      'Mendelevium', 'Nobelium', 'Lawrencium', 'Rutherfordium', &
      'Dubnium', 'Seaborgium', 'Bohrium', 'Hassium', &
      'Meitnerium']
end module elements

type(c_ptr) function get_elem_name(i) bind(C, name="get_elem_name")
   use ELEMENTS, only: elemname, nbelem
   use ISO_C_BINDING
   implicit none
!DEC$ ATTRIBUTES DLLEXPORT :: get_elem_name
!gcc$ ATTRIBUTES DLLEXPORT :: get_elem_name
   integer(c_int), INTENT(IN), value :: i
   interface
      function malloc(size) bind(C, name='malloc')
         use ISO_C_BINDING
         implicit none
         type(C_PTR) malloc
         integer(C_SIZE_T), value :: size
      end function malloc
   end interface
   integer(C_SIZE_T) returnlen
   character(len=21,kind=c_char) :: returnval
   returnval=''//C_NULL_CHAR
   if(i<=nbelem .and. i>0) then
      returnlen = len_trim(elemname(i))+1
      get_elem_name = malloc(returnlen)
      BLOCK
         character(returnlen), pointer :: returnval
         call C_F_POINTER(get_elem_name,returnval)
         returnval=trim(elemname(i))//C_NULL_CHAR
      END BLOCK
   else
      returnlen = 1
      get_elem_name = malloc(returnlen)
      BLOCK
         character(returnlen), pointer :: returnval
         call C_F_POINTER(get_elem_name,returnval)
         returnval=C_NULL_CHAR
      END BLOCK
   end if
end function get_elem_name

Compiled with

gfortran -shared get_elem_name.f90 -oget_elem_name.dll -Wl,--out-implib=libget_elem_name.a

#include <stdio.h>
#include <stdlib.h>

char *get_elem_name(int i);

int main()
{
   char *result;
   int i;

   for(i = 0; i <= 100; i++)
   {
      result = get_elem_name(i);
      printf("%s\n",result);
      free(result);
   }
   return 0;
}

Compiled with

gcc test.c -L. -lget_elem_name -otest

And the whole thing did print out the elements :)

[jimdempseyatthecove]

R.O.

While I agree that the allocation should be made using the application selected allocator, I think it is naïve to assume that the allocator is malloc. It could be any arbitrary allocator unknown to the Fortran code. Therefore, the allocation may be safer to be made in the scope of the calling application as opposed to by the Fortran subroutine. This said, if the allocation has to be made inside the Fortran code, then the safest way to do this is to pass in the allocator as part of the function call (but then your allocation interface must also be known).

Jim Dempsey

[Petros]

Well, the problem with allocating inside my code is that then I would need to deallocate somehow and keep track inside Python of the elements allocated in Fortran to then call another fortran (or C) subroutine to deallocate them.

Moreover, I don't see the benefit of this over what I did in the second example, where I allocate in python and pass the string to Fortran to be filled and returned. This way, the garbage collection of Python will take care of deallocating the string when not needed anymore (e.g. all references to the object removed).

Can you elaborate on why (or when) one should allocate inside the fortran code?

Thanks.

[mecej4]

There is a third way, which you have not yet considered. This approach is the same as that used by the ctime() function of the standard C/Posix libraries. In this, only one string is allocated, and allocated statically (as in Fortran attribute SAVE). Each call to the function puts a different string into the string, and the pointer to the string remains the same. If saving the string is necessary, the caller of the routine has to do the saving before making the next call to the function.

The function code is greatly simplified in this approach and, if the returned string is printed out (to screen or file) and not needed thereafter, there is no more to do. If the string is needed for use later, the programmer has a free hand in saving it in the caller using whatever facilities are provided by the language of the calling routine (C, Python, etc.).

[jimdempseyatthecove]

I am not suggesting allocation be made inside the Fortran code. What you did in the Python code is fine, with the possible exception of fixing the size

import ctypes
mylib = ctypes.CDLL("mylib.dll")
mylib.init() # initialize simulator
name=ctypes.create_string_buffer(21)
mylib.get_elem_name(1,name,sizeof(name)) # get the name of the first element
name_str=name.value

May be a better solution (assuming your Fortran code has been changed too). Now, if your make a change in the Python code, you need not change the Fortran code.

The Fortran code might be required to perform the allocation should the size of input not be known until the call. e.g. data located inside a Fortran written data file.

Jim Dempsey

[JVanB]

Well, when you return data of size unknown before the call, there are potential problems. If you allocate a buffer to hold the data before the call, the buffer might not be big enough. I recall a previous thread where an example was posted that returned Windows error messages, and for most a couple of hundred bytes was serious overkill. The biggest messages turned out to be kilobytes. Windows itself has several APIs where you call them and tell them how big your buffer is and if it turns out to be too small, they return an error and give the required number of bytes in an INTENT(OUT) argument.

The alternative is for the callee to provide the buffer to hold the data. The Fortran way would be to have a deferred-length allocatable result, which Fortran cleans up by itself after the expression containing the function reference is evaluated. Obviously python isn't going to be able to figure out how to do that. It has already been seen what happens when the buffer is provided in a location which is out of control of the caller.

If a static buffer is used to return all results in, the function is not thread-safe and your code doesn't actually have to be multithreaded to expose thread-unsafe usage. For example, if a recursive procedure invokes the static-buffer-returning function, one instance can invisibly overwrite the data of another instance in a relatively difficult to debug fashion.

Another method would be to have an array of NUL-terminated strings and just return a C pointer to the appropriate one, and this would work in my current example, but I don't know if you are just indexing into a table of constants or creating data more dynamically in the real code you want to use.

You get around the above problems by allocating in the function that produces the data, but as you pointed out python has its own memory allocator (but you asked in the title how to return a string to C, so I just assumed that your program would know what to do with it...) but it seems there is a version with C interface identical to malloc() which is spelled PyMem_Malloc() which does in fact use python's memory management. So I might suggest that you change malloc to PyMem_Malloc in my example and perhaps link the *.DLL with some appropriate python run-time library and see if it then plays nice with your python code. Python should at least be able to free the memory itself; you would have to read some docs yourself to find out whether its garbage collector can do anything useful with it.

[Petros]

Thanks R.O. I'll take a look at PyMem_Malloc! It looks interesting.

[FortranFan]

Repeat Offender wrote:

Well, when you return data of size unknown before the call, there are potential problems. If you allocate a buffer to hold the data before the call, the buffer might not be big enough. .. there is a version with C interface identical to malloc() which is spelled PyMem_Malloc() ... Python should at least be able to free the memory itself; you would have to read some docs yourself to find out whether its garbage collector can do anything useful with it.

Just noticed this thread: unless one has a need to really handle the memory management with fine-grained control of garbage collection, I recommend a much simpler approach using the enhanced interoperability with C feature in Fortran 2015 which Intel Fortran now largely supports (kudos to Intel!).  One can then perhaps write a C-processor based wrapper function that encapsulates the processing C-Fortran interface descriptor (CFI_cdesc_t type).  I've not used Python much, but I'd think one should be able to take the example below based on C and make use of standard functions like CFI_establish and CFI_deallocate to handle data on unknown size as well as free up the memory when done with Fortran dummy arguments having the ALLOCATABLE attribute: all one needs is the ISO_Fortran_binding.h file and the libraries from Intel Fortran compiler.

module m

   use, intrinsic :: iso_c_binding, only : c_char, c_null_char, c_int

   implicit none

   private

   integer, parameter :: MAXELEM = 5
   character(kind=c_char,len=*), parameter :: Elements(MAXELEM) = [ character(kind=c_char,len=40) ::&
      c_char_"Earth", c_char_"Fire", c_char_"Water", c_char_"Wind",                                 &
      c_char_"That mysterious one called ether!" ]

   public :: get_elem_name

contains

   pure subroutine get_elem_name(Ielem, ElemName, ErrorCode) bind(C, name="get_elem_name")

      !.. Argument list
      integer(c_int), value, intent(in)                      :: Ielem
      character(kind=c_char,len=1), allocatable, intent(out) :: ElemName(:)
      integer(c_int), intent(inout)                          :: ErrorCode

      !.. Local variables
      integer :: I
      integer :: LenName

      if ( (Ielem >= 1).and.(Ielem <= size(Elements)) ) then
         LenName = len_trim(Elements(Ielem))
         allocate( ElemName(LenName+1), stat=ErrorCode )
         if (ErrorCode == 0) then
            forall (I = 1:LenName)
               ElemName(I) = Elements(Ielem)(I:I)
            end forall
            ElemName(LenName+1) = c_null_char
         end if
      else
         allocate( ElemName(1), stat=ErrorCode )
         if (ErrorCode == 0) then
            ElemName(1) = c_null_char
         end if
         ErrorCode = 1
      end if

      return

   end subroutine get_elem_name

end module m

#include <stdio.h>
#include "ISO_Fortran_binding.h"

// Prototype for C wrapper function
char* getelem(int i, CFI_cdesc_t * dv);
// Prototype for Fortran getter function
void get_elem_name(int i, CFI_cdesc_t * dv, int *irc);

int main()
{

   CFI_CDESC_T(1) fortran_object;
   char* elem_name;
   int irc;

   // Call C wrapper function
   elem_name = getelem(5, (CFI_cdesc_t *)&fortran_object);

   if (elem_name != NULL) {
      printf("element name: %s\n", elem_name);
   }
   else {
      printf("failed to get element name.\n");
   }

   // Free the CFI decriptor object for Fortran-C interoperability
   irc = CFI_deallocate((CFI_cdesc_t *)&fortran_object);

   return 0;

}
// Wrapper function to Fortran get_elem_name procedure
char* getelem(int i, CFI_cdesc_t * dv)
{

   int irc;
   CFI_rank_t rank;

   irc = 0;
   rank = 1;
   irc = CFI_establish(dv, NULL,
      CFI_attribute_allocatable,
      CFI_type_char, sizeof(char *), rank, NULL);
   if (irc != CFI_SUCCESS) {
      return NULL;
   }

   // Call Fortran function
   get_elem_name(i, dv, &irc);

   if (irc == 0) {
      return (char *)dv->base_addr;
   }
   else {
      return NULL;
   }

}

Upon execution,

element name: That mysterious one called ether!
Press any key to continue . . .

Of course, if the data sizes are known up front or when the caller only wants to deal with data of certain sizes, it's best if the Fortran interface receives the buffer along with the buffer length information and the Fortran interface has the dummy argument(s) for data with assumed size attribute; also, I prefer using subroutines (i.e., void C functions) over a function returning the data (as in the original post which is a no-go in interoperating schemes) or the common C practice of returning an error code as function result and data via the function parameter.

[Petros]

Hi @FortranFan. Wow, I didn't know about this functionality. I always thought interoperability was kind of one-direction: A way to wrap Fortran so it plays nice with the "now-a-days" standard language C.

It's good you posted your solution here for completeness and future reference. However, I believe it's an overkill for what I need. The data structure is well known beforehand (character len=20), so I prefer to pass a buffer and get the info, as in my 2nd post.

If I was programming in C, then I would need to allocate, keep track, and deallocate anyway, so your proposed approach makes sense. It adds functionality and robustness without any extra cost.

Nevertheless, in Python, you don't need to do it yourself (otherwise why opt for a slower language). It takes care to free memory automatically (e.g. once an object is not referenced anywhere in the program anymore, etc.). So, to manually track memory, allocate and deallocate is not really the "Pythonic" way.

[Steven_L_Intel1]

FortranFan wrote:
, I recommend a much simpler approach using the enhanced interoperability with C feature in Fortran 2015 which Intel Fortran now largely supports (kudos to Intel!).  

"largely"? As far as I know, we support it entirely. What do you think we're missing? (Not counting the inevitable bugs, though I know of only one minor one at this time.)

[FortranFan]

Steve Lionel (Intel) wrote:

Quote:

FortranFan wrote:

, I recommend a much simpler approach using the enhanced interoperability with C feature in Fortran 2015 which Intel Fortran now largely supports (kudos to Intel!).  

"largely"? As far as I know, we support it entirely. What do you think we're missing? (Not counting the inevitable bugs, though I know of only one minor one at this time.)

Steve,

My choice of the word wasn't all that deliberate, I suppose I meant it suppositionally since the standard itself is unofficial yet (sure, members might feel TS 29113 aspect is not going to change but you know the cliche, it ain't over til..) and I'd a vague impression about some gap/issue (I could be wrong on this) with the assumed type, shape, and rank aspect either in Intel Fortran or in the standard itself.

Anyways, I'm happy to see Intel take the lead on this and strive to get all of TS 29113 aspect of Fortran 2015 implemented when it did; I like the approach and hope it can be extended quickly to other aspects of the standard too!  But first, I would like to see the rest of Fortran 2008 implemented - it's that of the season again: can we wish for ALL of Fortran 2008 in year 2016 and ALL of Fortran 2015 within a year of standard being "finalized", (year 2018?!)

[Steven_L_Intel1]

Yes, assumed type and rank are supported now. Nothing material is going to change for F2015 - I did discover some editorial issues that we (standards committee) took care of - no technical changes. I'm not aware of any gaps in our implementation. I know that gfortran has some of the TS but not the part that deals with C descriptors (yet).

Getting all of F2008 is our (Intel's) primary goal at this point, though some things from F2015 (such as the Interop TS) are too popular/important to put off.