How to free memory returned as c_ptr

Dong_Zheng · ‎06-17-2019

I have a lib which is written in Fortran and is intent to be used by both Fortran and C using iso_c_binding. Basically, in the lib, a subroutine allocates a Fortran array (may be of any type) and return the allocated memory as a c_ptr. After the data is used, the memory is freed either by using the C routine "free" or Fortran "deallocate". This works fine for Parallel studio XE 2016.5 (the current version we are using). However, it does not work for Parallel studio XE 2019.4. I get error message when I try to free the memory and the program crashes. I uses Linux system. Any one have similar problem?

Juergen_R_R · ‎06-17-2019

I think this depends on the companion C processor. The C library function void free(void *ptr) deallocates the memory previously allocated by a call to calloc, malloc, or realloc. The C pointer here has been allocated from the Fortran side via the bind(C) interface, so it is not obvious to me that this is indeed implemented by one of the C operations above. I think this depends on the relation of Fortran and C pointer. With gfortran it works for me, but it fails with ifort v17/18/19, nagfor and PGI fortran. Others with more experience on bind(C) interfaces could give you a better insight.

Steve_Lionel · ‎06-17-2019

The fundamental issues are 1) Knowing whether to use free or DEALLOCATE, and 2) Convincing the compiler to let you use DEALLOCATE. The Fortran standard has words about allowing DEALLOCATE of "the whole" of an allocated object, regardless of how you came by the pointer. Intel Fortran, though, tries to keep track in array pointers of whether you're allowed to deallocate or not, and if you use C_F_POINTER to reconstruct a pointer from a C_PTR, it will assume that this is not something you can deallocate. Most of the time this is the correct choice, but not always.

I would argue that the library interface is defective, and that it would be better if there was a separate interface for use with Fortran pointers rather than trying to make C_PTR work for both. Another alternative is to have only a Fortran pointer interface and use the CFI_descr_t type from the Fortran 2018 ISO_Fortran_binding.h header (Intel Fortran currently supports this) and have the C code either request that the library do the deallocate or call CFI_deallocate to do it. This latter is probably more trouble than it's worth here, though.

Ideally, Intel would revise its mechanism for determining when it is safe to deallocate a pointer, but I don't see that happening anytime soon.

FortranFan · ‎06-17-2019

@Dong Zheng,

You may want to consider a "rule" and its companion one (pun intended!):

objects allocated using the Fortran processor (for example, using the ALLOCATE statement) must be deallocated using the Fortran processor,
objects allocated using the companion C processor (for example, using 'malloc' in C) must be 'free'd' using that C processor

Note you can break the above "rule" and perhaps get away with it in many situations and with many compilers; the above is suggested as a safer option!

Another "rule" you may consider along with its companion:

Avoid the POINTER attribute in Fortran as much as possible
Investigate a coding approach involving the ALLOCATABLE attribute in Fortran whenever the POINTER attribute appears necessary. Note there are exceptions to this (as with all rules I suppose) such as data structures involving linked lists, trees, etc. In many circumstances, ALLOCATABLE can be a safer, cleaner approach.

I don't know anything of your actual library code and its interfaces, but your simple example can be refactored with above "rules" in mind. See this thread at comp.lang.fortran: https://groups.google.com/d/msg/comp.lang.fortran/ZSvJyDxVfZI/yGJi2KDJAgAJ

Dong_Zheng · ‎06-17-2019

Thanks for Joergen and Steve for your quick replies. It seems that with ifort v17/v18/v19, I'm facing a rewriting of my code which will also take some time. For the mean time, I can still use ifort v16 (and gfortran) which WORKS with my correct code. I'v actually made another example with just one Fortran file (test2.f90) to show the problem. It seem that if a subroutine allocates an array and returns it as a c_ptr, there is NO WAY to free the memory no matter in Fortran and C. It sounds not right.

Steve_Lionel · ‎06-17-2019

My recommendation is that, at least based on your examples, there is no good reason to use ALLOCATE and Fortran pointers here. Just use malloc (available as an intrinsic in Intel Fortran). You can then use free when needed. You can keep your C_PTR interface if you want - just TRANSFER it to/from an address-sized integer (INTEGER(C_INTPTR_T). You can always create a POINTER from it locally when you need to.

FortranFan · ‎06-17-2019

Dong Zheng wrote:
.. one Fortran file (test2.f90) to show the problem. It seem that if a subroutine allocates an array and returns it as a c_ptr, there is NO WAY to free the memory no matter in Fortran and C. It sounds not right.

I suggest you submit a support request with Intel OSC (https://supporttickets.intel.com/supportrequest?lang=en-US) for clarification. It appears Intel Fortran implementation restricts the use of DEALLOCATE of array objects with POINTER attribute of intrinsic data types when the compiler is unsure whether it is a whole object. The run-time error is not raised with scalar objects.

My hunch is Intel Fortran is following up on statement in the standard, "Deallocating a pointer .. whose target was not created by an ALLOCATE statement causes an error condition in the DEALLOCATE statement" and/or "If a pointer appears in a DEALLOCATE statement, it shall be associated with the whole of an object that was created by allocation." Intel Fortran team can inform you better.

By the way, you may want to think further as to what happens in your second 'test2.f90' example in line 47 with the statement, 'allocate(cc(n))'. Do note it is as if the Fortran processor sets up an anonymous object of TARGET attribute in global memory and points the local variable 'cc' with the POINTER attribute to point to this target. You then capture its C_LOC address for use with a companion C processor in 'cp' which is all you retain. If the 2 processors cannot function correctly to 'free' the memory at that C_LOC address due to some fault or restriction, you have a memory leak issue.

A better option you can consider in your library design is to always work with NAMED targets (e.g., 'cc' with ALLOCATABLE attribute in the modified example below) and interoperate with it in Fortran, C, C++, etc. as desired; you will be only limited by the capabilities of the companion C processor which will likely not be an issue for you on Linux:

module m
   
   use iso_c_binding
   
   ! NAMED target
   character(len=1), allocatable, target, save :: cc(:)
   
contains

   subroutine getdata(cp, n) bind(c, name="getdata")
      type(c_ptr) :: cp
      integer(c_int), intent(in) :: n

      integer :: i
      ! Checks elided re: cc allocation status
      cc = [( achar(47+i), i=1,n)]
      print *, 'allocated cc'
      cp = c_loc(cc)

   end subroutine getdata

   subroutine c_free(cp) bind(c, name='c_free')

      type(c_ptr), intent(inout) :: cp

      ! Checks elided re: cc allocation status
      if ( c_associated(cp, c_loc(cc)) ) then
         ! Deallocate the target if cp is indeed pointing to it
         deallocate(cc)
         cp = c_null_ptr
      end if

   end subroutine c_free

end module

program test2

   use iso_c_binding, only : c_ptr, c_null_ptr, c_f_pointer
   use m, only : getdata, c_free

   character(len=1), pointer :: s(:)
   type(c_ptr) :: cp
   integer :: n
   cp = c_null_ptr
   n = 5
   call getdata(cp, n)
   call c_f_pointer(cp, s, shape=)
   print *, "In main: ", s
   s => null()

   call c_free(cp)
   print *, 'deallocate cc done'

end program test2

Upon execution,

 allocated cc
 In main: 01234
 deallocate cc done

FortranFan · ‎06-17-2019

@Dong, Zheng:

Also with the example in Quote #7 with the module variable 'cc', the Fortran standard extends you further 'safety' with its rules around objects with the ALLOCATABLE attribute - you only need to DEALLOCATE it if you need to return the memory back to the system for other needs during run-time; otherwise, the memory with 'cc' is effectively 'garbage collected' once the program finishes.

Steve_Lionel · ‎06-17-2019

It's not that gfortran isn't enforcing the standard here, it's that it uses a different technique for detecting the issue, one that is (less?) susceptible to improper complaining.

When DEC (!) Fortran added F90 pointers, the notion that one could legitimately create a pointer out of an address was far in the future. The implementation chosen was (mostly) good enough, but it has had its share of bugs over the years. In this case, it's just a situation never anticipated by the design and one that can't be solved without either choosing to not report legitimate errors or a redesign of how pointer allocation and deallocation is done. This could be done entirely in the run-time library, I think, but it's far from trivial.

jimdempseyatthecove · ‎06-17-2019

Unlike C/C++, a pointer to an array can be NULLIFIED or ASSOCIATED. When the association is made by allocation, then the pointer does not directly contain the allocation, rather it points to an array descriptor that in turn points to the allocation. As to if the array descriptor is separate from the storage of the allocation or prepended (e.g. like a header) this is an implementation detail. C/C++ just points to a blob and has no descriptor. While there are some interoperational features to inject an array descriptor into C/C+, it is otherwise incompatible to allocate in one domain and then deallocate in the other domain.

Jim Dempsey

FortranFan · ‎06-17-2019

Steve Lionel (Ret.) (Blackbelt) wrote:
It's not that gfortran isn't enforcing the standard here, it's that it uses a different technique for detecting the issue, one that is (less?) susceptible to improper complaining.
When DEC (!) Fortran added F90 pointers, the notion that one could legitimately create a pointer out of an address was far in the future. The implementation chosen was (mostly) good enough, but it has had its share of bugs over the years. In this case, it's just a situation never anticipated by the design and one that can't be solved without either choosing to not report legitimate errors or a redesign of how pointer allocation and deallocation is done. This could be done entirely in the run-time library, I think, but it's far from trivial.

I miswrote in Quote #7: in the first paragraph, I meant to differentiate between scalars and arrays, not between intrinsic and derived types. Quote #7 is edited

It appears to me the Intel Fortran compiler at run-time can do the needful to figure out the whole object of an array is indeed being deallocated and thus avoid the run-time error it throws now.

Dong_Zheng · ‎06-18-2019

Thanks a lot for your help. You guys are great. Based on your suggestions, I made a small change to my example (in mymod.f90). Basically, I use C "malloc" instead of Fortran "allocate" to allocate memory. In this way, the c_ptr returned by the sub can be freed both in Fortran and C. The new test code works for all version of ifort and for gfortran. The test files are attached. Thanks again.

Steve_Lionel · ‎06-18-2019

You don't need "c_malloc". As I wrote earlier, malloc is an intrinsic in Intel Fortran. Just use it as you would in C. Same for free.

FortranFan · ‎06-18-2019

Steve Lionel (Ret.) (Blackbelt) wrote:
You don't need "c_malloc". As I wrote earlier, malloc is an intrinsic in Intel Fortran. Just use it as you would in C. Same for free.

But 'malloc' ain't an intrinsic in standard Fortran. OP mentions use of gfortran compiler now and perhaps may need to work with other processors in the future. OP's approach with 'c_malloc' appears more portable therefore.

Dong_Zheng · ‎06-18-2019

@Steve, I actually tried "malloc" and "free" first. But I found out that the data type has to be "integer(C_INTPTR_T)" for malloc/free (instead of "type(c_ptr)"). With this data type, parameters like "c_null_ptr" and functions like "c_f_pointer" etc. can not be used. This means large changes for my real code. To minimize my code changes, I opted to define "c_malloc" and "c_free" and stick with iso_c_binding. I'm actual trying to write a subroutine "c_allocate". With it, I can simply replace "allocate" and "call c_allocate" in the lib. Then my main code should work without any change.

Dong_Zheng · ‎06-18-2019

@FortranFan,

gfortran actually also has malloc/free. However, their manual states "The MALLOC intrinsic is an extension intended to be used with Cray pointers, and is provided in GNU Fortran to allow the user to compile legacy code. For new code using Fortran 95 pointers, the memory allocation intrinsic is ALLOCATE. "