Memory leak with derived-type allocatables and realloc_lhs

[Wukie__Nathan]

I have a derived-type with an allocatable array component defined as:

module type_one

    type, public :: one_t
        integer, allocatable :: i(:)
    end type one_t

    public operator (+)
    interface operator (+)
        module procedure add
    end interface

contains


    elemental function add(u,v) result(res)
        type(one_t),    intent(in)  :: u,v
        type(one_t) :: res

        res%i = u%i + v%i
    end function

end module type_one

Compiling with -assume realloc_lhs and executing the addition operation as follows results in a memory leak:

program mem_leak_test
    use type_one
    implicit none

    type(one_t), allocatable    :: a(:), b(:), c(:)
    integer :: i

    allocate(b(4), c(4))


    do i = 1,4
        allocate(b(i)%i(5))
        allocate(c(i)%i(5))

        b(i)%i = 1.0
        c(i)%i = 2.0
    end do

    a = b + c

    print*, a(1)%i
end program mem_leak_test

This was first observed on OS X using ifort v15.0.3 confirmed on Linux ifort 15.0.2. Valgrind --tool=memcheck produces the following:

==3411== LEAK SUMMARY:
==3411==    definitely lost: 60 bytes in 3 blocks
==3411==    indirectly lost: 0 bytes in 0 blocks
==3411==      possibly lost: 0 bytes in 0 blocks
==3411==    still reachable: 1,136 bytes in 16 blocks
==3411==         suppressed: 0 bytes in 0 blocks

This seems related to the following topic:
https://software.intel.com/pt-br/forums/topic/269437

Reproducer is attached.

[Kevin_D_Intel]

Thank you for the report. I will investigate and reply after I know more.

[Kevin_D_Intel]

I confirmed your Valgrind findings with the 15.0.3 compiler and found a regression with the program suffering a segmentation fault with our upcoming 16.0 compiler. It appears the leak may be addressed in 16.0. If you allocate a in main() (e.g. allocate(a(1)) ) then Valgrind does not report any leak with 16.0, only with 15.0.

I directed both issues to Development (see internal tracking id below) and will update this post about what I hear from them.

(Internal tracking id: DPD200374645)

[Wukie__Nathan]

Thanks for the confirmation, Kevin. I'll give things another try on my end once the 16.0 compiler is released.

Nathan

[Wukie__Nathan]

I am still observing a memory leak with this example using ifort 16.0.1.

A previous comment was made regarding the provided example:

If you allocate a in main() (e.g. allocate(a(1)) ) then Valgrind does not report any leak with 16.0, only with 15.0.

But, actually the point of '-assume realloc_lhs' would be that you do not need to allocate 'a' in main. It should be allocated at the assignment 'a = b + c'.

[Steven_L_Intel1]

There's no implicit deallocate at the end of the program. Try putting the code into a subroutine and calling it from the main program and see what you get.

[Wukie__Nathan]

Hi Steve,

Actually, in my local file I do have an explicit deallocate and still observe the memory leak. Per your recommendation, I have relocated the code to a subroutine and call it from the main program and valgrind still reports memory lost.

I am fairly confident that there is a memory leak somewhere based on experience with my larger application. I strictly use allocatables. The application compiles with gfortran and runs without growing memory usage. Compiling with ifort -assume realloc_lhs I rapidly exceed the machines memory resources. I spent some time tracking down where I see the memory grow and I believe it is related to the example in this post.

I have attached the main.f90 file I am using with the extension renamed to .txt because .f90 was unselectable for upload.

ifort -assume realloc_lhs main.f90 
valgrind --tool=memcheck ./a.out

==39290== LEAK SUMMARY:

==39290==    definitely lost: 321 bytes in 4 blocks

==39290==    indirectly lost: 0 bytes in 0 blocks

==39290==      possibly lost: 0 bytes in 0 blocks

==39290==    still reachable: 0 bytes in 0 blocks

==39290==         suppressed: 22,630 bytes in 194 blocks

[jimdempseyatthecove]

Do you not think it odd that the leak summary is not reporting bytes lost as a multiple of the granularity of your heap manager?

Depending on the heap manager (and 32/64 bit application) this might be 8, 16, 32, 64 bytes (or potentially in multiples of intptr_t).

Jim Dempsey

[Wukie__Nathan]

Well, I will admit my own limitations here and say I don't know if it is odd or not. Maybe a bit odd, but there are also quite a few things going on in the example with allocatables, derived types, and an overloaded operator, so, I am not making any assumption on what is happening behind the scenes. 

Also, I am not an expert on tracking down and diagnosing memory leaks. If there is a better way to interrogate the program and report results then I would be glad to take advantage of a few recommendations. My understanding is that if valgrind is reporting memory 'definitely lost' then that is a very good indicator of a memory leak.

Nathan

[jimdempseyatthecove]

Bear in mind that Valgrind is reporting peculiar numbers. I am stating that this is odd, bearing in mind that the typical heap is a linked list of nodes each having at least one node pointer and one size entry (and potentially additional flags). I've never seen an implementation where the size of the nodes are not a multiple of the size of a pointer. The two sizes you've shown (321 and 22,630) are not multiples of size of pointer (4/8 bytes). Perhaps valgrind keeps track of the size passed into malloc exclusive of the heap node headers.

Jim Dempsey

[FortranFan]

Nathan W. wrote:

Hi Steve,

Actually, in my local file I do have an explicit deallocate and still observe the memory leak. ..

@Nathan W.,

It's been a while since I've used Valgrind and I don't remember the details, but can it point out the instruction in source code that might be the cause of the memory leak?  You may need to debug compile and make the object file available to Valgrind.

One suggestion: perhaps you can have someone at Intel try out your code in Message #7 (the one in Main.txt) file using Intel Inspector XE tool and report the Memory Analysis result.  Or you can try it out yourself using your Intel Parallel Studio license (if you have it) or with a 30-day free trial version.

FWIW, I tried your code on Windows with a couple of other proprietary utilities and they show memory leak on code generated using Intel Fortran compiler for "res%i = u%i + v%i" instruction in function add.  The same utilities do not show any leak with code compiled using gfortran (GCC 6.0 development trunk).  Based on my experience, I reckon Intel's own Inspector XE tool will flag a "Memory leak" error for the same instruction.  So if you can get such analysis completed for you by Intel or by yourself, you may succeed in making a case for a resolution to this memory leak problem i.e., such an approach might be of help if the Valgrind results are being questioned and their validity is placed under doubt.  To me, though, your concern of memory leak appears real.

[Kevin_D_Intel]

I'll recheck this issue with 16.0.1 and Inspector.

[Wukie__Nathan]

@FortranFan
Thanks for running the example and providing some feedback from your own tests. I appreciate your effort and recommendations. Yes, valgrind seems to be able to report the location of the instruction that is causing the leak. valgrind reports 60 bytes definitely lost corresponding to "res%i = u%i + v%i" as your own test has also shown. This is also the location that I have been suspecting is causing the issue.

@jimdempseyatthecove
After investigating a bit more, the original memory leak report showing the odd '321 bytes definitely lost' was resulting from two sources. One, as reported above, is 60 bytes lost from the overloaded addition operator, which I am most concerned about. The other is 261 bytes from the 'print*, a(1)%i' statement. I do not know if the result from the print statement is expected, significant, or worth investigating. 

@Kevin Davis
Thanks for looking back into this. Please keep me posted on the status as well as any possible work-arounds or modifications to the code that may alleviate the issue until a fix is implemented. To-date, I have not identified any work around while still maintaining the desired functionality.

[jimdempseyatthecove]

Steve,

I note that the operator (+) function add is elemental, implying (requiring) that the result(res) be elemental as well. In the sample code, when the operator is executed, a temporary is created, but now being elemental, might this now interfere generating code to delete the temporary object (thus producing a side effect)?

Nathan, I am unable to test here. Can you remove elemental and run a test. This may provide you with a work around until a formal fix is made.

Jim Dempsey

[Wukie__Nathan]

Hi Jim,

Thanks for the recommendation. It seems you are on to something. I did remove the elemental and instead put a loop in the function to handle incoming array arguments. For that case, valgrind no longer reports a memory leak. I attached the file containing the modified version without the elemental. This is a good work around in general. However, I personally would need to make some substantial changes and reverify a number of things. gfortran works with my current implementation so I will wait for a formal fix.

!    elemental function add(u,v) result(res)
!        type(one_t),    intent(in)  :: u,v
!        type(one_t) :: res
!
!        res%i = u%i + v%i       ! valgrind says the leak is here
!    end function

!
!   Removed elemental attribute and added loop in function for array arguments
!
function add(u,v) result(res)
    type(one_t),    intent(in)  :: u(:), v(:)
    type(one_t), allocatable :: res(:)

    integer :: i

    allocate(res(size(u)))

    do i = 1,size(u)
        res(i)%i = u(i)%i + v(i)%i
    end do

end function add

[Steven_L_Intel1]

I can reproduce the leak, though as I suggested earlier if you convert the main program to a subroutine, a lot of the leak goes away since a, b and c get deallocated at the end. But indeed a chunk of data equivalent to the size of a+b is being leaked. I will ask the developers to look into this.

[Steven_L_Intel1]

I was able to simplify it somewhat - a, b and c didn't need to be allocatable, but the component of one_t did. There are multiple temporary arrays allocated, one for each element of a, but only one gets deallocated. I also removed the need for -assume realloc_lhs. It's very puzzling since I don't see why any temporary at all is necessary.

[jimdempseyatthecove]

Nathan,

What does your original implementation expect to do when size(u) /= size(v)

Jim Dempsey

[Wukie__Nathan]

@steve

Thanks for investigating further. I consider it an honor to have puzzled Dr. Fortran.

@jim

Essentially, one should not do such a thing in my original implementation. The case when size(u) /= size(v) for the version using an elemental function I expect to be caught as an array bounds mismatch using some compiler flag in a debug build (-check bounds   or -fcheck=bounds).

Testing this, gfortran -fcheck=bounds main.f90 produces a runtime error indicating an array bounds mismatch.
However, ifort -check bounds -assume realloc_lhs main.f90 executes without reporting an error and seems to just pick a size, which seems wrong given the check bounds flag.

[Steven_L_Intel1]

Intel Fortran doesn't do shape checking on assignments. Bounds checking is for subscripting.

[jimdempseyatthecove]

Steve,

The shape checking on assignment wasn't what I was concerned about (with realloc_lhs), rather it was the shape checking on operator (+). In particular what size the temporary becomes (or if error is emitted).

Jim Dempsey