topic My test program in post 10 in Intel® Fortran Compiler

Memory leak

qolin — Wed, 20 Nov 2013 16:34:48 GMT

I have a memory leak. See attached example, which is an adapted portion of a much larger code.

This code (and the rest of the application) was doing everything with pointers. Over the past few months I have been replacing them all with allocatables, on the assumption that memory leaks are not possible with allocatables. ...Well clearly I was mistaken on that, so what am I doing wrong?

The leak occurs in copy_branch_plot_results, when it calls construct_branch_plot_results. In its original form with pointers, the code was a lot busier, with explicit tests all over to de-allocate the pointers where necessarry. I assumed this was not required with allocatables, so removed most of it, and was dismayed to find that my changes introduced leaks.

Qolin

qolin — Wed, 20 Nov 2013 18:42:46 GMT

Sorry I should have specified the compiler version:

Intel(R) Inspector XE 2013 (build 250094)
Intel(R) Parallel Studio XE 2013
Copyright (C) 1985-2013 Intel Corporation. All rights reserved.
Intel(R) Composer XE 2013 Update 5 (package 198)

C:\Program Files (x86)\Intel\Composer XE 2013>

It is not clear what you mean

mecej4 — Wed, 20 Nov 2013 20:40:49 GMT

It is not clear what you mean by "memory leak".

A local variable (scalar or array) with the ALLOCATABLE attribute and whose status is "allocated" stays allocated unless it is used in a DEALLOCATE statement or the subprogram to which it belongs as a local variable executes a RETURN statement.

If you allocate a large array in the main program and never deallocate it, that array occupies a chunk of memory until your program terminates. This is not what should be called a memory leak. If you wish to release memory earlier, you can write DEALLOCATE statement at appropriate places.

How are you detecting the

IanH — Wed, 20 Nov 2013 21:48:51 GMT

How are you detecting the "leak"?

Task manager and debugger.

qolin — Wed, 20 Nov 2013 22:14:56 GMT

Task manager and debugger.

Breakpoint at the start of the loop and on the continue statement, this allows you to note memory used at start and end. For good measure you can breakpoint inside the loop to see the memory usage increase every time round. On Win7 processees tab, the column of interest is "Commit size".

Try the code as posted, and compare the behaviour with the alternative form of the code in copy_branch_plot_results.

Qolin

I don't think that's

IanH — Wed, 20 Nov 2013 22:28:25 GMT

I don't think that's confirmation of a leak (if you have a leak you will see that behaviour, but seeing that behaviour doesn't necessarily mean that you have a leak) - for example you could see progressively increasing commit in the face of memory fragmentation or perhaps some design decisions around the runtime allocator behaviour.

I have a little module somewhere that lets Fortran programs query the underlying C runtime allocator for debug builds or the underlying opreating system allocator. I've used that to find leaks before. If time permits later today I'll see if I can find them and "instrument" your code.

You can use tools such as vmmap.exe from sys internals to look at the operating system's idea of memory allocation for your process. That doesn't necessarily tell you what the C or Fortran runtime thinks is going on though (but those runtimes don't do much thinking, if memory serves me correctly).

OK what I haven't explained

qolin — Wed, 20 Nov 2013 22:52:16 GMT

OK what I haven't explained is how I found this.

On the actual, much larger, code, we were clearly seeing memory leaks, as evidenced by task manager. The Commit size was staring off around 250 MB, and after some hours had risen through 8 GB. We knew that the problem had been introduced recently, like in the last 6 weeks or so. Our code is rebuilt every night and the resulting builds copied to a server for possible historical reference. So I ran the problem dataset on various overnight builds, and was able to pin down the particular build which first exhibited the leak, and thence from the source control history, the changeset that introduced it. As I said earlier, I have been swapping pointers to allocatables, and it was one of my changes. But I couldn't understand how that change could produce a leak.

Luckily we have access to (drum roll) Intel Inspector, and pointing it at the code produced a clear report, pointing to the allocate in construct_branch_plot_results_appv. Of course Inspector doesn't tell you the line of code responsible for loosing the memory that constitutes the leak; it only tells you which line allocated it originally, and you have to work out how it gets lost.

So after about 10 minutes detective work and experimental code modification, I found that changing to the alternative form in copy_branch_plot_results makes it work correctly. Task manager no longer showed memory increasing, and inspector gave it a clean bill of health.

Ok. I found my routines,

IanH — Thu, 21 Nov 2013 02:18:00 GMT

Ok. I found my routines, which show a "leak". But it is a programming error - not a compiler problem. There are type mismatches in your code.

res1 and res2 in the main program are of type branch_plot_results_type. In the procedure copy_branch_plot_results, these arguments are of type branch_plot_results_ppv_type.

These two types are sequence types, but the sequence of components (considering attributes) in branch_plot_results_type is not the same as the sequence of components in branch_plot_results_ppv_type.

Edit - no - I'm fibbing. Let me look at this further.

It looks like a problem with

IanH — Thu, 21 Nov 2013 03:42:32 GMT

It looks like a problem with intrinsic assignment of derived types with allocatable components? What???

And to think that I was about to post a lengthy rant about Inspector and false positives...

(Instead I'll rant about the forum double spacing syntax highlighted bits in posts, and clicking upload in the file attachment bit creating another "Drag files here" section.)

Attempt twenty two.

IanH — Thu, 21 Nov 2013 03:45:19 GMT

Attempt twenty two.

[fortran]

program testy
USE CrtMemoryUtilities
USE, INTRINSIC :: ISO_C_BINDING, ONLY: C_INT

implicit none

type branch_plot_results_ppv_type
    real(8)       , allocatable :: v(:)
end type branch_plot_results_ppv_type

type branch_plot_results_type
    type(branch_plot_results_ppv_type), allocatable :: appv(:)
end type branch_plot_results_type

TYPE(CrtMemState) :: before, after, diff
INTEGER(C_INT) :: rc

! This also conveniently gets any initialization of the IO
! runtime out of the road.
PRINT "('Before:')"

! Reports for warnings go to stdout.
rc = CrtSetReportMode(CRT_WARN, CRTDBG_MODE_FILE)
rc = CrtSetReportFile(CRT_WARN, CRTDBG_FILE_STDOUT)

! Get and report the state of the heap at the start.
CALL CrtMemCheckpoint(before)
CALL CrtmemDumpStatistics(before)

! Put the action in a procedure so local allocatables will be cleaned up.
call main

! Get and report the state of the heap after.
CALL CrtMemCheckpoint(after)
PRINT "('After:')"
CALL CrtmemDumpStatistics(after)

! Report the difference.
rc = CrtMemDifference(diff, before, after)
PRINT "('Difference:')"
CALL CrtmemDumpStatistics(diff)

contains
SUBROUTINE main
    type(branch_plot_results_type), allocatable :: res1, res2

    allocate(res1)
    allocate(res1%appv(1))       ! #A
    allocate(res1%appv(1)%v(30))

    allocate(res2)
    allocate(res2%appv(2))       ! size must differ from #a above.
    allocate(res2%appv(1)%v(40)) ! This not cleaned up.

    res2 = res1
END SUBROUTINE main
end program[/fortran]

[plain]

>ifort /check:all /warn:all /Fe:Testy.exe /standard-semantics /MTd CrtMemoryUtilities.f90 Testy.f90 && Testy.exe
Intel(R) Visual Fortran Intel(R) 64 Compiler XE for applications running on Intel(R) 64, Version 14.0.1.139 Build 20131008
Copyright (C) 1985-2013 Intel Corporation. All rights reserved.

-out:Testy.exe
-subsystem:console
CrtMemoryUtilities.obj
Testy.obj
Before:
0 bytes in 0 Free Blocks.
8499 bytes in 7 Normal Blocks.
16194 bytes in 97 CRT Blocks.
0 bytes in 0 Ignore Blocks.
0 bytes in 0 Client Blocks.
Largest number used: 24693 bytes.
Total allocations: 34522 bytes.
After:
0 bytes in 0 Free Blocks.
8850 bytes in 8 Normal Blocks.
16194 bytes in 97 CRT Blocks.
0 bytes in 0 Ignore Blocks.
0 bytes in 0 Client Blocks.
Largest number used: 25998 bytes.
Total allocations: 36002 bytes.
Difference:
0 bytes in 0 Free Blocks.
351 bytes in 1 Normal Blocks. <<<<< Badness.
0 bytes in 0 CRT Blocks.
0 bytes in 0 Ignore Blocks.
0 bytes in 0 Client Blocks.
Largest number used: 1305 bytes.
Total allocations: 1480 bytes.[/plain]

Thanks Ian.

qolin — Thu, 21 Nov 2013 10:43:25 GMT

Thanks Ian.

Thanks, we'll check this out.

Steven_L_Intel1 — Thu, 21 Nov 2013 15:26:41 GMT

Thanks, we'll check this out.

Steve, Ian:

FortranFan — Mon, 25 Nov 2013 21:30:00 GMT

Steve, Ian:

It'll be most useful if both of you can comment on the general applicability of the code posted by Ian to test "badness" during variable allocation and deallocation in derived types.

I'll be very curious to learn what Steve finds with the example provided by Ian on the intrinsic assignment of derived types with allocatable components.

I'm working toward creating derived types with allocatable components (as well as type-bound procedures) for computational calculations by many other users. I assumed there will not be any "memory leak" issues associated with it since I'm adhering strictly to the 2003 and 2008 standards. The consumption of these types in user code will involve both intrinsic and defined assignments as allowed by the standard.

However when I try out Ian's code on a couple of these types, I get "badness" in one of the types similar to what Ian shows with the type defined by the OP. Whereas in my other type, I get a "Debug Error!" from Microsoft Visual C++ Debug Library:

[plain]

[Red circle with x in the middle] Debug Error!

Program: C:\\test\\TestMemLeak.exe

Damage before 0x00183FA0 which was allocated by aligned routine

(Press Retry to debug the application)

[/plain]

Does the above mean there is something seriously wrong with my code, or a compiler bug, or is it an artifact of Microsoft's C run-time? The above error occurs right when the RETURN statement is executed in the "CONTAIN"ed procedure (i.e., SUBROUTINE MAIN in Ian's example).

Unfortunately I currently don't have access to Intel Inspector (I'd tried out a trial version a while ago but couldn't convince the management to get me a fully licensed copy and I don't think the system will allow me to download it again) to see what it finds on my derived types.

But I'm now really worried about memory issues with the use of my derived types.

Thanks much,

Steve, any advance on this?

qolin — Sat, 01 Feb 2014 09:18:06 GMT

Steve, any advance on this?

Steve,

qolin — Fri, 14 Feb 2014 09:34:50 GMT

Steve,

I see the leak I described in my other thread (http://software.intel.com/en-us/forums/topic/493597, "Another memory leak") has been assigned an Issue ID. Is this thead's problem the same, or different? Please give us an update, even if its "sorry, still looking".

Qolin

P.S. Thanks for the nice new green belt (does my bum look big in it?)

I don't think these are

Steven_L_Intel1 — Fri, 14 Feb 2014 18:54:00 GMT

I don't think these are related. Using Ian's test code under 14.0.2 I see no leaks. I don't really see the leak you described in your original source either. Please try your real application with 14.0.2 and see what you find.

I do still see the internal compiler error for the move_alloc call, though. That is fixed in the 15.0 compiler.

Quote:Steve Lionel (Intel)

FortranFan — Fri, 14 Feb 2014 20:30:00 GMT

Steve Lionel (Intel) wrote:

I don't think these are related. Using Ian's test code under 14.0.2 I see no leaks. I don't really see the leak you described in your original source either. Please try your real application with 14.0.2 and see what you find.

I do still see the internal compiler error for the move_alloc call, though. That is escalated as issue DPD200253435.

Steve,

Is 14.0.2 same as Service Pack 1, Update 2 - the latest version you said became available this week?

When you say, "Using Ian's test code under 14.0.2 I see no leaks." - do you mean the "351 bytes - badness" illustrated by Ian doesn't occur anymore? If so, does that imply there was a compiler issue in the previous Intel version (say the one used by Ian) that has now been fixed?

By the way, have you analyzed code containing derived types with allocatable components extensively in Intel Inspector/VTune Analyzer? Do you feel confident about the ability of these tools to catch any potential memory issues with the latest Fortran enhancements involving derived types and bound procedures that may heavy use of allocatables (I know technically there shouldn't be issues with allocatables unless one messes up royally, but what if there are compiler bugs)?

I ask because I'm making heavy use of allocatables with derived types and TBPs and when I run Ian's utility on my code, I get the "badness". But my version is not up-to-date and I'm trying to convince management to upgrade my compiler plus get me these other tools.

Thanks,

When I ran Ian's test with 14

Steven_L_Intel1 — Sat, 15 Feb 2014 02:18:31 GMT

When I ran Ian's test with 14.0.2 (yes, 2013 SP1 Update 2) it showed no leak. I do use Inspector to identify memory leaks. It's pretty reliable at saying whether or not there is a leak, and if so, where in the program the memory was allocated. But finding the cause of the leak takes more effort. I have not used VTune Amplifier XE in such investigations.

Steve

qolin — Tue, 18 Feb 2014 11:33:38 GMT

Steve

OK thanks for all the updates. This time is my turn to apologise for not getting back to you sooner.

Alas I am not in a position to try this on 14.0.2. So ...

You say you ran ianH's test on 14.0.2 and it showed no leak, which sounds like very good news. Also you say "I cannot really see the leak in your original source either." ... more good news, hopefully this means the problem does not occur on 14.0.2.

To vertify this, please amend my original test so it uses more memory: in the last call to MAKE1 (line 220), change the first 2 arguments: from 40 and 50, to 400 and 500. I have attached the source again with this change made.

When I run this on 13.1.3, it pretty quickly exhausts available memory for a 32-bit process.

Well, this program does run

Steven_L_Intel1 — Tue, 18 Feb 2014 17:50:02 GMT

Well, this program does run out of memory. I'll take another look.