Memory increasing during fortran program execution

[hydrodyn]

Hi,
I have a program in which all variables are dynamically allocated.
It is a big hydrodynamic and water quality model. We are running the program to simulate 18 scenarios running simultaneously. All the console windows start with fixed memory and then the memory increases linearly with the length of the simulation. After a while I am running out of memory and the computer hangs up.

Here is my question.
1) Why memory increases linearly with time when all variables allocated in the beginning of the program.

Since additional memory cost quite bit, I want to find out if we can do anything in the code to avoid memory increase.
So far I have tested this in Compaq Visual Fortran. My intel fortran code is not ready yet.
But I am assuming intel fortran users can provide some kind of help to my question.

Thanks
Venkat S. Kolluru

[GVautier]

hello

Beware of hidden allocations: File buffers , console buffer ...

[mecej4]

Memory leaks and inefficient garbage collection, coupled with heavy use of allocatable variables, can cause depletion of free memory.

It would help if you can construct a small example, but I realize that doing so can be difficult.

[jimdempseyatthecove]

This is not necessarily the case of a memory leak.

The MS C/C++ heap manager has different configurations. The "standard" configuration seems to grow the Virtual Memory footprint eventhough allocated memory has been deleted. Effectively this becomes an allocate from untouched address space in Virtual Memory. And this results in hitting a page file limitation, which in turn is fatal to the application. My guess for this default choice is a poor fix for sloppy programming as this will avoid a buffer being used after being deleted causing unexpectedmemory modifications to memory allocated in the same address space. Allocation at a new/different Virtual Address will avoid the "corruption" but it does not fix the underlaying problem.

Look at enabling the Low Fragmentation Heap (LFH). See:

http://msdn.microsoft.com/en-us/library/aa366750(VS.85).aspx

Jim Dempsey

[anthonyrichards]

Jim, this is serious IMHO.

Are you saying that, contrary to the impression given by the Fortran manual, DEALLOCATE 'frees' memory but that freed memory is not actually useable by that process for a later ALLOCATE?

Surely this behaviour negates the whole point of using ALLOCATE/DEALLOCATE to make the most efficient use of limited memory?

Does it not also imply that the Windows memory management that goes on under the hood and that we rely on is actually pretty poor?

[GVautier]

Hello

As far as I know, memory blocks that has been freed can be reused for further allocations. But, memory space is never optimized so you can only allocate (in a freed block) an array with a size inferior or equal to the previous size of the freed block if there is a non free block above it. So to avoid this, you must take care in a program to allocate arrays in order of life time in your program. First allocate persistent arrays and last volatile or reallocatable arrays. It is also preferable to allocate small blocks first in order to avoid increase of memory size in case of frequent reallocations.

[jimdempseyatthecove]

Quoting anthonyrichards

Jim, this is serious IMHO.

Are you saying that, contrary to the impression given by the Fortran manual, DEALLOCATE 'frees' memory but that freed memory is not actually useable by that process for a later ALLOCATE?

Surely this behaviour negates the whole point of using ALLOCATE/DEALLOCATE to make the most efficient use of limited memory?

Does it not also imply that the Windows memory management that goes on under the hood and that we rely on is actually pretty poor?

FORTRAN does not perform the low level allocation. The C runtime library performs the low level allocation.

This is a"it depends" situation.From msdn.microsoft.com (http://msdn.microsoft.com/en-us/library/aa366750(VS.85).aspx):

"[The information in this topic applies to Windows Server2003, WindowsXP, and Windows2000 with Service Pack4 (SP4) and hotfix KB 816542. Starting with WindowsVista, the system uses the low-fragmentation heap (LFH) as needed to service memory allocation requests. Applications do not need to enable the LFH for their heaps.]"

Therefore, if you are pre-Vista, and I assume pre-Windows 7, then LFH is NOT enabled by default.

Unfortunately, the links to LFH tells youwhat LFH does for you, but it does not tell youhow the heap allocations behave prior to enabling LFH. So there is some guesswork (or digging into the C Run Time Library source code). Note, you can replace the heap manager to perform as you require. e.g. TBB scalable allocator.

One of the problems encountered in the earlier multi-threaded application environments was scenarios like this:

Thread A allocates a buffer
Thread A issues an asynchronousrequest supplying buffer allocated by thread A
Thread A continues to execute while O/S or other thread is handling requiest by A
while request pending
User clicks on to cancel operation
Thread A returns buffer
some time later
O/S or other user thread completes request for A and writes data to buffer

Clearly the above sequence is a programming error. Thread A should have issued a cancel request to O/S or other thread. Due to unavailable API or sloppyness of the programmer the cancel operation could not be issued. The bad code is already out in millions of locations so what do you do? One thing you can do is for these buffers (or anything that potentially looks like one of these buffers), is upon delete/free to return the buffer to a return after some long dwell time or return to a pool that gets drawed upon only after virgin heap data exhausted.

This is a nice idea except when due to some other programming error by that guy standing behind the tree over there(finger pointing somewhere else, the not me syndrome), that your code expects/requires the Pageing File to expand to meet your needs, but the O/S, or your code is inhibiting page file expansion. And in which case the failure is propigated all the way back to the application or app is ab-ended due to ignoring NULL returned or exception not handled.

Page File expansion errors generally occure on x64 platforms where Virtual Memory address space can exceed physical disk (or available page file restriction). On x32 platforms, the behavior is different but end results are the same. On x32, the memory node being returned but deferred for re-use later, generally results in the large memory blocks getting chopped up resulting in the agregate free memory being large enough but no one piece being large enough for your allocation. You can correct for this by creating allocator pools for like sized objects (or by enabling the LFH or both).

Jim Dempsey

[jimdempseyatthecove]

I forgot to mention:

When using Windows Task Manager the Process Mem Usage reflects the Page File requirement and does not represent the allocated memory requirements. Untouched Virtual Memory consumes no Page file pages (unless API called to preallocate page). Previously allocated, used, and delete/free'd memory has been touched and therefor thoseVirtual Address Pages are assigned pages in the Page File. Therefore, as your allocations/deallocations creap along Virtual Address space, so does your Page File footprint grow. By creating memory pools (or persistant and resuasable arrays) you can reduce the memory creep of your applicaiton.

Jim Dempsey

[onkelhotte]

Thanks for the insights of how memory allocations work and what do you have to consider.

Markus

[anthonyrichards]

Are we talking disk memory here rather than DRAM fast access memory? How to ensure that you get dRAM memory? Or is that impossible?

[jimdempseyatthecove]

Quoting anthonyrichards

Are we talking disk memory here rather than DRAM fast access memory? How to ensure that you get dRAM memory? Or is that impossible?

We are talking about both.

Sketch of typical heap implementation:

struct HeapNode
{
struct HeapNode* next;
void* block;
size_t size;
};

HeapNode* HeapHead = NULL;
...
// CRTL initialization
HeapHead = FirstUnusedVirtualMemoryAddressAboveProgramLoad;
HeapHead->next = NULL;
HeapHead->block = HeapHead + 1; // point at empty space following head node
HeapHead->size = HigestPossibleVirtualMemoryAddress
- HeapHead->block; // (with appropriate casts)
...

At this point you have a heap consisting of one memory node referencing all of unused Virtual Address space.
Additionaly, this address space is NOT committed to Page File (or at most only part of the first page of this address space has been mapped to Page File). The Page File pages are mapped into these Virtual Addresses upon writing to any Virtual Address within a page (2KB or 4MB). When first write occures it is to unmapped memory a page fault occures. The O/S responds to the page fault by allocating a page from the page file and mapping it to the Virtual Memory address "directories" and then backs up the instruction pointer to try again... .OR. in the event of Page File full or quota exceeded your application receives an error. This creates a secondary problem; a Heap Node declairedmemory was available but page file allocation fails. If this error condition is not properly handled then your program crashes.

It is possible to have in-use Virtual Address that exceeds the physical memory of the system. In this case, portions of your program are in DRAM and other portions reside in(were written to)Page File pages. Upon access to non-resident portions of Virtual Address a mapped page is evicted from the Virtual Memory "directory" (which includes a disk write when page had been modified), followed by a Page File read, thenVirtual Memory "directory" update then restart of faulting instruction.

The above is a sketch of what happens, actual coding is more complex.

Jim Dempsey