Problem with allocatable arrays

[michaelgreen1]

Greetings,

I have code like this:

integer(4),allocatable:: grid(:,:),temp(:,:)

if(allocated(grid))deallocate(grid)
if(allocated(temp))deallocate(temp)

allocate(temp(0:ew+1,0:ns+1),grid(ew,ns))

call ZeroMemory(loc(temp),sizeof(temp))
call ZeroMemory(loc(grid),sizeof(grid))

the sizes of ew and ns are typically 4000 to 7000 so grid and temp can be quite large.

On the first time through the above code everything works well, but on the second time through the attempt to allocate the arrays gives me "insufficient virtual memory" despite the fact I have just deallocated the arrays.

Cam anybody suggest what might be going on?

Many thanks

Mike

[jimdempseyatthecove]

Are you building for 32-bit environment?

If so, change to 64-bit code generation.

A 32-bit program has a Virtual address capacity of 4GB. However, not all of this is available to an application. Half of it, the upper 2GB is reserved for system space. The lower 2GB is available for process space (your program, stack, heap, code, initialization data, and a few other things).

Now then, had you experienced Stack Overflow's in prior runs, you may have been inclined to set a humongous stack size to resolve this problem. The stack size takes away from the available memory for the heap. (system page file size can also restrict available memory to the process as well).

With a small heap, you have to be aware of, and avoid memory fragmentation. Fragmentation can result in: while the total available heap could satisfy the allocation, there is no fragment (free node) sufficiently large enough to be used for the allocation.

64-bit mode will reduce the likelihood of an allocation failing (though fragmentation can occur without the allocation failing, process size may get ungainly large).

If you are unable to switch to 64-bit and reducing the stack size doesn't help. Then (on Windows) consider enabling the Low-fragmentation Heap.

I'd suggest the 64-bit route.

Jim Dempsey

[jimdempseyatthecove]

FWIW

allocate(integerArray(7000,7000)) requires approximately 200MB. On a 32-bit application, and tiny code and other resources, and first allocations are of these arrays. You'd be lucky to get 9 of these.

Deleting say the 1st one, then allocating something smaller might take part of the former 200MB node.

Deleting this something smaller will not (necessarily) piece back together the former 200MB node. An thus a subsequent allocation of the 200MB would fail.

IOW through allocations and deallocations you may be whittling down the sizes of the available free nodes.

Jim Dempsey

[michaelgreen1]

Thank you very much for your reply. Since receiving it I have tried numerous options without success though I have learnt a lot.

Unfortunately, I am unable to build my application in 64 bits so instead I have spent a lot of time adjusting the algorithm and making certain I do not have memory leaks. I have also ascertained that the low-fragmentation heap is in operation already. (All this, by the way, is on a server - does that make any difference?)

Since I have made the program more efficient, I have found that the memory problem only seems to occur in Debug mode - I presume that is the fault of the debugger taking up large amounts of memory. In Release mode, I can run through the loop as many times as I wish.

With respect to your explanation referring to the whittling down of available memory, why does this not happen if the program goes to completion, and I start it again? Presumably something happens to clean up the memory I used when the program exits and makes it available for the next run - whatever this is, can it not be called upon within the program without the program having to exit first?

Many thanks

Mike.

[JohnNichols]

it is called garbage collection and it is not always efficient, this is a major problem with some languages.  

What does zero memory subroutine look like and why deallocate and reallocate just zero out and reuse.  

But I am doing this with some arrays and have no problems. 

Why can you not use 64 bit?

post a sample there are people here who will play with it.  

[jimdempseyatthecove]

0>x|code|data|------------- heap -------------------------------------------------------------|stack|---------- system ----------|
1>x|code|data|aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaabbcdddd--------------------------------|stack|---------- system ----------|
2>x|code|data|                                         bbcdddd--------------------------------|stack|---------- system ----------|
3>x|code|data|eee                                      bbcdddd--------------------------------|stack|---------- system ----------|
4>x|code|data|eee                                      bbcdddd--------------------------------|stack|---------- system ----------|
                 aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa    aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

1 state at program start. x is generally reserved virtual page 0, protected against access. Above is for Windows, Linux is different.

2 First pass of system, large array is a's, other allocations are b, c, d.

3 At completion of first pass large array is returned to heap.

4 Some other allocation occurs (e's) occupying former place of large array

Now when large array needs to be allocated, while there is sufficient free heap for the allocation, there is no single empty node that can satisfy the allocation.

On a 64-bit system, the size of the virtual address space for the heap many times larger than the runtime requirements. And thus the insufficient memory (available free node) is not observed. You can run into a different issue with your page file being too small, but that is often not seen.

As to what you can do to work around this. One method is to keep your largest array(s) persistent. iow do not deallocate it(them). Reuse the first allocation. Your code does not have to use the entire size of the array. And you can use a pointer that has the same rank/sizes as that of what would have been your allocation.

The recommended way would be to place the pseudo allocations into a module contained procedure as this will make for cleaner usage.

Additionally, code in a manner that avoids array temporaries. 

Jim Dempsey

[jimdempseyatthecove]

Here is an example:

module alloc
    real, allocatable, target :: blob(:)
    real, pointer :: array(:,:,:)
    contains
    subroutine allocInit(maxBlobSize)
        implicit none
        integer :: maxBlobSize
        allocate(blob(maxBlobSize))  ! largest working size required
    end subroutine allocInit
    function allocArray(sz1, sz2, sz3) result(ret)
        implicit none
        integer :: sz1, sz2, sz3, ret
        if(sz1*sz2*sz3 <= size(blob)) then
            call helper(blob)
            ret = 0 ! no error
        else
            ret = 1 ! error code here
        endif
    contains
        subroutine helper(x)
            implicit none
            real, target :: x(sz1, sz2, sz3)
            array => x
        end subroutine helper
    end function allocArray
end module alloc
program Console14
    use alloc
    implicit none
    call allocInit(666666)
    ! ...
    if(allocArray(12,34,56) /= 0) stop "Allocation error"
    print *,loc(blob), loc(array)
end program Console14

Output:  25403440 25403440

I tried using TRANSFER to avoid the helper subroutine, but that gave me a compile time error.

You should be able to rework the code and embellish it. ( add a deallocArray that nullifies array, and error in allocateArray should array be associated).

TBD what is the largest working array size.

Jim Dempsey