Yet Another Virtual Memory Problem

[iatel]

Hi,

I have seen a lot of posts on this issue, but can't quite figure out everything about my problem. I am running a large application written in Fortran, on a 2-Gb RAM 32-bit system, and have finally run into a situation where dynamic memory allocation breaks down due to "insufficient virtual memory", while writing the same arrays to non-dynamic allocation causes the program not to run due to exceeded storage space. (2G is not a lot, I guess). The program is large and allocates a lot of arrays in the process, although most are deallocated by the time this part of the program comes; here, I have one 25-million-point array allocated dynamically, and seven such arrays written to non-dynamic storage memory (was dynamic, but I removed it for the same reason), and it seems to halt the program. Since I read people stumbling over billion-point arrays, I am wondering whether I'm missing something.

My paging file was at 3G or so, which is what is listed as "recommended" (why was it, by the way, if the limit is 2G??). It is NOT system-managed. Is it likely that I have run into a "terminal" wall with this?

Incidentally, what's the max paging file for a 64-bit system? I have one of those too; the paging file is at 2G right now, but it recommends more like 12. Is that attainable?

Thank you!!
Irina

[iatel]

And a follow-up problem/question:

I tried compiling and running the same program as I was mentioning above, on the 64-bit system. Virtual memory should be no issue there, right? (I have max at 4 gigs right now, and that should do for the time being...) But I ran into a different error message, having to do with kernel32.dll being relocated from memory (or something to that effect) and the application being unable to run correctly as a result.

It's a bad week for me. Any suggestions? Thanks!!!

[Steven_L_Intel1]

Yes, 2GB address space is the maximum on 32-bit Windows (and I don't want some smart-aleck to talk about the /3GB option...) The reason for a bigger pagefile is that your system runs many processes, and you need enough pagefile for everyone.

I know of no maximum pagefile size on 64-bit Windows. Note that you can have multiple pagefiles.

The kernel32 message is a new one to me. I suggest reproducing that and then doing a web search on the exact message text. I often find that leads me to a solution.

[jimdempseyatthecove]

Irina,

Try allocating your largest array first (at program start). If that does not get past the allocation then check your statically allocated memory (common or module data) to see that you do not have some huge aggregate of unused stuff left over from earlier versions of the program.

If you get past the allocation and initial few iterations of the application and then it fails, then I would suspect that some expression using the large array or slice of the large array is creating a temporary copy of the array or slice of the array. After enough iterations memory is fragmented and the allocation for the temporary copy fails. If this is the case then rework the expression such that no temporary array (slice)is created. This will not only fix the problem but get your program to run faster.

Jim Dempsey

[iatel]

Jim,

Thanks for your advice, but I think I need a little further help. It's possible that my program is not at its most efficient regarding memory use. But I'm not sure what you mean by unused stuff (or aggregate stuff) left over from earlier versions of the program. What would earlier versions of the program leave except earlier versions of output files that I write? Where would I check for this stuff?

What I have is that the current version of the program runs a bunch of subroutines, and each one allocates some arrays in turn. Some arrays are allocated at the beginning of the program, and are kept allocated all throughout; most, though, are deallocated. Some subroutines write some output files. A couple of these are several hundred MB. But this should be unrelated, no? Then we reach the subroutine that has the millions-of-points arrays. Here I find that I cannot allocate them either dynamically or statically, as I run into constraints either way (storage or virual memory). It seems unreasonable - what you suggest about "unused aggregate stuff" may well be true - but I'm not sure what I'm looking for. In other words: i'm not sure what's using up all 4 gigs of memory... (ram + virtual).

Back to basics, in other words :)
Thanks!
Irina

[jimdempseyatthecove]

>> I'm not sure what you mean by unused stuff..

Many of us are working with Fortran code that has evolved over 30 years, some maybe longer. All of this old code typically had COMMON blocks which were gradually migrated into MODULES. During this migration the COMMON blocks may have been left in (due to requirements of the evoloutionary process). At some point in time the COMMONs could have been removed but weren'tdue to an oversight, historical reasons, or backwards compatibility. If this is the case with your code, then you may have sizable amounts of memory that is reserved but never used.

I would suggest that you determine the largest size of array to be used during program run and do this atprogram start. Then use an allocatable arrayin amodule (global instance of a working array). Allocate to the largest sizes you intend to use later on in the applicaiton.

During the run of the application where you are working with smaller arrays then explicitly reference the shorter sizes. Array(1:ArrayInUse).

If this is too cumbersome (i.e. you have many instances of "Array" without subscripts) then try using a pointer

module foo
real, target,allocatable :: BigArray(:)
real, pointer :: Array(:)
...
end module foo

program YourProgram
allocate(BigArray(BiggestSizeYouWillEncounter))
...
c Where you would allocate(Array(SomeSize))
if(size(BigArray) .lt. SomeSize) STOP ! Bug
Array => BigArray(1:SomeSize)

Jim Dempsey