Memory Use and Allocation

[marti_anderson]

I have been using Compaq Visual Fortran to produce relatively straightforward Console applications (for multivariate applications in statistics). However, the methods I am programming usually require quite a lot of memory for doing matrix calculations. I have recently begun using Dynamic memory allocation, which helps me to manage memory. A "stack overflow" error, nevertheless, may still occur when attempting to run my program(s) with large data sets. I desperately need answers to the following:
(a) How do I work out the circumstances that will produce the stack overflow error?
(b) As I distribute my programs to others, I also need to know whether the circumstances in (a) are machine-specific (governed by RAM?) or not?
(c) Most IMPORTANTLY, I wish to know (or learn) how to produce programs that will not "blow up" like this with memory problems when sizes of matrices increase. Will this involve using virtual memory? How is this done?

I tried reading in the help files about something called "VirtualAlloc", but this apparently requires things like addressing particular places and sizes of memory and that basically read like greek to me. I would like to write programs that basically expand into the space they need for any particular use and on any particular machine.

Please help?
Dr Marti Anderson

[Jugoslav_Dujic]

Judging on the errors you get, you are using automatic arrays. Automatic arrays are created on the stack and they're dimensioned on the basis of variables (dummy arguments, commons or module variables). However, stack size is by default limited to 1MB (it's a linker setting), so that's the total limit of all living stack variables. You can increase it in Project/Settings/Link/Reserve. (Another possibility of "Stack overflow" error is infinite or very deep recursion, but that's seldom used).

However, a better approach for huge arrays are allocatable arrays. See ALLOCATABLE attribute, ALLOCATE and DEALLOCATE statements. In this way, you're limited only by swap file size (or cca 1.5 GB total).

Jugoslav

[Steven_L_Intel1]

It's possible that the compiler is creating temporary copies of arrays on the stack for certain array operations. There is no way at present to get the copies allocated elsewhere. If you run the program in the debugger, you should be able to see what statement is executing when the stack overflows - this can give you a clue as to what to do next.

Steve

[Intel_C_Intel]

I agree with Jugoslav. We have a multivariate statistical program that uses large arrays which we ALLOCATE. We tend to use NAG algorithms because they are pretty good at telling you how much workspace they require to solve - so you can dummy-call them to get the size of the arrays you need to allocate, allocate the arrays and then solve the problems.

[hbell]

Some follow-up ideas on the allocatable array thread:
1. I think you want to avoid allowing Windows to thresh around with virtual addressing when doing large matrix calculations. My vote is to stick to the limits of physical memory to the extent possible, even if it means doing the matrix calculations with lower dimension sub-matrices.
2. Before allocating arrays, first examine how much memory you have using
type(T_MEMORYSTATUS) lpMstMemStat
...
call GlobalMemoryStatus ( lpMstMemStat )
Bytes of physical memory available is now in:
lpMstMemStat.dwAvailPhys
3. If you want really fast calculations using allocatable arrays you might want to consider using or modifying my MASM inner product example (in the CVF distribution kit in folder C:Program FilesMicrosoft Visual StudioDF98SAMPLESMIXLANGMASMALLOCABL ).
4. If you don't mind inferior screen graphics you can start up Windows in "Safe Mode" (by pressing F8 immediately after the beep) and get a lot more available memory.
5. The ARPACK package seems to work pretty well for large symmetric eigenvalue problems (only fixed dimension arrays though). See http://www.caam.rice.edu/software/ARPACK/.
Authors are Rich Lehoucq, Kristi Maschhoff, Danny Sorensen, and Chao Yang.

Harry Bell