Force immediate allocation of large arrays

[Oscar_B_]

Hi everyone,

In a number of scenarios, we allocate fairly large arrays, i.e. a 4 million by 800 matrix of double precision numbers, which we fill column by column during an iterative process.

On several occasions we have noted that the memory used as reported by the operating system does not correspond to what we would expect by simple counting (4 million * 800 * 8 bytes), but is much lower and is then slowly increased as the iterations continue. This suggests that the compiler keeps an eye on how much memory is actually used - a very cool feature, since our iterations often stop before we reach the last column.

However, it would be nice to have a compiler option, where one could for debugging purposes force the compiler to immediately allocate the entire array. A search in the man page did not bring up anything, so therefore I am asking you guys/girls if you know of such an option?

[Steven_L_Intel1]

No, the compiler does no such thing. Rather, you are seeing the normal effects of virtual memory allocation. The address space is reserved for your array by the call to the operating system, but the pages are not brought into physical memory until you touch them. This is how nearly all virtual memory operating systems work.

You cannot depend on physical memory use as a gauge of virtual memory use. The OS manages the physical memory.

What exactly are you trying to do here? How would bringing the entire array into physical memory help debugging?

[Oscar_B_]

Dear Dr. Fortran

Thanks for your reply. I am looking for memory leaks. Currently, I am doing this by calling a function at a number of key points in the code. This function returns the currently allocated physical memory by my program, and I can thus compare what I expect the memory increase to be against what is actually allocated. But you are saying that I should consider both the physical and the virtual memory allocated by my program?

Do you have any tips on simple ways of getting the physical and/or virtual memory, allocated by my program, from a Fortran program compiled by ifort? 

Oscar

[Steven_L_Intel1]

You should ignore physical memory use - it's virtual memory you are concerned with. I recommend Intel Inspector XE which has excellent memory leak analysis tools, plus a profile of memory use over time. You can get a 30-day free trial.

[jimdempseyatthecove]

allocate(YourLargeArray(4000000,800))
YourLargeArray = 0 ! This may take a while to map pages and then wipe
! At this footprint will be indicative of size

Note, the allocation might succeed, but the =0 may fail if there is insufficient page file space.

Jim Dempsey

[Izaak_Beekman]

If you’re on Linux, you can also use valgrind to check for memory leaks, although the interface is not quite as nice as inspectorXE (no GUI at all for valgrind). On Mac OS X you can use instruments to check for memory leaks. (There is also a program called leaks on OS X that can be called from the command line, but AFAICT it needs to attach to an already running process, which could be problematic for a large simulation code.)