insufficient virtual memory error trapping

[larryscheier]

I develop engineering applications using 32-bit Intel 10.1 fortran compiler on Visual Studio 2005. Some of my customers still run 32-bit windows so I can't just update to 64-bit. Dynamically allocated arrays in 32-bit programs are limited to 2GB (usually much less given memory fragmentation, etc.). I have no problem with this since I write any data that exceeds the memory limit to disk. However, as I have no way of knowing how much memory my customers have installed (nor how many other programs or DLL's are running in their memory space), the problem is determining how large I can dynamically allocate an array before reaching the memory limit (which then crashes the program during runtime with error 41: "insufficient virtual memory"). Obviously, the program runs much faster when running with a huge array than it does reading/writing to disk so I want to dynamically create as large an array as I can. Calling the windows function GlobalMemoryStatusEx seems to vastly overestimate the available memory required by the fortran ALLOCATE function when dynamically allocating very large arrays. Looking through the forum it seems the only solution mentioned was to create an little subroutine that gradually increases the size of a dummy dynamic array until error 41 occurs and that defines the upper limit. So I guess I need help in writing an error trap to do that unless someone has a better idea.

Many thanks!
Lawrence

[nvaneck · Accepted solution]

Quoting - larryscheier

I develop engineering applications using 32-bit Intel 10.1 fortran compiler on Visual Studio 2005. Some of my customers still run 32-bit windows so I can't just update to 64-bit. Dynamically allocated arrays in 32-bit programs are limited to 2GB (usually much less given memory fragmentation, etc.). I have no problem with this since I write any data that exceeds the memory limit to disk. However, as I have no way of knowing how much memory my customers have installed (nor how many other programs or DLL's are running in their memory space), the problem is determining how large I can dynamically allocate an array before reaching the memory limit (which then crashes the program during runtime with error 41: "insufficient virtual memory"). Obviously, the program runs much faster when running with a huge array than it does reading/writing to disk so I want to dynamically create as large an array as I can. Calling the windows function GlobalMemoryStatusEx seems to vastly overestimate the available memory required by the fortran ALLOCATE function when dynamically allocating very large arrays. Looking through the forum it seems the only solution mentioned was to create an little subroutine that gradually increases the size of a dummy dynamic array until error 41 occurs and that defines the upper limit. So I guess I need help in writing an error trap to do that unless someone has a better idea.

Many thanks!
Lawrence

Use the stat=ichk option in allocate to trap the error.

[larryscheier]

Quoting - nvaneck

Use the stat=ichk option in allocate to trap the error.

Thanks! I'll try that!

Lawrence

[jimdempseyatthecove]

Quoting - nvaneck

Use the stat=ichk option in allocate to trap the error.

This will tell you when the heap cannot fulfill the allocation request.

I haven't tested to see if this works properly when the swap file (paging file) is consumed. My gut feel is it does not. The reason being is the pages in the page file are not bound to the virtual memory until the time of first touch. IOW a heap expansion and allocation might succeed to establish the virtual address entries but not the mapping to the page table (which occures later at first touch).

If the stat=ichk seems to work, it should be riggorously tested to verify it will not bomb out at the customer site.

If necessary Larry should be able to use C++ code to probe the heap (with exception code) to determine if the page file will accept the following Fortran allocation. e.g. prior to allocate call your C++ function with allocation size, the function uses _heapwalk and other heap functions to verify a node of the size (to be) requested exists, if not it calls the new heap allocaton routine, when node of sufficient size is available, it then touches each page (inside an exception loop). If the touching succeeds then upon return, the Fortran ALLOCATE should succeed .AND. be accessible. If the touch fails, you are in a low page file situation and the largeFortran allocation will succeed.AND. .NOT. be fully accessible. At this point your code should try to work through your data on file path.

A variation on this would be to replace ALLOCATE/DEALLOCATE with your own functions calling C++ code. This would perform the allocation .AND. touch page to verify page table has of sufficient remaining capacity. If not then pop-up request to expand page file, or Close other applications, or Abort, etc...

It might be easier to request your users to upgrade to x64 platform.

Jim Dempsey

[dannycat]

Quoting - jimdempseyatthecove

This will tell you when the heap cannot fulfill the allocation request.

I haven't tested to see if this works properly when the swap file (paging file) is consumed. My gut feel is it does not. The reason being is the pages in the page file are not bound to the virtual memory until the time of first touch. IOW a heap expansion and allocation might succeed to establish the virtual address entries but not the mapping to the page table (which occures later at first touch).

If the stat=ichk seems to work, it should be riggorously tested to verify it will not bomb out at the customer site.

If necessary Larry should be able to use C++ code to probe the heap (with exception code) to determine if the page file will accept the following Fortran allocation. e.g. prior to allocate call your C++ function with allocation size, the function uses _heapwalk and other heap functions to verify a node of the size (to be) requested exists, if not it calls the new heap allocaton routine, when node of sufficient size is available, it then touches each page (inside an exception loop). If the touching succeeds then upon return, the Fortran ALLOCATE should succeed .AND. be accessible. If the touch fails, you are in a low page file situation and the largeFortran allocation will succeed.AND. .NOT. be fully accessible. At this point your code should try to work through your data on file path.

A variation on this would be to replace ALLOCATE/DEALLOCATE with your own functions calling C++ code. This would perform the allocation .AND. touch page to verify page table has of sufficient remaining capacity. If not then pop-up request to expand page file, or Close other applications, or Abort, etc...

It might be easier to request your users to upgrade to x64 platform.

Jim Dempsey

I have noticed similiar behavior, i.e even when the stat=ichk returns 0 from an ALLOCATE statement implying success my program will soon crash or hang. Thisseems like the mapping to page table is where program fails. I'm notfamiliar withC++ so could you point me in the right direction with regard to creating the check function you mention in your reply?

Thanks.

For Intel:
Should this not be something thatis included as an option on the ALLOCATEstatement in Fortran or will it become superfluous when you move across to 64 bit?

[Steven_L_Intel1]

Quoting - dannycat

For Intel:
Should this not be something thatis included as an option on the ALLOCATEstatement in Fortran or will it become superfluous when you move across to 64 bit?

I'm not sure I understand the question. If the operating system returned an error due to insufficient virtiual memory, then the STAT= value will reflect that. Sometimes, though, if you have already used very close to the limit, the operating system may not have sufficient additional virtual memory to properly report the problem (say, if you tried to display an error message.) I have seen this across many different compilers and operating systems.

Even with a 64-bit environment, you can still run out of virtual memory. It just (usually) takes longer.

[jimdempseyatthecove]

>>If the operating system returned an error due to insufficient virtiual memory, then the STAT= value will reflect that.

Steve, your statement is true with respect to virtual memory address space, which is different from virtual memory in your address space mapped to page file pages. The mappingof virtual memory page tofile page occurs on touch. The problem lies in when the user (or heap manager) touches an address in a page that has not yet been mapped to the page file only to find out then that there is insufficient room in the page file.

Consider an x32program with a "simple" heap manager. The code and static data is 100MB, the application space Virtual Address space max is 2GB (we won't consider system or 3GB in this example). The heap manager creates an initial heap free nodeof 2GB - 100MB - StackSize starting at Virtual Address 100MB+StackSize.

*** but it does not touch this memory aside from inserting heap integrity signatures in front of and after the ~2GB free node.

Virtual memory is not assigned to the page file at initialization time, since all apps presumably will be doing the same thing. 100 apps would require 200GB and this would be an unnecessary acquisition of disk space.

Virtual memory pages are mapped to the page file at time of first touch. When allocations are small in size, the heap manager will tend to be the first to touch the memory (but not in all cases). When the allocations are large, say many pages in sized, the heap manager will touch the memory immediately preceding the node and touch the memory immediately following the node. When pages exist in between, these pages will not be touched until the application does so (or if running under Debug build the Debug heap manager presets the data on allocation).

The result of the above is in release build, you can have allocations that succeed (to consume address space) yet fail (later) to obtain pages inside the page file.

Jim Dempsey

[jimdempseyatthecove]

Steve,

I forgot to add.

What Intel could do is provide an option for ALLOCATE to touch the memory in a application crash-free manner and if page file insufficient, return the failing node to the heap, then return allocation failure (or two options, the second to initialize to value as well as test for page file full condition).

Jim Dempsey