insufficient virtual memory problem

[lukc_clement]

Hi,

I'm working on a project that will use a sparse matrix solver in Fortran. When I need to allocate a large chunk of memory (1.3Gb), it will cause the DLL to blow up and print an error message saying 'insufficient virtual memory'. Actually, there are still a lot of physical memory available and the machine has 2Gb physical memory and 4Gb virtual memory (set at control panel).

However, if I just write a dummy console program to allocate 1.3Gb memory, the program works.

So, my question is how to windows determine if there is enough memory? And if there is any work around?

I do some web search and found out that Windows will only allocate if it see contiguous memory. This really confuses me. Because I thought OS will always create contiguous virtual memory space for the user, and the virtual memory is more or less acting as a pointer. If that piece me memory is not avaiable in RAM, it will cause a page fault. Is that windows need to see contiguous physical memory?

Code detail
------------------------------------------------
Before calling this DLL this program use about 100Mb

Complex*16, allocatable :: A(:)

(from outside NK is about 3.2M)

LA = 25*NK // about 1.3G
ALLOCATE (A(LA))

-----------------

If LA is set to
LA = 17*NK

it could allocate. It semes to me as long as total is less than 1G it will still work? Is this a Windows DLL limit?


Clement

[Steven_L_Intel1]

There is no "DLL limit". DLL code is just code. 32-bit Windows has a fixed limit of 2GB for process virtual address space (can be raised to 3GB on some configurations, but with restrictions.) In reality, the practical limit is around 1.75GB. You also have to take into account code, static data and any dynamic allocated data your application uses.

It sounds to me as if your application is already using enough VM so that a 1.3GB allocation pushes you over the limit.

[kavalero]

Clement,

Here are two suggestions for you to deal with large memory allocations. Both of them require going down to the level of system API (Win32 or other) and I don't know how well it will mesh with Fortran so you might need to write it in C:

1) use memory mapped files. The file system is 64-bit so the file can be much bigger than 4 GB. However, the views you open to that file are limited to 4 GB. Since you have a lot of physical memory, the actual disk writing may never occur, esp. if you point your file to the swapper. The memory accesses within the view are done just like regular memory accesses.

2) go to the OS to allocate and commit memory for your application rather than use the default heap. This will still be still subject to availability of the contiguos block. But you can grab that block as soon as you start the process. If you need to do a lot of reallocations on top of that, you could use a third party heap manager, or maybe create a user heap using the runtime. (VisualAge C++ makes it rather trivial)

Vadim