insufficient virtual memory

[rswy]

Hi,
I wish to run an application that does dynamic array allocation.I find that when the size of the array becomes an 8 digit value(eg:12046320) I am not able to allocate as many arrays of that size as I want.
More specifically I get a Visual Fortran error saying "insufficient virtual memory" when I try to allocate t(n,3) where n is an 8 digit value.
Am I doing something wrong or is there and upper limit to the total memory I can use for dynamic allocation.If there is an upper limit then how do I overcome this limitation and make my application get as much memory as it needs?
--rswy

[Steven_L_Intel1]

The amount of virtual memory you can allocate is limited by several things. First, there is a hard limit in Windows of 2GB of process virtual address space, but you have to subtract a bit from that for code, data, stack and other things in your address space. If you want more than that, buy an Itanium system!

Second, is that Windows insists that you have enough swapfile space to be a backing store for the allocated memory. So to get that 2GB, you need a 2GB swapfile.

Another limit depends on the OS and the CVF version - over the years, we've had to change the way we do dynamic allocation to get around limits in the Visual C run-time library on certain operating systems. Windows 95/98/Me are particularly problematic. Given the size you mention, I am guessing you may be running into this problem, which can prevent allocations of larger than 256MB. Maybe not, though...

There is no CVF-imposed limit - we just ask for the memory and see what answer we get back.

Steve

[tlillys]

We are running into a similar problem on the following platform: Win 2K SP2, DVF 6.6 Pro, 1.5GB RAM, 1700MB VM.

The attached test code depicts the allocation problems we are having. With ISUM = 250800000, the code allocates both arrays (X & Y), but with ISUM = 250900000 I get an error message. With ISUM= 250800000, memory usage is about 1.99 G while with ISUM = 250900000 the error message of 41 indicates lack of virtual memory. However, given the amount of physical and virtual memory, this should not be happening. For instance, when we run 2 instances of this code simultaneously, comment out the allocation of Y and with ISUM = 250900000, both work simultaneously, accessing a total of 3.1 G of virtual memory.

Are we missing something? Have any suggestions? Is the 2G ceiling Windows imposes that you speak of on a per process basis?

[Steven_L_Intel1]

Yes, the 2G is per-process, but the actual limit will depend on your swapfile size and what else is in your process address space. We have seen 1.75GB readily attainable.

Steve

[tlillys]

Is that 2GB/process limit documented anywhere? Would starting another process (ie, multithreading) be a way around the limit?
Thanks,
Ted

[Steven_L_Intel1]

The 2GB limit is a fundamental part of the Windows design. See this MSDN article

Threads run within a single process. If you started a second process, it would get its own 2GB, but you'd have to use some mechanism to communicate across the processes (such as sending messages).

Steve

[tlillys]

I realized that would be hairy to implement, I was curious if it would work in principle. Thanks for the article.
Ted

[Steven_L_Intel1]

No. You can have as much swapfile as you have disk space (well, I suppose Windows has some limit, but it is quite large). The 4GB total virtual address space is a limit of the 32-bit address size, but a given system could theoretically have hundreds of processes, each filling its 2GB of process address space. As long as there was sufficient swapfile to "back" all of it, it would work. Of course, if the system didn't have a lot of RAM, it would run slowly, but it WOULD run.

Steve

[knasterbax1]

The /3GB switch in boot.ini might help you when our OS is WinXP. It allows you to allocate 3 GB instead if 2 GB for one process. You must compile your fortran application with /largeaddressaware linker switch. that way I got about 2.6 GB for fortran arrays.