Insufficient Virtual Memory on Allocation - again

[YertleTheTurtle]

Hello:

I have a user who managed to create a very large case and hit the "Insufficient Virtual Memory" (error 41 on Allocate) error.

So, I made up a small test program that allocates and deallocates successively larger arrays, until it hits the allocation error limit.

That limit seems to be about 2GB, which is consistent with everything I've read.

Then I tried combinations of the following variations to try to get around the limit:

1. /3GB using BCDEDIT /set IncreaseUserVa 3072.... no change

2. Set Largeaddressaware ...  no change

3. Compiled with both ia32 and SSE3 ..  no change

4. set file Pagefile.sys to 10 GB ...  no change

The only other hint I could find is in the Fortran Help pages where it says:

"See the product Release Notes for more information on memory limits for large data objects."

I can find nothing in the product Release Notes.

I am using Intel® Visual Fortran Compiler Professional Edition 11.1.

My test program runs on Windows7, 4GB RAM, but my user has access to all sorts of different machines and they all generate the same problem.

Although my test program just allocates one large array, the real program is allocating several smaller (but still large) arrays. That doesn't seem to make a difference.

Questions:

1. Does this compiler (version??) have some intrinsic limit of 2 GB?

2. Is there some way around this that escapes me?

Thank you for any advice.

[IanH]

The obvious solution is to move to 64 bit compilation and operating system.  Otherwise you are pretty much hard up against the fundamental limits of virtual memory for 32 bit windows.
 

[YertleTheTurtle]

I am running 64 bit Windows 7, and I thought I was doing 64 bit compilation by using

ifort /arch:sse3

but it made no difference to the test case.

There doesn't seem to be a flag for something like arch:ia64, so how does one guarantee 64-bit compilation? I have searched the help documents looking  for a "64-bit compilation" option and found nothing.

To make things clear, the problem is not that I have a >2GB array. I have more than 10 arrays each of which is about 0.2GB, and I am hitting the 2GB limit when I try to allocate more or less the 10'th one.

Thank you.

[IanH]

You get 64 bit compiles if you compile in a "64 bit compiler" command prompt.  That's called "Intel 64" today, it may have been "em64t" or some such back in 11.1 days.  In Visual Studio you get it if you have created and selected the x64 platform (create via Build > Configuration Manager or similar).

If the compiler accept /arch:ia32, then you are not compiling for 64 bit. 

(Equally telling - if your program runs out of address space allocating allocatable arrays after only 2GB of allocations... you are not compiling for 64 bit.)

If you can't find the appropriate command prompt or if VS doesn't appear to know about 64 bit compilation, then note that for some VS editions/versions there is (was?) the requirement to manually select installation of 64 bit compiler components when installing.  In ifort's case you may have needed to select the correct combined ia32/x64 installation package.  Can't remember details now, sorry.

[mecej4 · Accepted solution]

If the compiler that you use does not generate 64-bit code, all that you are going to have available to run is a 32-bit program limited to 3/4GB, regardless of whether you will run on another machine with 32-bit Windows or 64-bit Windows. If you have a compiler that generates 64-bit code, the resulting EXE/DLL will only run on a machine with 64-bit Windows.

Finding out the version of your compiler is easy. Just open the command window from the Intel Compiler 11.xx start menu, and type "ifort"; the output banner line will tell you the target type. For example,

     "Intel(R) Visual Fortran Intel(R) 64 Compiler XE for applications running on Intel(R) 64, Version..."

OR

     "Intel(R) Visual Fortran Compiler XE for applications running on IA-32, Version..."

Some earlier versions of Intel Fortran were cross-compilers, but yours is probably not such a one.

Finally, you do not want /arch:ia64 unless you have a machine with an Itanium CPU.

[Steven_L_Intel1]

There is no /arch:ia64. /arch is an instruction set selector that omits the check for a supported Intel CPU when the program is run. /arch:ia32 restricts the instruction set to what the Pentium II supported.

I agree that you need to build a 64-bit application using the "Intel 64" compiler.

For information on memory limits, see https://software.intel.com/en-us/articles/memory-limits-applications-windows

[YertleTheTurtle]

Now that it is explained how to compile for 64-bit applications, I found a lot of information in the help documents.

(Once you know what you need to know, it's easy to find. Just a matter of already knowing where to look.)

I had previously compiled a 64-bit version using Visual Studio x64 configuration, without understanding that it was invoking a different compiler, not just a few different compilation switches.

But the problem persisted, because I hadn't increased pagefile.sys.

Hence the test program.

Thank you all - that solved it.

BTW - version 11.1 was new when I bought it, and that doesn't feel like so long ago.

Is there something in the newest version that I really need?

[Steven_L_Intel1]

Well, the newer version is supported, has more language standard features, supports newer Visual Studios and has bugs fixed.

[John_Campbell]

When creating a 32-bit executable, the largest array size for a single allocatable array is < 2gb. With a 64-bit executable, this limit no longer applies.

Using Windows 7_64 OS on another 32-bit Fortran compiler, I can create multiple allocatable arrays, each less than 2gb, with a combined allocation of up to about 3.7gb. It is like defaulting to a /4gb switch. I am not sure if ifort_32 allows this but it should support /3gb functionality. 

I (very) rarely use ifort_32, as I find ifort_64 to be a much more flexible solution. However if you are testing ifort_64 to see how big an array can be allocated, this can be deceptive, as you can allocate a larger virtual array than can be practically used, when the limits of the paging file size apply as the program uses this allocated memory.

John

[Steven_L_Intel1]

The stack size setting is unrelated to errors on an ALLOCATE.