Single array memory allocation limit in 64-bit systems

[kostastse]

I read somewhere in the forum (can't find the post), that in 32-bit systems, if you use the /3GB switchin the boot.ini and the /largeaddressaware option in the linker,you can have about 3GB of memory available for allocation, but a single array can't allocate more than about 2GB of them.

Does the same thing apply in 64-bit systems as well?

I mean, in a 64-bit system one can alloccate much more memory than 3GB, but what's the limit for a single array?

In all the cases I am referring to dynamic allocation, using ALLOCATABLE arrays.

Thanks!

Kostas

[Steven_L_Intel1]

The compiler does not impose its own limit - on the 64-bit operating systems, you should be able to allocate arrays as large as the OS is willing to handle. I know we've tested up to 6GB at least.

[knasterbax1]

If there is no specific memory limit for single arrays, how comes the following effect:

Im using Win Server 2003 x64 (Athlon64) and newest Intel Fortran Compiler 9 for EM64T.

I allocate a single char-array of, say 2.5 GB. Allocation works, but when I access elements beyond ca. 2100 MB I get access violations. If I allocate two separate arrays of less than 2100MB it works. So it seems not to be the case, that the process is limited to 2GB, but single array seem to be...

[Steven_L_Intel1]

I can't explain the results you are seeing - especially if allocation of the larger array succeeds but you cannot access it. Please send an example demonstrating the problem to Intel Premier Support.

[jim_dempsey]

This sounds like the index you are using to access the array is 32-bits. Check to see ifyour index variable is 64-bits. (As well as any variables used to produce the index.)

Note, F90 does not have "unsigned integer". Therefore any 32-bit expression exceeding 2,147,483,647 will be negative.

Jim Dempsey

[Intel_C_Intel]

Hello.

Increasing the stack size using the option /stack, like for example "/link /stack:64000000" may help to enable verylarge arrays.

Lars Petter Endresen

[marshall-l-buhl]

I bought a server last spring that runs Win2k3 EM64T and have spent tens of hours since then trying to get IVF 9 to compile a program with arrays over 2 GB. That's why I bought the server. I've have been working with Wendy using the Premier support system and finally, yesterday, I got this answer from her:
----
Q2: How do I access larger memory models with the linker?

You can not do this on Windows (we do offer the ability to access larger memory models on Linux where the underlying tools/OS supports it - this is supported by the mcmodel switch).
----

This seems to be a direct contradiction to what Steve Lionel said on September 13:
----
"The compiler does not impose its own limit - on the 64-bit operating systems, you should be able to allocate arrays as large as the OS is willing to handle. I know we've tested up to 6GB at least."
----

Who is correct?

I must say, however, that the "compiler" did not impose the 2 GB limit but the linker did. Here's the output from my compile and link:
----
D:DesignCodesTestTest64>ifort test64.f90 /link /stack:64000000
Intel Fortran Compiler for Intel EM64T-based applications, Version 9.0 Build 20051130 Package ID: W_FC_C_9.0.028
Copyright (C) 1985-2005 Intel Corporation. All rights reserved.

Microsoft Incremental Linker Version 8.00.40310.39
Copyright (C) Microsoft Corporation. All rights reserved.

-out:test64.exe
-subsystem:console
-entry:mainCRTStartup
/stack:64000000
test64.obj
LINK : fatal error LNK1248: image size (BEC44000) exceeds maximum allowable size (80000000)

D:DesignCodesTestTest64>
----

Please note that the error is a linker error. The program had compiled sucessfully, but could not link. I was *not* dynamically allocating the 3 GB array, but set the array size on the declaration statement. Please also note that I tried to increase the stack as Lars suggested on December 12, but it did not help.

This has been an extremely frustrating experience for me. I've wasted thousands of dollars of the government's money for the server and tens of thousands for my time trying to build large-memory programs for Windows. If only someone or the documentation had just said it can't be done.

Marshall

Message Edited by Marshall-L-Buhl on 12-21-2005 08:30 AM

[Steven_L_Intel1]

Both are correct.

Windows, even the 64-bit versions, have a 2GB limit for static code and data. You can exceed this with allocatable arrays, but not static arrays (eg. COMMON.)

[marshall-l-buhl]

Wow! Quick response. Thanks Lionel.

OK, so I tried to dynamically allocate the array. Here's my test program:
----
program Test64
implicit none
integer(8), parameter :: arysize = Z'C0000000' ! this is 3 GB
integer(8) :: i
real(8), allocatable :: ary( : )
write(*,'(z12,i12)') arysize, arysize
allocate ( ary(arysize) )
end program Test64
----

Here's my build:
----
D:DesignCodesTestTest64>ifort test64d.f90 /link /stack:64000000
Intel Fortran Compiler for Intel EM64T-based applications, Version 9.0 Build 20051130 Package ID: W_FC_C_9.0.028
Copyright (C) 1985-2005 Intel Corporation. All rights reserved.

Microsoft Incremental Linker Version 8.00.40310.39
Copyright (C) Microsoft Corporation. All rights reserved.

-out:test64d.exe
-subsystem:console
-entry:mainCRTStartup
/stack:64000000
----

Here's my execution:
----
D:DesignCodesTestTest64>test64d
C0000000 3221225472
forrtl: severe (41): insufficient virtual memory
Image PC Routine Line Source
test64d.exe 000000000045576F Unknown Unknown Unknown
test64d.exe 0000000000452B29 Unknown Unknown Unknown
test64d.exe 000000000041411D Unknown Unknown Unknown
test64d.exe 00000000004074DF Unknown Unknown Unknown
test64d.exe 0000000000403D80 Unknown Unknown Unknown
test64d.exe 0000000000403C9B Unknown Unknown Unknown
test64d.exe 000000000040112E Unknown Unknown Unknown
test64d.exe 0000000000457B3C Unknown Unknown Unknown
test64d.exe 0000000000444E38 Unknown Unknown Unknown
kernel32.dll 0000000078D5965C Unknown Unknown Unknown
----

Normally, I wouldn't do an allocation without checking the status. When I tried that, I got a return status of 41, which I suspect means insufficient memory. My computer has 6 GB of RAM and another 6 GB of virtual memory--far more than is needed to hold a 3 GB array.

As you can see, I'm using the latest EM64T compiler. I downloaded the latest version of PSDK-amd64.exe yesterday. and applied it before installing the compiler.

So, Lionel, what am I doing wrong?

Marshall

Message Edited by Marshall-L-Buhl on 12-21-2005 09:51 AM

[Steven_L_Intel1]

The amount of RAM you have is not directly relevant. That you got an "insufficient virtual memory" error means that the operating system returned an error. We have successfully allocated much larger arrays - you have not run into a Fortran constraint.

[marshall-l-buhl]

Steve,

This is a pretty simple program. Can you offer any advice? When I look at Task Manager, I'm told my Commit Charge is 232M / 11870M. I definitely have much more than 3 GB of available memory.

Does it sound as if the compiler is incorrectly installed?

Any hints as to how I can get past this roadblock would be greatly appreciated.

Marshall

[Steven_L_Intel1]

I can't think of any installation-related issue that would help. You are using the compiler "for Intel EM64T-based applications", yes?

It would be interesting to see if a call to the Win32 API routine VirtualAlloc for the same number of bytes succeeds.

[marshall-l-buhl]

Steve,

As you can see from the screen output quoted in my earlier posting, I am using the EM64T compiler.

I'll try the VirtualAlloc call just for grins if I can find documentation for it.

Thanks for helping,

Marshall

[marshall-l-buhl]

Oops! My apologies. In the example program I mentioned in my earlier message, I allocated an array of 3x2^30 elements. I forgot to account for the fact the elements were 8 bytes each, which means I was trying to allocate 24 GB--more than I had available. I can now allocate and use a 6 GB array.

Thanks for your help! The real tip I needed was the bit about the stack size.

Marshall

[Steven_L_Intel1]

Glad to hear it!

[criga]

Hey Marshall,

could you explain how the maths work and show your corrected code. I'm having the same problem ....

Also, I was wondering, if there is a tool around that keeps track of allocated memory. The task manager only seems to show, what is used, but not what is allocated only. Are there any other tools around that show how much memory is currently reserved for a certain process?

Thanks Chris

[Steve_Nuchia]

Memory management under Windows (and indeed any OS with a rich process model) is a tricky business. There is no single right answer to the question of how much memory a process is using or has reserved.

In general, anything task manager can do process explorer, freefrom SystInternals (now owned by MSFT)can do better. But you need to frame the question more precisely -- in terms that match the real process model of the OS -- to get an answer. "Working set size" is what you want to compare against your physical memory resource. Be aware that windows slows WAY down when you start to cramp its disk cache, leave 15% or so for that.

But for non-insane amounts of memory on 64-bit windows the limit will be the unused swap space. If the swap file is expandable and you allocate something larger than it can hold you may get a temporary failure, at least on XP. That's a wrinkle you don't find in most other operating systems.

If you try to allocate a truly insane amount of virtual memory -- like say you have terabytes of SAN storage configured as your swap file -- it is conceivable you could hit some other limit, like say in the operating system's page table management code. You also have to have sufficient contiguous linear (process virtual) address space available but that won't be a problem in x64 unless you've deliberately fragmented your space.

At the margins, on classic Unix systems the swap space was the limit for system-wide total allocated virtual memory. On Windows and Linux the limit is physical memory - kernel overhead + swap space.

I'm not clear what "maths" you (Chris) are asking about. I'll watch for followup questions and try to provide less enigmatic answers next time.

Finally, if you're trying to force your allocated array to be resident in physical memory you're off into a whole other arena of pain. You're telling the operating system's memory management that you know better than it does how to manage the resource. You may be right but you can expect it to get huffy about it and not be terribly helpful.

[veena_h_]

HI sir,can you help me to implement 2d array like below code

include<stdio.h>
#include<malloc.h>
int main()
{
  int ch,*a,data,*old,n=0,i=0;
  a=NULL;
printf("\n you want to add data");
scanf("%d",&ch);
while(ch)
{
    old=a;
        printf("\n enter");
        scanf("%d",&data);
    a=(int *)malloc(sizeof(int)*(n+1));
        if(n>0)
{
    for(i=0;i<n;i++)
{
a=old;

}

free(old);
}
a=data;
n++;
printf("\ndo you want to add another data");
scanf("%d",&ch);
}

printf("\n elements are");
    for(i=0;i<n;i++)
{
printf("%d\n",a);
}
}