topic Re: +2 GB memory access in Intel® Fortran Compiler

+2 GB memory access

snfper19 — Wed, 24 Sep 2003 21:50:12 GMT

I am developing an application that requires several large arrays. Each of the arrays is below the 2 GB limits, but the total memory needed for all the arrays are close to 3 GB.

I am using a Pentium III dual processor workstation with 2 GB RAM and 2 GB of swap space (upgrading shortly to Xeron based workstation with 4 GB RAM)

The current system is:
Os: Linix Redhat 7.1 (kernel 2.4.2-2smp)
Glibc: glibc-2.2.4-31
Ifc: Intel Fortran Compiler for 32-bit applications, Version 7.1 Build 20030521Z
Ld: GNU ld version 2.10.91 (with BFD 2.10.91.0.2)

The ulimit -a reports the following settings:
core file size (blocks) 1000000
data seg size (kbytes) unlimited
file size (blocks) unlimited
max locked memory (kbytes) unlimited
max memory size (kbytes) unlimited
open files 1024
pipe size (512 bytes) 8
stack size (kbytes) 8192
cpu time (seconds) unlimited
max user processes 65533
virtual memory (kbytes) unlimited

I have used the following program to test the amount of memory I can access:

Program memtest

type array
integer, pointer :: elem(:)
end type array

type(array), allocatable, dimension(:) :: a
type(array), allocatable, dimension(:) :: b
type(array), allocatable, dimension(:) :: c

integer n_elem,ierr
real mem,block

mem=0.0
n_elem=(2**26)
block=n_elem*4.0/(1024**3)

allocate(a(10),b(10),c(10),stat=ierr)
if (ierr /= 0) then
print *,'Allocation of arrays failed with status=',ierr
stop 16
endif

do i=1,4
allocate(a(i)%elem(n_elem),stat=ierr)
if (ierr /= 0) then
print *,'Allocation of A(',i,' ) failed with status=',ierr
stop 16
endif
mem=mem+block
print *,'A(',i,' ): ok - memory allocated (GB) = ',mem
a(i)%elem=1

allocate(b(i)%elem(n_elem),stat=ierr)
if (ierr /= 0) then
print *,'Allocation of B(',i,' ) failed with status=',ierr
stop 16
endif
mem=mem+block
print *,'B(',i,' ): ok - memory allocated (GB) = ',mem
b(i)%elem=1

allocate(c(i)%elem(n_elem),stat=ierr)
if (ierr /= 0) then
print *,'Allocation of C(',i,' ) failed with status=',ierr
stop 16
endif
mem=mem+block
print *,'C(',i,' ): ok - memory allocated (GB) = ',mem
c(i)%elem=1
enddo

print *,'EOJ'
end

I have tried both with and without the compiler options -stack_temps, but get always the same results:

A( 1 ): ok - memory allocated (GB) = 0.2500000
B( 1 ): ok - memory allocated (GB) = 0.5000000
C( 1 ): ok - memory allocated (GB) = 0.7500000
A( 2 ): ok - memory allocated (GB) = 1.000000
B( 2 ): ok - memory allocated (GB) = 1.250000
C( 2 ): ok - memory allocated (GB) = 1.500000
A( 3 ): ok - memory allocated (GB) = 1.750000
Allocation of B( 3 ) failed with status= 494
FORTRAN STOP 16

It seems that I can not exceed 2 GB of memory.

Is this a limit set by Linux or is it set by ifc?

Per

Re: +2 GB memory access

Steven_L_Intel1 — Wed, 24 Sep 2003 22:49:33 GMT

I am not as familiar with Linux as Windows, but it was my understanding that, as in Windows, the default process address space was 2GB. I don't know if there's a way to change that in Linux - others might.

Steve

Re: +2 GB memory access

arturodigioia — Thu, 25 Sep 2003 14:51:10 GMT

You might find this link useful. It's a summary of all the informastions a user with the same problem obtained from the lkml.

http://www.spack.org/index.cgi/LinuxRamLimits

I don't think upgrading to a Xeon workstation will solve the problem, no matter of the amount of RAM installed. It's a limit of 32bit processors and Linux kernel.
32 bit -> Max 4GB
Linux kernel -> Reserves 1GB
(Theoretical) max per process -> 3GB (the real limit depends upon many factors)

Apparently on Windows the limit is 2GB per process, as Steve said.

I would think twice before buying a shared memory SMP 32bit workstation for memory intensive scientific computing, at this moment. We've reached the limits of this architecture.
I would think about Itanium or the other chip which I don't name on an Intel forum ;-) .

Re: +2 GB memory access

Steven_L_Intel1 — Thu, 25 Sep 2003 22:48:28 GMT

I would think about Itanium or the other chip which I don't name on an Intel forum ;-)

Alpha? :-)

Steve

Re: +2 GB memory access

snfper19 — Fri, 26 Sep 2003 14:14:32 GMT

The information?s on: http://www.spack.org/index.cgi/LinuxRamLimits and on similar pages shows that it should be possible to access up to 64 GB of RAM on an IA-32 running Linux 2.4.x by supporting Intel's PAE (Physical Address Extension) features which are in all Pentium Pro and newer CPU's. Without PAE compiled into the kernel the OS can address a maximum of 4GB of RAM. With 2.4 kernels (with a large memory configuration) a single process can address up to the total amount of RAM in the machine minus 1GB (reserved for the kernel), to a maximum 3GB.

My kernel is configured for large memory support (CONFIG_HIGHMEM4G=y and CONFIG_HIGHMEM=y). Therefor, I expect that it should be possible to allocate close to 3 GB of memory. But my test Fortran program can only allocate ~2 GB.

Too me, this indicates that the limit is in the ifc compiler and not related to Linux nor to the IA-32 architecture. If this the right conclusion right?

Per

Re: +2 GB memory access

arturodigioia — Fri, 26 Sep 2003 15:41:26 GMT

From the page I cited above:
-------------------------
> we had problems with process dying right around 2.3GB (according
> to top).

Out of 3 GB, you had 2.3 GB used and 0.7 GB of tiny chunks of
memory that were smaller than what you tried to allocate.
-------------------------
So it seems that the theoretical limit of 3GB can be very far from the actual limit. Tweaking the kernel is an option, but it's not an easy step.
We've got a 2GB RAM+8GB swap RedHat 8 box, but I don't think the IT staff would be happy if I run your test code on it. I'll try to ask and I'll post the results if I can.

However, it's quite strange that these issues are very little documented on the net, since this limit is starting to cause many headaches in the field of scientific computing.

Re: +2 GB memory access

Steven_L_Intel1 — Fri, 26 Sep 2003 21:35:25 GMT

You're confusing virtual memory with physical memory.

PAE is a method, using specific OS calls, to map regions of extended physical memory. It does not allow the virtual address space to be extended and normal virtual memory calls cannot use PAE.

It is not a restriction of the compiler. The compiler will allocate whatever the OS tells it is allowed.

Steve

Re: +2 GB memory access

snfper19 — Fri, 26 Sep 2003 21:36:22 GMT

It seems to be ifc that sets the limit of 2 GB.

I have been running my test program compiled with the Portland Group pgf90 compiler. With this compiler I can allocate and access at least 2.5 GB of memory.

Here is the output from the program compiled with ifc:

A( 1 ): ok - memory allocated (GB) = 0.2500000
B( 1 ): ok - memory allocated (GB) = 0.5000000
C( 1 ): ok - memory allocated (GB) = 0.7500000
A( 2 ): ok - memory allocated (GB) = 1.000000
B( 2 ): ok - memory allocated (GB) = 1.250000
C( 2 ): ok - memory allocated (GB) = 1.500000
A( 3 ): ok - memory allocated (GB) = 1.750000
Allocation of B( 3 ) failed with status= 494

And here, when compiled with pgf90:

A( 1 ): ok - memory allocated (GB) = 0.2500000
B( 1 ): ok - memory allocated (GB) = 0.5000000
C( 1 ): ok - memory allocated (GB) = 0.7500000
A( 2 ): ok - memory allocated (GB) = 1.000000
B( 2 ): ok - memory allocated (GB) = 1.250000
C( 2 ): ok - memory allocated (GB) = 1.500000
A( 3 ): ok - memory allocated (GB) = 1.750000
B( 3 ): ok - memory allocated (GB) = 2.000000
C( 3 ): ok - memory allocated (GB) = 2.250000
A( 4 ): ok - memory allocated (GB) = 2.500000
Allocation of B( 4 ) failed with status= 1

Re: +2 GB memory access

Steven_L_Intel1 — Sat, 27 Sep 2003 03:41:23 GMT

I did a search of our support database, and it appears that indeed it is ifc that is limiting to 2GB. This is fixed in the next release, due out at the end of the year.

Steve

Re: +2 GB memory access

fudd_elmer — Tue, 04 Nov 2003 06:27:38 GMT

Steve,

Does the icc compiler under linux have the same 2GB limit?...our code runs out of memory at around 2.4GB although we have 4GB physical on a Xeon system.

John D'Angelo
dangelo@ieee.org

Re: +2 GB memory access

Steven_L_Intel1 — Tue, 04 Nov 2003 23:05:15 GMT

It looks as if I missed something when writing my earlier response. The compiler issue that was resolved was for Itanium systems only. My apologies for the confusion.

As I understand it, Linux on IA-32 allows at most 3GB for the process address space, but by default it reserves 1GB of that space for shared libraries, making the effective limit 2GB.

I don't think this is a compiler issue, but I don't pretend to be as familiar with the Linux side of the world.

Steve

Re: +2 GB memory access

lop — Tue, 27 Jul 2004 04:23:35 GMT

I'd like to report that the test program posted on this forum completes SUCCESSFULLY, with no static linkage or anything, on a Fedora Core 2 system. FC2 uses 2.6 kernels, and without looking I'm pretty sure PAE. The same program bombs out at 2GB on an FC1 system.

-Luke

Re: +2 GB memory access

lop — Wed, 28 Jul 2004 04:37:48 GMT

Ok, I've got more details.

On FC2, I am not successful with a kernel.org kernel, but I am with a stock Fedora kernel. Therefore, one of the mods redhat likes to stick in there must be helping.

I tracked down the exact one. I haven't had much luck getting this patch to apply cleanly to a kernel.org kernel, but enabling/disabling this option in the fedora kernel source (from rpm) directly affects how much memory I can allocate.

http://lwn.net/Articles/39283/

Re: +2 GB memory access

Steven_L_Intel1 — Wed, 28 Jul 2004 07:38:18 GMT

Thanks for the information.

Re: +2 GB memory access

florinm1982 — Sun, 08 Aug 2004 09:33:34 GMT

Hi!
Can you please tell me the link where I can get that patch that would fix the problems with generating/accesing files larger than 2 Gb?

Thanks,
Florin M.