Out of memory in reading large file after allocating large arrays

[luca_sbordone]

Hi all,
I am running into memory problems when trying to run a code reading large files into allocatable arrays. Please keep in mind I'm not a software engineer but an astrophysicist, and my programming/computing skills are not deep nor formal. So my apologies if the question is trivial (I actually hope so).

I'm reading a grid of suynthetic stellar spectra from a large unformatted file. The file giving me problems is about 1.3 GB, and contains basically some scalar variables at the beginning and a large array of 3240 spectra, each one represented as a real vector of more than 100000 elements. Since both these values are variable, they are written into some of the first variables in the file. So the readin section is supposed to work as follows: I open the file, read the initial variables which give me the array dimension, then allocate the array, then read it. When I try this with a smaller grid, say a file of 300-400 MB, roughly 800 spectra, everything works. When I try with the large file, it allocates the array properly, then dies with an "forrtl: severe (98): cannot allocate memory for the file buffer - out of memory" error.

Now, what puzzles me is that if I define the big array statically instead that making it allocatable, the file is read without problems. So I guess that somehow the memory is allocated differently for statically defined arrays and allocatable arrays, so that the latter share some limited space with the file buffer memory... but as I said, I know nearly to zero about this business. I hope somebody can enlighten me!

Thanks

L. Sbordone

P.S. Everything is happening on a SUSE 10 box with 2 GB ram.

[Steven_L_Intel1]

What does "uname -a" say? Is this a 32 or 64-bit OS?

[luca_sbordone]

Steve, thank you for the quick reply. I was not in the office over the week end, so here it goes the uname -a output:

Linux ####### 2.6.16.54-0.2.5-default #1 Mon Jan 21 13:29:51 UTC 2008 i686 athlon i386 GNU/Linux

where the #### are the machine name. The system is 32 bit.

Thank you again!

L.

[Steven_L_Intel1]

Ok. All I can think of is that you're right at the edge of available virtual memory, though I do find the behavior puzzling as I would not expect the run-time library to need to allocate another buffer. Can you show me a short program that just declares the array, allocates it, opens the file and reads the file - showing the problem?

[luca_sbordone]

Steve,
thank you again. Sure I can. Here it goes:

program testpro
!just to test the virtual memory problem
!on mygisfos/broadgrid codes...

implicit none
character(len=50) :: gridco,filename
character(len=56) :: outname
integer :: frn,maxpigu,ten,metn,grn,vtn,aln,nalp,i,j,yfra,ti,gi,mi,vi,ai,fi,k,totsp,cosp
real(kind=4)::meti,mets,tei,tes,gri,grs,vti,vts,ali,als,sunabu(92),bro,broad
real(kind=4):: tef,gra,vtu,alp,fer
real(kind=4),allocatable::gridu(:,:,:,:,:,:,:),passgrid(:),vstep(:)
integer,allocatable :: npixu(:),alps(:,:)
real(kind=8),allocatable::wl_gr(:,:),passwave(:)
character(len=2) :: names(92)

open(12,file='grid.bin',form='unformatted')
!This is all the "header" part: many variables descriving the synthetic spectra grid
!are read from the beginning of grid.bin. Some of them are needed to define the size of the allocatable
!arrays.
read(12)gridco,frn
allocate (npixu(frn),vstep(frn))
read(12)npixu
read(12)maxpigu
read(12)vstep
read(12)tei,ten,tes
read(12)meti,metn,mets
read(12)gri,grn,grs
read(12)vti,vtn,vts
read(12)ali,aln,als
read(12)sunabu
read(12)nalp
allocate(alps(nalp,3))
read(12)alps
read(12)broad
!these are the two big arrays, the one giving problems (and by far the largest) is gridu
allocate (wl_gr(frn,maxpigu),gridu(frn,maxpigu,ten,metn,grn,vtn,aln))
read(12)wl_gr
read(12)gridu
close(12)
write(6,*)"Done."
stop
end

this one just reproduced the error. On the other hand, I had now a chance to test on a 64 bit Linux machine with 4 GB ram and there it runs. Now, I wonder whether it is the 64 bit or just the larger memory...

Thank you again!

L.

[Steven_L_Intel1]

I'd guess it was the 64-bit address space.