recl and open

[marc_hufnagl]

Hello everyone,
I have a problem concerning reading a file. The file is structured as follows
365 days
a grid of x = 301 (M), y=254(N)with z =21 (K) layers
for each day Temperatures and Salinity is stored and each value has a length of 4 byte. the file therefore has a size of
364x2x301x254x21 entries times 4 bytes = 4688.16 Megabytes

If I read the file with the following code (kmax = 21, M = 301 and N=254):

for day = 1,365

open(16, file = 'TS.081', &
form = 'unformatted', &
access = 'direct', &
recl = M*N*4, &
iostat = ios)
if (ios /= 0) then
print *, ' !! access to 16 not possible !!'
stop
endif

RecBaseTS = (Day-1) * (2 * kmax)

do k = 1, kmax
read (unit = 16, &
rec = RecBaseTS+k, &
iostat = ios) array3D(:,:,k)
T(:,:,k) = transpose(array3D(:,:,k))
enddo

do k = 1, kmax
read (unit = 16, &
rec = RecBaseTS+k+01*kkMax+00, &
iostat = ios) array3D (:,:,k)
S(:,:,k) = transpose(array3D (:,:,k))
enddo

I get the error message ios = -1 after 28 days.
I cannot find the problem. If I chose a recl of M*N*8 (8 Byte per entry) the iof = -1 is returned at day 14. A recl of M*N is not possible and even if would only increase the days by a factor of 4 (120 days and not 365)

I am thankful for every hint an help. Cheers Marc

[Steven_L_Intel1]

Add /assume:byterecl to the options (if you are using VS, this is Fortran > Data > Use bytes as RECL= unit...)

[Paul_Curtis]

Quoting - marc.hufnagl

... Temperatures and Salinity is stored and each value has a length of 4 byte.

If it's not too late for some systems analysis, the nature of your measurements suggests that their actual dynamic range and precision may allow storage in one byte each (ie, 0 < value < 255, no decimal component), so a single I*2 could hold both datums. This would make for great ease and speed of manipulation, as well as maximal concision in storage, and each set of merged data would only require only one spatial array.

[marc_hufnagl]

Quoting - Steve Lionel (Intel)

Add /assume:byterecl to the options (if you are using VS, this is Fortran > Data > Use bytes as RECL= unit...)

Thx for your reply, I already did this and also converted to big endian but the problem still exists

[marc_hufnagl]

Quoting - Paul Curtis

If it's not too late for some systems analysis, the nature of your measurements suggests that their actual dynamic range and precision may allow storage in one byte each (ie, 0 < value < 255, no decimal component), so a single I*2 could hold both datums. This would make for great ease and speed of manipulation, as well as maximal concision in storage, and each set of merged data would only require only one spatial array.

Thx for the reply. I have files for 40 years that already exist and therefore unfortunately I cannot change the size or shape of the file. It would be nicer to handle if everthing was stored like you suggested and files would be much smaller but I have no influence.

[tropfen]

Quoting - marc.hufnagl

Hello everyone,
I have a problem concerning reading a file. The file is structured as follows
365 days
a grid of x = 301 (M), y=254(N)with z =21 (K) layers
for each day Temperatures and Salinity is stored and each value has a length of 4 byte. the file therefore has a size of
364x2x301x254x21 entries times 4 bytes = 4688.16 Megabytes

If I read the file with the following code (kmax = 21, M = 301 and N=254):

for day = 1,365

open(16, file = 'TS.081', &
form = 'unformatted', &
access = 'direct', &
recl = M*N*4, &
iostat = ios)
if (ios /= 0) then
print *, ' !! access to 16 not possible !!'
stop
endif

RecBaseTS = (Day-1) * (2 * kmax)

do k = 1, kmax
read (unit = 16, &
rec = RecBaseTS+k, &
iostat = ios) array3D(:,:,k)
T(:,:,k) = transpose(array3D(:,:,k))
enddo

do k = 1, kmax
read (unit = 16, &
rec = RecBaseTS+k+01*kkMax+00, &
iostat = ios) array3D (:,:,k)
S(:,:,k) = transpose(array3D (:,:,k))
enddo

I get the error message ios = -1 after 28 days.
I cannot find the problem. If I chose a recl of M*N*8 (8 Byte per entry) the iof = -1 is returned at day 14. A recl of M*N is not possible and even if would only increase the days by a factor of 4 (120 days and not 365)

I am thankful for every hint an help. Cheers Marc

Hello,

question 1:
are you sure that the following line is correct?
RecBaseTS = (Day-1) * (2 * kmax)
it looks like your calculation of the position is wrong.


question 2:
is it needed that you are openning your file in each 'day' cycle again?


Frank

[marc_hufnagl]

Quoting - tropfen

Hello,

question 1:
are you sure that the following line is correct?
RecBaseTS = (Day-1) * (2 * kmax)
it looks like your calculation of the position is wrong.


question 2:
is it needed that you are openning your file in each 'day' cycle again?


Frank

Hi Frank,
concerning question 1: the position is the right position. At day 1 RecBaseTS is 0 and reading starts at the beginning of the file. At day 2, 2*21 layers are already read and thefore reading starts at position 2*kmax and so on

concerning question 2: it is easier opening the file every day, as I not always need all data. Otherwise I would have to read 4 GB to the memory and slow down the whole calculation.
Cheers Marc.

[Steven_L_Intel1]

It would be helpful if you could determine which data is being read and from what positions in the file. This may require some experimentation with code and examination of the data file.

[marc_hufnagl]

Quoting - Steve Lionel (Intel)

It would be helpful if you could determine which data is being read and from what positions in the file. This may require some experimentation with code and examination of the data file.

It seems to be a compiler problem. Until now I used gfortran as followed:
fortran -o read.exe read.f90 -frecord-marker=4 -fconvert=big-endian -Werror -fbounds-check
what produced the described errors.

I now tested the xlf90 compiler on our UNIX machine as
setenv F_RECLUNIT BYTE
xlf90 read.f90 -C -qextchk -qsigtrap -qinitauto -bnoquiet
and it worked without complications (even without setenv and the extensions).

I now read and write thedata with the xlf compiler transfer them to my computer and read them there with the gfortran code.Thanks for your suggestions and help.

[Steven_L_Intel1]

I see no evidence yet that it is "a compiler problem". Can you attach a sample data file? (Zip it if it is reasonably large). You're getting an end-of-file error.