Reading large binary files

joao_n_ · ‎08-08-2016

Dear all,

I have a raw file (just a plain sequence of 0's and 1's) that a need to assign to a 3D matrix. The number of elements is very large (~2000^3) and I'm running into some issues when using ifort.

I've tried two ways:

Option 1-

sizeofdouble = 1
sizerecl = imax*jmax*kmax*sizeofdouble

OPEN( UNIT=1113, FILE=mesh,FORM='unformatted',access='direct',RECL=sizerecl)

READ (1113,REC=1) f0
CLOSE (1113)

gives me: forrtl: severe (118): The 'RECL=' value in an OPEN statement for unit 1113, (....) exceeds the maximum allowed for the file's record type.

Option 2-

sizeofdouble = 1

OPEN( UNIT=1113, FILE=mesh,FORM='unformatted',access='direct',RECL=sizeofdouble)
       DO k = 1,kmax
           DO j = 1,jmax
                   DO i = 1,imax
                               READ (1113,REC=i+(j-1)*imax+(k-1)*imax*jmax) f0(i, j, k)
                    END DO
                END DO
       END DO
CLOSE (1113)

gives me: forrtl: severe (25): record number outside range, unit 1113

imax, jmax and kmax are the number of elements of the matrix f0, which I'm populating from the file mesh, consisted by 0's and 1's.

I understand that RECL has an upper limit in fortran and that's why option 1 does not work. Something similar appears to be case with option 2 as well.

So, my point is: How to circumvent this ?

btw, I'm already compiling with -assume byterecl and the code works when the number of elements is small enough. I've also tried using a formatted input and still got the same errors.

Many thanks !

Steven_L_Intel1 · ‎08-08-2016

Use ACCESS='STREAM' and unformatted READs. You may also want to play with BUFFERED and BLOCKSIZE to adjust performance.

joao_n_ · ‎08-09-2016

It works !

Thanks !