F 2003 File Stream Access with pos = n

john3 · ‎07-20-2006

I have a project where I'm solving up to 350,000 complex unknowns on a PC. Even in compressed matrix storage, the data must reside on disk (files sizes up to 30GB) and this data must be read back in for each new LU block factor. On this part of the problem, a simple rewind, read data, write new data rewind and repeat works OK (binary sequential file, direct access will not work due to variable record lengths).

For the solve part of the problem, I have to start, not at the begining of the file, but a various locations down from the file start. I presently do this by closing the fortran file, reopening it as a C file and use the C routine to position the read pointer and to then read the next xxx number of bytes.

Metcalf and Reid in F95/2003 (page 327) say that F 2003 will have the stream access file position capability as : read(unit, pos=position) for files opened with access='stream'.

My question is: When will IVF have this capability as an extension or do we have to wait until F2003 arrives?

jimdempseyatthecove · ‎07-20-2006

I've had good success using Fortran binary I/O where the I/O is performed in "blobs" where all the blobs are the same size and some of the blobs indicate where the other blobs are locate and what is in the blob, how it is compressed, sparse fill filtering, etc..

Everything can be done in Fortran

--------> snip from open

! Use: OPEN (u, FILE='AVRDBnn',RECL=BlobSize,FORM='BINARY',ACCESS='DIRECT', STATUS=sta)
! sta = 'UNKNOWN' for new file on creation
! sta = 'OLD' for old file on post processing

call GETPTH(DBROOT, FileName, DBPATH)
OPEN(IOUnit,FILE=DBPATH,RECL=sizeof(DataBase.Header),FORM='BINARY',ACCESS='DIRECT', ACTION='READ', STATUS='OLD')
READ(IOUnit, REC=1) DataBase.Header
! perform consistency check on header record here
if(DataBase.Header.Name(1) .ne. 'AVRDB - ') call DOSTOP('AVRDB_Name(1)')
if(DataBase.Header.VersionMajor .lt. 0) call DOSTOP('AVRDB_VersionMajor')
if(DataBase.Header.VersionMajor .gt. AVRDB_VersionMajor) call DOSTOP('AVRDB_VersionMajor')
if(DataBase.Header.VersionMinor .lt. 0) call DOSTOP('AVRDB_VersionMinor')
if(DataBase.Header.VersionMinorRevision .lt. 0) call DOSTOP('AVRDB_VersionMinorRevision')
if(mod(DataBase.Header.BlobSize, 512) .ne. 0) call DOSTOP('AVRDB - BlobSize')
! initial consistency check is fine
! close and reopen for reading through blob read functions
CLOSE(IOUnit)
OPEN(IOUnit,FILE=DBPATH,RECL=DataBase.Header.BlobSize,FORM='BINARY',ACCESS='DIRECT', STATUS='OLD')

------------------- Read (next) blob

subroutine
AVRDB_ReadBlob(bc)
USE IFPORT
use MOD_AVRDB
type(TypeAVRDBBlobContext) :: bc
bc.ByteCount = 0
bc.BlobNumber = bc.BlobNumber + 1
read(bc.IOUnit,REC=bc.BlobNumber) bc.Blob(1:bc.BlobSize)
UNLOCK(bc.IOUnit)
end subroutine AVRDB_ReadBlob

--------------------

In my scheme all the blobs within a given database are of uniform size. Blobs from different databases may be of different sizes. The front of the first blob on all databases contains a header that is supposed to be of same size for all databases of a given version and revision of the database software.

Essentially the unknown database is open using header sized records and the header is read. Thensanity checks are performed on the header information. If all is OK then the database (with header sized records) is closed and the information in the header is used tospecify the record sizeas that of the blob size and the file is reopened.

TheAVRDB_ReadBlob sequentially reads through database, however, elsewhere the application can perform direct access by setting the BlobNumber to read as 1 less than the one desired.

My read routine has an unlock following the read because for my use the database is shared as on of my applications writes the database while another of my applications may be reading the same database (e.g. displaying intermediary values as my simulation is running).

In my implementation my header contains information as to the number of blobs (including blob 1) that contain a map and translation database which describes how the database is "compressed".

I can declare in the map file which portions of the data (snapshot of a Finite Element simulation) is written as well as the precision in which it is written (in binary). My simulations run using real(8) variables and I can select which variables to:

a) include/exclude (reads back as 0)
b) demote to real(4)
c) demote to a compressed real(8)
d) record as INT(4)
e) record as INT(2)
f) record as INT(1)
... other forms as well

There are a few flavors of Format c) for compressing the real(8) depending on the content of the data. The compression may be lossy. I simply look at the array to be written, find the MAX and MIN, determine the range, divide the range into 32-bits (or 16-bits, or 8-bits) toproduce a delta, then store into the portion of the blob the real(8) midpoint, real(8) delta, then the array count of INT(4) or INT(2), INT(1) worth of deltas.

You haven't given us much detail on your database but I am sure you could implement a similar technique. From you brief description I would think a blob and sparse matrix scheme would be suitable.

Jim Dempsey

dbruceg · ‎07-20-2006

If you can maintain a vector of row (or column) lengths in core, and you access the file with form='binary', then you can calculate the position of any row (or column) in the file and position the file with FSEEK (an IFPORT function). I use this all the time - it works fine.

Bruce

john3 · ‎07-21-2006

fseek is exactly what is needed. Went back to my bookshelf and did find it in the portability library routines. But those were not the sections I read very close. It was also in CVF and even Microsoft Fortran.

Thanks.