EOF-function on direct access files

[sander]

Hallo,

for a long time we have used the following code to get the last record of a direct access file:

open (nr_unit,file=pfm_file,form='FORMATTED',access='DIRECT',IOSTAT=rt,recl=192)

irec=1

do while (.not.EOF(nr_unit))

read(nr_unit,'(a10)',REC=irec) cwa

irec=irec+1

enddo

max_rec=irec-1

With Intel Compiler 2017 we get an error severe(36): attempt to access a non existing unit. Is there a Change in the behaviour? What is the best and ensured way to get the last record of such a file?

Thank you!

Thomas

[Steve_Lionel]

Would you please post a short but complete program that demonstrates the problem? My experience is that snippets such as what you posted don't show where the problem really is.  That said, I would not expect EOF to do what you are looking for, as it has meaning for sequential access only. You could instead do this, which works:

do 

read(nr_unit,'(a10)',REC=irec, IOSTAT=rt) cwa
if (rt /= 0) exit
irec=irec+1

enddo

If there are a lot of records you might want to try some sort of binary search technique instead. You can also get a pretty good value on the number of records by obtaining the file size in bytes (GETFILEINFOQQ) and dividing by the record length.

[sander]

Hallo Steve,

thank you very much for the help. Nice to hear from you again after years! Please excuse the bad form, it was my first post here.

One question left to your solution: Is there any danger that the EOF state respectively the result which gives an EOF matchable state is covered by other errors? But I think your solution is everytime reliable because it gives the last readable record which should be in most cases the real last record. Otherwise it gives the usable number of records as an even suitable result.

At other locations we already use GETFILEINFOQQ. At least there it is of course useful to calculate the number of records according to your proposal.  

[andrew_4619]

standard logical function IS_IOSTAT_END
example.

PROGRAM iostat
  IMPLICIT NONE
  INTEGER :: stat, i
  OPEN(88, FILE='test.dat')
  READ(88, *, IOSTAT=stat) i
  IF(IS_IOSTAT_END(stat)) STOP 'END OF FILE'
END PROGRAM

[Steve_Lionel]

EOF is not safe to use for files opened for direct access. With direct access files, as long as you don't try to read a record number higher than was ever written to the file, all record numbers are valid. Yes, it's possible that some other error could occur. If you wanted to protect against that you could check for the implementation-specific error number, FOR$IOS_ATTACCNON from module FORIOSDEF or 36. I would not depend on getting an end-of-file condition for a file opened as direct access.

[sander]

Yes, the EOF condition is not set on direct access files, therefore the test with IS_IOSTAT_END is not working. The error number is 36, like you wrote. I will apply your code and check for FOR$IOS_ATTACCNON. Thank you once more! 

[mecej4]

I am slightly puzzled by your attempt to apply sequential access notions such as end-of-file, last-record, etc., to direct access files. Once a file has been opened for direct access, you are not allowed to do I/O to that file as if it were a sequential access file. This is a restriction at the Fortran language level, regardless of the physical attributes of the file and the I/O device. It is best to think that a direct access file has no beginning or end, as far as your program is concerned. Therefore, if you wish to use direct access, and need to be on the guard against attempting to access a non-existing record number, you should make the maximum record number available to your program in some way.

One way of doing this is to make Record-1 contain the record count of the file, or to make Record-1 contain only the record count of the file.

Another way is to write the file name, description and record count to a small sequential access file at the same time as when you create the large direct access file. Subsequently, to read this direct access file, you can first read the record count from the sequential access file and use that information so as to avoid accessing invalid record numbers in all reads from the direct access file.

[LRaim]

In some applications I create direct access files (i.e. files with records of the same length) as sequential files. When the records have been written the file is closed and re-opened as directs access file.  In such a case I know in advance how many records have been written.
​You may try to count the records by opening the file in sequential mode..

[FortranFan]

(name withheld) wrote:

Yes, the EOF condition is not set on direct access files, therefore the test with IS_IOSTAT_END is not working. The error number is 36, like you wrote. I will apply your code and check for FOR$IOS_ATTACCNON. Thank you once more! 

@Thomas Sander,

Not sure you can quite rely on the legacy stuff like FOR$IOS_ATTACCNON: you may have as much success or better by sticking to standard Fortran and using the INQUIRE statement on the unit opened for direct access and asking for RECL and SIZE results.  On Windows OS, you can expect the line endings to be CR+LF, thus taking up 2 bytes and for the returned size to be the record length times the number of records plus two.  And on Linux with just LF line ending, the size to be the record length times the number of records plus one.

   character(len=*), parameter :: fname = "dac.dat"
   integer :: lrec
   integer :: lun
   integer :: istat
   character(len=256) :: imsg
   integer :: nrec

   print *, "Enter number of file records to use in the test:"
   read (*,*, iostat=istat, iomsg=imsg) nrec
   if ( istat /= 0 ) then
      print *, "Read of number of records failed: istat = ", istat
      print *, imsg
      stop
   end if
   if ( nrec <= 0 ) stop

   print *, "Enter record length to use in the test:"
   read (*,*, iostat=istat, iomsg=imsg) lrec
   if ( istat /= 0 ) then
      print *, "Read of record length failed: istat = ", istat
      print *, imsg
      stop
   end if
   if ( lrec <= 0 ) stop

   blk1: block
      character(len=LREC) :: s
      integer :: i
      open( newunit=lun, access="sequential", file=fname, status="replace", &
         form="formatted", iostat=istat, iomsg=imsg )
      if ( istat /= 0 ) then
         print *, "Open to write sequentially failed: istat = ", istat
         print *, imsg
         stop
      end if

      do i = 1, nrec
         s = repeat( achar(33+mod(i,128)), ncopies=len(s) )
         write( unit=lun, fmt="(a)" ) s
      end do

      close( unit=lun )

   end block blk1

   blk2: block
      integer :: rl
      integer :: fs
      open( newunit=lun, access="direct", recl=LREC, file=fname, status="old", &
         form="formatted", iostat=istat, iomsg=imsg )
      if ( istat /= 0 ) then
         print *, "Open to read direct-access failed: istat = ", istat
         print *, imsg
         stop
      end if
      inquire( unit=lun, recl=rl, size=fs )
      print *, "RECL = ", rl, ", SIZE = ", fs
      print *, "Number of records = ", fs/(rl+2), "; expected = ", nrec
      close ( unit=lun, status="delete" )
   end block blk2

   stop

end

Upon execution,

 Enter number of file records to use in the test:
666
 Enter record length to use in the test:
42
 RECL =           42 , SIZE =        29304
 Number of records =          666 ; expected =          666

[jimdempseyatthecove]

FF,

CRLF vs LF

I suppose one could open the file with 'STREAM' access, read a chunk of data, and scan for the line terminator (also get the line length as verification of record length). Then use INQUIRE to get the file size, estimate the number of records, back off a few/several records, use the POS=nn specifier to position near the end of file (recheck line terminator position), then sequentially read until EOF keeping count of records advanced (verified record count). When you have record count, close file and reopen in desired (direct) access mode.

Jim Dempsey

[FortranFan]

jimdempseyatthecove wrote:

FF,

CRLF vs LF

I suppose one could open the file with 'STREAM' access, ... 

Indeed, Jim, that's another option.  It's up to OP what approach to follow but almost any logical methodology using standard Fortran will be better I think.

Separately, given how fast 'STREAM' access is for file reads and considering the ever-increasing availability of memory on most devices, setting up a simple Fortran 'class' to process even several GB worth of data can prove to be a more user-friendly option than DIRECT access processing.  That is, such a Fortran 'class' can have a data 'load' method which in one fell swoop reads all the file data using STREAM access into an ALLOCATABLE CHARACTER component of of the derived type, it can then split the data into equal chunks corresponding to the record length and take into account whatever line endings (CRLF vs LF) are present in the file, and then parse each chunk as appropriate for the data.

[sander]

Thank you all for the helpful and interesting advices, which gave me stimulation in excess of my question! I will apply two solutions:

• Mixture of sequential and direct access and
• getting the record number by INQUIRE statement and RECL and SIZE.

dependend on the files and the furthermore file handling.

I wish you a good start in a healthy and succesful new year!

[reidar]

Maybe problem solved?

With direct access files you may write to any line number, ok? And I never add an EOF mark (maybe it is put there when closing the file.  When reading the file b ack I use:READIDEV,REC=LINE1,ERR=300) ..List of variables here... In your case you do not tell the program what should happen in case of an error, so why not add an err=? See below..  

read(nr_unit,'(a10)',REC=irec, ERR='A valid label outside the loop  here') cwa

Regards..