Strange run-time errors while reading an UNFORMATTED file, IOSTAT=39 && IOSTAT=67

[peng__hanbing]

Dear friends from the ifort community,

recently I have been confused by 2 run-time errors while reading an UNFORMATTED file which I opened first with following statement

open(iUnit,FILE='lsq_tmp', ACCESS='SEQUENTIAL', ACTION='READWRITE', CARRIAGECONTROL='NONE', FORM='UNFORMATTED', ORGANIZATION='SEQUENTIAL', RECL=40000000, STATUS='UNKNOWN')

After opening this file, I write  records into it one by one. Those records are in different length, ranging from 2-character string to several thousands of real*8 float numbers. After all records have been written into the file, I begin to read those records out one by one from the end to the beginning with the assistance of  BACKSPACE statement.

During this reading procedure, sometimes (not always) 2 types of run-time error might happen: IOSTAT=39 && IOSTAT=67. I can't get enough information from the description about these run-time errors on  List of Run-Time Error Messages webpage. As it said there, IOSTAT=39 means "The Intel® Fortran RTL I/O system detected an error condition during execution of a READ statemen" But about what? Any possible to be more specific? And IOSTAT=67 means "input statement requires too much data" . This seems more clear. But for debugging, I also write the same content  into an ASCII file with the following open statement 

open(iUnit1,FILE='lsq_tmp1',FORM='FORMATTED',STATUS='UNKNOWN')

Thus I can check what it looks like exactly at the place where those run-time error happen. But everything seems normal. There does exist enough data in the record as the reading format requires.  Those two errors have been torturing me for a while.

Does anyone happen to meet similar problem as me? Any advice about it would be highly appreciated.   

[mecej4]

Please provide a small test program that we can compile and run to reproduce the problems that you described. In such situations, "the devil is in the details".

Why do you bother with the CARRIAGECONTROL option for an unformatted file? Is there a good reason to read the file backwards?

Perhaps, you have not realized that, when reading a file backwards, you need two BACKSPACE statements after each READ? This is because the first BACKSPACE takes you back to the beginning of the record just read, and you need another BACKSPACE to move the position to the beginning of the previous record?

program hanbing
implicit none
integer iunit,i,ij(5)
real x(10)
character(4) str
!
iunit=11
open(iUnit,FILE='lsq_tmp', ACCESS='SEQUENTIAL', ACTION='READWRITE', FORM='UNFORMATTED', ORGANIZATION='SEQUENTIAL', RECL=40000000, STATUS='UNKNOWN')
do i=1,5
   ij(i) = 6-i
end do
call random_number(x)
str = 'xyz '

write(iUnit)str
write(iUnit)ij
write(iUnit)x

x = 0
ij = 0
str = ''

backspace(iUnit)
read(iUnit)x
backspace(iUnit)
backspace(iUnit)
read(iUnit)ij
backspace(iUnit)
backspace(iUnit)
read(iUnit)str
close(iUnit)
print *,str
print *,ij
print *,x
end program

[peng__hanbing]

Hi mecej4,

thanks for your reply.

1) For this moment, it is not easy to provide a small test program specifically for this problem. Because this problem does not occur every time while another different data set is processed, it is hard to create the same environment that would definitely cause this problem. 

2) I don't understand what's wrong with the CARRIAGECONTROL option. I set this option while opening the file just because I want to make all related options for this file as explicit as possible. According to the other options in my case, I think the default value of  CARRIAGECONTROL option would be NONE if not set. So I just set it to NONE for clearness.

3) Yes,  there is a good reason to read the file backwards. Because otherwise the data flow of the program should be re-designed, which might take a lot of time.

4) I did realize that the BACKSPACE statement should be executed twice for stepping backward one record.

And for your notice, I also write and read the same content via an ASCII file in the same way. But no similar errors occur for the ASCII file. As via ASCII file there might cause accuracy loss for float-point numbers and performance degradation for speed, I don't think I should use that instead of binary file. Is that right?

[jimdempseyatthecove]

I suggest that you verify that the data within the file is what you expect it to be (verify number of records and record lengths) to assure that you do not have:

...| text | blob | text | text | |blob | blob | ...

when you require alternating ... | text | blob | ...

STATUS='UNKNOWN' implies you are writing and reading the file.

Check your code for a missing FLUSH(YourUnitNumber) that must either:

a) Follow a WRITE than can potentially be followed by a BACKSPACE, or,
b) Precede a BACKSPACE that can potentially follow a WRITE

Could you provide the version number of your Fortran and the version (and vendor) of the program that wrote the original data file should the file have been imported from a different system/program.

Jim Dempsey

[jimdempseyatthecove]

Also, see that you do not have the file open on multiple units, in particular writing on one and reading on another.

If all else fails and you need to move on (while bug fixed)

re>>But for debugging, I also write the same content  into an ASCII file with the following open statement... via ASCII file there might cause accuracy loss for float-point numbers

Consider writing/reading the REAL(8) as text in Z16 format (comma/space separated). While this won't be as fast as binary, it will satisfy no loss in precision.

The fact that the ASCII test did not fail, does not mean the ASCII formation will always succeed.

With some versions of IVF there were some issues with BUFFERED sequential I/O and BACKSPACE. You could easily test this by using BUFFERED='NO' on your OPEN.

Jim Dempsey

[peng__hanbing]

jimdempseyatthecove wrote:

I suggest that you verify that the data within the file is what you expect it to be (verify number of records and record lengths) to assure that you do not have:

...| text | blob | text | text | |blob | blob | ...

when you require alternating ... | text | blob | ...

STATUS='UNKNOWN' implies you are writing and reading the file.

Check your code for a missing FLUSH(YourUnitNumber) that must either:

a) Follow a WRITE than can potentially be followed by a BACKSPACE, or,
b) Precede a BACKSPACE that can potentially follow a WRITE

Could you provide the version number of your Fortran and the version (and vendor) of the program that wrote the original data file should the file have been imported from a different system/program.

Jim Dempsey

Hi Jim,

thanks for your reply. Yes, I have checked the file content via an ASCII file. But I didn't find anything abnormal. I use ifort (IFORT) 12.1.3 20120212 on Linux. As you will see in my sample code, the write and read part are in the same program.

Here is the sample code, which has the same structure and processing procedure as the actual program.

program sample
    
implicit none

integer*4     iUnit,iUnit1
character*2   cid
real*8        r1,...,rn
real*8        s1,...,sn

!Open or creat the file
iUnit=11
open(iUnit,FILE='lsq_tmp',ACCESS='SEQUENTIAL',ACTION='READWRITE',CARRIAGECONTROL='NONE',&
     FORM='UNFORMATTED',ORGANIZATION='SEQUENTIAL',RECL=40000000,STATUS='UNKNOWN')
!This record is for tagging the beginning of the file, which would also tag the 
!end of the file while reading backwards
write(iUnit) '00'
!for debug, also open an ASCII file to do the same thing
iUnit1=12
open(iUnit1,FILE='lsq_tmp1',FORM='FORMATTED',STATUS='UNKNOWN')
write(iUnit1,'(A2)') '00'
!for debug

do while(...)
    ...
    ...
    ...
    !Write an 'ob' record when test1 successes
    if(test1) then
        !The real data record is not really like r1,...,rn (just scalar float-point numbers)
        !but mixed with scalar intergers, integer arrays, scalar float-point, float-point arrys,...
        write(iUnit) 'ob',r1,r2,...,rn
        !for debug
101     format(A2,...)        
        write(iUnit1,101) 'ob',r1,r2,...,rn
        !for debug
    endif
    ...
    ...
    ...
    !Write a 'pa' record when test2 successes
    if(test2) then
        !Same as the "ob" record above, s1,...,s2 is just for example.
        write(iUnit) 'pa',s1,s2,...,sn
        !for debug
102     format(A2,...) 
        write(iUnit1,102) 'pa',s1,s2,...,sn
        !for debug
    endif
    ...
    ...
    ...
enddo
!Reading backwards
do
    !Back to the previous record
    !for debug
    backspace(iUnit1,IOSTAT=iErr,IOMSG=String)
    !for debug
    backspace(iUnit,IOSTAT=iErr,IOMSG=String)
    !Read the record TAG to judge which type it is
    !for debug
    read(iUnit1,'(A2)',IOSTAT=iErr,IOMSG=String) cid
    !for debug
    read(iUnit,IOSTAT=iErr,IOMSG=String) cid
    
    !If it is "00", exit the loop (which means the beginning of the file) 
    if(cid.eq.'00') exit
    
    !Otherwise, go back to the last record again and really to read it
    !for debug
    backspace(iUnit1,IOSTAT=iErr,IOMSG=String)
    !for debug
    backspace(iUnit,IOSTAT=iErr,IOMSG=String)
    
    if(cid.eq.'ob') then
        !for debug
        read(iUnit1,101,IOSTAT=iErr,IOMSG=String) cid,r1,r2,...,rn
        !for debug
        !Sometimes, the run-time error occurs in the next line, i.e. the read statement
        read(iUnit,IOSTAT=iErr,IOMSG=String) cid,r1,r2,...,rn
    else if(cid.eq.'pa') then
        !for debug
        read(iUnit1,102,IOSTAT=iErr,IOMSG=String) cid,s1,s2,...,sn
        !for debug
        read(iUnit,IOSTAT=iErr,IOMSG=String) cid,s1,s2,...,sn
    else
        write(*,'(a)') '***ERROR: When read file'
        call exit(1)
    endif
    !Go back to the position of the record that just was read
    !for debug
    backspace(iUnit1,IOSTAT=iErr,IOMSG=String)
    !for debug
    backspace(iUnit,IOSTAT=iErr,IOMSG=String)
enddo
!for debug
close(iUnit1,status='delete',iostat=iErr)
!for debug
close(iUnit,status='delete',iostat=iErr)

end program sample

[mecej4]

The "sample code" cannot be compiled and run. Since the issue is whether the compiler is putting out correct code, sorry, the sample code is of little use.

[jimdempseyatthecove]

>>Yes, I have checked the file content via an ASCII file.

You mentioned in your first post that the file that produces the error is written as (I assume)

... | some sort of header, possibly ASCII + binary blob of REAL(8) | ...

You must verify the above file and not a different file that you believe is a ASCII-only equivalent file. You will need to use a hex editor for this. This will show you not only your data, but also the pre-pended Leading Length Field and appended Trailing Length Field.

From: https://hpc.icc.ru/documentation/intel/f_ug1/format_of_record_types_.htm

Variable-length records are prefixed and suffixed by 4 bytes of control information containing length fields. The trailing length field allows a BACKSPACE request to skip back over records efficiently. The 4-byte integer value stored in each length field indicates the number of data bytes (excluding overhead bytes) in that particular variable-length record.

The character count field of a variable-length record is available when you read the record by issuing a READ statement with a Q format descriptor. You can then use the count field information to determine how many bytes should be in an I/O list.

One other thing comes to mind. Some of the system calls where an error status can be returned (IOSTAT=iErr), that the error code is written only on error. IOW not overwritten when no error occurs. Might iErr have been an earlier (known and handled) error code? If so, simply initialize iErr to 0 prior to your read.

Jim Dempsey

[peng__hanbing]

mecej4 wrote:

The "sample code" cannot be compiled and run. Since the issue is whether the compiler is putting out correct code, sorry, the sample code is of little use.

Yes, I know. Because the original program is a large system, it depends on a lot other subroutines, functions and also data sets. For this moment it is not easy to provide it here. But this sample code descripts exactly the whole writing forwards and reading backwards procedure in it. Anyway, thanks a lot for your continual reply. 

[peng__hanbing]

jimdempseyatthecove wrote:

>>Yes, I have checked the file content via an ASCII file.

You mentioned in your first post that the file that produces the error is written as (I assume)

... | some sort of header, possibly ASCII + binary blob of REAL(8) | ...

You must verify the above file and not a different file that you believe is a ASCII-only equivalent file. You will need to use a hex editor for this. This will show you not only your data, but also the pre-pended Leading Length Field and appended Trailing Length Field.

From: https://hpc.icc.ru/documentation/intel/f_ug1/format_of_record_types_.htm

Variable-length records are prefixed and suffixed by 4 bytes of control information containing length fields. The trailing length field allows a BACKSPACE request to skip back over records efficiently. The 4-byte integer value stored in each length field indicates the number of data bytes (excluding overhead bytes) in that particular variable-length record.

The character count field of a variable-length record is available when you read the record by issuing a READ statement with a Q format descriptor. You can then use the count field information to determine how many bytes should be in an I/O list.

One other thing comes to mind. Some of the system calls where an error status can be returned (IOSTAT=iErr), that the error code is written only on error. IOW not overwritten when no error occurs. Might iErr have been an earlier (known and handled) error code? If so, simply initialize iErr to 0 prior to your read.

Jim Dempsey

Thanks for your sharing of the reference. As you may find from the sample code, the writing and reading statements are in the same program. It is not so difficult to check that whether there is a miss-match between writing and reading.

Besides, I found another strange  issue. If a file is opened as the statement I wrote, the RECORDTYPE may depend on the compile option `-fpscomp ioformat`. If complied with `-fpscomp ioformat`, the RECORDTYPE would be 'STREAM' (returned by a following INQUIRE statement); Otherwise (i.e. compiled without `-fpscomp ioformat`), the RECORDTYPE would be 'VARIABLE'. I am not sure whether this issue is related.

[jimdempseyatthecove]

>>It is not so difficult to check that whether there is a miss-match between writing and reading.

It is not that the sequence of operations, nor the argument lists vary, but rather the contents of the on-disk data differ. IOW if you performed a binary difference on the files you would have a glaring difference between the file written to iUnit and iUnit1.

Your problem is receiving an I/O error after READ, BACKSPACE, BACKSPACE, READ.

One of the causes for this is corrupted data in the file format on the disk (media). Data corruption can occur not only amongst the variables you read (which may go unnoticed), but also in the  pre-pended Leading Length Field and appended Trailing Length Field. Should the pre-pended Leading Length Field be OK .AND. should the Trailing Length Field be corrupt, in say one record, then you will be able to correctly read this file forward, but you will not be able to backspace correctly over or from within this record. Note,  backspace correctly is not the same as backspace without receiving an I/O error. It means repositioning the file correctly such that the subsequent READ gets the expected data. Be mindful that you can get incorrect data on that READ without receiving an I/O error. These records do not have a checksum nor MD5 hash..

Did you try disabling buffered I/O on your OPEN statement?

Jim Dempsey