Read Error for binary file with implied/explicit do-loop

[hdankowski]

Hi,

I get a strange behaviour at reading a binary file. A simple test case is attached.

If the data from the file is read with an implied-do-loop (line 37) AND the code is compiled in debug (-g), the read is successful and no error is returned.

On the other hand, if the same code is compiled in opt-mode OR if the loop is written explicitly like in line 39-42, an error is returned that the end-of-file is reached.

It should be mentioned that this piece of code worked for over 20 years without any problems and several other compilers including older versions of the ifort.

Could anyone explain the behavior to me and give me a solution how to use the code also in the opt-version?

Best Regards
Hendrik

Compiler: ifort (IFORT) 15.0.2 20150121, IA-32, OpenSUSE

[mecej4]

The file "test.mos" is a big-endian Fortran unformatted file with a single record. The record markers at the beginning and end of the file contain 00 00 25 3C, which gives the record length in bytes as 9532. This length plus the 8 bytes for the two record markers makes the file length equal to 9540. To match the record length, strictly speaking, you would need LBUF = 9532/4 = 2383, not 20. I find that allocating BUF with this size allows the program to run without error, so it is conceivable that when the READ attempts to read the full record into the under-allocated array an access violation can result.

Fortran unformatted sequential files have several restrictions on how they may be read. In particular, you cannot read a record with a length different from that used to write the file. An attempt to do so will cause data bytes in the record to be read as record lengths, or vice versa, causing unpredictable errors. This is why the code in your lines 39 to 42 will never work. Likewise, the buffer into which you attempt to read a full record has to be large enough.

The test program has another error: if EOF or ERR occurs, control passes to statement 100, and an attempt is made to print SIZE before it has a value assigned to it. Finally, I don't understand why you specified POSITION='ASIS', unless you had opened the file previously and positioned it at the beginning.

Apart from the issues that I just noted, with Intel Fortran 15.0.3 and 15.0.2 you get not only the expected I/O errors but also other errors. Code compiled with 15.0.2 on Windows 8.1 aborts with an illegal access on the line with call cs_ldms(), and code compiled with 15.0.3 results in "forrtl: severe (8): internal consistency check failure, file for_exit_handler.c, line 418". Earlier versions and other compilers may appear to work without error, but the general rule is that with "nonconforming code" the run-time behavior is "undefined".

[hdankowski]

Thank you very much for the detailed explanation.

However, the code in the test case I pusblished here is only an extract from a larger contents. This is the reason for the smaller side errors you mentioned.

The purpose of the part of source code I've shown here, is actually to determine the size of the data set i.e. the record length. The variable SIZE is later used to allocate an array of that size. This is illustrated in the following lines of code:

      SIZE = BUF(5) - 1
      IF (p_cs%ms(1) .LT. SIZE) THEN
!------------------------------------------------ TOO SMALL
        CALL CS_RESIZE(P_CS,SIZE,ERROR)
        IF (ERROR.NE.0) THEN
          ERROR=1
          RETURN
        ENDIF
      ENDIF

!------------------------------------------------ READ REST
      OLDSIZE = p_cs%ms(1)
      rewind(chan, err=220)
      read (chan, iostat=error, end=120, err=120) (p_cs%ms(i), i=1, size)

Is there any other possiblity to read only the header of an unformatted file? Or do you have any other suggestion to have a conforming source code?

One comment: It is not very convenient that a source code which worked over 20 years without any problems and still works with other compilers like gfortran and older versions of ifort results now in these problems. And I think it is quite usual that a lot of especially older Fortran code is "nonconforming" but is still supported by many compilers.

This is a very crucial part of a larger project (database access), so I must be sure that any corrections made to this part are only done with great care and only if it is absolutely necessary.

It is also very interesting that the Windows version throws some errors, because on Linux even with -warn all, no warnings or erros are issued.

One last remark: If I execute the program in Linux, it stops during the read statement and doesn't even reach the other write statements.

[mecej4]

I understand your position, and the issue can be construed as continuation of vendor support for non-standard code. I note that on Windows the IFort 11.1.070, Gfortran 4.8, CVF 6.6C and Lahey compilers allow your program to run as intended   Perhaps one of the Intel personnel will respond, but I remember seeing a note from Dr. Fortran that he will be on leave for a couple of weeks.

Since you asked about a standard-conforming way, here is a contribution to doing that. To keep the code short, I have used the convert = 'BIG_ENDIAN' feature, for without it the four bytes of each integer read would need to be reassembled in reverse order to obtain the little-endian version before interpreting the value. Having read the array sizes this way, you can allocate the arrays, then reopen the file with FORM='UNFORMATTED', convert='BIG_ENDIAN' (and rewind). This method is probably feasible only if you need to read a handful of unformatted big-endian records to ascertain allocation requirements.

Note that the ascertaining of the record size can be further simplified if you can assume that the record length stored in the file will always equal the size information in the fifth 32-bit word of the record, i.e., rsiz/4 = dsiz-1.

program hdrsiz
!
! Read big-endian unformatted file, unknown record size, have to read bytes
! at beginning of file to ascertain buffer size needed for reading file.
! First record contains buffer length+1 in 4th 32-byte integer
!
implicit none
integer i,rsiz,dsiz,skip
open(unit=11,file='test.mos',access='stream',convert = 'BIG_ENDIAN')
rewind(11)
read(11)rsiz,(skip,i=1,4),dsiz
close(11)
write(*,*)rsiz/4,dsiz-1
end program

The output from running this on your test.mos file:

        2383        2383

[pbkenned1]

mecej4, thanks for the nice standard-conforming example and detailed explanations. 

hdankowski, your original program runs correctly up through the 14.0 compiler.  On Linux with 15.0, indeed the program never gets past:

 open                   21           0

With the changes suggested my mecej4, the program runs correctly with ifort 15.0.x

INTEGER, PARAMETER                   :: LBUF=2383 ! suggested by mecej4

INTEGER :: I
INTEGER :: BUF(LBUF)
 

I don't know what changed in 15.0, but it's not likely the developers will restore the old behavior.

A possible workaround in 15.0 is to compile for runtime bounds checking:

$ ifort -V
Intel(R) Fortran Intel(R) 64 Compiler XE for applications running on Intel(R) 64, Version 15.0.3.187 Build 20150407
Copyright (C) 1985-2015 Intel Corporation.  All rights reserved.

$ ifort main.f90 -check bounds -o main.exe; ./main.exe
 open                   21           0
 regular end            21           0        2383
 size                 2383
[$

Patrick

[mecej4]

Hendrik, from what Patrick Kennedy wrote it appears that you may need to make two passes through the file. In the first pass, you open with access='stream', and read those numbers in the file that give the array sizes. Needless to say, you need to know the byte offsets in the file where those numbers are stored. Then, you can allocate arrays with that size information, following which you reopen the file as an unformatted file and read the records into those arrays.

If you are able to make slight changes to the mainframe program that produces the big-endian file, it may be possible for you to write all the array size numbers into the first record of the file, which may be regarded as your own file header. Reading this record does not require you to know the array sizes beforehand in your data consumer program, nor would you need to use access='stream' or make two passes over the file. You then allocate the arrays, and read the remaining records as usual.

[hdankowski]

Thank you all for your helpful answers.

I will discuss the issue with the other developers.

But I think the most feasible solution would be to first open the file with stream access and close it again before reading the rest. The -check bounds flag will probably decrease the performance.

Another possibility I'm thinking of is to use the 'size' option from the inquire statement to determine the file size.

[pbkenned1]

On second thought the prudent course is to consider this a regression and report the issue to the developers (internal tracking ID DPD200371245).  If it's not a true regression WRT 14.0, we should at least get a good explanation of what changed in 15.0 that breaks the original test case.

Patrick

[pbkenned1 · Accepted solution]

The Fortran runtime team lead apologizes for this bug.  It's fixed in the current 16.0b compiler, and is also planned to be fixed in the next 15.0 update.  I'll check with the latter once released and disposition this thread accordingly.

Patrick

[U558474]$ ifort -V
Intel(R) Fortran Intel(R) 64 Compiler for applications running on Intel(R) 64, Version 16.0.0.069 Beta Build 20150527
Copyright (C) 1985-2015 Intel Corporation.  All rights reserved.

[U558474]$ ifort main.f90 -o main-if-16.0.0.069.exe
[U558474]$ ./main-if-16.0.0.069.exe
 open                   21           0
 regular end            21           0        2383
 size                 2383
[U558474]$

[hdankowski]

Dear Patrick

these are good news! Please let me know as soon as the update for the 15.0 compiler is ready.

Best Regards

Hendrik

[hdankowski]

Dear All,

the new update from today for the Intel Fortran Compiler in Version 16 finally solved the problem.

The code now compiles and runs without any problems in both debug and opt mode.

Thanks and Kind Regards

Hendrik

[pbkenned1]

Hello Hendrik,

Thanks for confirming the fix!

Regards,

Patrick