BACKSPACE issue with different Intel Compiler version

Lorenzo_W_ · ‎03-01-2018

Hi all,
I notice a difference between two different version of the same Fortran77 code compiled with 2011 and 2017 Intel Compiler.

The code is quite simple:

it opens an ASCII file and read it until a specific line, containing an integer number, N
the data from the next N lines are stored into an array
now - with the backspace command - we return to the position where data were stored and overwrite data with new values (elaborated somewhere else)

I can't post the full code here, but I prepare a simplified example below.

          open(2,file=filename,status='old')
c
		  read(2,*) N
		  do ii=1,N
		     read(2,*)  X(ii),Y(ii),Z(ii),T(ii)
		  enddo
c
		  do ii=1,N
			backspace(2)
		  enddo
c
		  do ii=1,N
			write(2,*) X(ii),Y(ii),Z(ii),T2(ii)
		  enddo

With a version of the code compiled with Intel 2011 compiler, the output is good, while if I compile with 2017 version, backspace does not work, and data are not overwritten, but stored "under" old data.

Sorry for my English, I hope that someone can help me.

Lorenzo

Steve_Lionel · ‎03-01-2018

I took your snippet and turned it into a complete program - it seems to work correctly in the 2018 (18.0.1) compiler, which is all I have available. What does this do for you? If it works, please modify it so that it shows the problem you describe.

program test
    implicit none
    integer, dimension(10) :: X, Y, Z, T
    integer :: ii,N
    character(80) :: filename = 'file.txt'
    character(100) :: line
    open(2,file=filename,status='replace')
    write (2,*) 5
    do ii=1,5
        write (2,*) ii,ii,ii,ii
    end do
    rewind(2)
          
    do
        read (2,'(A)',end=800) line
        print *, trim(line)
    end do
800    close (2)
    
    open(2,file=filename,status='old')
!
		  read(2,*) N
		  do ii=1,N
		     read(2,*)  X(ii),Y(ii),Z(ii),T(ii)
          enddo
          
          X = X+3
          Y = Y+4
          Z = Z+5
          T = T+6
!
		  do ii=1,N
			backspace(2)
          enddo
!
		  do ii=1,N
			write(2,*) X(ii),Y(ii),Z(ii),T(ii)
          enddo
          
          rewind(2)
          
          do
              read (2,'(A)',end=999) line
              print *, trim(line)
          end do
999       close(2)
          end program test

            5
            1           1           1           1
            2           2           2           2
            3           3           3           3
            4           4           4           4
            5           5           5           5
            5
            4           5           6           7
            5           6           7           8
            6           7           8           9
            7           8           9          10
            8           9          10          11

mecej4 · ‎03-01-2018

Your example code is incomplete. I constructed my own example programs based on your description. The first one filled x, y, z and t with ten random numbers each, and wrote the data to a file, 'xyzt.txt'. The second program does what you described:

program bksp
implicit none
integer :: i,n
real :: x(10),y(10),z(10),t(10)
!
open(2,file='xyzt.txt',status='old')
read(2,'(i4)')n
read(2,'(4ES13.5)')(x(i),y(i),z(i),t(i),i=1,N)
call random_number(x)
call random_number(y)
call random_number(z)
call random_number(t)
do i=1,n
   backspace(2)
end do
write(2,'(4ES13.5)')(x(i),y(i),z(i),t(i),i=1,N)
close(2)
end program

The data file xyzt.txt written by the first program:

  10
  7.81992E-01  2.82940E-01  2.84832E-01  3.35540E-01
  9.91179E-01  2.87988E-01  5.43921E-01  1.28007E-01
  5.01391E-01  1.28984E-01  1.73896E-01  8.19105E-01
  8.05520E-02  3.27284E-01  1.86924E-01  3.60910E-01
  9.18913E-01  5.46430E-02  7.88861E-01  5.54644E-01
  8.43133E-01  7.95269E-01  1.75804E-01  4.21319E-01
  2.63455E-01  9.98151E-01  9.60265E-01  9.16240E-01
  5.10197E-01  7.75644E-01  2.89306E-01  1.43285E-01
  9.59975E-01  9.43003E-01  7.42298E-01  1.05501E-01
  5.62531E-01  3.07533E-01  4.48210E-01  4.03696E-01

After running the second program, the file contained:

  10
  3.92087E-07  3.45043E-01  4.03134E-02  9.94574E-01
  2.54804E-02  8.71184E-01  8.50248E-02  7.78250E-01
  3.52516E-01  8.99184E-02  5.58821E-01  1.96650E-02
  6.66914E-01  8.88284E-01  9.26452E-01  1.69864E-01
  9.63055E-01  7.00979E-01  7.56408E-02  9.94743E-01
  8.38288E-01  7.34553E-01  9.11790E-01  7.53692E-01
  3.35355E-01  3.00176E-01  9.18222E-02  2.17974E-01
  9.15327E-01  4.97177E-02  6.37766E-01  6.59293E-01
  7.95864E-01  9.08189E-01  8.52292E-01  5.39018E-01
  8.32693E-01  9.76586E-02  1.21078E-01  4.80063E-01

I used the Intel 17.0.6 compiler. A different version may generate different random numbers, but in your program the arrays are probably calculated deterministically, so you should have no problem.

Lorenzo_W_ · ‎03-01-2018

Steve, mecej4,
I want to thank you for your effert for reproducing my example.

I compile your programs both with a Windows compiler (version 17.0.0.109 Build 2016.07.21) and a Linux compiler (version 11.1 Build 2009.11.30), compiling directly from command line.
Results are pretty strange, and I'm going to summarize them here below.

Steve's program produce a different output between the two versions, and the Windows version 17 produce the (wrong) following output:

While in Linux - Intel Fortran version 11.0 - the output is the same reported by Steve.

The strange thing is that these differences can not be seen in the program written by mecej4, because the output here is the same between the two version.

I sincerely don't understand what's going on and I don't know why compiling Steve example, my compiler produce different results respect to the one reported by him.

Thank you again for helping me

Steve_Lionel · ‎03-01-2018

Can you try a more recent version? The one you have is a year and a half old.

Lorenzo_W_ · ‎03-02-2018

I perform several tests, thanks to the help of my colleagues.

Here are the results (version - outcome):

12.04.196 - OK
17.00.109 - WRONG
17.01.143 - WRONG
17.02.187 - WRONG
18.00.124 - OK

And I also have your OK result with v18.01. It's not a problem for me to use version 18, but I don't see why your code is different from "macej4" code, since both of you use the syntax

do i=1,N
   backspace(2)
enddo

What's going on?
Thank you again

mecej4 · ‎03-02-2018

Do the results in #6 relate to Steve's program in #2?

Note that the Fortran standard forbids backspacing in one context:

Backspacing over records written using list-directed or namelist formatting is prohibited.

See section 9.5.1 of the Fortran 95 standard, or section 9.8.2 para 4 of the Fortran 2008 standard. When list-directed WRITEs are used, the number of lines that are output is implementation-dependent, and may change even with different versions/releases of the same compiler.

What happens when a program in violation of this is compiled and run is probably unspecified. The codes in #1 and #2 do use list-directed output, but that should not be a problem since those example codes write no more than four integers or reals per record, which should fit comfortably within a single line.

If you take Steve's program and provide a format instead of '*' in the I/O statements, does that change take care of the problem with the 17.0.x compiler versions?

In your original program (not the example code fragment that you posted) were the WRITE statements list-directed?

jimdempseyatthecove · ‎03-02-2018

Additional note on mecej's comment:

If using list-directed write (* formatting), line break position as well as numeric formatting is implementation dependent, Therefore, if you perform BACKSPACE(n), and the subsequently WRITE fewer records than the record count to the end of file, then you may corrupt the remaining (numbered) records (or potentially overstrike fewer records).

Jim Dempsey

Lorenzo_W_ · ‎03-02-2018

The problem is surely linked to list-directed I/O.

Modifying Steve's program adding or removing "*" in read and or write statement, I was able to produce different results.
So, I have to be careful in the future about this.

My question is: what if I don't know how the file is being written? (By user-input or by another unknown program?) I think that in this scenario the use of backspace should be carefully considered.

Thank you all for the support,

Lorenzo Winchler

mecej4 · ‎03-02-2018

Lorenzo W. wrote:
My question is: what if I don't know how the file is being written? (By user-input or by another unknown program?) I think that in this scenario the use of backspace should be carefully considered.

In general, if you do not know how a file was written, it may be difficult or impossible to read it. There needs to be some coordination between the process/program/person that creates the file and the process that reads the same file. The simple answer is : in such circumstances, do not write the file using list-directed output.

The rules in the Fortran standard have reasons behind them, and the standard rarely describes those reasons. Sometimes you may write code that is in violation and still works the way you wanted, but you cannot count on the persistence of such good fortune.

Steve_Lionel · ‎03-02-2018

I had missed the significance of the number of records, and this also explains (in part) the version difference. Version 18 is the first (I think) to make /assume:noold_ldout_format the default. This changes the formatting of integers from a width based on the kind to I0, resulting in a shorter record. It also changes the width of real values to be the minimum needed. If each list-directed WRITE statement spreads the values over two or more records, that will affect how many BACKSPACEs you need.

I agree that you can't depend on where records break in list-directed output.