Strange errors with stream I/O

[Arjen_Markus]

Hello,

I was experimenting with different ways to read a CSV file and found that this program terminates on
a rather unconspicuous location in the file:

! checkstream.f90 --

! Read a CSV file, using stream access

!

program checkstream

implicit none

integer :: i, k, kn, count

integer :: ierr

integer :: start, stop, clock_rate

real :: x, y, z, sum

character(len=20) :: xstr, ystr, zstr, string

!

! Read the file as a stream

! NO success with Intel Fortran 11.0

!

do i = 1,2

open( 10, file = 'input.csv', access = 'stream', form = 'formatted' )

count = 0

call system_clock( start, clock_rate )

do

read( 10, *, iostat = ierr ) x, y, z, string

! read( 10, * ) x, y, z, string

! write( *, * ) count, x, y, z, string

if ( ierr /=0 ) then

exit

endif

count = count + 1

sum = sum + x + y + z

enddo

call system_clock( stop )

close( 10 )

enddo

write(*,*) 'Method 4 (formatted stream):', (stop-start)/real(clock_rate), count

end program checkstream


I have generated the file with this program:

! createfile.f90 --

! Program to generate a large CSV file

!

program createfile

implicit none

real :: x, y, z

character(len=20) :: string

integer :: i

integer :: k

open( 10, file = 'input.csv' )

string = 'abcdefghijklmnopqrst'

do i = 1,1000000

call random( x )

y = 1.0 - x

z = 1000.0 * x

k = min(20, int(1 + 20.0 * x) )

write( 10, '(f0.3,a,f0.5,a,f0.2,a,a)' ) x, ',', y, ',', z, ',', string(1:k)

enddo

end program createfile

and using Intel Fortran 11.0 (on either Windows XP or Linux) the first program simply stops
after several thousand lines.

Here is a part of the output:

2384 0.5490000 0.4509600 549.0400 abcdefghijk

2385 0.4890000 0.5111500 488.8500 abcdefghij

2386 0.1370000 0.8629200 137.0800 abc

2387 0.9820000 1.8090000E-02 981.9100 abcdefghijklmnopqrst

2388 0.6190000 0.3813700 618.6300 abcdefghijklm

2389 0.7670000 0.2331600 766.8400 abcdefghijklmnop

forrtl: severe (59): list-directed I/O syntax error, unit 10, file .../input.csv

Image PC Routine Line Source

checkstream 080996FD Unknown Unknown Unknown

checkstream 08098C75 Unknown Unknown Unknown

checkstream 0807ACF8 Unknown Unknown Unknown

checkstream 0806CF03 Unknown Unknown Unknown

checkstream 0806C860 Unknown Unknown Unknown

checkstream 08055869 Unknown Unknown Unknown

checkstream 08049F5C Unknown Unknown Unknown

checkstream 08049E01 Unknown Unknown Unknown

libc.so.6 00144E8C Unknown Unknown Unknown

checkstream 08049CF1 Unknown Unknown Unknown

The line it is apparently trying to read looks like:

0.250,0.74994,250.06,abcdef

I can not see anything wrong with that line or the program.
Can anyone shed some light on this?

Regards,

Arjen

[Flavio_Almeida]

I think you need to include sum=0 initialization on checkstream, after count=0.

I do not know if this will work, but sometimes reported runtime errors are side effects.

Question: Why did you use do i=1,2 on checkstream?

[Arjen_Markus]

Hi Flavio,

do i = 1,2 is a left-over from a larger program.

I read the file twice to exclude caching effects in the timings.

I have some further information - reading the line into a string
and then reading the data from that string enabled me to print
what exactly is being read from the file.

It turns out that the line read from the file is incomplete:
994,250.06,abcdef

instead of:
0.250,0.74994,250.06,abcdef

(the first 10 characters aremissing)

That explains why the program stops with this error, but the new puzzle is: why
does it read only a part of the line?

Regards,
Arjen

[Flavio_Almeida]

After posting my comment, I began to duplicate your problem. (curiosity)

Apparently the problem was not with sum=0 (neither with the stop variable - a keyword that I changed to sstop).

I do not know why but my program stopped on count=977. As I need to leave my computer now, I will try to see this later.

The input.csv is big (33,385,387 bytes) and I am thinking about what text editor to open it.

Did you find out why the csv generator produced an incomplete line?

Regards

Flavio

[Flavio_Almeida]

Excuse me (you said checkstream.f90 wrongly read the line and not that the line is incomplete)

-I am in a hurry.

[Arjen_Markus]

Well, it is not the program that generates the file! It is the reading program that suddenly
decides to skip some characters. Really puzzling!

Regards,
Arjen

[Steven_L_Intel1]

It appears to work fine in 11.1.060 (Update 5).

[Flavio_Almeida]

Can we conclude this is a bug of Intel Fortran (previous versions) ?

Will we have a patch ?

In my opinion software vendors should do as auto makers - free recalls for bugged components mainly for standard parts (i.e. I am not talking about new features) and notice thatIntel Fortran for Windows with IMSL (my version) is expensive enough to justify such a policy.

If that was the law, many people would not hate Microsoft (but they would have to write solid code,always).

Regards

Flavio

[Steven_L_Intel1]

When you buy Intel Fortran, you get free support for a year, including maintenance updates and new versions. All software has bugs, and vendors do fix them. Support policies vary - we're more liberal than some I know.

[Steven_L_Intel1]

The program here works in 11.0.074.

[Arjen_Markus]

Thanks for clarifying that.

Under Linux I have version 11.0-1569, as reported by the -what option (though
from the installation directory structure I would have thought a version 11.0.081 would have been more appropriate).

Under Windows I have 11.0.072, so I guess that is slightly older than 11.0.074.

Perhaps it is time to ask my IT department to upgrade.

Regards,

Arjen

[Ron_Green]

I am able to reproduce the error with multiple versions of the compiler. HOWEVER, I think the error is in the ACCESS='STREAM' - remove that specifier and checkstream.f90 runs fine.

I am not sure that ACCESS='STREAM' and FORM='FORMATTED' are compatible. Stream, to me, implies a raw data output to mimic the binary mode in the C language. In this mode, there is no file formatting ( CR-LF for example, as you have in this common separated file). I can see where the runtime is getting confused and reading CR LF at the line ends.

I'll open a bug report since there may be some bug with STREAM. But to me, I would disallow the open with ACCESS=STREAM and FORM=FORMATTED - these 2 are IMHO incompatible.

ron

[Steven_L_Intel1]

Yes, ACCESS='STREAM' and FORM='FORMATTED' are compatible. You just read or write the bytes without record delimiters.

[Ron_Green]

This bug has been fixed in our 12.0 aka "Composer XE 2011" compilers. It will not be fixed in 11.1.

ron