derived-type IO with larger arrays (error)

Ferdinand_T_ · ‎11-10-2014

Dear All,

with the unformatted IO of (larger) arrays throgh user-defined derived-type transfer stamenets, I get the following error (upon reading the array back):

forrtl: severe (67): input statement requires too much data

The error appears at different points in a larger transfer (including nested child transfer statements), depending on the sizes of the data. I managed to reproduce the error with the followign code on ifort 14.0.3:

[fortran]

module m

    implicit none
     ! global IO data
     ! critical size: 4 * 2**15 bytes (131.07200 kB)

integer, parameter :: ARR_SIZE = 2**15 ! = 32768; no crash with 2**15 - 1

integer, dimension(ARR_SIZE) :: arr ! crash with/without dynamic allocation
! writing data through user-defined dt-transfer

type :: t

        logical :: non_empty
    contains
        ! unformatted read/write

procedure :: read_unformatted

procedure :: write_unformatted

generic :: read(unformatted) => read_unformatted

generic :: write(unformatted) => write_unformatted

end type
contains
subroutine read_unformatted(this, unit, iostat, iomsg)

implicit none

class(t), intent(inout) :: this

integer, intent(in) :: unit

integer, intent(out) :: iostat

character(len=*), intent(inout) :: iomsg
! read data, crash

! forrtl: severe (67): input statement requires too much data, unit -130

read(unit, iostat=iostat, iomsg=iomsg) arr

end subroutine
subroutine write_unformatted(this, unit, iostat, iomsg)

implicit none

class(t), intent(in) :: this

integer, intent(in) :: unit

integer, intent(out) :: iostat

character(len=*), intent(inout) :: iomsg
! write data

write(unit, iostat=iostat, iomsg=iomsg) arr

end subroutine
end module
program p

use m ! load data and type(t) IO-transfer statements

    implicit none
    type(t) :: my_t
    ! I/O

integer :: unit

integer :: iostat

character(len=999) :: iomsg
! write ---------

open( &

newunit = unit, &

action = 'write', &

file = 'io.dat', &

form = 'unformatted', &

iostat = iostat, &

iomsg = iomsg &

)

if (iostat /= 0) then

stop 'error opening file for writing'

end if
! I. write data

write(unit, iostat=iostat, iomsg=iomsg) arr

if (iostat /= 0) then

print *, 'iomsg = ', iomsg

stop 'error writing to file (I)'

end if
! II. write same data through dt-transfer

write(unit, iostat=iostat, iomsg=iomsg) my_t

if (iostat /= 0) then

print *, 'iomsg = ', iomsg

stop 'error writing to file (II)'

    end if
    close(unit)

   ! read ----------

open( &

newunit = unit, &

action = 'read', &

file = 'io.dat', &

form = 'unformatted', &

iostat = iostat, &

iomsg = iomsg &

)

if (iostat /= 0) then

stop 'error opening file for reading'

end if
! I. read data

read(unit, iostat=iostat, iomsg=iomsg) arr

if (iostat /= 0) then

print *, 'iomsg = ', iomsg

stop 'error reading from file (I)'

end if
! II. read data through dt-transfer - crash

read(unit, iostat=iostat, iomsg=iomsg) my_t

if (iostat /= 0) then

print *, 'iomsg = ', iomsg

stop 'error reading from file (II)'

end if
close(unit)
end program

[/fortran]

No error: Reading and writing the array 'arr' with a classical IO-statement (I) works fine, even reading back with the dt-transfer (II) is ok.
Only when writing with dt-transfer (II), there is no way (neither (I) nor (II)) reading back the array without error.
No error: for arrays smaller than a critical size of 4 * 2**15 bytes
No error if the read/write statement transfers multiple variables (e.g. as in read(unit) A,B,...) with total size (A+B+...) >= the critical size, but individual sizes (A, B, ..). smaller.

Compile-line and output:

$ ifort test.f90 && ./a.out
iomsg =
input statement requires too much data, unit -130, file /home/s_ftschi/Desktop/
test/io.dat

error reading from file (II)

Any suggestions on how to avoid the error are very welcome!

(preferably on a higher level, since the error appears in variuos deeply nested transfer statements)

Best regards
Ferdinand

Ferdinand_T_ · ‎11-12-2014

Corrected formatting: [fortran]

module m

implicit none

! global IO data

! critical size: 4 * 2**15 bytes (131.07200 kB)

integer, parameter :: ARR_SIZE = 2**15 ! = 32768; no crash with 2**15 - 1

integer, dimension(ARR_SIZE) :: arr ! crash with/without dynamic allocation

! writing data through user-defined dt-transfer

type :: t

logical :: non_empty

contains

! unformatted read/write

procedure :: read_unformatted

procedure :: write_unformatted

generic :: read(unformatted) => read_unformatted

generic :: write(unformatted) => write_unformatted

end type

contains

subroutine read_unformatted(this, unit, iostat, iomsg)

implicit none

class(t), intent(inout) :: this

integer, intent(in) :: unit

integer, intent(out) :: iostat

character(len=*), intent(inout) :: iomsg

! read data, crash

! forrtl: severe (67): input statement requires too much data, unit -130

read(unit, iostat=iostat, iomsg=iomsg) arr

end subroutine

subroutine write_unformatted(this, unit, iostat, iomsg)

implicit none

class(t), intent(in) :: this

integer, intent(in) :: unit

integer, intent(out) :: iostat

character(len=*), intent(inout) :: iomsg

! write data

write(unit, iostat=iostat, iomsg=iomsg) arr

end subroutine

end module

program p

use m ! load data and type(t) IO-transfer statements

implicit none

type(t) :: my_t

! I/O

integer :: unit

integer :: iostat

character(len=999) :: iomsg

! write ---------

open( &

newunit = unit, &

action = 'write', &

file = 'io.dat', &

form = 'unformatted', &

iostat = iostat, &

iomsg = iomsg &

)

if (iostat /= 0) then

stop 'error opening file for writing'

end if

! I. write data

write(unit, iostat=iostat, iomsg=iomsg) arr

if (iostat /= 0) then

print *, 'iomsg = ', iomsg

stop 'error writing to file (I)'

end if

! II. write same data through dt-transfer

write(unit, iostat=iostat, iomsg=iomsg) my_t

if (iostat /= 0) then

print *, 'iomsg = ', iomsg

stop 'error writing to file (II)'

end if

close(unit)

! read ----------

open( &

newunit = unit, &

action = 'read', &

file = 'io.dat', &

form = 'unformatted', &

iostat = iostat, &

iomsg = iomsg &

)

if (iostat /= 0) then

stop 'error opening file for reading'

end if

! I. read data

read(unit, iostat=iostat, iomsg=iomsg) arr

if (iostat /= 0) then

print *, 'iomsg = ', iomsg

stop 'error reading from file (I)'

end if

! II. read data through dt-transfer - crash

read(unit, iostat=iostat, iomsg=iomsg) my_t

if (iostat /= 0) then

print *, 'iomsg = ', iomsg

stop 'error reading from file (II)'

end if

close(unit)

end program

[/fortran]

Steven_L_Intel1 · ‎11-12-2014

I can reproduce the problem. The error is actually with the DTIO Write, where it is writing the file with an invalid record length in the header and trailer of the record. I will do a bit more analysis and then pass it on to the developers. Issue ID is DPD200363176.

jimdempseyatthecove · ‎11-12-2014

Steve,

FWIW in this case, the size(userDerivedType) does not represent the size of the data to be written, which is an array in a module, not in the type. Perhaps the buffer allocation looks at larger of "current buffer" or "size(userDerivedType)", and adjusts the buffer size in the case of the latter, and in either case the resulting buffer would be smaller that the data written.

Jim Dempsey

Steven_L_Intel1 · ‎11-12-2014

There's no requirement that the size of the data in the I/O statement match the amount of data the DTIO routine writes. This is simply a bug in our DTIO support.

Steven_L_Intel1 · ‎12-12-2014

The bug has been fixed - I expect the fix to appear in Update 2, planned for February. The fix is in the run-time library, so if you linked to the DLL libraries (the default nowadays) you won't have to recompile.