Fastest way to read/write binary data?

[david_sallngc_com]

Hi!

I am reading/writing large datasets of double precision real data and single precision integer data and would like to know if there is a way I can speed up the I/O. I do my I/O using binary data and am using Intel FORTRAN 11.1. In particular, I open my files by,

open(unit=fid, file = 'foo.dat', form = 'unformatted', access = 'stream')

I READ and WRITE my data as,

do i = 1,N
read(fid) x(i)
write(fid) x(i)
end do

Is there a way to use OpenMP to help speed up the I/O?

Thank you very much for your help.

Sincerely,

David

[Steven_L_Intel1]

You're reading an element at a time, which has a lot of overhead. Can you read a whole array at a time, process it and then write a whole array?

[david_sallngc_com]

Hi Steve!

Yes, I can write an entire vector at one time. It is a very large vector though. Is there a way I can check, apriori, via some code, to make sure I have enoughmemory to write it in one chunk or is this not even a concern?

Is is possible to use OpenMP in some way to write different chunks of data to a file at the same time?

Is writing entire vectors at one time the fastest I can do?

Thanks Steve.

David

[Steven_L_Intel1]

Since you're using ACCESS='STREAM' you can decide how much to read at a time. If you can overlap the reading, computation and writing you might want to look into using ASYCHRONOUS='YES' option on opening the file and doing the I/O. Read more about that in the documentation, though I'd advise you to use version 12 (Composer XE 2011) for best results.

It is not clear to me that OpenMP will help you here, but there are many experienced parallel applications programmers in this forum who I am sure will offer opinions.

How do you know that it's the I/O that is the limiting factor?

[jimdempseyatthecove]

>>Is is possible to use OpenMP in some way to write different chunks of data to a file at the same time?

real :: YourData(BigSize)
integer, volatile :: ElementsInsertedIntoBuffer
logical :: InsertionsFinished
integer, volatile :: LastElementWritten
logical :: WriteError
ElementsInsertedIntoBuffer = 0
InsertionsFinished = .false.
LastElementWritten = 0
WriteError = .false.

!$omp parallel sections
do while((.not. InsertionsFinished) .and. (.not. WriteError))
InsertionsFinished = AppendDataToBuffer(YourData, ElementsInsertedIntoBuffer)
end do
!$omp section
do while((.not. InsertionsFinished) .or. (ElementsInsertedIntoBuffer .gt. LastElementWritten))
if(ElementsInsertedIntoBuffer .gt. LastElementWritten) then
WriteError= WriteDataFromBuffer(YourData, LastElementWritten)
if(WriteError) exit
else
sleepqq(100)
endif
end do
!$omp end parallel sections

Something like that will do. I will let you write the functions.

Note, you can also use two data arrays as a double buffer or use a smaller large array as a circular buffer (with logic to avoid overrunning the writes).

A similar thing can be done for reading the data.

Note, the writes in your WriteDataFromBuffer would be normal writes.

Jim Dempsey

[david_sallngc_com]

Dear Jim and Steve!

Thank you both very much for your suggestions. Sounds fantastic.

Now for some fun coding!!

Sincerely,

David