Software Archive
Read-only legacy content
17061 Discussions

Large array IO optimization

dpbozarth
Beginner
424 Views
I have large array (8x3000000 is typical) from data acq system to process. Unfortunately, the req'd processing can be accomplished in one pass so the entire array must be held in memory..I can allocate an array for the whole thing and do an unformatted read, but that ties up the machine for the entire i/o operation...my question is What is the most efficient way to do the read piecewise to allow for an occasional progress indicator update and perhaps a user request to cancel the operation?

So far, I've discovered that if I ALLOCATE buf(nChn, nScans), the READ(ioun) buf takes several times as long as the write did under Visual Basic. This, of course, is risky business, but the data acq box is an old 90MHz P-I while the processor is a 200+MHz PII....I had expected the FORTRAN read to be at least on the same order as the VB PUT....however, this may just be demonstrating the weakness of IDE vs SCSI as the data acq box has a brand new UW SCSI adapter/drive whereas the analysis box is run of the mill brand D notebook. Both are running NT4, if that has any bearing on it.

Anybody have suggestions of proper coding form to at least minimize the pain? I'm going to go try a direct fixed allocation although all data was not collected under the same conditions so variable sizes will have to be handled eventually.
Thanks
0 Kudos
1 Reply
Steven_L_Intel1
Employee
424 Views
If I recall from your posting on comp.lang.fortran, you are using FORM='BINARY' for your Fortran I/O. Try adding BUFFERED='YES' to the OPEN - that should help a lot.

Steve
0 Kudos
Reply