Intel® Fortran Compiler
Build applications that can scale for the future with optimized code designed for Intel® Xeon® and compatible processors.

Writing to binary file is extremely slow

Ioannis_K_
New Contributor I
986 Views

Hello,

I have built a program which has to print large output files. When I print the output in a formatted ASCII file, the printing is almost instantaneous, taking a couple of seconds. However, when I run the program and try to print the SAME EXACT output to a binary file, the printing is extremely slow, taking more than 5 minutes 

I wanted to ask if there is an explanation for such discrepancy, and whether there is something I could do to speed up writing to a binary file.

Many thanks in advance.

0 Kudos
7 Replies
Steve_Lionel
Honored Contributor III
986 Views

Probably a mistake you made somewhere. An example program showing the issue would be helpful.

One thing I often saw was where the program OPENed the file for unformatted I/O and specified a RECL in bytes, with the user not knowing that the Intel (and DEC-heritage) default was that this value was in 4-byte units. This would cause four times as much data to be written, which was slower. This can be adjusted with the /assume:byterecl option.

0 Kudos
Ioannis_K_
New Contributor I
986 Views

I cannot find anything wrong with my code. Also, the RECL SHOULD be 8 bytes. The printed output is to be read by another app (for which I do not have the source), and this other app uses a RECL of 8 bytes.

 

Here is a small sample from my code:

do i1 = 1,numnp1
        write(9,pos=filposit*(iWORDsiz1)+1) i1
        filposit = filposit + 1 
      end do !i1

 

The value of the integer iWORDsiz1 is equal to 8, and fileposit is an integer parameter that gives the position of each variable to be printed.

 

0 Kudos
Ioannis_K_
New Contributor I
986 Views

I also wanted to provide the following additional information:

The problem could be hardware-related(?). When I run the program in a different computer, the speed is reasonable: this means that the print to the binary file is not slow at all (it is only very slightly slower than the corresponding ASCII print, which is expected in my opinion). On the other hand, when I run the program in the PC for which I reported the issue, the binary print is extremely slow. By "extremely slow", I mean that, instead of taking 3-4 seconds, it takes about 10 minutes!!! 

More interestingly, when run the program IN THE SAME COMPUTER (where binary print was very slow) but in a folder located in a DIFFERENT disk drive, the binary print is fast! Could this be attributed to the hardware somehow? I cannot imagine that there is a hardware malfunction in the disk drive, because I cannot see how a malfunctioning drive would only have slow write in binary files (and fast write in ASCII files). Could the slow speed have to do with, e.g., the way in which the hard disk is formatted?.

0 Kudos
garylscott1
Beginner
986 Views

Perhaps I'm missing something, but "binary" reads and writes or ordinarily MUCH faster than ASCII reads/writes, depending on the details of the data content somewhat but nearly always, binary is faster.  To read/write ASCII requires an added layer of conversion to/from internal representation that is not need typically for binary IO.

0 Kudos
jimdempseyatthecove
Honored Contributor III
986 Views

When you incorporate pos=, and depending on sharing mode, if shared it implies

Read (buffer), Modify (buffer), Write (buffer)

If on open you specified a large buffer size, this can take considerable time.

If (when) the file is opened for exclusive use (no sharing), the Read and Write occur only when necessary.

Also the flushing can be exacerbated when the file is located on a network share.

Can you post your open statement?
Can you tell us the difference between file destinations?
Can you tell us if Antivirus scans one destination folder and not the other?

Jim Dempsey

0 Kudos
jimdempseyatthecove
Honored Contributor III
986 Views

Also,

Use NOSHARED specifier on OPEN

And, try pre-extending the file. IOW write a dummy last record first, then write the sequence of records.

Jim Dempsey

0 Kudos
CTOptimizer
Beginner
986 Views

For good measure, I would also try BUFFERED='YES' in your OPEN() statement (of the binary output file).  There have been some recent changes in the buffering (including a possible performance regression) so that could affect your results.

0 Kudos
Reply