Best way to speed slow disk writes in Fortran

[van_der_merwe__ben]

We have a Windows native C++ application that calls Fortran code, and that Fortran code writes to an output text file. When done, the application reads in the file contents. All good, but such file activity can be costly and we are looking for a way to avoid it totally. And there is a lot of complex Fortran code, not just a few files which you could convert. I need some help with that please. I know of these options:

Option A:  Call to a C function instead of the Fortran write command.

Instead of this code:

          write (JOT, 101) data

You can do this:

          character*80 CARRAY

          write (CARRAY, 101) data

          call writec(CARRAY,JOT)

And that works beautifully, you use a script to convert all your files and there you go. BUT the following complications arise:

- Carray has to be large enough and the c function writec does not receive the actual size, it just always gets 80. Minor but not ideal.

- For worse, if the format statement contains / (means start new record or line) or the data has arrays or the format state is repeated for lots of data, you get the dreaded runtime error 27 "too many records in i/o statement file internal format write". So you discover that you need something like:

character (len=120), dimension(50) :: CARRAY

And then it works. But in some situations you still get error 27 no matter what you do.

Any ideas around this? Is there a way to direct the write to a character array and suppress or disable records to make error 27 got away?

Option B: When you open the file use STATUS='SCRATCH'.

Here I hope Lionel or someone can comment on how well Intel Fortran supports this. Does it merely create a real temporary file which is deleted on close (which is not going to help here), or does it create a temporary in memory only file? Is there a way to create an in memory only file via a suitable status setting in the OPEN command?

Option C: Use a RamDrive

Yes, that would help, but you would need to install drivers and configure it which is problematic.

Option Intel Fortran had option.

You could implement code so that write statements would call you code, and there is an item in these forums about that somewhere, but I do not think it works anymore and I wonder if it is supported in the IFX compiler? Does anyone know? That would be ideal.

Option E: Fortran open buffer options.

What options would give you the best performance? Something like this perhaps: Use OPEN with BUFFERED='yes', BLOCKSIE=131072,BUFFERCOUNT=10

Option F: USEROPEN

Intel Fortran lets you use USEROPEN with OPEN so you can direct the open to your C function, which should then be able to call CreateFile with the FILE_ATTRIBUTE_TEMPORARY | FILE_FLAG_DELETE_ON_CLOSE flag.

See: https://www.intel.com/content/www/us/en/docs/fortran-compiler/developer-guide-reference/2025-0/open-useropen-specifier.html

Summary

I have been working on option a, but error 27 is complicating matters. Now I am contemplating trying  option F, USEROPEN. Any insights or advice would be appreciated.

[andrew_4619]

Well the whole writing then reading files is very suboptimal however without knowledge of the details it is hard to suggest and to understand the work needed. Just as a point you can make carray big and pass trim(carray) to writec

[Steve_Lionel]

1) STATUS='SCRATCH' writes a temporary disk file that is then deleted on close

2) Formatted I/O has overhead that can be significant at times - might unformatted I/O work for you in this case? If you're going to share the data with C++, you might do well with ACCESS='STREAM'.

3) For calling C, you'd need to null-terminate the strings so that C knew the length. I don't recommend going down this path

My usual advice when dealing with perceived performance issues is to use a profiler, such as Intel VTune, to find out just where the time is being spent.

[ChrisC2]

Based on my personal experience, BUFFERED='yes' makes a huge difference when writing out large arrays.  (100MBs of file size, etc).     5X-10X speed improvement.   So by all means, try that first.   I don't specify anything else, so blockszie, buffer count, other variables, they all stay as default in my programs.