Intel® Fortran Compiler
Build applications that can scale for the future with optimized code designed for Intel® Xeon® and compatible processors.

Scratch File Written To Ram Only

ScottBoyce
Beginner
900 Views

I have several support routines that only work with file unit numbers to load information from a file. 

My issue is that I have a character array, [ e.g. CHARACTER(700), DIMENSION(500) ] that needs to be processed by those support utilities.

What I do now is make a scratch file with a large buffer value and then pass the unit number to the utility.

Ideally it would be great if there was a way of making either scratch file that resided entirely in ram (similar to a Linux tmpfs) or have a character array be processed as if it was a file. Python has a feature that does this called StringIO to make a memory file.

Any suggestions would be greatly appreciated since this is a frequent problem I have.

0 Kudos
10 Replies
andrew_4619
Honored Contributor II
900 Views

From your description it seems to me you have that large character array full of stuff what you write to a temp file and then read is back to 'process the data'. Why not cut out the slow file stuff and make reads directly from the character array?

0 Kudos
ScottBoyce
Beginner
900 Views

Character array is actually distilled from an input file for a simulation model. To process the input information it requires a series of subroutines that parse it. However the subroutines only work with files themselves and any particular line can have a redirect input option that points to a separate file. Writing a new subroutine that mimics the file version for the character array is not feasible (it would require cloning a lot of code and there would still be pitfalls).

Also the data structure is actually not a true character array, but rather a linked list of strings of variable length that represent each line of an input file that was pre-processed. What I do is just write the list to a scratch file and pass that unit number to the other read utilities. What be better is if I could create a memory file to write the list too and then pass that unit number over.

0 Kudos
mecej4
Honored Contributor III
900 Views

Files can be formatted or unformatted, and access to the contents of the file can be sequential or otherwise. On the other hand, there is no Fortran file type that closely matches a linked list, binary tree or a heap. Before we can make suggestions, you need to provide us a description of the structure of the data and your patterns of repeated access. Questions that need to be answered, for example: "Why use a linked list at all?"

The simplest solution (by which I mean the one for which the least amount of reprogramming would be needed) at this point would be for you to create a Ramdisk with a capacity of, say a few megabytes and store your data file on it.

0 Kudos
ScottBoyce
Beginner
900 Views

You guys are missing the point. I am just looking for a way of making some sort of internal file that is more than a single character string, pretty much do the same that python has with the module StringIO for making a memory based file.

I would not mind doing a RamDisk, but the problem there is that it is difficult to set up on windows and this software is distributed to other people that only want a single executable and do not want to do any additional windows set up.

0 Kudos
Steven_L_Intel1
Employee
900 Views

A CHARACTER array with sufficient length and bounds can be used with internal formatted I/O. Of course, each READ or WRITE starts at the beginning so maybe this won't be what you want.

Is there a reason not to use a STATUS='SCRATCH' file?

0 Kudos
andrew_4619
Honored Contributor II
900 Views

There is no build in way to replicate file i/o to memory. You can do formatted i/o to characters strings and sub-strings.

700x500 is quite a small amount of data

If you do not want to restructure your program to reflect that memory size has moved on since this was written then leave it alone an accept it runs a lot slower than it needs to. How big is this program BTW?

 

0 Kudos
ScottBoyce
Beginner
900 Views

Its more for neatness of the code and to avoid making a .tmp file on the system (my end users tend to not like those). I put a farily large buffer value so I think the tmp file should mostly reside in RAM.

What I do now is what you suggested is writing to a scratch file and passing that unit number to the subroutines. I just was hopeful there was something in Fortran that I was not aware of that could do multiline internal reads.

Thanks all for your comments.

0 Kudos
Steven_L_Intel1
Employee
900 Views

Sure - a character array is treated as a series of records, one record per array element. Maybe not what you want, though.

0 Kudos
IanH
Honored Contributor II
900 Views

I was wondering whether you could the ifort USEROPEN extension to open a pipe reading from memory or somesuch, rather than a file on disk.

But I can't get the useropen example that comes with the compiler docs (see https://software.intel.com/en-us/node/678417) to run with ifort 17.0 on IA32, using my usual command line options of /check:all /warn:all /standard-semantics.  It blows up with an access violation when the user function is called.  (I assume the interface for useropen with 64 bit program is a bit different to that shown in the docs?)

(You can create ramdisks on windows using third party products, but the need to install a device driver will probably not thrill your users.)

0 Kudos
Steven_L_Intel1
Employee
900 Views

There is indeed something wrong with USEROPEN - it had many bugs fixed in 16, but I agree that the example doesn't work. It's been a background task of mine to figure out why (especially since I wrote and tested this example in the first place.)

You don't need USEROPEN to open a pipe. Again, it seems very simple to use STATUS='SCRATCH' - the file won't show in the user's directory and gets deleted on close.

0 Kudos
Reply