How to store simulated data efficiently?

[jkfzis]

Hallo, my name is John, I just started to do some programming in VISUAL FORTRAN and I have the following problem:

Due to Monte Carlo simulations I want to produce many realizations of different random variables, and I want to store them to use the same data set several times in an iteration process. In smaller programs, I always stored the simulated data through e.g. an external *.txt file.[OPEN(100,FILE='DATA.TXT');WRITE(100,*)RE;.. Now there is so much data to store that this approach becomes very inefficient. How could I store the simulated data more efficiently?
Thank you in advance,

John from Switzerland

[pcurtis]

You don't say how much data, of what sort, you need to store, but it would certainly be much more efficient to save the data in a file as an exact "core image" of the memory block holding the array, rather than in text or any applied format which requires extra space and extra processing. This is difficult using antiquated standard Fortran i/o, trivial using modern Win32 file i/o routines which allow you to directly write/read a specified byte count, at a specified offset location within a file, from/to any location in memory. email for details, sample codes using Win32 fileio from CVF.
Paul Curtis
pcurtis@kiltel.com

[Intel_C_Intel]

HDF5 is a file format and library (with Fortran bindings) that is designed for scientific data. You can find more information about it at: http://hdf.ncsa.uiuc.edu/. One advantage of using HDF5 is that the files are portable to other machine architectures.

The Intel Array Visualizer provides a library that simplifies using HDF5 and also a Viewer for visualizing data files. See: http://www.intel.com/software/products/compilers/fwin/array_vis.htm.

John

[kdkeefer]

John,
Even in "antiquated" Fortran, an unformatted, sequential I/O is much faster than formatted. (The amount of computation required to convert a string of characters to a binary number is very significant). You can write out one value or an entire array. However you should know the number of values you've written so that you can read them back (If your WRITE I/O list is the same as your READ, it's all taken care of). Each write statement generates one record, the length of the I/ list. You may read back the first part of a record, if the I/O list is shorter than the one that wrote it, but if the I/O list is longer, you get an end of record error.
This can be implemented quickly and easily: open the file for sequential, unformatted access and remove the format statement labels from the read/write statements and you're done. It will be ~10 times faster.
The other recommendations are undoubtedly faster for huge amount of data, but so is direct access I/O.
Keith