Conditional Output Buffer

[christian_rackl]

My program calculates values for many thousand points.

Each calculation loop writes result values to a log file.

A log entry for such a point is about 100 lines in length.

So you can imagine that the file grows very large....

My idea was to create a filter: the value of the end result decides if the block is written to the log file. Any ideas to do that efficiently?

If i write the data first to a temporary file and copy it afterwardsto the log file the performance is quite bad.

So i'm searching for a way to write only to diskwhen absolute nessecary.

can i writemy data somehowon an internal memory, and copy it then to the log if the value is greater then my filter-value?

thank everyone for helping!

[greldak]

Basically preceed your write statements with an IFstatement such as.

IF OutputLevel.GT.Threshold THEN

WRITE(DestinationUnit,*)StringToWrite

END IF

What I have used in the past is a wrapper routineto do this and replacing all theWRITES in the main code with a call to it.It makes it a lot easier to read afterwards than preceeding all your write statements with an IF statement.

This is also easy to expand to allow different locations to send output including multiple ones such as screen and file.

You may need to consider the file size limitations of 2Gb if you are using a FAT partition as well (NTFS is a lot higher though) if your files get this large

[christian_rackl]

well, the problem actually is that the condition (if something has to be written) depends on the end result.

so i cannot decide it the way you mentioned it.

because there are many functions and alsosubcalculation results it is not possible for me to store all information until i have the end result

what dou you think of a very large character variable in C style so i can also use carriage returns?

[christian_rackl]

No ideas?

[ingo_berg]

Allocate a buffer of strings large enough to hold your information in memory and write to this buffer. Once you know if it is going to be written or not either write the buffer to file or discard it.

[christian_rackl]

Here is my sample Script for this purpose:
program buffertest
implicit none
character*512:: buffer

open(10, file='buffertest.txt')
write(buffer, '("Line1")')
write(buffer, '("Line2")')
write(10, '(a)') buffer

close(10)
end program

When I write to the buffer the data is overwritten.
Is there a way that the buffer is always added at the end?
It would be great if this works like an external file: Each line is appended to the preceding and always ends with an CRLF

[ingo_berg]

Allocate the lines dynamically and Store their pointers in a dynamic array. But be warned Fortran does a pretty poor job when it comes to pointers and dynamic allocation compared to C/C++.

It does not support arrays of pointers, this is the reason for the buf_type stunt. It does not support reallocation of arrays this is the reason for the realloc subroutine. Dont forget to release the buffer afterwards.



program buftest

implicit none

! Variables
integer, parameter :: MAXLEN = 1024

! Type for an array of pointers to strings
type :: buf_type
character(LEN=MAXLEN), pointer :: ptr
end type

integer :: i
type(buf_type), allocatable :: buffer(:)
character(LEN=MAXLEN), pointer :: sLine

do i=1, 1000
! reallocate array of pointers to the lines
call realloc(buffer, i)
allocate(sLine)
write(sLine, "(""This is Line "", I2)") i
buffer(i)%ptr => sLine
end do

! Do whatever you want with the buffer
! ...
! ...
! ...

! deallocate all strings
do i=1, size(buffer,1)
deallocate(buffer(i)%ptr)
end do
deallocate(buffer)

contains

subroutine realloc(a_Data, a_iNewSize)

integer, intent(in) :: a_iNewSize
type(buf_type), allocatable, intent(inout) :: a_Data(:)
type(buf_type), allocatable :: Buf(:)
integer :: iAlloc = 0, i = 0

if (a_iNewSize<=0) return

allocate(Buf(a_iNewSize))
Buf = buf_type(null())

iAlloc = size(a_Data, 1)
forall (i=1:min(a_iNewSize, iAlloc))
Buf(i) = a_Data(i)
end forall

if (allocated(a_Data)) deallocate(a_Data)
allocate(a_Data(a_iNewSize))
a_Data = Buf

deallocate(Buf)

end subroutine

end program buftest

[engineerik]

Onesimple method would be to write your output to the log file and track how many lines were added. If you later determine that you do not want to save those lines in the log file then backspace the correct number of lines.

Erik

[ingo_berg]

What happens if you backspace 20 lines and add 15 new lines instead. Will the 5 lines difference remain in the file or will they be deleted?

Ingo

[Steven_L_Intel1]

Any time you write a record in sequential I/O, that record becomes the last record in the file. Therefore, in the situation you describe, the last five records would be lost.

[christian_rackl]

I found a way to do it now:
I always used a variable called LOG for describing the output cannel of write statements.
Therefore it was easy to redirect the output to a new temporary file (only changing the value of LOG)
after i got the end result i conditionally copied the temporary file to the original log file

works fine so far but the performance is very poor

any ideas on increasing read/write performances on formatted files?

[ingo_berg]

The only way to accelerate writing to a file is not to write to the file hence use an offline buffer. There is no magical button that you push and suddenly the I/O mechanism is much faster.

You could try binary I/O which ist "faster" since the ammount of data written is significantly lower.

For instance:
"3.14159265" -> 10 bytes as text + additional bytes for the formatting (Spaces, tabs, line breakes). The same number takes only or 4/8 bytes as a floating point value. My guess would be that reading is faster too since it can be copied into a memory location from file whilst ascii has to be recalculated into a floating point value.