2 programs communicating via files

[mamey4]

Hi there,

I'm working on a task right now where I'd like to get two running programs exchange data with each other by writing and reading files. So each of the programs does some computations, then write to its own file, then reads from the file the other program wrote, and then starts over.
The code for both of my programs looks like this:

[bash]    program p2

    implicit none
    integer i,sum,ioerror,received
    
    sum=1001   
    
    do i=1,50
    
        sum=sum-1
        
        !wait until file doesnt exist any more
        2 open(40, file='D:/p2_p1.txt', status='OLD', iostat=ioerror)
        if(ioerror.eq.0) then
            close(40)
            goto 2
        else
            close(40)
        end if
        
        !write new data
        open(40, file='D:/p2_p1.txt', status='NEW', iostat=ioerror)
        write(40,*) sum
        close(40)
        write(*,*) 'data sent.'
        
        !read new data
        1 open(40, file='D:/p1_p2.txt', status='OLD', iostat=ioerror)
        if(ioerror.ne.0) goto 1
        read(40,*) received
        close(40)
        write(*,*) 'data read from p1:',received
        
        !delete
        call system('del D:\p1_p2.txt')
        write(*,*) 'file deleted.'
        write(*,*) ' '
             
    end do

    end program p2[/bash]

Program p1 looks exactly like this, except for:
- "p2" is replaced by "p1"
- the filenames p2_p1.txt and p1_p2.txt are swapped
- file channel 20 is used instead of 40

Now the problem is that despite the tests I inserted so that the programs do not read from exchange files that have not been written yet by the other program, these tests don't seem to work everytime. The loop in each program is 50 iterations; sometimes the I/O error comes after the 5th, sometimes after the 21st iteration - it's unpredictable.
Can you give me a hint on how to improve my programs so that the intented effect is achieved?

Thanks in advance for your thoughts!
mamey

[JohnNichols]

1. Can I suggest that you use a routine that creates a new file name each time you run the program, the accelorometer program I buy does that . That is not hard to do.
2. See below
3. Have one program with two common blocks, and three loops, outside loop controls whihc code executes, two inside loops take turns. Commons are mighty handy although freaking dangerous to use if you are not strict in how you set them up.

---------------------------------------------------------------------------------------------------------------------------

Use the following code to check and see if new files exist in the directory and return its name. I stole this from somewhere recently, I think the Intel help files:

[bash]!  WRITE (*, 900) ' Enter wildcard of files to view: '
      !  900 FORMAT (A, )
 
      !  length = GETSTRQQ(files)
        handle1 = FILE$FIRST

        files = '*.csv'
 
        DO WHILE (.TRUE.)
            length = GETFILEINFOQQ(files, info, handle1)
            IF ((handle1 .EQ. FILE$LAST) .OR. &
                (handle1 .EQ. FILE$ERROR)) THEN
                SELECT CASE (GETLASTERRORQQ( ))
                    CASE (ERR$NOMEM)
                        WRITE (*,*) 'Out of memory'
                    CASE (ERR$NOENT)
                        EXIT
                    CASE DEFAULT
                        WRITE (*,*) 'Invalid file or path name'
                END SELECT
            END IF
 
            permit = ' '
            IF ((info%permit .AND. FILE$HIDDEN) .NE. 0) &
                permit(1:1) = 'H'
            IF ((info%permit .AND. FILE$SYSTEM) .NE. 0) &
                permit(2:2) = 'S'
            IF ((info%permit .AND. FILE$READONLY) .NE. 0) &
                permit(3:3) = 'R'
            IF ((info%permit .AND. FILE$ARCHIVE) .NE. 0) &
                permit(4:4) = 'A'
            IF ((info%permit .AND. FILE$DIR) .NE. 0) &
                permit(5:5) = 'D'
 
                lengName = Len_Trim(info%name)
      
                do 20 I = 1, 80

                    FileString(i) = ''
                    !write(*,*)FileString(i)
    20          end do

                do 40 I = 1, lengName-4

                    FileString(i) = info%name(I:I)
                    !write(*,*)FileString(i)
    40          end do

    !            WRITE (*, 9000) lengName, info%name, info%length, permit
    9000        FORMAT (1X,I4, '   ', A90, I9, ' ',A6)
               ! Write(*,*)FileString
[/bash]

[jimdempseyatthecove]

The problem you have is the file delete will fail whenever the other program has the file open for the purpose of detecting the presense/absence of the file. Nichols suggest you use a directory search technique instead.

A third option is to not use file exists to indicate change in data, use change in data in shared file. You can use a message file in combination with a sequence number written as first data. This is using combination of

! p1 init
sequenceNumber = 0
otherSequenceNumber = 0
open(fileUnit, file='yourSharedFileName.txt',status='NEW',iostat=ioerror)
if(ioerror.ne.0) goto yourErrorLabel
write(fileUnit,*) sequenceNumber
call StartYourOtherProgram()
! other program running and waiting for written sequenceNumber to advance
do ... ! your process loop
call doYourWorkAndWriteOutputFiles()
if(sequenceNumber .gt. 0) then
do while(otherSequenceNumber .le. sequenceNumber)
REWIND(fileUnit) ! add errror test
FLUSH(fileUnit) ! may be implicit with REWIND
READ(fileUnit,*) otherSequenceNumber ! add error code
end do
sequenceNumber = otherSequenceNumber
endif
sequenceNumber = sequenceNumber + 1
WRITE(fileUnit, *) sequenceNumber ! add errror test
end do


Note, the above code permits the "producer" program to build the next output data files for the consumer program while the consumer program is working on the prior data.

This may require you to use file names containing the sequence number. .OR. recode to produce the data into internal arrays, wait for consumer program to complete prior processing, then write new data set, then update shared file sequence number. Both of these techniques provide for a degree of double buffering.

Jim Dempsey

[bmchenry]

Is there any particulalr reason why you are using files and I/O to exchange the data/information instead of a doing it in memory via a dll?
Why not create a dll which contains a module that contains the data both program use/share.
Then this same dll could be the control program to call each program in sequence/loop or a flag in the module is set in each program to transfer control to the other program.
I think it wouldspeedthingsand get you away from the slow I/O file solution you currently are using.
(and at any time the control program or one of the other programs could write output if and when you need it)

brian

[jimdempseyatthecove]

>>Is there any particulalr reason why you are using files and I/O to exchange the data/information instead of a doing it in memory via a dll?

You would do this when the two programs are running on seperate systems. On multi-core single system you could use the DLL, a memory mapped file, a pipe, interprocess events, or files. Or, if possible, link the two programs as one and run as two threads using memory buffers.

Jim Dempsey

[Steven_L_Intel1]

I will comment that I have seen many users tripped up by Windows not immediately "releasing" a file when it is closed or deleted.

We provide a worked example of using a DLL for communication in the sample DLL\DLL_Shared_Data

[JohnNichols]

Use the DLL, that is significantly better than any other method.

Although one would be tempted to try a parallel approach and run them together

After all you are using the world's fastest compiler

JMN

[mamey4]

I'v tried a couple of variants now, all of which include files as the format for data exchange. None of them work for every run of the two programs, they just differ in how many times out of 100 runs an IO error (or a deadlock at an internal loop which checks for IO errors) occurs and crashes the workflow.
I would like to use the DLL variant, since I also believe that it should be faster than using files. But the data to be exchanged by my two programs are not just values, but most definitely allocatable arrays whose size is only determined during the programs' runtime. I think I've read somewhere that this kind of data can't be shared in the way it's show in the DLL example. Do you know a way to circumvent this?

[JohnNichols]

Make it one program. It is a lot easier to control.

JMN

[mamey4]

You can be sure that if that were an option I'd have gone with it...

[GVautier]

Hello

Have you tried with a "handshake" file that can be created by p1 program when it finish to write to p1_p2.txt. P2 program will only wait the "apparition" of this file to know that the data file writing has been completed. The same process can be used in the other way.

[mamey4]

Mh I guess that should look something like this:

[bash]program p2

    implicit none
    integer i,sum,ioerror,from_p1
    
    sum=1000   
    
    do i=1,50
    
        sum=sum-1
        
        2 open(40, file='D:/p2_p1.txt', status='OLD', iostat=ioerror)
        if(ioerror.eq.0) then
            close(40)
            goto 2
        end if
        
        open(40, file='D:/p2_p1.txt', status='NEW', iostat=ioerror)
        write(40,*) sum
        close(40)
        write(*,*) 'data sent.'
        
        !Signalfile
        open(40, file='D:/p1_to_delete.txt', status='NEW', iostat=ioerror)
        write(40,*) 'signal'
        close(40)
        write(*,*) 'signal file created.'
        
        !wait for Signalfile
        3 open(40, file='D:/p2_to_delete.txt', status='OLD', iostat=ioerror)
        if(ioerror.ne.0) goto 3
        close(40)
        call system('del D:p2_to_delete.txt')
        
        1 open(40, file='D:/p1_p2.txt', status='OLD', iostat=ioerror)
        if(ioerror.ne.0) goto 1
        read(40,*) from_p1
        close(40)
        write(*,*) 'data read from p1:',from_p1
        
        call system('del D:p1_p2.txt')
        write(*,*) 'file deleted.'
        write(*,*) ' '
             
    end do

    end program p2[/bash]

It gives me the same kind of errors like the version before.

[mamey4]

Mh I guess that should look something like this:

[bash]program p2

    implicit none
    integer i,sum,ioerror,from_p1
    
    sum=1000   
    
    do i=1,50
    
        sum=sum-1
        
        2 open(40, file='D:/p2_p1.txt', status='OLD', iostat=ioerror)
        if(ioerror.eq.0) then
            close(40)
            goto 2
        end if
        
        open(40, file='D:/p2_p1.txt', status='NEW', iostat=ioerror)
        write(40,*) sum
        close(40)
        write(*,*) 'data sent.'
        
        !Signalfile
        open(40, file='D:/p1_to_delete.txt', status='NEW', iostat=ioerror)
        write(40,*) 'signal'
        close(40)
        write(*,*) 'signal file created.'
        
        !wait for Signalfile
        3 open(40, file='D:/p2_to_delete.txt', status='OLD', iostat=ioerror)
        if(ioerror.ne.0) goto 3
        close(40)
        call system('del D:p2_to_delete.txt')
        
        1 open(40, file='D:/p1_p2.txt', status='OLD', iostat=ioerror)
        if(ioerror.ne.0) goto 1
        read(40,*) from_p1
        close(40)
        write(*,*) 'data read from p1:',from_p1
        
        call system('del D:p1_p2.txt')
        write(*,*) 'file deleted.'
        write(*,*) ' '
             
    end do

    end program p2[/bash]

It gives me the same kind of errors like the version before.

[GVautier]

in P2

Create stop-p1.txt (to prevent p1 to start)
wait until stop-p2.txt exists
read p2-p1.txt
delete p2-p1.txt
write p1-p2.txt
delete stop-p1.txt

in P1

Create stop-p2.txt (to prevent p2 to start)
wait until stop-p1.txt exists
read p1-p2.txt
delete p1-p2.txt
write p2-p1.txt
delete stop-p2.txt

I think that in that configuration p1 and p2 cannot execute at the same time

[bmchenry]

sharing of allocatable arrays may be an issue unless you know the max possible allocation you would need
i find allocate/dealloc can be a major time delay in code so...

1) Icreatearrays of type and maximum size need in a dll as static memory (memory is fast and cheap!)

2) I set a variable to the max size and aflag to catch the situation when it wants larger than the largest size set and halt (or useallocate/dealloc for the rare situation of exceeding the max, if you want to handle all instances)

3) then you only use the portion of the arrays you need

It willspeed up your code 1st because you aren't using file sharing and 2nd because you aren't alloc/dealloc for every new dataset exchange

obviously the need to estimate the max array may be an issue.
just another possible approach

brian

[mfinnis]

This works for me:

[fortran]  program p2  

  use ifport    

  implicit none 
   
  integer i,sum,received,iostat
  logical :: there
    
  sum = 1001     
    
  do  i = 1,50000
      sum = sum - 1  
        
! Wait until file doesn't exist any more
      do
          inquire(file = 'p2_p1.txt',exist = there)
          if (.not.there) exit
      end do
      call SleepQQ(1)

      do
          open(40,file = 'p2_p1.txt',status = 'new',iostat = iostat)
          if (iostat == 0) exit
      end do

! Write new data
      write(40,*) sum
      close(40)
      write(*,*) 'data sent'
                
      do
          inquire(file = 'p1_p2.txt',exist = there)
          if (there) exit
      end do
      call SleepQQ(2)
  
! Read new data and delete
      open(40,file = 'p1_p2.txt',status = 'old')
      read(40,*) received  
      close(40,status = 'delete')  
      write(*,*) 'data read from p1:',received 
  end do  
    
  end program p2  
[/fortran]

I had to tweak the parameters to SleepQQ to get it 100% reliable on my machine (note the loop count). Depending on how much file opening you need to do and how much time you can afford to take over it you could up the values or have them as input data for tweaking 'in the field'. Maybe not the most efficient way of communicating between processes and/or machines over a network but nice and simple.

[mamey4]

Very nice, it works fine for me as well; thanks for that! You're right, the SleepQQ value maybe has to be adjusted for each machine, maybe it also depends on how many other programs are running simultaneously when the two communicating programs are executed - but I'll figure that out.