Intel® Fortran Compiler
Build applications that can scale for the future with optimized code designed for Intel® Xeon® and compatible processors.

Interprocess communication

Don_D_
New Contributor I
828 Views

My program spawns child processes.  I need to tell them when to start have them tell me when they are finished.  I am trying to do this with disk files, writing and reading certain values that encode the information.  Problem is, often one process wants to read while another is still writing.  I have tried the INQUIRE function, to see if the file is currently open, and waiting until it is not before doing any more IO on it.

But this often fails with a file read or write error.  Is there any elegant way for a child to communicate with a parent, and vice-versa?  These are separate processes, not threads.

0 Kudos
1 Solution
Don_D_
New Contributor I
828 Views

The most elegant approach seems to be to use the common data array to store an array of flags that tell the other processes when to start and where they post a flag saying they are done.  So I tried this method, and it works well.  No file writing and reading; no collisions.

View solution in original post

0 Kudos
9 Replies
Steve_Lionel
Honored Contributor III
828 Views

See the DLL_Shared_Data sample in the Intel Parallel Studio XE for Windows Sample Bundle. But if the spawned programs exit when done, you can just do a WaitForSingleObject on the process.

0 Kudos
jimdempseyatthecove
Honored Contributor III
828 Views

In addition to the DLL_Shared_Data method, suitable only for Windows .AND. on single CPU or SMP (multi-socket/CPU in single system), you might consider investigating using MPI (Intel's or OpenMPI). This is a inter-process message passing facility that can run on single CPU or SMP (multi-socket/CPU in single system), or cluster, or network. As well as mix of Windows and Linux.

Note that most of the initial description of MPI relates to running the same executable on each process...
... this is not a requirement. You can run different executables in each process if you so desire.

Note, while the setup of MPI (learning curve) is cumbersome at first, the flexability of your runtime configuration is (may be) worth your effort. For example: from your office notebook or workstation, you can launch your application that runs the presentation portion(process) of your application on your office notebook or workstation and computational portions(processes) on remaining capacity of your office notebook or workstation together with that of an available cluster (or second workstation in your office).

Jim Dempsey

0 Kudos
jimdempseyatthecove
Honored Contributor III
828 Views

Second option for file sharing.

Background: One of the older networking standards was called Token-Ring (https://en.wikipedia.org/wiki/Token_ring).

You could implement this using files. Whereby the "token" is a signaling file that contains no data, rather it is a signaling means as to which process in your case has ownership of the data file or files. The ownership is held for as long as the token holder has the token. When the token holder is finished, it passes the token on to another process. You would decide which process this is to be. In a ring setup, it would be simply the next process number with wrap at end. You can also set this up in star format. The application holding the token can do whatever it wants with the data file(s).

As an example. In the folder that contains your data file, you could create a folder named Token. Within the Token folder, you (initially) create a file, who's file name is the holder of the token. Assume you have process MAIN, A, B, C. Initially you delete all files in folder Token (unless the only file is MAIN), then create file MAIN. Next start your other processes. The other processes, at initialization time wait for file MAIN to exist in folder Token. After the MAIN process completes initialization it deletes Token\MAIN, then creates Token\A. Process A observes that it holds the token, gets its work information, performs any initial shared file work, then deletes Token\A and then creates Token\... where ... could be MAIN or B or C. In star arrangement it would be MAIN, in ring it would be B.

Note, you can also have an aborting process create Token\ABORT to alert other processes to shut down (or Token\CONVERGED, etc...)

Jim Dempsey

0 Kudos
jimdempseyatthecove
Honored Contributor III
828 Views

It should be obvious that I am not suggesting you use the hardware of the Token-Ring setup, rather to use a token passing concept via use of file name.

Also, not described. The MAIN process  (or any/all processes for that matter) could contain a timeout capability. IOW should any single process hold the token past the timeout, then it is assumed that process crashed or is in deadlock. You would determine the timeout and the corrective action (e.g. terminate application or kill and restart failing process).

Jim Dempsey

0 Kudos
jimdempseyatthecove
Honored Contributor III
828 Views

If the file access patterns is non-deterministic, then you could have an additional folder in the data file folder named TokenWanted. When a process desires file access it creates its process name inside TokenWanted. When the Token holder enters its release token function, it chooses a process from the TokenWanted folder (deleting the chosen file), then creates that file in (or moves the file to) the Token folder.

In the event that your MAIN process has significant work to do (iow no process is in need of holding the token) then consider adding a monitoring process that periodically monitors the Token and TokenWanted folders. When Token folder empty, it selects and moves file (if present) from TokenWanted folder to Token folder. The token owner need then only delete token file from the Token folder when done with the files. When no move is necessary, the monitoring thread/process sleeps a bit.

Jim Dempsey

0 Kudos
Don_D_
New Contributor I
828 Views

I see many cogent suggestions above, some suggesting that the common data array can contain flags that start and stop processes.  I thought of this, but at the moment want to pursue a different method, rather than manage 128 flags, one for each process, all with the same name.

So I'll try to avoid file errors with exit routes all over the place, and if that works, I'm all set.  If not, I'll educate myself about those other suggestions.  Many thanks to the contributors.

0 Kudos
jimdempseyatthecove
Honored Contributor III
828 Views

>>each process, all with the same name

While the executable may have the same name, the process can have any internal name. You likely have one already. This could be the System Process ID (aka PID), or launch sequence, or system tick time of launch, or arg on command line when launched, etc....

The following is quickie batch hack:

: Interlockutor.bat
mkdir Token
echo y | del Token\*.*
mkdir TokenWanted
echo y | del TokenWanted\*.*
:Loop
: Add delay using ping to known non-existant (local) IP address
: vary the -w arg to vary wait time. this is in approximately units of ms
ping 192.168.199.199 -n 1 -w 100 > nul
echo first for
for %%x in (Token\*) do ( echo %%x has token && goto loop )
echo second for
for %%x in (TokenWanted\*) do ( echo %%x wants token && move %%x Token && goto loop)
echo no for
echo No token holder
goto loop

Remove the undesired ECHO commands, add ECHO OFF at top.

Also, just prior to :loop, you could insert a series of:

START YourProgramNameHere

for the number of instances (consider adding an argument for instance number and then using

FOR /L%%I IN(1,23,1) DO START YourProgramNameHere %%I

That will launch 23 instances of your YourProgramNameHere with 1, 2, ... 23 as 1st argument.

Jim Dempsey

0 Kudos
FortranFan
Honored Contributor II
828 Views

Don D. wrote:

..  Is there any elegant way for a child to communicate with a parent, and vice-versa? ..

In the context of Fortran, a forceful argument is nothing quite surpasses the 'elegance' offered by standard-conforming code which then works 'effectively' the same with all standard-conforming processors!

if one then buys the above argument, the use of the Fortran standard facility toward COARRAYs is definitely worth a consideration:

program p

   implicit none

   integer :: i

   ! Simple array-like syntax to manage program data
   real :: r
  • ! Let each 'image' (think of it as a process) do its work r = real( this_image(), kind=kind(r) ) ! Simple synchronization when each 'process' is finished sync all ! Do the post-processing if ( this_image() == 1 ) then print *, "On image 1: r = ", (r, i = 1, num_images() ) print *, "Expected is ", (i, i=1, num_images() ) end if stop end program
  • C:\Temp>type p.f90
    program p
    
       implicit none
    
       integer :: i
    
       ! Simple array-like syntax to manage program data
       real :: r
  • ! Let each 'image' (think of it as a process) do its work r = real( this_image(), kind=kind(r) ) ! Simple synchronization when each 'process' is finished sync all ! Do the post-processing if ( this_image() == 1 ) then print *, "On image 1: r = ", (r, i = 1, num_images() ) print *, "Expected is ", (i, i=1, num_images() ) end if stop end program C:\Temp>ifort /Qcoarray:shared /Qcoarray-num-images=4 p.f90 Intel(R) Visual Fortran Intel(R) 64 Compiler for applications running on Intel(R) 64, Version 19.1.0.085 Pre-Release Beta Build 20190522 Copyright (C) 1985-2019 Intel Corporation. All rights reserved. Microsoft (R) Incremental Linker Version 14.21.27702.2 Copyright (C) Microsoft Corporation. All rights reserved. -out:p.exe -subsystem:console p.obj C:\Temp>p.exe On image 1: r = 1.000000 2.000000 3.000000 4.000000 Expected is 1 2 3 4 C:\Temp>
  •  

    0 Kudos
    Don_D_
    New Contributor I
    829 Views

    The most elegant approach seems to be to use the common data array to store an array of flags that tell the other processes when to start and where they post a flag saying they are done.  So I tried this method, and it works well.  No file writing and reading; no collisions.

    0 Kudos
    Reply