IFORT and CNTRL/C

[jscott]

I often run MPI programs which do a lot of scientific calculations, then write data to files, followed by many subsequent repeats of this cycle. For different reasons, I sometimes abort these programs using CNTRL/C. They are set up so they can restart the calculation where it left off, but I am worried about the case in which the CNTRL/C occurs in the middle of a write. I have not, as yet, experienced this, but would like to know if there are ways to avert this problem.

[andrew_4619]

One way would be to embed a quit routine in your main loop so it gets called regularly.  That would have a peekcharqq to check the input buffer and if there is content do a getcharqq. So you would designate a quit character e.g. "Q" and if you get it do something sensible like ask user if they really want to quit, flush the file buffers (if you are doing buffered i/o), set a short delay maybe, and then do a normal exit. Other methods exist....

[jscott]

Thank you for your suggestion, which I tried out on one of my programs. Sadly, PEEK/GETCARQQ did not work: the program did not terminate or show any signs of recognising the given character, which may be due to it running on multiple processes using MPI.  I have not tried it without MPI because that is not a configuration of interest to me.

You mention other methods... Please tell.

Finally, I am surprised that it does not seem possible to create a CNTRL/C handler, such as one might do using C.

[Ron_Green]

MPI complicates the matter.  Any input to stdin goes only to rank 0, not the other ranks in the application.  And all stdin first goes through your rank 0 launcher and process control daemon.  The MPI process daemon for rank 0 first gets the Ctrl-C.  So it will probably at this point begin to tear down your application - the Ctrl-C never gets to your Rank 0.  So writing a handler just isn't going to work.

In any event, use your cluster resource manager/scheduler to abort your application.  It's much better than anything you can write to handle Ctrl-C, nothing personal.  Run your program under batch, not interactive.  If you determine you need to kill the app, use the resource manager to do that. Most resource managers will 'do the right thing' to shutdown and clean up your app.  That includes letting any in-flight MPI transfers complete AND it will flush any IO buffers before it kills off your ranks.  This is why I say you simply cannot do better with clean up than your resource manager. 

If you are not using a resource manager to allocate nodes and launch your app, and you are running MPI raw with 'mpirun', Ctrl-C should get captured by the MPI daemons and they'll also clean up all the ranks, waiting for in-flight MPI to complete, etc.  Again, don't worry about it.

Have you seen a case where you only get partial writes?  Perhaps you are worrying unnecessarily.  You can use calls to FLUSH or the FLUSH statement after your writes.  This usually prevents any partial writes. 

[hakostra1 · Accepted solution]

Disclaimer: I develop exclusively for Linux, and if you are on Windows things might work differently.

You can implement signal catching in your program. Both GNU and Intel Fortran compilers have a separate implementation that works sufficiently similar. My code is something like:

received_signal = 0  ! This is a variable inside a module

! Done once to set up the signal handler, in some kind of "init" routine
#if defined __INTEL_COMPILER
ierr = SIGNAL(signum(i), sighandler, -1)
#endif
#if defined __GFORTRAN__
CALL SIGNAL(signum(i), sighandler, ierr)
#endif

! Function within the same module as received_signal
INTEGER(int32) FUNCTION sighandler(sig)
    INTEGER(kind=int32) :: sig
    received_signal = sig
    sighandler = 0
END FUNCTION sighandler

Then you can check the value of "received_signal" to check if a signal has been received. I would generally do an MPI_Allreduce on the received_signal to make sure your program stays in sync when only one MPI rank receive the signal.

The "signum" is the signal to catch, and you can do it for several sinals if you like. Examples: SIGINT = 2, SIGUSR1=10, SIGXCPU=24

With for instance slurm you can give sbatch the argument "--signal=USR1@300" to have SIGUSR1 sent 5 minutes before the job ends (allowing your program to gracefully finish before the walltime is out), or you can send an arbitrary signal to the job with "scancel --signal=USR1", this will just send SIGUSR1 to the job.

If you set a signal handler for SIGINT that will catch the Ctrl+C and make the program shut down in a nice way when you kill it. A second Ctrl+C will usually the kill the program instantly...

Ref:

https://slurm.schedmd.com/sbatch.html

https://slurm.schedmd.com/scancel.html

[jscott]

Thanks,

I have not yet tried your suggestion. However, it seems to disagree with Ron_Green, who said "So writing a handler just isn't going to work.". That was because of MPI. Have you actually used this approach with MPI?

[hakostra1]

I use this with PBS and Slurm. The command for sending a signal to the job in PBS is "qsig" if you want to look up the manpage.

The crux is to check for signals every now and then (e.g. every timestep, iteration or whatever you are doing) and then do an MPI_Allreduce on the value, then you know your processes are in sync before doing any further actions. Then it does not matter which process receive the signal also.

[jscott]

Thanks,

Julian