Skip to content

Commit

Permalink
Update signal_handling.h
Browse files Browse the repository at this point in the history
Edited comments.
  • Loading branch information
ugermann committed Aug 4, 2020
1 parent 0a406d8 commit 521560b
Showing 1 changed file with 7 additions and 8 deletions.
15 changes: 7 additions & 8 deletions src/common/signal_handling.h
Original file line number Diff line number Diff line change
Expand Up @@ -3,25 +3,24 @@

// SIGNAL HANDLING

// The Marian signal handlers set global flags that thread can
// The Marian signal handlers set global flags that threads can
// consider when a signal is received. This can be used for a graceful
// shutdown instead of a hard abandonment, e.g. after receiving
// SIGTERM during training.

// When SIGTERM is received, the global (static member) flag sigterm_
// (false by default) is set to true by signalHandler(). When sigterm_
// is true, keepGoing() returns false, and the current state of
// training models is saved prior to exiting. This functionality is
// helpful when training on clusters with time limits on compute
// When SIGTERM is received, the global bit flag for SIGTERM
// (false by default) is set to true by setSignalFlag. Threads involved in training
// (batch generator, scheduler) periodicly check this flag and gracefully exit;
// the current state of training models is then saved prior to exiting the program.
// This functionality is helpful when training on clusters with time limits on compute
// slots, e.g., on s clusters managed by slurm. Slurm can be asked to
// sending a (custom) warning signal to a process at a given point in
// time prior to the hard "time's up".
//
// Correspondingly, fetchBatches in the batch generator checks the flag
// frequently and quits after the overall process receives a SIGTERM.


namespace marian {
bool getSignalFlag(int sig); // return true if sig was received, false otherwise
void setSignalFlag(int sig); // set custom handler (set flag) for sig
void setSignalFlag(int sig); // custom handler (set flag) for sig
}

0 comments on commit 521560b

Please sign in to comment.