How to auto-resume torchrun multi-node training after hitting SLURM time limit?

[amorehead]

Hello. I have recently been trying to set up fault-tolerant (and time limit-tolerant) multi-node training on a SLURM script I have access to. I can successfully train a model using 2 nodes and 4 GPUs each node. However, when my SLURM job times out (e.g., after 1 hour), my SLURM job does not get resubmitted automatically (like would happen when using the SLURMEnvironment) nor does my rendezvous node automatically restart the timed out workers. My question is, what is the standard process for setting up auto-restarts with torchrun and PyTorch Lightning on a SLURM cluster?