Graceful exit from hpc #12
Description
If a training hits the hpc time limit it is terminated and data is not saved. Need to have some saving of parameters at checkpointing to be able to restart the training.
{{ message }}
If a training hits the hpc time limit it is terminated and data is not saved. Need to have some saving of parameters at checkpointing to be able to restart the training.