Max runtime for k8s job

[Steboss]

This PR is a simple test:

• make sure each k8s job can stop after 3 hours
• in this way we're preventing clogging EKS queue
• and avoiding having ghcr secret deleted

[Steboss]

@olupton I'd like to have your point of view on how to work out a solution here.
The idea is that each job has a 3 hours of max execution time, in order to get its secret and K8s job correctly deleted, without stopping all the other jobs. Ideally we should lower this execution time to 1 hour, but it may be not enough for some job we're running.

[olupton]

I think the runtime limit can be added to .github/eks-workflow-files/mpi-nccl-test.yml too.

This seems like an improvement in principle. Did you do any experiments to make sure the error case handling actually works (e.g. run a job with the limit set to a tiny value)?

3 hours is a lot, but we might have to tune the value down in a second pass.

[Steboss]

@olupton
I did actually let the job to run, to see if everything worked correctly, and it looks like the AXLearn job that was hanging was correctly deleted, while all the other jobs managed to run without having errors in the ghcr secret deletion.