triage-tool: support multi-node triage

[olupton]

• Add --{passing,failing}-commits option, which allows the commit-level search to be manually restricted in scope
  • This also allows single-container triage, as you can pass --{passing,failing}-container and --{failing,passing}-commits
• With the Pyxis backend, re-use the same container instance within a single triage tool process
  • This reduces the amount of time spent in container creation
• Support multi-node/multi-process triage with the Pyxis backend
  • This is implemented by annotating the various commands run inside the containers as once (run once, in one container instance), once_per_container (run once per container instance, i.e. once per node), and default (run without extra srun arguments -- the caller must make this do the right thing e.g. by passing --ntasks-per-node to salloc.
    • once example: getting the JAX commit from a container
    • once_per_container example: building JAX
    • default example: running the test case

[gpupuck]

Is there anything else needs to be addressed?

[gpupuck]

This has been tested in practice, let's ship it!