Found while wiring the LongMemEval runner through retrieve() (PR #204).
benchmarks/eventqa_runner.py refuses to run when the checkout does not support the requested lever (for example --graph-expansion on a build where retrieve() has no graph_expansion parameter). That refusal is correct: the alternative is silently producing two identical arms and reporting a null result that is really an instrument failure.
The problem is how it refuses. It returns from main() rather than exiting non-zero, so the process exits 0. An overnight chain that runs arms in sequence and checks $? between them reads "I refused to run" as "this arm succeeded." The chain proceeds, and the missing arm shows up later as an absent or empty result file rather than as a failure at the point it happened.
This is the same fails-open shape as #197: the error path is present and correct, and then the signal that the error occurred is thrown away.
Fix
sys.exit(1) on refusal, as PR #204 does for the LongMemEval runner. Worth also asserting the abort in the EventQA tests, which currently do not cover it: non-zero exit, dataset not opened, and a supported checkout not aborting.
Related and worth checking at the same time
Whether any other runner under benchmarks/ refuses or errors without a non-zero exit. The pattern is easy to repeat and invisible until a chain silently skips an arm.
Found while wiring the LongMemEval runner through
retrieve()(PR #204).benchmarks/eventqa_runner.pyrefuses to run when the checkout does not support the requested lever (for example--graph-expansionon a build whereretrieve()has nograph_expansionparameter). That refusal is correct: the alternative is silently producing two identical arms and reporting a null result that is really an instrument failure.The problem is how it refuses. It returns from
main()rather than exiting non-zero, so the process exits 0. An overnight chain that runs arms in sequence and checks$?between them reads "I refused to run" as "this arm succeeded." The chain proceeds, and the missing arm shows up later as an absent or empty result file rather than as a failure at the point it happened.This is the same fails-open shape as #197: the error path is present and correct, and then the signal that the error occurred is thrown away.
Fix
sys.exit(1)on refusal, as PR #204 does for the LongMemEval runner. Worth also asserting the abort in the EventQA tests, which currently do not cover it: non-zero exit, dataset not opened, and a supported checkout not aborting.Related and worth checking at the same time
Whether any other runner under
benchmarks/refuses or errors without a non-zero exit. The pattern is easy to repeat and invisible until a chain silently skips an arm.