3.14.2.2 - #216
Merged
Merged
Conversation
PalNilsson
commented
Sep 3, 2026
PalNilsson
commented
Collaborator
- Improved payload setup verification
- A payload setup verification that hangs and is timed out by the pilot is now reported with the new error code 1390 (SETUPTIMEDOUT, "Payload setup verification timed out") and diagnostics naming the time-out, instead of the generic 1110 ("Failed during payload setup"). Previously the pilot discarded the in-memory stderr holding the time-out reason whenever the on-disk setup.stderr was empty, substituted a placeholder that was itself one of the apptainer/singularity stderr patterns, and consequently reclassified the time-out and logged a spurious "found apptainer error in stderr" warning twice, since a reverse look-up of the error-pattern maps by exit code also fabricated a match for an empty payload.stderr. That reverse look-up is removed, and an exit code that is already a pilot error code is no longer overwritten with the generic PAYLOADEXECUTIONFAILURE. The two setup.stdout/setup.stderr file objects leaked on the setup-failure return path are now closed
- Issue spotted on UNI-SIEGEN-HEP which currently is suffering from setup time-outs.
- Job monitoring loop counter
- The job monitor loop counter now restarts at 1 for each job of a multi-job pilot rather than continuing to count for the lifetime of the pilot, and the internal job state is set to starting when the payload executor is entered, so the monitor no longer reports an empty state while the setup is being verified.
- Looping job detection
- Fixed a defect that prevented looping job detection (error code 1150) entirely. The diagnostic snapshot series added with the looping process dumps is written into the job work directory, which is the same directory the looping algorithm scans for payload activity, and no experiment plugin filtered it out. Because the series starts at half the looping limit and is written at every looping verification, each snapshot reset the time since the last file touch, so that time could never reach the looping limit and no payload could ever be identified as looping. The artifacts written by the diagnostics (snapshot series, core files and their analysis companions) are now excluded centrally from the file list the algorithm measures, and the snapshot file's modification time is pinned to the payload's own last touch so that it cannot appear as the newest file in the work directory under any circumstances. In addition, the looping error code and the failed job state are now set before the core dump and the kill rather than after them, so that a failing diagnostic can no longer cancel a detection, and the looping check gate in the job monitoring now requires both the job's loopingCheck attribute and the experiment plugin to allow the check.
…DOUT) A setup verification that hung and was killed by execute() after 600 s was reported as SETUPFAILURE (1110) with the diagnostics "General payload setup verification error (check setup logs)", accompanied by a misleading "found apptainer error in stderr" warning that was printed twice (job 7291003889, UNI-SIEGEN-HEP, 2026-09-02). Three separate defects compounded: - run() rebuilt the diagnostics only when *both* stdout and stderr were non-empty, and re-read both files unconditionally. On a time-out execute() returns an empty stdout with the reason for the kill in stderr, while the container had written everything to setup.stdout, leaving setup.stderr empty - so the re-read overwrote the only description of the failure. The collection is now factored out into collect_setup_diagnostics(), which reads a stream from file only when it is missing from memory. - The resulting placeholder text was itself a key in the error_map of resolve_transform_error(), so the pilot pattern-matched its own placeholder and reclassified COMMANDTIMEDOUT (1367) to SETUPFAILURE. The entry is removed: it never was an apptainer/singularity message. - resolve_transform_error() reverse-looked-up the pattern maps by exit code and returned the matching key as the "error message found in stderr", fabricating one for any exit code that happened to equal a mapped pilot error code. That produced the duplicate warning from perform_initial_payload_error_analysis() on an empty payload.stderr and downgraded 1110 to PAYLOADEXECUTIONFAILURE. The lookup is removed, and an exit code that is already a pilot error code is now passed through. A timed-out setup verification is reported as the new SETUPTIMEDOUT (1390) with the time-out duration preserved in the diagnostics. The setup.stdout/ setup.stderr file objects are also closed before the failure return, which used to leak both. The same diagnostics defect is fixed in the rubin event-service copy. Two unrelated job-monitor issues from the same log: the monitor loop counter was a single counter for the lifetime of the job_monitor thread (so it did not restart for the second job of a multi-job pilot, and advanced even while no job was being monitored) and is now keyed by job id; and job.state stayed empty from job acquisition until run_payload(), since send_state() only sets job.serverstate and a job without input files never gets the 'stagein' state - execute_payloads() now sets the internal 'starting' state as well.
The looping algorithm decides whether a payload is alive from the modification time of the most recently modified file in the job work directory. The diagnostic snapshot series introduced with the looping dumps is written into that same directory as looping_snapshots.log, and is not excluded by remove_unwanted_files() in any experiment plugin, so the pilot's own write looked exactly like payload activity. Since the series starts at looping_snapshot_fraction (0.5) of the looping limit and is written at every looping verification, the time since the last file touch was pinned at roughly half the limit plus one verification interval - it could never reach the limit. Detection was not degraded but impossible, which matches the observation that the new pilot reported no looping job at all while the pilot it replaces reports them continuously. Two independent guards are added, since one filter that stops being applied must not be able to disable detection again: - is_looping_diagnostic_file()/remove_diagnostic_files() name every artifact the diagnostics write (the snapshot series, core.<pid> and the .analysis.txt companions), and get_time_for_last_touch() drops them from the measured file list. This is done centrally rather than in each experiment plugin because the names belong to the pilot, and because seven separate plugin filters are what failed to catch it; - store_snapshot() pins the modification time of the snapshot file to the payload's own last touch, so the file cannot be the newest file in the work directory regardless of any name based filtering. Two further defects in the same path are fixed. The error code was assigned after create_core_dump() and kill_looping_job(), inside the same try block, so an exception in either left the job running with a single logged warning and no error code; the outcome is now recorded first and each diagnostic step is guarded separately. kill_looping_job() sets the error code and the failed state before the diagnostics for the same reason. In verify_looping_job(), 'not job.looping_check and runcheck' let an experiment plugin that disallows detection be overruled and is corrected to 'not (job.looping_check and runcheck)'. kill_looping_job() and looping_job() are split into helpers to keep the McCabe complexity within the pylint limit after the added guards. Adds pilot/test/test_looping_detection.py: the reported scenario end to end, each guard tested on its own, the plugin/central filter composition for all seven experiments, the detection surviving a failing core dump or kill, and the corrected gate.
The looping algorithm decides whether a payload is alive from the modification time of the most recently modified file in the job work directory. The diagnostic snapshot series introduced with the looping dumps is written into that same directory as looping_snapshots.log, and is not excluded by remove_unwanted_files() in any experiment plugin, so the pilot's own write looked exactly like payload activity. Since the series starts at looping_snapshot_fraction (0.5) of the looping limit and is written at every looping verification, the time since the last file touch was pinned at roughly half the limit plus one verification interval - it could never reach the limit. Detection was not degraded but impossible, which matches the observation that the new pilot reported no looping job at all while the pilot it replaces reports them continuously. Two independent guards are added, since one filter that stops being applied must not be able to disable detection again: - is_looping_diagnostic_file()/remove_diagnostic_files() name every artifact the diagnostics write (the snapshot series, core.<pid> and the .analysis.txt companions), and get_time_for_last_touch() drops them from the measured file list. This is done centrally rather than in each experiment plugin because the names belong to the pilot, and because seven separate plugin filters are what failed to catch it; - store_snapshot() pins the modification time of the snapshot file to the payload's own last touch, so the file cannot be the newest file in the work directory regardless of any name based filtering. Two further defects in the same path are fixed. The error code was assigned after create_core_dump() and kill_looping_job(), inside the same try block, so an exception in either left the job running with a single logged warning and no error code; the outcome is now recorded first and each diagnostic step is guarded separately. kill_looping_job() sets the error code and the failed state before the diagnostics for the same reason. In verify_looping_job(), 'not job.looping_check and runcheck' let an experiment plugin that disallows detection be overruled and is corrected to 'not (job.looping_check and runcheck)'. kill_looping_job() and looping_job() are split into helpers to keep the McCabe complexity within the pylint limit after the added guards. Adds pilot/test/test_looping_detection.py: the reported scenario end to end, each guard tested on its own, the plugin/central filter composition for all seven experiments, the detection surviving a failing core dump or kill, and the corrected gate.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.