[safe-output-health] Safe Output Health Report - 2026-08-22

[github-actions[bot]]

🏥 Safe Output Health Report — 2026-08-22

Executive Summary

• Period: Last 24 hours (2026-08-21 ~04:30 UTC → 2026-08-22 ~04:20 UTC)
• Coverage: 841 workflow runs analyzed (~23 of 24 hours; a ~1h05m gap from 04:29 to 05:34 UTC on 2026-08-21 was not fetched due to log volume — this repo runs an unusually large number of scheduled agentic workflows)
• Runs that attempted a safe output: 364
• Safe output job failures: 2 (both in the same workflow)
• Safe output job success rate: 99.45%
• Error clusters identified: 1

This audit is scoped strictly to safe-output job health (the job/step that processes and writes agent-declared outputs back to GitHub — comments, issues, PRs, etc.), per this monitor's mandate. Agent-job and detection-job failures are explicitly out of scope and are tracked by other monitors; 51 non-safe-output failures were observed in the sampled window (Code Simplifier, Design Decision Gate, Smoke Codex, Code Scanning Fixer, Daily Observability Report, Issue Arborist ×3, Agent Job Health Monitor, Daily BYOK Ollama Test, Duplicate Code Detector, AI Moderator, Agentic Workflow Audit Agent, and others) — all of these had safe_items_count = 0, meaning no safe-output write was ever attempted, so they are agent/detection failures, not safe-output failures, and are excluded from the analysis below.

Architecture note: github/gh-aw does not run separate create_issue / add_comment / create_pull_request jobs as isolated GitHub Actions jobs. Each compiled workflow has a single consolidated safe_outputs job that processes every safe-output type declared by that workflow (add-comment, create-issue, approve-workflow-run, resolve-pull-request-review-thread, etc.) in one Process Safe Outputs step. So the statistics below are reported per-run/per-workflow rather than per-output-type, since output-type-level pass/fail isn't independently observable from the data this monitor can access.

Safe Output Job Statistics

Scope	Total	Failures	Success Rate
All runs with ≥1 safe output attempted	364	2	99.45%
PR Sous Chef only (8h sample window)	24	2	91.7%
All other workflows with safe outputs	340	0	100%

Error Clusters

Cluster 1: safe_outputs job fails at "Process Safe Outputs" step — isolated to PR Sous Chef

• Count: 2 occurrences (both observed instances of this workflow failing in the sample)
• Affected job: safe_outputs (the consolidated safe-output processing job)
• Affected workflow: PR Sous Chef only — no other workflow showed this pattern
• Affected runs:
  • §32543132845 — 2026-08-22 01:20 UTC, 9 safe items declared, job failed after 1.4m
  • §32526302837 — 2026-08-21 20:59 UTC, 1 safe item declared, job failed after 23.0s
• Sample structural evidence (from job/step conclusions, both runs identical shape):

  job: safe_outputs
    Download agent output artifact ....... success
    Setup agent output environment ....... success
    Download patch artifact .............. success
    Process Safe Outputs ................. FAILURE   <-- fails here
    Upload Safe Outputs Items ............ success   <-- still runs (already-valid items get uploaded)
  

• Root cause: Not directly determinable — the audit/logs MCP tools expose job/step conclusions and aggregate metrics but no artifact set maps to the raw GitHub Actions console stderr for the safe_outputs job, so the exact validation error text could not be retrieved in this session. However:
  • PR Sous Chef declares an unusually wide set of safe-output types in a single run: add-comment, approve-workflow-run (with a strict cjs.yml/cgo.yml/CWI.yml allowlist plus a dynamic eligible-PR-number allowlist), resolve-pull-request-review-thread (requires exact PRRT_... node IDs), dismiss-pull-request-review, update-pull-request, push-to-pull-request-branch, and create-issue.
  • The workflow's own prompt (.github/workflows/pr-sous-chef.md) already contains multiple explicit, emphatic warnings to the agent about exactly the failure modes that would break this step — e.g. "never pass a review comment node ID (PRRC_...)", "Every add_comment must include pr_number" (stated twice), "never fetch [reviewThreads] via gh pr view". These read as prior patches for recurring validation failures, suggesting the underlying issue is that a single malformed safe-output item (e.g., a bad thread ID format, a missing required field, or an out-of-allowlist workflow run) aborts the entire consolidated Process Safe Outputs step rather than being rejected/skipped per-item while valid items still go through.
  • This is consistent with Upload Safe Outputs Items still succeeding after the failure — the items that were validated got uploaded as artifacts, but something in server-side processing of the batch (likely one bad item) caused the step to exit non-zero.
• Impact: Low blast radius — PR Sous Chef runs every 15 minutes, so a failed run simply delays that cycle's nudges/approvals by ~15 min until the next successful run. No data loss observed. But it is a real, repeatable ~8% failure rate for this specific workflow, worth root-causing since it's one of the more actuation-heavy workflows in the fleet (comments, branch pushes, workflow-run approvals).

Root Cause Analysis

API-Related Issues

None observed. Firewall/network telemetry for both failing runs showed 0% blocked requests; GitHub API rate limits were nowhere near exhausted (14,914–12,941 of 15,000 core requests remaining at failure time).

Data Validation Issues

The one confirmed cluster (above) is best explained as a data-validation issue: one or more items in a large, multi-type safe-output batch likely fails server-side schema/allowlist validation, and that failure appears to blow up the whole Process Safe Outputs step instead of degrading gracefully.

Permission Issues

None observed in this window.

Network Errors

None observed in this window (0 blocked requests, no timeout indicators in either failing run's audit).

Other Issues

None — no missing_tool, no missing_data, no parsing errors were recorded in the aggregate metrics (total_missing_tools: 0, total_missing_data: 0 across both batches).

Recommendations

Critical Issues (Immediate Action Required)

None. The observed failure rate (2/364 safe-output attempts, 0.55%) is low and self-recovers on the next scheduled run.

Bug Fixes Required

1. Make Process Safe Outputs degrade per-item instead of failing the whole batch, for PR Sous Chef-style multi-type runs

   • File/Location: The safe-outputs processing logic shared by all compiled workflows (safe_outputs job / Process Safe Outputs step — likely in the gh-aw compiler's generated safe-output handler, not in pr-sous-chef.md itself, since this step is common infrastructure).
   • Problem: When a workflow declares many safe-output types in one run (as PR Sous Chef does — 7 types), a single invalid item appears to fail the entire step, even though Upload Safe Outputs Items afterward shows valid items were already captured.
   • Fix: Investigate whether Process Safe Outputs can catch and log a per-item validation failure (e.g., a malformed resolve-pull-request-review-thread ID, an add-comment missing pr_number, or an approve-workflow-run outside the allowlist) and continue processing remaining valid items, only failing the step for genuinely fatal errors (auth, API outage). This would turn workflow-level failures into partial-success logs, which is both more informative and reduces false "workflow failed" signals.
   • Affected jobs: safe_outputs (all workflows using multiple safe-output types are theoretically at risk, but only PR Sous Chef manifested it in this window — likely because of its unusually high type/volume combination).

2. Surface the actual validation error text for safe_outputs job failures in a place the audit tooling can read

   • Problem: This monitor could not retrieve the specific error string for either PR Sous Chef failure — the agenticworkflows audit/logs MCP tools only expose job/step conclusions, not raw step console output, and none of the supported artifacts sets (activation, agent, all, detection, evals, experiment, firewall, github-api, mcp, usage) map to the safe_outputs job's own logs.
   • Fix: Either add a safe_outputs-scoped artifact set to the audit/logs MCP tool, or have the Process Safe Outputs step write a structured error summary (which item failed, which safe-output type, why) to the job's step summary / an uploaded artifact, so future health audits (and maintainers) can see root cause without needing raw Actions log access.

Configuration Changes

None required at this time — no rate-limit, permission, or firewall configuration issues were found.

Process Improvements

1. Add safe-output-type-level success telemetry
   • Current State: safe_items_count and job conclusion are the only per-run signals available; there is no breakdown by output type (e.g., how many add-comment vs resolve-pull-request-review-thread items succeeded/failed within a run).
   • Proposed: Emit a per-type count (attempted/succeeded/failed) in the run summary, similar to the per-job breakdown already used elsewhere in this report's data model.
   • Benefits: Would have let this audit directly confirm which of PR Sous Chef's 7 safe-output types caused the failures, instead of relying on prompt-text inference.

Work Item Plans

Work Item 1: Root-cause and fix Process Safe Outputs step failures in PR Sous Chef

• Type: Bug Fix / Investigation
• Priority: Medium
• Description: Two runs of PR Sous Chef (§32543132845, §32526302837) failed at the Process Safe Outputs step with no other observable cause (no rate limiting, no firewall blocks, no missing tools/data). Given Upload Safe Outputs Items still succeeds afterward, the failure likely originates in validating one of the several safe-output types this workflow emits per run.
• Acceptance Criteria:
  • Raw logs/error text for a reproduction of this failure are captured (may require enabling verbose logging or reproducing locally with a crafted invalid safe-output payload for each of PR Sous Chef's 7 declared types).
  • Root cause identified and documented.
  • Either the offending validation is fixed, or the processing step is made to degrade per-item so one bad item doesn't fail the whole batch.
• Technical Approach: Start by testing each of PR Sous Chef's safe-output types in isolation against the compiler's safe-output schema validator, focusing on resolve-pull-request-review-thread (strict PRRT_ ID format) and approve-workflow-run (dynamic allowlist substitution from needs.approval_allowlist.outputs.eligible_pull_request_numbers), since those are the two types with the most complex/fragile input contracts per the workflow's own defensive prompt text.
• Estimated Effort: Medium
• Dependencies: Access to raw Actions job logs for the two failing runs (not available via current audit tooling — see Work Item 2).

Work Item 2: Extend audit/logs tooling to expose safe_outputs job raw logs

• Type: Enhancement
• Priority: Low
• Description: The agenticworkflows audit and logs MCP tools currently support artifact sets activation, agent, all, detection, evals, experiment, firewall, github-api, mcp, usage — none of which surface the safe_outputs job's own console output or the specific safe-output validation error. This blocked deeper root-causing in this audit.
• Acceptance Criteria:
  • A safe_outputs (or equivalent) artifact set is added, or the audit report's jobs[] array includes a steps[].error_excerpt field for failed steps.
• Technical Approach: Extend whatever service backs the agenticworkflows MCP bridge to fetch and truncate the failed step's log excerpt (similar to how GitHub's own Actions UI shows the last N lines of a failed step).
• Estimated Effort: Small–Medium
• Dependencies: None.

Historical Context

No prior safe-output health audits were found in cache memory (/tmp/gh-aw/cache-memory/safe-output-health/ was empty before this run) — this is the first audit of this kind. Today's findings have been persisted to 2026-08-22.json and index.json in that directory to enable trend comparison on the next run.

Trends

• Error rate trend: N/A (first audit, no baseline)
• Most common recurring issue: safe_outputs job failure in PR Sous Chef (2 occurrences, only workflow affected)
• Improvement since last audit: N/A

Metrics and KPIs

• Overall Safe Output Success Rate: 99.45% (362/364 runs that attempted a safe output)
• Most Reliable Job Type: All workflows other than PR Sous Chef (100% success, 340/340)
• Most Problematic Workflow: PR Sous Chef (91.7% success, 22/24, in an 8-hour sample window)
• Average Time to Failure: Not meaningfully comparable — one failure occurred 23.0s into the safe_outputs job, the other 1.4m in.

Next Steps

• Investigate PR Sous Chef's 7 safe-output types for a validation edge case that aborts the full Process Safe Outputs batch (Work Item 1)
• Consider making safe-output validation degrade per-item rather than per-batch across all workflows, not just PR Sous Chef
• Extend audit/logs tooling to expose raw safe_outputs job error text for future audits (Work Item 2)
• Re-run this audit in 24h to check whether the PR Sous Chef failure rate persists or was a transient blip

References:

• §32543132845 — PR Sous Chef, safe_outputs job failure, 9 safe items
• §32526302837 — PR Sous Chef, safe_outputs job failure, 1 safe item
• §32541284863 — PR Sous Chef, nearest preceding successful run (baseline comparison)

Generated by 🔒 Safe Output Health Monitor · agent · 233.5 AIC · ⌖ 9.24 AIC · ⊞ 10.8K · ◷

• expires on Aug 22, 2026, 8:50 PM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-08-23T04:50:29.733Z.

Closed by Workflow