Skip to content

Show the normalizer failure in the triage outcome recap - #53

Merged
tarekziade merged 1 commit into
mainfrom
fix/triage-recap-normalizer
Jul 30, 2026
Merged

Show the normalizer failure in the triage outcome recap#53
tarekziade merged 1 commit into
mainfrom
fix/triage-recap-normalizer

Conversation

@tarekziade

Copy link
Copy Markdown
Collaborator

Why

In transformers#47631 the recap explained the longcat_flash group as:

LLM returned unparseable output (finish_reason=stop, 62 LLM turns · 60 tool calls · 82.3s · 2014266 in / 16210 out tokens)

That is the last symptom. The run was actually killed by the repo normalizer failing on check_docstrings — a failure the patch could not have caused or fixed (the runner image's baked transformers predated a model that had landed on main 51 minutes earlier). A reviewer reading the issue had no way to tell those two situations apart without opening the Serge dashboard.

The resnet row had the opposite problem: its reason was the normalizer, but the cell ended at the colon (…before a clean patch was found:) because the recap keeps only the first line — the checker name was cut off.

What

Consumes the new normalizer_error field from Serge's /status (huggingface/serge#84):

  • the failing checker's name is lifted into the Reason cell — … — normalizer: 1 failed: docstrings
  • the output tail goes in a collapsible under the table, bounded to 3k chars (an issue body is capped at 65536 and may carry one block per group)

Falls back to Serge's Normalizer failed (exit N) preamble when there is no checker summary (a make style crash, a timeout), and is a no-op against a Serge build that does not send the field.

Rendered against the real longcat_flash payload:

| Group | Reason | LLM | Tokens (in / out) |
| --- | --- | --- | --- |
| `longcat_flash` | LLM returned unparseable output (finish_reason=stop, …) — normalizer: 1 failed: docstrings | `moonshotai/Kimi-K2.7-Code` | 2,014,266 / 16,210 |

<details><summary>`longcat_flash` — normalizer output (tail)</summary>
… ModuleNotFoundError: Could not import module 'GraniteMoeSWAConfig' … 1 failed: docstrings
</details>

Test

tests/test_integration_failure_triage.py: the error-path shape above, the exit-line fallback, block placement after the table, tail preservation under the bound, and a fence longer than any backtick run in the captured output. 88 passed.

Pre-existing failures in test_export.py / test_resource_plugin.py are a missing opentelemetry in the local venv, identical at HEAD.

🤖 Generated with Claude Code

When a dispatched group opens no PR, the recap showed only Serge's
terminal message. On the error path that names the last symptom, not the
cause: the 2026-07-29 longcat_flash group read "LLM returned unparseable
output" when what actually killed it was the normalizer failing on
check_docstrings against a checker the patch could not influence. A
reviewer had to open the Serge dashboard to learn that.

Consume the new `normalizer_error` field from Serge's /status: lift the
failing checker's name into the Reason cell ("normalizer: 1 failed:
docstrings") and render the output tail in a collapsible under the table,
so a reviewer can tell a patch the model got wrong from a normalizer
failure the patch never caused.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@tarekziade
tarekziade force-pushed the fix/triage-recap-normalizer branch from 7c5d142 to 92738bc Compare July 30, 2026 07:17
@tarekziade
tarekziade merged commit 893324e into main Jul 30, 2026
@tarekziade
tarekziade deleted the fix/triage-recap-normalizer branch July 30, 2026 07:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant