Triage skill: evidence lives in issue comments; suggest fixes at high confidence - #344
Open
sibson wants to merge 4 commits into
Open
Triage skill: evidence lives in issue comments; suggest fixes at high confidence#344sibson wants to merge 4 commits into
sibson wants to merge 4 commits into
Conversation
Skill runs pass 22/22 assertions across the three cases; baselines pass 19/22. Both baseline failures are would-close draft comments — the boundary the skill encodes — while investigation quality was equal. Case 3's baseline read the skill from the working tree, so its column is recorded as contaminated, and the README now tells future runs to isolate baselines in a skill-free worktree. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Outcome A gains a fix-suggestion path with an explicit confidence bar (executed repro, small fix with in-codebase precedent, green suite, wire repro flipping to passing) — suggest in the comment, build the fix PR only on request, on a fresh branch that supersedes the repro PR. Artifact lifetime rules from the #90 run: logs are quoted inline and must let the comment stand alone; screenshots live only on the triage branch and expire with it; no binary evidence is ever merged to main. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The repro PR is designed to be merged, so anything committed on it eventually lands on main — the #90 run committed a scripted server, logs and screenshots there before the maintainer caught it. New rule: the repro PR carries exactly the expectedFailure test; scripts and logs go in the triage comment as folded text; screenshots are described in verifiable terms (size, pixel extrema) rather than committed or embedded anywhere. Also point wire-level repro work at the tests/servers docker fleet before scripting a fake server, now that the compatibility harness exists. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-ups to the merged skill (#333), from the first live run on #90:
references/decision-table.md: the repro PR carries exactly one file (theexpectedFailuretest); scripts and full logs go in the triage comment as<details>-folded text, which survives branch deletion and costs the repo nothing; screenshots are described in verifiable terms (size, pixel extrema) instead of being committed or embedded anywhere. test: reproduce #90 — capture completes black on desktop-size-only update #342 has been restructured to match.claude/fix-issue-NNNbranch that supersedes the repro PR. (Fix black captures from servers announcing DesktopSize before pixel data (#90) #343 followed this shape.)tests/servers/(the docker compatibility fleet from Spike: Docker Compose VNC test servers for compatibility testing #334–Graduate server compatibility spikes into CI (Phase 0) #339) before scripting a fake server.evals/results/2026-08-13-iteration-1.json): skill 22/22 assertions, baseline 19/22 — both baseline failures were would-close comments; case-3's baseline read the skill from the tree, noted as contamination with the isolation fix in the README.The branch was restarted from
mainafter #333 merged; these three commits are the only delta.Generated by Claude Code