bench: pre-run correctness fixes — promote sink read-back, audit-signal integrity, grader/merge/env guards (Closes #977)

[cjimti]

Closes #977

Hardens the bench harness ahead of the first real cold-start knowledge-growth run (#958, #963): promotion is verified all the way to the sink, the coverage and audit signals cannot silently degrade, metric redefinitions refuse incomparable baselines, and the grader, merge, environment, and output-isolation guards close the remaining paths by which a paid run could produce wrong, unattributable, or destroyable numbers. The bench module's checks also now run locally in make verify, mirroring CI's harness job.

Promotion is verified at the sink, not only in the platform's own records

promote.Reviewer.Apply (bench/internal/promote/promote.go) follows the insights/changesets API verify with a sink read-back: for a datahub-sink promotion it reads the entity through datahub_get_entity and requires the effective description to carry the applied fact; for a page-sink promotion it lists the portal knowledge pages and requires a page with the applied slug and summary. An API-confirmed apply whose sink does not show the change is a harness error, never a reported promotion, so a silent sink-write loss (the resulting_state defect class fixed in v1.94.1) aborts the run before episodes are spent instead of publishing a flat learning curve misattributed to agent behavior. The read-back window is configurable end to end (-sink-timeout, default 15s, plumbed through the lifecycle, supersede, and cold-start runners), and its failure message states that the promotion may be live despite the error — approve and apply already succeeded — so the run is treated as contaminated rather than retried in place. Covered by TestReviewerApplySinkReadBack (both sinks, both the landed and lost-write cases) and exercised against the live platform by the scripted smokes.

Audit-signal integrity on both claude-cli paths

A claude-cli episode that reports successful MCP calls but surfaces no dps_ handle is a harness error in both the cold-start and S5 lifecycle runners (matching the S1-S3 pipeline's contract): with no handle a successful data call is impossible, so the inconsistency — a stream-shape drift or an unparseable platform_info result — surfaces instead of contributing unflagged zeros. A failed audit read-back on a graded attempt is recorded per attempt as audit_read_error in results.json, totaled as audit_read_failures, and printed as an integrity warning in the human summary of both suites — enrichment coverage climbing is a documented cold-start pass criterion, so audit-signal loss is visible in the kept artifact rather than only in a log line. Covered by no-handle and audit-loss tests in both packages.

The duplicate rate measures the supersede gate, not capture noise — and old baselines are refused

The S5 supersede stage verifies the correction capture before spending the post-update recall episode (bench/internal/lifecycle/runner.go): a missed correction skips the recall (no staleness dilution of update correctness, no wasted paid episode), is measured on the new update_capture_rate, fails pass^k, and leaves the duplicate outcome unscored — with no correction on the platform the supersede gate never ran, so counting the attempt as duplicated would inflate the duplicate rate by up to the capture-miss rate, more than the regression gate's 5-point tolerance. When the transcript is inconclusive (a claude-cli stream can drop the paired tool_result of a call that really executed), the insights API is the arbiter, with the teach-stage insight excluded by ID because the skew-widened since window can reach back past it. BaselineCompatible refuses a baseline that scored duplicates but carries no update-capture data: such a baseline predates this gating, its duplicate numbers are definitionally inflated, and gating against it could absorb a real supersede regression. The isolated supersede sub-benchmark reports a per-protocol update_capture_missed count. Covered by TestRunSupersedeUpdateCaptureMiss, TestCorrectionCapturedAPIArbiter, and the extended TestBaselineCompatible; the test fakes now honor the since parameter exactly as the admin API does.

Entity grading matches aliases on word boundaries

grade.Entity (bench/internal/grade/grade.go) matches aliases and wrong-alias vetoes on word boundaries (letters, digits, and underscores are word characters; dots remain separators, so bench.orders still matches inside memory.bench.orders). A correct answer phrased "West, at least by net revenue" is no longer vetoed because "at least" contains "east", and "West" no longer matches inside "southwest". Covered by TestEntityWordBoundary.

Merge, subscription, and parser guards

-merge refuses to fold passes from different LLM providers into one scorecard (sameConfig compares the provider, not the exact CLI version, mirroring BaselineCompatible), so an anthropic pass and a claude-cli pass of the same model can no longer blend into a k=2 artifact the regression gate itself declares incomparable. The claude-cli child environment strips ANTHROPIC_AUTH_TOKEN, CLAUDE_CODE_USE_BEDROCK, and CLAUDE_CODE_USE_VERTEX alongside ANTHROPIC_API_KEY, so no exported metered credential can silently bill a subscription run. MCPCalls is derived from the deduplicated tool_use id set, keeping it consistent with the success/error tallies and preserving the audit read-back's overcount detection under replayed stream events. An episode that drove tools but reports zero token usage warns loudly, so a stream-shape change in a claude release cannot silently zero a run's self-reported cost basis.

Every run's output is kept

bench-lifecycle and bench-supersede write into per-run timestamped directories under build/bench-results/ (matching bench-cold-start); all three run-dir names carry a PID suffix and are created with a plain mkdir, so two runs launched in the same second abort loudly instead of silently sharing a directory. benchrun refuses an existing -out in all three timestamped-run modes, and the report targets take RESULTS= with a listing of available run dirs when unset. The S1-S3 task mode deliberately keeps its fixed per-arm slot (results-<arm>.json) — the contract bench-compare and the CI regression gate consume — and the guard's documentation says so explicitly.

Bench module checks run locally

make verify now runs bench-test and the new bench-lint (a full-module golangci-lint of bench/, mirroring CI's Harness module checks job). bench/ is a separate Go module the root lint and test targets never reach, and the root lint's patch scoping cannot see a finding anchored on an unchanged line, so a bench-only finding previously surfaced first in CI. The cold-start Aggregate folds lesson and attempt accumulation through helpers, keeping it under the gocognit ceiling that lint enforces.

Documentation

bench/README.md documents the sink read-back, the update-capture gating of the duplicate rate, the new pass-criteria warnings, the RESULTS= report usage, and which bench targets belong in verify (pure module checks) versus not (stack-dependent runs). docs/reference/benchmarks.md documents the per-run directories and RESULTS= invocation. The claude-cli cache-token section states the real-run confirmation is pending: every kept claude-cli results file predates the current parser, so the first real run on it supplies that verification, and the zero-usage warning makes a silent field move loud in the meantime.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.06%. Comparing base (7355523) to head (8383c08).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #978   +/-   ##
=======================================
  Coverage   89.05%   89.06%           
=======================================
  Files         424      424           
  Lines       48598    48598           
=======================================
+ Hits        43280    43283    +3     
+ Misses       3545     3543    -2     
+ Partials     1773     1772    -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.