bench(cold-start): pre-run hardening - promoted-page summary delivery, run isolation, baseline preflight, cache settle (Closes #975)

[cjimti]

Closes #975

Pre-run hardening for the cold-start knowledge-growth suite (#963, part of #958) so the first paid run measures knowledge delivery instead of harness gaps. All changes are bench-side; no pkg/ changes.

1. Promoted knowledge-page facts are deliverable through search

Search renders a page hit as title plus summary, and the a3 tool surface has no page-body fetch tool, so a promoted page without a summary is a title-only hit whose fact never reaches an evaluator. This was the root cause of the weak page-sink channel in the #958 transfer data.

• protocol.PagePayload carries a Summary field, and promote.BuildApplyArgs sends it on the apply_knowledge page payload (the platform already stores and renders it).
• Both loaders refuse a page-sink payload without a summary at load time (curriculum.validateSink, protocol.validateSink), so a run can never spend budget measuring a channel that structurally cannot deliver.
• The four a2 seed page summaries are shared constants in bench/internal/gen: the curriculum's three page-sink lessons promote pages byte-identical to the a2 seed rows (slug, title, summary, body), pinned by TestCurriculumPageLessonsMatchSeedPages. S5 protocol page payloads use their one-sentence fact as the summary (TestProtocolPagePayloadsCarrySummaries).
• The lifecycle diagnosis instrumentation's page-sink surfaced-needle is now the summary instead of the body, so future transfer diagnosis reports delivery instead of a guaranteed "not surfaced".
• cs-traps.yaml and bench/protocols/* regenerated via make bench-gen; drift tests green.

2. Per-run output isolation

• make bench-cold-start writes each invocation into its own build/bench-results/cold-start-a3-<timestamp>/results.json (transcripts land in the sibling results.json.transcripts/).
• benchrun -cold-start refuses to start when the -out file or its transcript directory already exists, with no override flag: a re-run can never overwrite a prior run's paid-for results or transcripts.
• make bench-cold-start-report takes the run to summarize via RESULTS= and lists the available run dirs when unset.

3. Capture verification bounded to the current run

Teacher identities are deterministic per lesson index and URNs are fixed by the curriculum, so a pending insight left by an interrupted earlier run could fake a capture and then be promoted. lifecycleapi.InsightFilter gained Since (emitted as the admin API's RFC 3339 since param), and promote.WaitForInsight takes the teach episode's start time minus a 30-second clock-skew margin. Both call sites (cold-start and S5 lifecycle) pass it.

4. Baseline-integrity preflight

An empty baseline cannot be restored by re-seeding: promotions write the editableDatasetProperties aspect and an a2 seed leaves editable column docs, tags, and deprecation, while bench_mces_empty.json upserts only datasetProperties. coldstart.Run now hard-fails before any episode is spent when:

• any lesson entity's effective description is non-empty, or it carries tags or a deprecation (read through the platform via datahub_get_entity on the admin session, concurrently per distinct URN);
• any insight exists at all (any capturer, any status), or any knowledge page exists at all: the contract is an empty enrichment layer, so the store-level checks assert emptiness outright rather than hunting curriculum-shaped leftovers;
• the failure message states the remediation explicitly (Postgres reset via the make target's TRUNCATE; DataHub requires a fresh quickstart because re-ingesting the empty seed cannot clear editable aspects).

5. Cache-settle window between promote and eval

a3 caches DataHub table context with a 5m TTL, so a cache entry populated by checkpoint N-1's evaluators could serve the stale pre-promotion description to checkpoint N. A -settle flag (default 5m, SETTLE= on the make target) pauses between a successful datahub-sink promote and the next checkpoint, respecting ctx cancellation. Page-sink promotes skip it (page hits are served live from the portal store; nothing is cached), as do unpromoted lessons, and every skip is logged. The window is recorded on the results manifest so archived runs stay comparable, the lesson record is flushed to disk before the pause so an interruption never discards a completed teach and promote, and the scripted smoke runs with SETTLE=0s. The a3 arm config is untouched.

6. Platform refusals in the reviewer promote path are harness errors

preAuditRefusal moved to a single shared mcpc.PreAuditRefusal (the coldstart, lifecycle, and pipeline copies are gone). promote.Reviewer.Apply returns a harness error when apply_knowledge fails with a pre-audit refusal code (session_expired, rate_limited, ...) instead of scoring it as a measured miss that would silently flatline the curve; a genuine tool-level refusal stays a measured miss.

7. Evaluator no-self-teach rule plus a validity signal

The eval scaffold forbids saving, capturing, or updating memories or knowledge during an eval session. Because the rule is prompt-level, each EvalAttempt records MemoryWrites, derived from the transcript identically on the loop and claude-cli paths: only executed writes count (memory_capture, and memory_manage with a mutating command: update, forget, consolidate), while budget-refused calls, error results, and read-only memory_manage commands do not. Metrics.EvalMemoryWrites totals them and HumanSummary prints a validity warning when non-zero.

8. Regression-blindness test gaps closed

• Identity math pinned to exact values (teacherSeq, evaluatorSeq, maxIdentitySeq), plus global evaluator-email uniqueness across all checkpoints.
• The OnCheckpoint flush snapshots are pinned to the exact (lessons, checkpoints) sequence, guarding the flush-after-append ordering.
• Every teachAndPromote branch (teach harness error, capture-verify error, capture miss, admin-session error, apply error, apply refusal) has a targeted test, and Run is proven to return non-nil on any harness failure.
• claude-cli credential rotation is pinned: the per-episode MCP config for seq 7 must authenticate as testkey-007.
• The episode exec closure's refusal and transport-error classification is covered end to end through the audit read-back bounds.
• benchrun cold-start wiring tests mirror the supersede set: -arm required, -baseline refused, factory errors, plus the refuse-to-overwrite paths for both results and transcripts.

9. Runbook

bench/README.md documents the fresh-quickstart requirement and why, the explicit pass criteria (lessons 6 (captured 6, promoted 6) with harness failures 0 and climbing enrichment coverage; a zero exit code alone is not a pass because capture misses and promote refusals are measured outcomes), both adapter paths and their cost (claude-cli is subscription-funded, roughly 181 episodes at ~50s for k=1; the anthropic adapter is estimated around USD 20 at claude-sonnet-5 pricing for k=1), background launch for long runs, and the per-run dirs, RESULTS=, and -settle workflow.

Validation

• make verify green (full CI-equivalent suite), bench module fully linted and tested with -race.
• $0 end-to-end validation against a live stack: fresh DataHub quickstart, make bench-up BENCH_ARM=a3 BENCH_SEED_PAGES=0, make bench-seed-datahub-empty, then make bench-cold-start-smoke passed with the preflight green and lessons 6 (captured 6, promoted 6), harness failures 0.
• Re-running the smoke without a DataHub reset produced a second, distinct timestamped run directory and was refused by the preflight, which identified the prior run's promoted description on a lesson entity and printed the remediation.

Scope note

The never-overwrite guard applies to the cold-start mode, per the ticket. The -lifecycle and -supersede modes keep their fixed -out paths: their documented smoke-then-real-run flows (and CI's repeated writes to results-a0.json) write to the same path by design, so extending the guard there requires per-run directories for those targets as well; that is a candidate follow-up.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.05%. Comparing base (75cee5d) to head (7c1a3c8).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #976      +/-   ##
==========================================
- Coverage   89.05%   89.05%   -0.01%     
==========================================
  Files         424      424              
  Lines       48598    48598              
==========================================
- Hits        43280    43278       -2     
- Misses       3545     3546       +1     
- Partials     1773     1774       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.