bench(s5): confidence intervals, larger protocol set, and power analysis (#965)

[cjimti]

Summary

Part 3 of 4 of the benchmark-improvement epic (#958). The S5 memory-insight-knowledge lifecycle metrics were noisy because the suite was small — 15 protocols, with applicable denominators as low as 7–10 on the supersede metrics — so several lifecycle numbers could only be reported as ranges across replicates rather than point estimates with stated uncertainty. This PR gives every S5 rate a bootstrap confidence interval, doubles the protocol set so those intervals tighten, and documents the identity-pool sizing and a power analysis for the budget-gated replicate run.

Confidence intervals

Every S5 lifecycle rate and every isolated-supersede rate now carries a 95% percentile-bootstrap confidence interval, reproducible from a fixed resampling seed, and it is rendered directly in the human summary:

  capture rate            90.0%  95% CI [84.0-96.0]  (90/100)
  personal recall         75.0%  95% CI [67.0-83.0]  (75/100)
  transfer rate           66.7%  95% CI [55.0-78.3]  (40/60)
    transfer surfaced     50.0%  95% CI [36.7-63.3]  (30/60)
  update correctness      80.0%  95% CI [67.5-92.5]  (32/40)
  duplicate rate          15.0%  95% CI [5.0-27.5]  (6/40)

The bootstrap primitives live in a new bench/internal/stats package shared by both the S1–S3 cross-arm report and the S5 lifecycle report, so the two suites draw their intervals from one implementation rather than two divergent copies. report/compare.go was refactored onto the shared package; its numeric output is unchanged (the report still resamples its actual per-attempt outcomes with the same seed).

A CI is shown only when the interval has width. A zero-width interval — an unexercised metric (empty denominator), a rate pinned at 0% or 100% whose bootstrap collapses to a point, or a pre-#965 results file whose stored metrics carry no interval — is omitted rather than printed as a meaningless [x.x-x.x] that the count already conveys. In the isolated-supersede report, SupersedeRate's interval is the exact reflection of DuplicateRate's ([1 - dupHigh, 1 - dupLow]), so the two stay exact complements exactly as their point rates already do.

Larger protocol set

The S5 protocol set doubles from 15 to 30 (20 promote + 10 supersede), which roughly doubles every metric's denominator — the supersede metrics most of all, since they were the noisiest. Each of the 15 new protocols teaches a novel definition mapped to a dataset-computed ground truth, collision-free against the seeded knowledge pages and against every other protocol, so recall and transfer remain clean measurements (the only way to answer is the taught memory or the promoted knowledge). Ground truth is generated from the fixed-seed dataset, never hand-typed, exactly as the S1–S3 truths are; the committed protocol YAML and the scripted no-model smoke are regenerated and gated by the existing reproducibility hash.

Power analysis and identity-pool sizing

bench/README.md documents both the sizing and the statistical power the grown set buys. Each attempt consumes two pool identities (teacher + learner), so a run needs 2 × protocols × k keys; the committed 264-key pool covers the 30-protocol lifecycle through k=4 (240), and a k=5 run needs the pool grown to ≥300 (recommend 320) with -identity-keys raised to match. A computed detectable-effect table (normal-approximation sample size at α=0.05, 80% power) shows the supersede/duplicate group is the binding constraint — its denominator is only 10 × k, so at k=3 it resolves ~26-point effects and at k=5 ~20-point ones, which is why the phase-4 supersede numbers read as noise. Resolving a 15-point transfer change — the size of the observed transfer ceiling — needs k=5 over the twenty promote protocols. Growing the pool is intentionally left to that budget-gated run rather than pre-committed here.

Scope

The larger real replicate run is the budget-gated evaluation deliverable and is not part of this PR. The published S5 result records in docs/reference/benchmarks.md and docs/llms-full.txt describe the earlier 15-protocol runs and are left unchanged, since those runs genuinely used 15 protocols.

Testing

All changes are in the bench/ submodule plus bench/README.md.

• make bench-test (build, vet, race test) — green
• golangci-lint run ./... in bench/ — 0 issues
• Protocol reproducibility hash gate — green (make bench-gen produces no diff)
• New/changed functions at 100% unit coverage, including the CI reproducibility, the exact-complement supersede relationship, and the zero-width-interval omission at both ends
• semgrep --config=p/golang on the refactored report file — 0 findings

Closes #965

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.03%. Comparing base (97280e8) to head (7183e6a).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #971   +/-   ##
=======================================
  Coverage   89.02%   89.03%           
=======================================
  Files         424      424           
  Lines       48581    48581           
=======================================
+ Hits        43251    43255    +4     
+ Misses       3552     3550    -2     
+ Partials     1778     1776    -2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.