Skip to content

feat(tier3): report pass@k uncertainty and paired outcomes - #63

Open
AdemVessell wants to merge 2 commits into
NVIDIA:mainfrom
AdemVessell:arkhe/tier3-paired-evidence
Open

feat(tier3): report pass@k uncertainty and paired outcomes#63
AdemVessell wants to merge 2 commits into
NVIDIA:mainfrom
AdemVessell:arkhe/tier3-paired-evidence

Conversation

@AdemVessell

Copy link
Copy Markdown

Summary

  • add two-sided 95% Wilson score intervals to each condition's case-level
    pass@k rate;
  • add direction-preserving paired outcome counts and paired rate delta;
  • add a two-sided exact McNemar diagnostic only for completely paired cases;
  • render the new evidence in the Tier 3 HTML report and document its limits.

Why

An aggregate with-skill minus without-skill rate does not reveal whether gains
and regressions occurred on the same cases, and a point estimate alone obscures
small-sample uncertainty. The additional fields make the existing result more
auditable without changing its pass/fail policy.

Scientific boundary

The intervals and exact test describe one configured run. They do not establish
task independence, correct leakage, prove generalization, or make the test set a
random sample. Partial/unidentified pairings receive no exact-test result.

Validation

  • PATH="$PWD/.venv/bin:$PATH" make test PYTHON=.venv/bin/python
    • 4,862 passed, 17 skipped, 4 deselected
  • make lint PYTHON=.venv/bin/python
    • passed
  • focused collector/report tests
    • 34 passed
  • make build PYTHON=.venv/bin/python
    • wheel and source archive built

Compatibility

This is an additive JSON/report change. Existing pass@k values and verdict
thresholds are unchanged.

Addresses #62.

Verification checklist

  • I am familiar with the Contributing Guidelines
  • Added or updated focused tests
  • Updated documentation for user-visible changes
  • Ran make lint
  • Ran make test
  • Ran make build
  • Did not add credentials, private datasets, or proprietary benchmark content

Release impact

  • Updated CHANGELOG.md

Signed-off-by: Adem Vessell <134353219+AdemVessell@users.noreply.github.com>
Signed-off-by: Adem Vessell <134353219+AdemVessell@users.noreply.github.com>
@AdemVessell

Copy link
Copy Markdown
Author

Pushed follow-up 444af99 after a hostile correctness review of the paired
pass@k evidence.

The follow-up:

  • prevents small nonzero exact McNemar probabilities from being rounded to
    zero and preserves decimal plus exact-rational representations (with a
    null numeric approximation instead of false zero on float underflow);
  • reports the minimum attainable p-value for the observed discordant-pair count
    and labels results that are resolution-limited at alpha 0.05;
  • derives the arm pass-rate delta from counts, avoiding contradictions caused
    by subtracting already-rounded rates;
  • renders the paired delta, suppresses the zero-pair row when pairing is
    unavailable, and keeps the exact test restricted to complete pairing; and
  • adds a collector-path regression where reward IDs are absent and case IDs
    are derived from Harbor trial metadata.

Validation on the final head:

  • full suite: 4,874 passed, 17 skipped, 4 deselected;
  • focused collector/report suite: 81 passed;
  • Ruff and diff checks: clean;
  • wheel and sdist build: passed;
  • Fern docs check: zero errors (redirect check skipped without Fern auth).

An independent cold re-review found no remaining actionable defect. The new
statistics remain diagnostic for one configured run; they do not establish
task independence, generalization, or causal skill benefit. This follow-up also
corrects an existing pass-lift rounding defect, so the stored delta may change
by one unit in the fourth decimal while verdict thresholds remain unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant