Skip to content

Commit 695a027

Browse files
usirinclaude
andcommitted
chore(fabrika): remove the eval tooling and every eval case folder (#5510)
Founder ruling: the eval layer goes, and the amendment on #5510 extends it to the case folders themselves — nothing eval-shaped survives in the plugin. Removed: - packages/fabrika-cli/src/eval/ in full (68 files, ~19.9k lines): graded runner, gate, spawner, post verb, deterministic floor, corpus and incident corpus. - the eval / grade verb group and its registry row. - packages/fabrika-cli/src/wire/eval-record.ts and its wire-registry row. - .github/workflows/fabrika-eval-gate.yml. - claude-plugins/fabrika/skills/eval-runner/. - every claude-plugins/fabrika/skills/*/evals/ folder. - claude-plugins/fabrika/reports/eval/ and docs/cost-baseline.md. Four fixture sets under those folders backed live unit tests and worked examples, so they moved to packages/fabrika-cli/test-fixtures/ rather than dying with the runner: glossary, write-pattern and adr corpora. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent 0fe7c74 commit 695a027

293 files changed

Lines changed: 127 additions & 30275 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.decisions/0146-graded-corpus-oracle-for-stochastic-model-swap.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -32,8 +32,9 @@ is the right rule, but its §3 oracle has no *statistical* form to apply it agai
3232
whose output is a distribution rather than a fixed artifact.
3333

3434
Epic [#1842](https://github.com/kamp-us/phoenix/issues/1842) built the apparatus that supplies
35-
that statistical evidence: the [`eval-harness`](../packages/fabrika-cli/src/eval)
36-
tool. This ADR records the durable *why* — what graded-over-corpus measurement is and why a
35+
that statistical evidence: the `eval-harness` tool, which lived at
36+
`packages/fabrika-cli/src/eval/` until [#5510](https://github.com/kamp-us/phoenix/issues/5510)
37+
removed it. This ADR records the durable *why* — what graded-over-corpus measurement is and why a
3738
stochastic model swap requires it — extending 0112's method to the stochastic case. It records
3839
**no** per-swap numbers (those live on the lever/decision children, per 0112's division of
3940
surfaces) and it does **not** decide the tiering policy — the tiering call that
@@ -55,7 +56,7 @@ stage and reduces the grades to two aggregate axes:
5556
representative-task-set discipline as 0112 §1 (reproducible-from-identifier, pinned to the
5657
recorded state, append-only growth) — scaled from n=1 to a set large enough that a rate is
5758
meaningful. This is the committed ground truth under
58-
[`eval-harness/corpus/`](../packages/fabrika-cli/src/eval/corpus).
59+
`eval-harness/corpus/` (removed with the harness, #5510).
5960

6061
2. **A graded oracle — the generalization of 0112 §3's binary oracle.** 0112 §3's per-stage
6162
oracle (did the stage reproduce the same decision artifact) is applied **per corpus entry**,

‎.decisions/0243-review-eval-stage-surface-discriminator.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ against the v1 shape, so the merge lands on a module whose stated central guaran
2424
Three source facts fix the problem, and they were read rather than assumed:
2525

2626
- **The two review label shapes genuinely differ.**
27-
[`packages/fabrika-cli/src/eval/corpus.ts`](../packages/fabrika-cli/src/eval/corpus.ts) declares
27+
`packages/fabrika-cli/src/eval/corpus.ts` (removed by #5510) declared
2828
`ReviewCodeEntry` with `label: {verdict, acFindings}` and `ReviewDocEntry` with
2929
`label: {verdict, findings}` — different field names over different rubrics (acceptance-criteria
3030
findings versus doc findings), not one shape spelled two ways.
@@ -35,7 +35,7 @@ Three source facts fix the problem, and they were read rather than assumed:
3535
*unrepresentable*". Collapsing `review-code` and `review-doc` onto a bare `review` key would make
3636
*both* label shapes legal under one key — the guarantee's exact negation.
3737
- **The grader dispatches on that same discriminator.**
38-
[`packages/fabrika-cli/src/eval/oracle.ts`](../packages/fabrika-cli/src/eval/oracle.ts)'s
38+
`packages/fabrika-cli/src/eval/oracle.ts`'s
3939
`gradeEntry` is a `switch (entry.stage)` selecting `gradeReviewCode` (compares `verdict` +
4040
`acFindings` as a set) or `gradeReviewDoc` (compares `verdict` + `findings` as a set). With one
4141
`review` case and no further key, two rubrics silently collapse onto one grader.

‎.decisions/0244-live-stage-key-vs-recorded-provenance.md‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -18,18 +18,18 @@ forever; the live vocabulary re-keys around them.
1818
fabrika re-keys the pipeline's stage vocabulary (ADR
1919
[0242](0242-fabrika-skill-nouns-redefine-build-and-review.md): `write-code` becomes `build`, the
2020
`review-*` family collapses to `review`). The eval harness in
21-
[`packages/fabrika-cli/src/eval/`](../packages/fabrika-cli/src/eval/) uses a stage name in two
21+
`packages/fabrika-cli/src/eval/` uses a stage name in two
2222
unrelated jobs, and the re-key forced the difference into the open:
2323

24-
- **As a live key.** `STAGES` in [`corpus.ts`](../packages/fabrika-cli/src/eval/corpus.ts) is what
25-
`--stage` accepts ([`command.ts`](../packages/fabrika-cli/src/eval/command.ts) rejects anything
24+
- **As a live key.** `STAGES` in `corpus.ts` is what
25+
`--stage` accepts (`command.ts` rejects anything
2626
else by name), what `CorpusManifest` groups entries under, and what
27-
[`runner.ts`](../packages/fabrika-cli/src/eval/runner.ts)'s `collectFromCapture` joins a capture
27+
`runner.ts`'s `collectFromCapture` joins a capture
2828
run to a corpus entry on.
2929
- **As a record.** A committed corpus row's own `stage` field says which pipeline produced the
3030
labeled artifact. The module's README states the label is what the baseline *actually produced*,
31-
and three rows in [`corpus/build.json`](../packages/fabrika-cli/src/eval/corpus/build.json) plus
32-
three in [`corpus/review.json`](../packages/fabrika-cli/src/eval/corpus/review.json)
31+
and three rows in `corpus/build.json` plus
32+
three in `corpus/review.json`
3333
are genuine v1 artifacts (issues/PRs #1223, #106, #1032, #1199, #1294, #1115).
3434

3535
Treating those as one thing gives two bad options. Retiring the rows throws away the only graded
@@ -139,7 +139,7 @@ manifest small enough to hand-edit is not worth a migrator. Two boundaries:
139139

140140
### 5. `incident-corpus/` is out of scope
141141

142-
[`incident-corpus/`](../packages/fabrika-cli/src/eval/incident-corpus/) — `evals.json`,
142+
`incident-corpus/` — `evals.json`,
143143
`provenance.json`, `ruled-keeps.json` — is a separate body of ground truth, decoded by
144144
`skill-eval-set.ts` / `incident-provenance.ts`, and carries **no stage key at all**. Where v1 skill
145145
names appear in it they are the *subject* of a recorded incident, not a key anything dispatches on.
@@ -155,7 +155,7 @@ re-key of `STAGES` — to those files.
155155
new writes and load-bearing for decoding history; the union member is what keeps "nothing new may
156156
be filed here" a type-level fact rather than a convention.
157157
- **The per-stage minimum-entry and seed assertions in
158-
[`corpus.data.unit.test.ts`](../packages/fabrika-cli/src/eval/corpus.data.unit.test.ts) are
158+
`corpus.data.unit.test.ts` are
159159
undisturbed by this ruling, and that is a consequence of keeping the rows.** Those assertions
160160
require ≥3 entries in each of the `triage` / `build` / `review-code` groups and the presence of
161161
the ADR 0112 §1 seed `inputRef`s (#1227, #1223, #1199). Because nothing is retired and `inputRef`

‎.decisions/0253-eval-record-is-an-eval-namespaced-pr-comment.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,7 @@ namespace, polarity, `@ <sha>`, trailing clause, with `bindToHead` answering
4848
and its own docblock records *why* each addition is a root and not a `review-<gate>` member — a
4949
verdict wearing another gate's namespace is the family confusion the partition ruling removed
5050
([#4891](https://github.com/kamp-us/phoenix/issues/4891)). The scorecard cell the record feeds is
51-
`ScorecardCell` in [`packages/fabrika-cli/src/eval/report.ts`](../packages/fabrika-cli/src/eval/report.ts),
51+
`ScorecardCell` in `packages/fabrika-cli/src/eval/report.ts`,
5252
carrying `stage`, `surface`, `model`, `gradedRuns`, `passedRuns`, `passRate`. Both were read at head.
5353

5454
This is engineering's lane: the eval harness is pipeline infrastructure, which ADR

‎.decisions/0257-platform-incidents-enter-only-as-artifact-checks.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -92,7 +92,7 @@ platform-originated incident:
9292
### 3. The tier is derived, so the rule is self-enforcing — a graded platform case cannot be authored honestly
9393

9494
A case's tier is not a field an author declares. `deriveTier`
95-
([`packages/fabrika-cli/src/eval/skill-eval-set.ts`](../packages/fabrika-cli/src/eval/skill-eval-set.ts))
95+
(`packages/fabrika-cli/src/eval/skill-eval-set.ts`)
9696
returns `deterministic` only when a case has assertions and **every one** of them classifies as
9797
mechanical, and `graded` otherwise — including for a case with no assertions at all. Mechanical means
9898
the assertion's text hits one of the cue phrases for an observable: an exit status, a file artifact, a

‎.github/workflows/fabrika-eval-gate.yml‎

Lines changed: 0 additions & 73 deletions
This file was deleted.

0 commit comments

Comments
 (0)