|
| 1 | +# Measurements |
| 2 | +How runs that spend money or call models are planned, compared, judged and recorded. |
| 3 | + |
| 4 | +## #brief_before_run |
| 5 | +**When:** about to launch anything that spends money or calls a model on the lab's behalf (a run, eval, benchmark). |
| 6 | +**Do:** First write `packages/director/docs/BRIEF-<ID>.md`; do not start the run until it exists. |
| 7 | +**Because:** The lab requires a brief before any run. |
| 8 | +**Links:** [[Measurements#brief_contents]] · [[Measurements#verdict_fixed_before_run]] · [[Measurements#headroom_first]] |
| 9 | + |
| 10 | +## #brief_contents |
| 11 | +**When:** writing a `BRIEF-<ID>.md` before a run. |
| 12 | +**Do:** Include: what is run, why, the model and provider, the verdict table, what would falsify it, the stopping condition and a spending cap. |
| 13 | +**Because:** These are the required parts of a brief. |
| 14 | +**Links:** [[Measurements#brief_before_run]] · [[Measurements#verdict_fixed_before_run]] · [[Measurements#name_planner_and_judge]] |
| 15 | + |
| 16 | +## #record_result_in_brief |
| 17 | +**When:** a run has finished (or been stopped) and its result is known. |
| 18 | +**Do:** Write the result into the same brief under `## Result` / `## Outcome`, dated, stating what was spent. |
| 19 | +**Because:** The brief is the single record of a run, and the public site checks its numbers against the briefs. |
| 20 | +**Links:** [[Writing#update_citing_docs]] · [[Measurements#flat_result_published_flat]] · [[Measurements#smoke_run_not_measurement]] |
| 21 | + |
| 22 | +## #verdict_fixed_before_run |
| 23 | +**When:** setting the bar that decides whether a run succeeded, or tempted to adjust it after seeing numbers. |
| 24 | +**Do:** Fix the verdict table in the brief before the run, and never move it after seeing the numbers. |
| 25 | +**Because:** The lab rule: the verdict table is fixed before the run. |
| 26 | +**Links:** [[Measurements#brief_contents]] · [[Measurements#flat_result_published_flat]] |
| 27 | + |
| 28 | +## #flat_result_published_flat |
| 29 | +**When:** a result came in below the bar of the verdict table and you are writing it up. |
| 30 | +**Do:** Publish it as flat; never reword it into a win. |
| 31 | +**Because:** The lab rule: a result below the bar is published as flat. |
| 32 | +**Links:** [[Measurements#verdict_fixed_before_run]] · [[Measurements#record_result_in_brief]] |
| 33 | + |
| 34 | +## #headroom_first |
| 35 | +**When:** about to measure whether a change improves something. |
| 36 | +**Do:** Measure the baseline first, before measuring the improvement. |
| 37 | +**Because:** If the baseline is already near the ceiling there is no room to show an improvement. |
| 38 | +**Links:** [[Measurements#stop_at_ceiling]] · [[Measurements#brief_before_run]] |
| 39 | + |
| 40 | +## #stop_at_ceiling |
| 41 | +**When:** the baseline you just measured is already near the ceiling. |
| 42 | +**Do:** Stop the measurement and say so in the write-up. |
| 43 | +**Because:** C0, D0O and PG were stopped there for this reason. |
| 44 | +**Links:** [[Measurements#headroom_first]] · [[Measurements#record_result_in_brief]] |
| 45 | + |
| 46 | +## #compare_against_replaced |
| 47 | +**When:** choosing what to compare a new version against in a measurement. |
| 48 | +**Do:** Compare against the version it replaces; never against a frontier model alone. |
| 49 | +**Because:** The lab rule for measurements. |
| 50 | +**Links:** [[Measurements#name_planner_and_judge]] · [[Measurements#brief_contents]] |
| 51 | + |
| 52 | +## #name_planner_and_judge |
| 53 | +**When:** writing up a measurement or its brief. |
| 54 | +**Do:** State which model planned and which model judged. |
| 55 | +**Because:** Results depend on the models used; since 2026-10-21 the person plans with Opus. |
| 56 | +**Links:** [[Measurements#label_sonnet_numbers]] · [[Measurements#compare_against_replaced]] |
| 57 | + |
| 58 | +## #label_sonnet_numbers |
| 59 | +**When:** reporting or citing numbers that were measured with Sonnet. |
| 60 | +**Do:** Label them as Sonnet's numbers. |
| 61 | +**Because:** Since 2026-10-21 the person plans with Opus, so Sonnet numbers must not be read as the current setup's. |
| 62 | +**Links:** [[Measurements#name_planner_and_judge]] |
| 63 | + |
| 64 | +## #smoke_run_not_measurement |
| 65 | +**When:** a run used one seed or only a handful of cases (a smoke run) and you are writing it up. |
| 66 | +**Do:** Treat it as deciding nothing, and say in the write-up that it is a smoke run, not a measurement. |
| 67 | +**Because:** One seed or a handful of cases decides nothing. |
| 68 | +**Links:** [[Measurements#record_result_in_brief]] |
0 commit comments