Skip to content

Commit dd83898

Browse files
committed
Elevate the V0 failure to a positive finding; record the gate's catch as a methods note
1 parent 691f8a8 commit dd83898

1 file changed

Lines changed: 59 additions & 4 deletions

File tree

scripts/run_theory_v2.py

Lines changed: 59 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -391,6 +391,52 @@ def flip_rate(variance: float) -> float:
391391
}
392392
results["V3_pooling"] = v3
393393

394+
# The V0 failure is a positive finding, not four separate negatives.
395+
shared = v0["shared_phi_check"]
396+
results["FINDING_misspecification_is_structured"] = {
397+
"claim": (
398+
"Model misspecification in scaling-law fits cannot be absorbed into a noise-inflation term. A "
399+
"constant multiplying squared residuals is the wrong functional object, because the misspecification "
400+
"is structured in compute rather than exchangeable across the ladder."
401+
),
402+
"why_it_matters": (
403+
"Variance inflation is the first thing a practitioner reaches for when a fit does not match its "
404+
"nominal noise. This says that reflex fails here, and says why, which is more useful than the "
405+
"extension would have been had it worked."
406+
),
407+
"evidence": {
408+
"1_misspecification_is_real": {
409+
"residual_scatter_over_seed_se": 4.22,
410+
"variance_inflation": 17.8,
411+
"note": "the power law does not fit DataDecide cell means within seed noise",
412+
},
413+
"2_a_shared_inflator_does_not_transfer_across_ladders": {
414+
"modelled_over_empirical_by_design": {k: e["ratio"] for k, e in shared.items()},
415+
"spread_factor": max(e["ratio"] for e in shared.values()) / min(e["ratio"] for e in shared.values()),
416+
"note": "one phi over-states variance by 2.7x to 8.9x depending on the ladder",
417+
},
418+
"3_the_inflator_tracks_design_geometry": {
419+
"slope_phi_vs_log_L": v0["V0a_trend"]["slope_phi_vs_log_L"],
420+
"slope_phi_vs_k": v0["V0a_trend"]["slope_phi_vs_k"],
421+
"monotone_in_log_L": v0["V0a_trend"]["monotone_in_log_L"],
422+
"monotone_in_k": v0["V0a_trend"]["monotone_in_k"],
423+
"note": "phi is not a property of the misspecification; it moves with the ladder it was fitted on",
424+
},
425+
"4_it_does_not_predict_held_out_structure": {
426+
"predicted_over_observed": v0["V0b_ratio"],
427+
"held_out_scale": v0["held_out_scale"],
428+
"note": "phi fitted on the ladder under-predicts residual structure at an unseen scale by ~12x",
429+
},
430+
},
431+
"single_reading": (
432+
"All four measurements say the same thing. If the residuals were exchangeable noise, one inflator "
433+
"would serve every ladder, would not track ladder geometry, and would extrapolate to a held-out "
434+
"scale. None of those hold. The residuals carry compute-dependent structure, so the correct object "
435+
"is a model of that structure, not a scalar."
436+
),
437+
"scope": "DataDecide 5xC ladder, C4-EN bits per token, three-parameter power law, three fit ranges",
438+
}
439+
394440
# V4: which pre-registered failure mode did the extension land in?
395441
results["V4_overall"] = {
396442
"gate_passed": gate["passed"],
@@ -410,10 +456,19 @@ def flip_rate(variance: float) -> float:
410456
"evidence that the variance model is right, and it does not rescue V1."
411457
),
412458
"next_step": (
413-
"The mixed noise model sigma_seed^2 + phi r^2 over-states projection variance by ~5x and fails to "
414-
"predict held-out residual structure by ~12x. Misspecification is real (residual 4.22x seed SE) but "
415-
"is not captured by scaling squared residuals: the residuals are structured in compute, not "
416-
"exchangeable noise. A correct treatment needs a model of that structure, not a variance inflator."
459+
"Modelling compute-structured misspecification properly is separate work, not a third derivation "
460+
"bolted onto this one. The theory line stops here: correct structure (quadratic in log L, correct "
461+
"signs, pooling via the leverage ratio), magnitudes wrong by 2-16x for an identified reason, and a "
462+
"demonstration that the obvious fix fails for a specific structural reason."
463+
),
464+
"methods_note_gate_found_a_shipped_error": (
465+
"The phi=0 gate was installed to guard the new work. On its first run it failed at ratio 2.06 and "
466+
"stopped the script, and the cause was an error already present in the two-parameter theory we had "
467+
"reported: the analytic side squared a mean of relative standard errors while the numeric side "
468+
"averaged their squares. With a noise coefficient of variation near 1.0 those differ by ~2x. The "
469+
"numeric estimator was the correct one; the analytic constant was the loose one, and it was "
470+
"corrected to match, after which the gate passed at exactly 1.0000. A gate guarding new work found "
471+
"a defect in work already shipped."
417472
),
418473
}
419474
_write_json_atomically(results, destination)

0 commit comments

Comments
 (0)