eval/finalize: load torchao-quantized models without a flag

[jdinalt]

Summary

Closes #41. `forgather eval` can now load and evaluate `forgather finalize --quantize`-produced artifacts with no extra CLI flag and no marker file. Issue #41 originally proposed both; investigation showed HF's existing config-based autodetection makes them unnecessary.

Two minimal changes:

1. Finalize writes a `quantization_config` block into the saved `config.json` (lines added after `_apply_quantize`). On reload, HF `from_pretrained()` instantiates a `TorchAoHfQuantizer`, pre-processes the model with the right quantized linear modules, then runs `load_state_dict` — so torchao tensor subclasses land in slots that know how to hold them.

2. Eval (`scripts/eval_script.py:resolve_checkpoint`) reads `config.json` at load time. If `quantization_config` is present, it forces the `from_pretrained()` path regardless of `--checkpoint` / `--no-checkpoint` flags. Without this, eval's default checkpoint-resume path uses `from_config()` + `load_state_dict()` which has no quantizer hook and fails with `'Parameter' object has no attribute 'tensor_data_names'` on quantized subclasses.

Bf16 models are unaffected — autodetect is a no-op when the config block is absent.

Three-way comparison

Run on the 4.43M Tiny Llama from PR #39 verification (RTX 3090, full Tiny Stories eval, recipe `int8-dynamic-act-int4-weight`):

Model	eval_loss	perplexity	Δ vs bf16
bf16 baseline	1.3656	3.918	—
PTQ on bf16 baseline	1.3917	4.022	+0.0262
QAT-trained + converted	1.3905	4.017	+0.0249

QAT shaves ~0.0013 eval-loss off PTQ at this scale — real but small. Doc is honest about it.

Same mechanism unblocks issue #42 (inference server already uses `from_pretrained()`; once the artifact carries `quantization_config` it picks up the optimized path automatically — #42 stays open for the webui-control + docs work).

Files changed

• `tools/finalize_model/finalize_model.py` — attach `config.quantization_config = TorchAoConfig(...)` after `_apply_quantize`.
• `scripts/eval_script.py` — `resolve_checkpoint()` reads config.json, forces `from_pretrained()` path on quantized models.
• `docs/trainers/qat-training.md` — new "Evaluating Quantized Models" section with autodetect mechanism + three-way comparison table.
• `docs/guides/finalize-model.md` — note on the new `quantization_config` write behavior.
• `docs/guides/evaluating-models.md` — note in "Model loading" about quantized-model autodetect.

Test plan

• Smoke: `forgather eval test tinystories -M <quantized_dir>` returns finite eval_loss (no `--no-checkpoint` needed).
• Autodetect log fires for quantized models, not for bf16 ones.
• Regression: `forgather eval test tinystories -M <bf16_dir>` unchanged.
• `config.json` after finalize contains a `quant_method: torchao` block with the right `quant_type`.
• Full three-way comparison (numbers above).
• Webui eval submit (user-side check, not on this branch).

🤖 Generated with Claude Code

[jdinalt]

Review pass complete

Two review agents ran in parallel. Both said no blockers.

Applied in commit be638c5

• Autodetect cache — `resolve_checkpoint()` now caches its result on `args`, so the "Detected quantization_config..." log fires once per run instead of twice. Both reviewers flagged the duplicate.
• Explicit `--checkpoint PATH` override warning — previously, `--checkpoint ` on a quantized model short-circuited before the autodetect check and crashed with the `tensor_data_names` error. Moved the quantization probe ahead of the checkpoint branch; explicit paths now log a WARNING naming the overridden path and route through `from_pretrained()`. (Reviewer Add comprehensive unit tests for forgather modules #1 nit 1.)
• Dry-run log — `--quantize --dry-run` now logs "Would write 'quantization_config' block to config.json". (Reviewer Add comprehensive unit tests for forgather modules #1 nit 2.)
• Doc clarifications:
  • Log message + prose changed from "instead of checkpoint resume" to "ignoring checkpoint flags" — accurate phrasing for the override behavior. (Both reviewers.)
  • s/perplexity premium/eval-loss premium/ in the Three-Way Comparison section — the +0.025 number is eval-loss, not perplexity. (Reviewer Add claude GitHub actions 1771648604662 #2 nit 4.)
  • Forward reference from the "Choosing a Recipe" table to the measured 0.0013 Δ in the Three-Way Comparison section, so the qualitative claim and the empirical datapoint don't read as contradictory. (Reviewer Add claude GitHub actions 1771648604662 #2 nit 1.)

Also factored out a tiny `_model_is_quantized()` helper while refactoring `resolve_checkpoint()` — keeps the function readable.

Filed as follow-up

• Quantization: backfill three-way comparison at larger model scale #47 — backfill the three-way comparison at a larger model scale. The 4.43M datapoint is honest about its scale, but a second rung would settle the "does this generalize?" question.

Skipped with reasoning

• Reviewer Add claude GitHub actions 1771648604662 #2's parallel inference-server-docs note — that's inference server: load torchao-quantized models #42's scope, not this PR.