Warm checkpoint weights at spawn: fill prewarm_paths from manifest records (#178)

[OwenPriceSkelly]

Consumes the #177 manifest weight records at spawn time — ladder rung 2 of the cold-start work (#160 → #167/#171 lineage). Closes #178.

What

The spawn-time prewarm covers the env tree, but a checkpoint's model weights load via mmap after it ends and fault in at ~0.35 MB/s on cold Lustre — on Delta, uma's 14 GB fairchem cache structurally blew every warm-up timeout its siblings fit (2026-07-29 field data). This fills spec["prewarm_paths"] (reserved since #171) so the existing prewarm machinery streams the weights too, tiered:

1. Manifest record (Track where each checkpoint's weights live: record weight files in the manifest at add/verify time #177) → warm exactly those files, joined to the cache root so split-cache installs (Frontier, Perlmutter) resolve correctly.
2. No record (worker spawns only — downloads use the record tier alone, since their record is only written after the download) → best-effort heuristic: top-level dirs of {cache_root}/cache, home/.cache, home whose names match a package the env ships (_-boundary prefix match: cache/orb ↔ orb_models, home/.dgl ↔ dgl) or the env's name. The huggingface/torch hub trees are never matched — they hold every family's weights side by side, and over-warming under a job's memory cgroup evicts exactly the pages the worker needs. Known transitional cost: the heuristic emits whole per-family dirs, so one unrecorded checkpoint over-warms its cached siblings (all mace ids share cache/mace) until one add/verify/smoke-test pass records it — the tier tag is the telemetry for retiring it. The manifest tier also requires at least one recorded path to still exist, so a scratch-purged record falls back a tier instead of logging weights: 0 MB (manifest) while the load cold-faults.
3. Nothing → today's behavior. :custom ids skip the heuristic (the user's weights already ride checkpoint_path into the prewarm), but would use a manifest record if one existed under the id (no production path writes one today — smoke-test's custom leg records only verifications).

Where

• The lookup runs in spawn_in_env — the choke point every spawn path (calculator, verify, serve, add) already goes through, and the same seam the node-local staging design (Node-local env staging: pack envs to single images, extract to local disk at spawn #180) will redirect later. A caller-supplied prewarm_paths is left alone.
• It reads manifest.json raw, not via load_manifest: this is the first time the calculator path reads the manifest, and an optimization hint must never print migration notes, take locks, or refuse a manifest written by a newer rootstock. Anything unparseable just falls back a tier; a failed lookup never fails the spawn.

Observability

• The prewarm summary tags the tier — weights: 14.2 GB (manifest) / (heuristic) / weights: none recorded — field data for retiring the heuristic once record coverage is universal, and the line keeps its filesystem-health-probe role.
• New up-front warning when the expected cold working set (env tree + weights, known before any read) exceeds the job's cgroup memory limit (v1 and v2, ancestor walk for SLURM's job-slice limits) — the eviction footgun that burned the 2026-07-29 uma run, reduced to one visible line printed before the reads so it survives a stalled warm-up.

Deployment

Ships client-side (the prewarm module is staged fresh at every spawn): deployed envs benefit from a client pin bump, no env rebuilds. Backfill on an existing install = one rootstock smoke-test pass to write the records (#177).

🤖 Generated with Claude Code

[OwenPriceSkelly]

Applied fixes from a 10-finding review pass (b51f056). Highlights:

• Non-string record paths no longer TypeError out of the fall-back-a-tier contract — entries require a non-empty str path.
• Download spawns resolve from the record tier only: a checkpoint's record is written after its download, so first-time rootstock add would otherwise always heuristic-match the whole family dir and cold-read every cached sibling's weights on a login node, before setup() even starts.
• The manifest tier now requires at least one recorded path to exist (one stat in the common case): a purged-behind-an-intact-record scratch sweep previously logged weights: 0 MB (manifest) while the load cold-faulted — misattributing exactly the stall this line exists to catch. Falls through to heuristic/none instead; records self-heal on the next add/verify/smoke-test.
• Label honesty: caller-supplied prewarm_paths tag (caller), not (custom).
• Known transitional cost, now named in the docstring: with no record, the heuristic emits whole per-family cache dirs, so one unrecorded checkpoint over-warms its cached siblings (all mace ids share cache/mace). Bounded by the family, gone after one recording pass; the tier tag is the telemetry for retiring it.
• Cleanups: cgroup walk probes only its hierarchy's limit file; summary total and weights use the same byte formatter; heuristic scan runs string filters before per-entry stats; -/_ normalized in dir matching.
• Tests: dead/partial-record fallbacks, download tiering, wrong-typed entries, and a reader test pinned against a manifest written by the production save_manifest path (the hand-rolled JSON fixtures would stay green across a schema relocation of weight_files). Also corrected a false test docstring: no production path writes weight_files under a :custom id today (smoke-test: exercise the custom-weights path and report :custom verification in the manifest #204 records only verifications).

🤖 Generated with Claude Code