Skip to content

Load checkpoint onto CPU to avoid duplicating the model in VRAM - #50

Open
tlancaster6 wants to merge 1 commit into
Parskatt:mainfrom
tlancaster6:fix/load-weights-on-cpu
Open

tlancaster6 wants to merge 1 commit into
Parskatt:mainfrom
tlancaster6:fix/load-weights-on-cpu

Conversation

@tlancaster6

@tlancaster6 tlancaster6 commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Problem

RoMaV2.__init__ fetches the ~1 GB checkpoint with map_location=device:

weights = torch.hub.load_state_dict_from_url(
    ".../v2.0.1/romav2.0.1.pt",
    map_location=device,
)
...
self.to(device)          # module -> device
...
self.load_state_dict(weights)

On CUDA this materializes the entire state dict in VRAM. By the time
load_state_dict runs, self.to(device) has already moved the module, so two
complete copies of the model are resident simultaneously.

This costs twice. The obvious cost is the initialization peak. The subtler and
more damaging one is that after weights is freed, the caching allocator
retains roughly a gigabyte of reserved-but-unallocated arena for the lifetime
of the process, which later blocks large contiguous allocations.

Fix

Stage the checkpoint on CPU. This is equivalent: load_state_dict copies each
tensor into the already-placed parameters, and self.to(device) continues to
handle placement for CUDA, MPS and CPU alike. device is still used elsewhere
in the module, so the import is unaffected.

Initialization memory

RTX 4070 Ti (12 GB), precise, anchor 512, torch 2.5.1+cu121:

peak allocated peak reserved resident after init
map_location=device (current) 2211.4 MiB 2296.0 MiB 1048.9 MiB
map_location="cpu" (this PR) 1162.2 MiB 1228.0 MiB 1047.6 MiB

1049 MiB less peak allocation, resident memory unchanged.

Equivalence

The resulting model is bit-identical. SHA256 over all 907 state-dict tensors
(425,510,400 elements), computed after construction on each path:

5dc2886d894fc472468fd4323ebc1dc287e7f59404fa5d640c5ebcb65d0a9717   # map_location=device
5dc2886d894fc472468fd4323ebc1dc287e7f59404fa5d640c5ebcb65d0a9717   # map_location="cpu"

End-to-end effect

Same machine, same two-frame dense-matching workload, only this line differing.
On v2.0.1 as it stands, the second frame dies inside local_correlation:

File "romav2/refiner.py", line 175, in forward
  local_corr = local_correlation(
File "romav2/local_correlation.py", line 74, in native_torch_local_corr
  (feature0[_, ..., None] / (c**0.5) * window_feature)
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 3.59 GiB.
GPU 0 has a total capacity of 11.72 GiB of which 3.61 GiB is free.
Of the allocated memory 6.50 GiB is allocated by PyTorch, and 1.15 GiB is
reserved by PyTorch but unallocated.

3.61 GiB free, 3.59 GiB requested, and it still fails — the 1.15 GiB of
reserved-but-unallocated arena is the freed checkpoint copy, and it is enough
fragmentation to break the request. With map_location="cpu" the same
allocation succeeds and both frames reconstruct.

frames reconstructed
map_location=device 1/2
map_location="cpu" 2/2

RoMaV2.__init__ fetches the ~1 GB checkpoint with map_location=device, so
on CUDA the full state dict is materialized in VRAM. By the time
self.load_state_dict(weights) runs, self.to(device) has already moved the
module, so two complete copies of the model are resident simultaneously.

Staging the checkpoint on CPU is equivalent: load_state_dict copies each
tensor into the already-placed parameters, and self.to(device) continues
to handle placement for CUDA, MPS and CPU alike.

Measured on an RTX 4070 Ti (12 GB), precise setting, anchor 512:

                        peak alloc    peak reserved    resident after init
  map_location=device   2211.4 MiB       2296.0 MiB             1048.9 MiB
  map_location="cpu"    1162.2 MiB       1228.0 MiB             1047.6 MiB

The resulting state dict is bit-identical -- SHA256 over all 907 tensors
(425.5M elements) matches across both paths -- so this is purely a memory
fix with no behavioural change.

The lasting cost is not the init peak but the fragmentation it leaves.
Once the duplicate is freed the caching allocator retains ~1.15 GB of
reserved-but-unallocated arena, enough to break a later large contiguous
allocation in local_correlation: 3.61 GiB free, 3.59 GiB requested, and
it still fails. On a two-frame dense-matching workload that is the
difference between reconstructing 1 of 2 frames and 2 of 2.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@tlancaster6
tlancaster6 force-pushed the fix/load-weights-on-cpu branch from c529471 to 29ee427 Compare September 14, 2026 17:30
tlancaster6 added a commit to McGrathLab/AquaMVS that referenced this pull request Sep 14, 2026
Upstream RoMaV2 v2.0.1 (95c9968) loads the ~1 GB checkpoint with
map_location=device. self.to(device) has already placed the module by the
time load_state_dict runs, so two full copies of the model sit in VRAM
during init: 2211 MiB peak versus 1162 MiB when the checkpoint is staged
on CPU (measured, RTX 4070 Ti, precise setting, anchor 512).

The lasting cost is not the init peak but the fragmentation it leaves.
Once the duplicate is freed, the caching allocator holds ~1.15 GB of
reserved-but-unallocated arena for the life of the process, which is
enough to break a large contiguous allocation later in
romav2.local_correlation:

  torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 3.59 GiB.
  GPU 0 has a total capacity of 11.72 GiB of which 3.61 GiB is free.
  Of the allocated memory 6.50 GiB is allocated by PyTorch, and 1.15 GiB
  is reserved by PyTorch but unallocated.

Verified end to end on this dataset, two frames, only that line differing:
upstream v2.0.1 reconstructs 1 of 2 frames, the fix reconstructs 2 of 2.
Note that the loss is silent -- runner.py catches the OOM per frame and
continues, so the run still exits 0 with frames missing depth_maps, mesh
and point_cloud.

Pin all three references to tlancaster6/RoMaV2@29ee427 (upstream v2.0.1
plus the one-line fix; loaded weights are bit-identical by SHA256 over
all 907 state-dict tensors). docs/cli_guide.md was previously unpinned
and silently tracked upstream HEAD, so it is now pinned as well.

Upstream PR: Parskatt/RoMaV2#50
Revert to Parskatt/RoMaV2 once it lands.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant