Load checkpoint onto CPU to avoid duplicating the model in VRAM - #50
Open
tlancaster6 wants to merge 1 commit into
Open
tlancaster6 wants to merge 1 commit into
tlancaster6 wants to merge 1 commit into
Conversation
RoMaV2.__init__ fetches the ~1 GB checkpoint with map_location=device, so
on CUDA the full state dict is materialized in VRAM. By the time
self.load_state_dict(weights) runs, self.to(device) has already moved the
module, so two complete copies of the model are resident simultaneously.
Staging the checkpoint on CPU is equivalent: load_state_dict copies each
tensor into the already-placed parameters, and self.to(device) continues
to handle placement for CUDA, MPS and CPU alike.
Measured on an RTX 4070 Ti (12 GB), precise setting, anchor 512:
peak alloc peak reserved resident after init
map_location=device 2211.4 MiB 2296.0 MiB 1048.9 MiB
map_location="cpu" 1162.2 MiB 1228.0 MiB 1047.6 MiB
The resulting state dict is bit-identical -- SHA256 over all 907 tensors
(425.5M elements) matches across both paths -- so this is purely a memory
fix with no behavioural change.
The lasting cost is not the init peak but the fragmentation it leaves.
Once the duplicate is freed the caching allocator retains ~1.15 GB of
reserved-but-unallocated arena, enough to break a later large contiguous
allocation in local_correlation: 3.61 GiB free, 3.59 GiB requested, and
it still fails. On a two-frame dense-matching workload that is the
difference between reconstructing 1 of 2 frames and 2 of 2.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
tlancaster6
force-pushed
the
fix/load-weights-on-cpu
branch
from
September 14, 2026 17:30
c529471 to
29ee427
Compare
tlancaster6
added a commit
to McGrathLab/AquaMVS
that referenced
this pull request
Sep 14, 2026
Upstream RoMaV2 v2.0.1 (95c9968) loads the ~1 GB checkpoint with map_location=device. self.to(device) has already placed the module by the time load_state_dict runs, so two full copies of the model sit in VRAM during init: 2211 MiB peak versus 1162 MiB when the checkpoint is staged on CPU (measured, RTX 4070 Ti, precise setting, anchor 512). The lasting cost is not the init peak but the fragmentation it leaves. Once the duplicate is freed, the caching allocator holds ~1.15 GB of reserved-but-unallocated arena for the life of the process, which is enough to break a large contiguous allocation later in romav2.local_correlation: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 3.59 GiB. GPU 0 has a total capacity of 11.72 GiB of which 3.61 GiB is free. Of the allocated memory 6.50 GiB is allocated by PyTorch, and 1.15 GiB is reserved by PyTorch but unallocated. Verified end to end on this dataset, two frames, only that line differing: upstream v2.0.1 reconstructs 1 of 2 frames, the fix reconstructs 2 of 2. Note that the loss is silent -- runner.py catches the OOM per frame and continues, so the run still exits 0 with frames missing depth_maps, mesh and point_cloud. Pin all three references to tlancaster6/RoMaV2@29ee427 (upstream v2.0.1 plus the one-line fix; loaded weights are bit-identical by SHA256 over all 907 state-dict tensors). docs/cli_guide.md was previously unpinned and silently tracked upstream HEAD, so it is now pinned as well. Upstream PR: Parskatt/RoMaV2#50 Revert to Parskatt/RoMaV2 once it lands. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
RoMaV2.__init__fetches the ~1 GB checkpoint withmap_location=device:On CUDA this materializes the entire state dict in VRAM. By the time
load_state_dictruns,self.to(device)has already moved the module, so twocomplete copies of the model are resident simultaneously.
This costs twice. The obvious cost is the initialization peak. The subtler and
more damaging one is that after
weightsis freed, the caching allocatorretains roughly a gigabyte of reserved-but-unallocated arena for the lifetime
of the process, which later blocks large contiguous allocations.
Fix
Stage the checkpoint on CPU. This is equivalent:
load_state_dictcopies eachtensor into the already-placed parameters, and
self.to(device)continues tohandle placement for CUDA, MPS and CPU alike.
deviceis still used elsewherein the module, so the import is unaffected.
Initialization memory
RTX 4070 Ti (12 GB),
precise, anchor 512, torch 2.5.1+cu121:map_location=device(current)map_location="cpu"(this PR)1049 MiB less peak allocation, resident memory unchanged.
Equivalence
The resulting model is bit-identical. SHA256 over all 907 state-dict tensors
(425,510,400 elements), computed after construction on each path:
End-to-end effect
Same machine, same two-frame dense-matching workload, only this line differing.
On v2.0.1 as it stands, the second frame dies inside
local_correlation:3.61 GiB free, 3.59 GiB requested, and it still fails — the 1.15 GiB of
reserved-but-unallocated arena is the freed checkpoint copy, and it is enough
fragmentation to break the request. With
map_location="cpu"the sameallocation succeeds and both frames reconstruct.
map_location=devicemap_location="cpu"