Summary viz galleries and RoMa checkpoint-on-CPU dependency fix - #2
Merged
Merged
Conversation
_collect_height_maps gridded each frame on bounds derived from that frame's own point cloud, so consecutive maps had different shapes and origins. The timeseries gallery shared a color scale across panels that were not spatially registered, and the maps could not be differenced at all. Grid every frame onto one lattice spanning the union of all frames' XY bounds. Clouds are read twice (bounds, then gridding) rather than held in memory at once. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CPmDvxdeKbBygYGj15c51o
render_delta_gallery plots one panel per consecutive frame pair (frame[i] - frame[i-1]) on a diverging colormap centred at zero with a single shared colorbar, so erosion reads blue and deposition reads red. The symmetric color limit uses the 99th percentile of |delta| so outlier cells do not flatten the scale. Wired into the summary viz stage as summary/delta_gallery.png. Needs at least two frames; single-frame and sparse-mode runs write nothing. Consecutive pairs whose grids disagree are skipped with a warning. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CPmDvxdeKbBygYGj15c51o
Both galleries were scaled by outliers rather than by signal. render_timeseries_gallery used raw min/max. The tank wall reconstructs up to the water surface, so a third of all cells sit hundreds of mm off the bed: the range spanned 695 mm while real bed relief is ~51 mm, leaving the surface in 7% of the colormap. Percentile clipping alone does not fix this -- the wall is a bulk population, not a thin tail. Replace it with an asymmetric window, since the contamination is one-sided. The shallow limit is median - 3*(p75 - median), sized from the upper half-spread that the wall never reaches. The deep limit is a generous p99.8, because the wall does not reach the deep end at all and sand pits dug against the wall form a thin but real tail there -- a symmetric window clipped 1.34% of sand, all of it at r/Rmax ~ 0.92 and up to 46 mm past the limit, flattening the pits into solid colour. Also add the missing colorbar and switch the panels to mm. render_delta_gallery pooled |delta| over all cells at p99, which the noisy rim set to 35 mm and which washed out every quiet pair. Drop to p90 (~12 mm here, close to the rim-masked p95) so frame-to-frame detail is visible; the loud pairs saturate, which is the intended trade. Mark both colorbars extended so clipping is visible rather than silent, and relabel the delta bar "Δz (mm, + = deeper)" -- Z is down, so the previous "Height Change" read backwards. Note: the delta colormap itself is still inverted with respect to its docstring (red is currently erosion, not deposition). Left as-is here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EKdvwupAznxKrsbk4kt4Kk
Upstream RoMaV2 v2.0.1 (95c9968) loads the ~1 GB checkpoint with map_location=device. self.to(device) has already placed the module by the time load_state_dict runs, so two full copies of the model sit in VRAM during init: 2211 MiB peak versus 1162 MiB when the checkpoint is staged on CPU (measured, RTX 4070 Ti, precise setting, anchor 512). The lasting cost is not the init peak but the fragmentation it leaves. Once the duplicate is freed, the caching allocator holds ~1.15 GB of reserved-but-unallocated arena for the life of the process, which is enough to break a large contiguous allocation later in romav2.local_correlation: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 3.59 GiB. GPU 0 has a total capacity of 11.72 GiB of which 3.61 GiB is free. Of the allocated memory 6.50 GiB is allocated by PyTorch, and 1.15 GiB is reserved by PyTorch but unallocated. Verified end to end on this dataset, two frames, only that line differing: upstream v2.0.1 reconstructs 1 of 2 frames, the fix reconstructs 2 of 2. Note that the loss is silent -- runner.py catches the OOM per frame and continues, so the run still exits 0 with frames missing depth_maps, mesh and point_cloud. Pin all three references to tlancaster6/RoMaV2@29ee427 (upstream v2.0.1 plus the one-line fix; loaded weights are bit-identical by SHA256 over all 907 state-dict tensors). docs/cli_guide.md was previously unpinned and silently tracked upstream HEAD, so it is now pinned as well. Upstream PR: Parskatt/RoMaV2#50 Revert to Parskatt/RoMaV2 once it lands. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Welcome to Codecov 🎉Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests. ℹ️ You can also turn on project coverage checks and project coverage reporting on Pull Request comment Thanks for integrating Codecov - We've got you covered ☂️ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four commits since
v1.6.2: three summary-visualization changes and onedependency fix.
Summary visualization
feat: add frame-to-frame height change gallery to summary vizfix: grid all summary height maps onto a shared latticefix: make summary gallery color scales show the actual surfaceDependency fix
fix(deps): pin RoMa to fork with checkpoint-on-CPU memory fixUpstream RoMaV2 v2.0.1 loads the ~1 GB checkpoint with
map_location=device.self.to(device)has already placed the module by the timeload_state_dictruns, so two full copies sit in VRAM during init — 2211 MiBpeak versus 1162 MiB when staged on CPU (measured, RTX 4070 Ti).
The lasting cost is the fragmentation, not the init peak. Once the duplicate
is freed the caching allocator retains ~1.15 GB of reserved-but-unallocated
arena, enough to break a later large contiguous allocation in
local_correlation— 3.61 GiB free, 3.59 GiB requested, and it still fails.Verified end to end on a two-frame run, only that line differing:
map_location="cpu"The loaded weights are bit-identical (SHA256 over all 907 state-dict tensors).
This repins all three references —
requirements-prereqs.txt,INSTALL.md,and
docs/cli_guide.md. The last of these was previously unpinned andsilently tracked upstream
HEAD, which is how the regression reached a cleaninstall in the first place. Both files carry a comment pointing at
Parskatt/RoMaV2#50 and instructing a revert to upstream once it merges.
Notes for merging
feat:commit forces a minor bump).messages into one; if that reads
fix:the result is 1.6.3 and thefeatis silently lost.
release.ymlhas noneeds:ontest.yml, so the two race on push tomainand a tag is cut regardless of test outcome. Worth confirming thisPR's checks are green before merging.
Unrelated follow-up worth tracking
runner.py:203catchesExceptionper frame and continues, so a CUDA OOMskips a frame and the run still exits 0. A local 10-frame run silently
produced 5 frames with no
depth_maps,mesh, orpoint_cloud. Notaddressed here.
🤖 Generated with Claude Code