[crazy-robotaxi] Live-edit abilities: skin swap, coins, spawned obstacles, weather (stacked on #463) - #494
Conversation
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Wire the flag-gated live_edit package into the composition root: - CLI: --live-edit-* flags registered in runtime_cli.build_parser and plumbed through cli._build_application / runtime_cli.run into CrazyRobotaxiApp(live_edit_config=...). - Session: install_style_ability_on_backend swaps in a CUDA-graph-free flashdreams session when the drift corrector is configured and attaches the text-edit LoRA + corrector after model warmup. - Presenter: CausalFrameAlignmentPresenter(LiveEditPresenter(inner)) so coin compositing sees frame-aligned poses; pixel-drawn SKIN / COINS chips; per-rollout coin ability bound via set_coin_ability. - Input: K cycles the world skin, C toggles coins, in both the native HUD and the MJPEG /control path; rising-edge requests live on CrazyRobotaxiKeyboardState.live_edit and are drained each tick by CrazyRobotaxiRuntime.process_events. - Coins: course laid along ALL driving-lane centerlines (previously only allows_taxi_stops lanes, which kept coins outside the pickup radius); pickups run on the authoritative per-frame trajectory states. - Style: duty-cycled re-swap (--live-edit-style-reswap-chunks, default 8) re-issues the active skin's replace_text so long holds stay crisp, respecting the finalize -> replace_text ordering. Validated end-to-end on GPU via the headless MJPEG app: photoreal start, mid-run key-cycle to cyberpunk at chunk 9, re-swaps at 17/25 keeping the style crisp through chunk 32, 57 coin pickups on-route. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Obstacle events (O key): clone a real moving vehicle track from the scene's perception tracks, retime it to now, and rigidly shift it ahead of the ego through the advance_frames dynamic-actor seam (synthetic boxes render in the conditioning but never materialize; clones do). Merging same-direction templates at city speed are preferred. Optional box-axis guidance (validated operating point s=2.0: in-box |diff| ~18 vs ~7 unguided, out-box clean) runs a shadow encoder cache over obstacle-free conditioning and doubles predict_flow during events. Weather events (V key): clear -> rain -> snow -> clear via plain two-prompt guided swaps (2.5/20, the PR NVIDIA#431 mechanism); the style LoRA is detached around weather-only swaps and the drift corrector is gated off during weather (its gate profile was calibrated on style v6). Skin + weather compose into one compositional prompt through the LoRA. Presenter gains WEATHER/OBSTACLE/HITS chips and an optional projected 3D-box annotation for the obstacle clone (evidence aid). 29 new CPU tests (weather cycle/LoRA-bypass/corrector policy, template extraction/placement/despawn/collision, guidance seams, key plumbing). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fe17c05 to
dd3ea05
Compare
…w, edge clipping Sizing keeps the FTheta vertical-extent projection for height, preserves the sprite's native aspect ratio, and carries the spin squash on width. Adds a light elliptical contact shadow (kept subtle so the coin pops) and edge-clipped compositing so large near-edge sprites can't raise. The default coin stays procedural; a custom RGBA sprite can be supplied with --live-edit-coin-sprite. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 2 px padding clipped the Gaussian falloff into a faint rectangle on flat road textures (visible in the cyberpunk demo stills). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Skin+weather combo prompts produced rain that read as no rain under the neon skins, and the corrector hard-off rule left over half the demo run drifting. Weather now only runs over the base world: the V key is rejected while a skin is active or pending, activating a skin clears an active weather, and compose_swap_target rejects the combo outright. WeatherPreset drops the combo clause and front-loads visible-rain cues (streaks in the air, droplets on the windshield/lens, tire spray) in the standalone prompt. The corrector's boolean dispatch becomes a per-state gain dispatch: the style gain rides the validated gated predict_flow, gain 0 short-circuits to the bit-clean base forward, and any other gain re-derives the alpha*(t) x gain LoRA scale before the base forward. New flags: --live-edit-weather-guidance (2.5 default; snow needed 3.0 in sweeps) and --live-edit-weather-corrector-gain (0 default = corrector off during weather, the calibrated-safe behavior). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Rewrite the snow preset with front-loaded falling-precipitation cues (heavy snowfall, flakes streaking past the windshield, accumulation on the hood/roadsides) after the 2.5-guidance capture read as a dusting; A/B probes picked guidance 3.0 over 3.5 (3.5 added sky smear without more snow). - Add an experimental hurricane-force 'storm' preset (appearance cues expected to land; dynamic wind effects documented as unlikely). - Add --live-edit-weather-first to rotate the V-key cycle so any preset is one confirmed press away (no pass-through transition weather). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s-live-edit Brings 671d9f99 (CUDA-graph-safe fused mode) and 39fe3af1 (DriftCorrectorDispatch + TextEditLoRA.release_targets) into the game branch; library tests live on omnidreams-live-edit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
corrector_mode=fused (default; LIVE_EDIT_CORRECTOR_MODE / --live-edit-corrector-mode fall back to unfused) deploys DriftCorrectorDispatch with per-(base|skin|weather) pre-merged sets: base keeps an optional photoreal corrector, the skin state composes the released self-attn LoRA delta with the style corrector in one copy_ source, weather reuses the style (or a dedicated) corrector at its own gain. compile_network + use_cuda_graph stay ON in fused mode; the graph-free session rebuild now happens only for unfused. The K/V state machine selects the dispatch state at chunk boundaries. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Greptile SummaryThe PR adds flag-gated live editing to Crazy Robotaxi and extends OmniDreams state management for runtime model edits.
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains. Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart LR
Input[Native or browser controls] --> Requests[LiveEditRequests]
Requests --> Runtime[CrazyRobotaxiRuntime]
Runtime --> Abilities[Skin / Weather / Coins / Obstacles]
Abilities --> Model[OmniDreams model state]
Abilities --> Presenter[LiveEditPresenter]
Model --> Frames[Generated frames]
Frames --> Presenter
Presenter --> Output[Native HUD or MJPEG stream]
Reviews (7): Last reviewed commit: "Add traffic events: multi-clone obstacle..." | Re-trigger Greptile |
| @@ -589,7 +589,7 @@ def _print_port_conflict_help(host: str, port: int, exc: OSError) -> None: | |||
| } | |||
There was a problem hiding this comment.
The changed browser control hint advertises K and C but omits the supported V weather-cycle and O obstacle-spawn bindings, leaving those two live-edit abilities undiscoverable to MJPEG users.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
be12871 to
8fd5362
Compare
…edit Signed-off-by: aidanfnv <aidanf@nvidia.com>
… off and guidance-window knobs Fixes the three perf problems reported against PR NVIDIA#494 + NVIDIA#496: - --live-edit-corrector-mode off disables every drift corrector even when checkpoints are configured: no DriftCorrectorDispatch is built, no weight sets are snapshotted, and the scheduler/finalize gate drivers are never installed (weights verified bit-identical in tests). - The native-DIT guard is now precise and actionable: it fires only for the prompt-swap abilities that genuinely need the Python transformer forward (replace_text_embeddings raises NotImplementedError under the native executor, and TextEditLoRA's copy_ toggles never reach the native fp8 weight snapshot), names the exact flags to drop, and only mentions corrector flags when a corrector is actually enabled. Coins and other pixel-only abilities never construct the ability and run under native_dit_acceleration untouched. - New knobs --live-edit-skin-guidance-chunks (default 6) and --live-edit-weather-guidance-chunks (default 20). The skin window is LoRA-realized single-branch (length is not a per-chunk cost); a GPU A/B on a cyberpunk swap showed 6 lands the style as fast and as strong as 20. Weather has no LoRA: its guided window costs ~2x per chunk while open, and the 8-chunk re-swap refresh re-opens it - now documented in the README with the knob to shorten it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…sident The presenter used to materialize the model frame to host numpy for PIL compositing, which on the native Vulkan HUD fast path (lazy CUDA uint8 HWC frames) forced a GPU->CPU->GPU round trip per frame (~10 fps observed). LiveEditFrameCompositor keeps the frame on device: coin sprites, contact shadows, and HUD chips are pre-rendered once (PIL) and cached as premultiplied per-size textures; the per-frame work is one float32 conversion, a fused lerp ROI blend or two per coin, and a single round/clamp/uint8 cast (plus an optional separable-Gaussian unsharp for skins). Measured on GB300 at 1280x704: 0.09 ms/frame with 0 coins, 0.59 ms with 9 coins (~4.7 ms per 8-frame chunk), issued from the presenter thread so it overlaps model compute. The composited result is re-wrapped as a LazyCudaFrame whose CUDA event is recorded after the blends, so both consumers stay ordered (Vulkan interop copy stream, MJPEG host prefetch) and the single-materialization numpy-XOR-cuda contract holds. Host/numpy sources keep the PIL path, as does the obstacle box-outline annotation (debug aid). All compositor math is device-agnostic and unit-tested on CPU tensors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
obstacle_ability/test_runner import-block ordering (repo ruff 0.12.7 --select I), C408 dict literals, PLR0402/RUF059/RUF046/UP037 from the current ruff default set on the live-edit files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nv/flashdreams-wq into robotaxi-463-live-edit
Every HTTP endpoint (page, /stream, /bev_stream, /control, /state, /scenes, /thumbnail, /scene/select, /taxi/name) now requires the shared token via ?token= query param or X-Stream-Token header when the flag is set; requests without a valid token get 403. The served page reads the token from its own URL and appends it to all of its requests. Omitting the flag keeps the historical open behavior. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Escalates storm along the cues that materialize (2026-08-21 A/B at guidance 3.5/4.0 vs the shipped storm frames): visibility collapse, walls of spray/mist, static debris lying on the flooded road, and a black-green emergency-gloom sky. Dynamic wind wording (bending trees, flying debris) is deliberately absent — it never materializes in the history-anchored model. Shipped at weather guidance 4.0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
aidanfnv
left a comment
There was a problem hiding this comment.
Awesome!
The style transfers probably need a follow-up to clear up some issues with how they look in motion (see my message in Slack), but the weather effects look incredible, and the coins and obstacles work well. My new map format will break the obstacle ability, but that will of course be on me to fix.
Thank you for this!
Aidan reported coins still hurting native-path perf after the CUDA compositor fix. Profiling at real course scale (4395 coins on the shipped suburb scene) found two costs the 9-coin measurement missed: - visible_sprites/advance_frames scanned all coins every frame (0.60 ms/frame CPU at 4400 coins). A static 32 m spatial hash with per-cell cached windows plus a vectorized projection filter makes both O(nearby): 0.20 ms/frame, pickup 0.10 -> 0.015 ms. - The compositor is CPU-launch-bound, not bandwidth-bound (GPU drain adds ~0 in every measurement), and dense areas put 211 sprites inside the 120 m render radius -> 11-14 ms/frame enqueue. Fixes: one blend per coin (shadow+coin+quantized fade pre-composited into one cached texture, 2.6x fewer kernels: 1.14 -> 0.44 ms at 9 sprites, 5.20 -> 1.84 ms at 48) and a nearest-N sprite cap (--live-edit-coin-max-visible, default 64) so dense courses cannot blow the budget. Also adds the ROI-only uint8 blend path (LIVE_EDIT_COMPOSITOR=roi) that never converts the full frame to float32 - exact parity with the float path (max diff 0) and minimal GPU traffic, but slower wall-clock on launch-bound machines, so the float canvas stays the default - and a perf self-report (--live-edit-perf-log N / LIVE_EDIT_PERF_LOG) that logs p50/p95 coin CPU ms, compositor enqueue CPU ms, and compositor GPU ms (lazy CUDA event pairs, no added sync) every N composited frames so remote users can send numbers instead of guesses. Validated headless MJPEG at course scale: chunk cadence unchanged (219-242 ms, matching the no-coins baseline), coin rendering parity vs the previous compositor within 1.5 LSB mean on sprite pixels (fade quantization + <=1 px shadow anchor). New tests cover the windowed culling, the sprite cap, ROI/float parity, and the perf log. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One O press now spawns --live-edit-obstacle-count clones (default 1 = unchanged single obstacle): distinct crossing/oncoming template tracks (never pace-matched lead cars, which render at ghost strength), staggered across a 16-40 m band ahead of the ego (spacing_m per slot) and --live-edit-obstacle-stagger-chunks apart in time, each retimed+rigidly shifted with ground-z correction and despawning after its own pass. Guidance already strips every OBSTACLE_ENTITY_PREFIX actor from the shadow branch, so the box-axis path handles simultaneous boxes unchanged. Mask-anchored validation (RNG-matched baseline, per-clone camera-projected boxes, 26-chunk headless runs): s=2.0 lands 2/3 clones at N=3 (52/61/18% of real-car in-box benchmark) and 4/5 at N=5 (52/60/15/69/83%), out-box clean (median 5.7-7.5); s=2.5 is stronger still at N=3 (63/79/22%) with no breakup, so the traffic demo ships at s=2.5. Simultaneous spawns (stagger 0) are stable; the 1-chunk default is for on-screen pacing. The one weak clone is the fastest template (~11 m/s) which holds distance until late. The presenter annotates every active clone and shows a TRAFFIC xN chip. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Stacked on #463 (Crazy Robotaxi standalone) — only the top 2 commits are new; implements the plan from #489 using the mechanisms from #431/#458. cc @aidanfnv
Adds an optional, flag-gated
live_edit/ability package to Crazy Robotaxi (vanilla behavior unchanged with flags off; all keys work in both the native HUD and the browser/MJPEG path):Validation: end-to-end headless MJPEG runs (65 chunks) with all abilities; 511 CPU tests pass (+62 new in
tests/test_live_edit*.py) against the synthetic fixtures; ruff clean. Known limits documented in code/tests: corrector currently requires CUDA graphs + compile off (graph-safe corrector is the follow-up), snow strength is honest-partial at the pinned guidance, revert-to-clear can't undo accumulated scene change (history-carrying model).Demo videos attached below.
robotaxi_liveedit_demo3_rain.mp4
robotaxi_liveedit_demo3_annotated.mp4
robotaxi_liveedit_demo3_cyberpunk.mp4
Update: real-time performance + weather refinements (head
a13d4c74)torch._foreach_copy_before each CUDA-graph replay — so the corrector, the style-LoRA delta, and the per-timestep gate all run with CUDA graphs on. This also makes the dispatch the sole writer of the self-attn projections, resolving the LoRA/corrector composition cleanly.--live-edit-corrector-mode unfusedkeeps the legacy path.--live-edit-base-corrector), reducing long-session drift with no features active.--live-edit-coin-sprite <path>accepts a custom RGBA sprite.Test suites: 87 live_edit CPU tests green; ruff clean.