Skip to content

[crazy-robotaxi] Live-edit abilities: skin swap, coins, spawned obstacles, weather (stacked on #463) - #494

Open
wenqingw-nv wants to merge 70 commits into
NVIDIA:dev/aidanf/game/crazy-robotaxifrom
wenqingw-nv:robotaxi-463-live-edit
Open

[crazy-robotaxi] Live-edit abilities: skin swap, coins, spawned obstacles, weather (stacked on #463)#494
wenqingw-nv wants to merge 70 commits into
NVIDIA:dev/aidanf/game/crazy-robotaxifrom
wenqingw-nv:robotaxi-463-live-edit

Conversation

@wenqingw-nv

@wenqingw-nv wenqingw-nv commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Stacked on #463 (Crazy Robotaxi standalone) — only the top 2 commits are new; implements the plan from #489 using the mechanisms from #431/#458. cc @aidanfnv

Adds an optional, flag-gated live_edit/ ability package to Crazy Robotaxi (vanilla behavior unchanged with flags off; all keys work in both the native HUD and the browser/MJPEG path):

  • K — world-skin cycle (base → arcade → comic → cyberpunk → pixel): multi-style LoRA + drift corrector attach at the session seam, prompt-swap at chunk boundaries (lands next chunk, ≈0.7–1.4 s), duty-cycled re-swap keeps the style crisp indefinitely. Attach is bit-clean at rest (validated against an unattached control at the rasterizer's reproducibility floor).
  • C — coin pickups: course laid on the driving-lane centerlines, projected per frame through the FTheta camera model, pickup by ego proximity, HUD counter chip.
  • O — obstacle events: a template-cloned vehicle crossing the lane ahead, materialized via box-axis guidance (s=2; crossing paths materialize fully opaque — pace-matched lead clones stay ghost, so events prefer crossing templates). Visual + HIT chip; physics integration left out deliberately.
  • V — weather cycle (clear → rain → snow → clear) via guided prompt swap; composes with skins (cyberpunk + rain validated as one compositional prompt).

Validation: end-to-end headless MJPEG runs (65 chunks) with all abilities; 511 CPU tests pass (+62 new in tests/test_live_edit*.py) against the synthetic fixtures; ruff clean. Known limits documented in code/tests: corrector currently requires CUDA graphs + compile off (graph-safe corrector is the follow-up), snow strength is honest-partial at the pinned guidance, revert-to-clear can't undo accumulated scene change (history-carrying model).

Demo videos attached below.

robotaxi_liveedit_demo3_rain.mp4
robotaxi_liveedit_demo3_annotated.mp4
robotaxi_liveedit_demo3_cyberpunk.mp4

Update: real-time performance + weather refinements (head a13d4c74)

  • Fused per-state drift corrector (rides the omnidreams fused mode from the live-edit PRs): pre-merged weight-sets per game state (base / skin / weather), swapped by torch._foreach_copy_ before each CUDA-graph replay — so the corrector, the style-LoRA delta, and the per-timestep gate all run with CUDA graphs on. This also makes the dispatch the sole writer of the self-attn projections, resolving the LoRA/corrector composition cleanly.
  • Measured in-game: 206.6 ms per 8-frame chunk model-side (38.7 fps), 237.9 ms end-to-end (33.6 fps) — vs ~1.4 s/chunk on the previous unfused path (~6.8×). Dispatch VRAM: 7.0 GiB (3 states). --live-edit-corrector-mode unfused keeps the legacy path.
  • Weather is now a base-world-only ability (V rejected while a skin is active; activating a skin clears weather) with per-state corrector gains; rain preset re-prompted for visible rain (guidance 3.0), storm preset added behind the same gating.
  • Base world now runs the photoreal clean-forcing corrector by default (--live-edit-base-corrector), reducing long-session drift with no features active.
  • Coin sprite is procedural by default; --live-edit-coin-sprite <path> accepts a custom RGBA sprite.

Test suites: 87 live_edit CPU tests green; ruff clean.

Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
aidanfnv and others added 4 commits August 14, 2026 19:42
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Signed-off-by: Aidan Foster <aidanf@nvidia.com>
Wire the flag-gated live_edit package into the composition root:

- CLI: --live-edit-* flags registered in runtime_cli.build_parser and
  plumbed through cli._build_application / runtime_cli.run into
  CrazyRobotaxiApp(live_edit_config=...).
- Session: install_style_ability_on_backend swaps in a CUDA-graph-free
  flashdreams session when the drift corrector is configured and attaches
  the text-edit LoRA + corrector after model warmup.
- Presenter: CausalFrameAlignmentPresenter(LiveEditPresenter(inner)) so
  coin compositing sees frame-aligned poses; pixel-drawn SKIN / COINS
  chips; per-rollout coin ability bound via set_coin_ability.
- Input: K cycles the world skin, C toggles coins, in both the native
  HUD and the MJPEG /control path; rising-edge requests live on
  CrazyRobotaxiKeyboardState.live_edit and are drained each tick by
  CrazyRobotaxiRuntime.process_events.
- Coins: course laid along ALL driving-lane centerlines (previously only
  allows_taxi_stops lanes, which kept coins outside the pickup radius);
  pickups run on the authoritative per-frame trajectory states.
- Style: duty-cycled re-swap (--live-edit-style-reswap-chunks, default 8)
  re-issues the active skin's replace_text so long holds stay crisp,
  respecting the finalize -> replace_text ordering.

Validated end-to-end on GPU via the headless MJPEG app: photoreal start,
mid-run key-cycle to cyberpunk at chunk 9, re-swaps at 17/25 keeping the
style crisp through chunk 32, 57 coin pickups on-route.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Obstacle events (O key): clone a real moving vehicle track from the
scene's perception tracks, retime it to now, and rigidly shift it ahead
of the ego through the advance_frames dynamic-actor seam (synthetic
boxes render in the conditioning but never materialize; clones do).
Merging same-direction templates at city speed are preferred. Optional
box-axis guidance (validated operating point s=2.0: in-box |diff| ~18
vs ~7 unguided, out-box clean) runs a shadow encoder cache over
obstacle-free conditioning and doubles predict_flow during events.

Weather events (V key): clear -> rain -> snow -> clear via plain
two-prompt guided swaps (2.5/20, the PR NVIDIA#431 mechanism); the style
LoRA is detached around weather-only swaps and the drift corrector is
gated off during weather (its gate profile was calibrated on style v6).
Skin + weather compose into one compositional prompt through the LoRA.

Presenter gains WEATHER/OBSTACLE/HITS chips and an optional projected
3D-box annotation for the obstacle clone (evidence aid).

29 new CPU tests (weather cycle/LoRA-bypass/corrector policy, template
extraction/placement/despawn/collision, guidance seams, key plumbing).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@copy-pr-bot

copy-pr-bot Bot commented Aug 20, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

wenqingw-nv and others added 6 commits August 21, 2026 00:59
…w, edge clipping

Sizing keeps the FTheta vertical-extent projection for height, preserves
the sprite's native aspect ratio, and carries the spin squash on width.
Adds a light elliptical contact shadow (kept subtle so the coin pops)
and edge-clipped compositing so large near-edge sprites can't raise.
The default coin stays procedural; a custom RGBA sprite can be supplied
with --live-edit-coin-sprite.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 2 px padding clipped the Gaussian falloff into a faint rectangle on
flat road textures (visible in the cyberpunk demo stills).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Skin+weather combo prompts produced rain that read as no rain under the
neon skins, and the corrector hard-off rule left over half the demo run
drifting. Weather now only runs over the base world: the V key is
rejected while a skin is active or pending, activating a skin clears an
active weather, and compose_swap_target rejects the combo outright.
WeatherPreset drops the combo clause and front-loads visible-rain cues
(streaks in the air, droplets on the windshield/lens, tire spray) in
the standalone prompt.

The corrector's boolean dispatch becomes a per-state gain dispatch: the
style gain rides the validated gated predict_flow, gain 0 short-circuits
to the bit-clean base forward, and any other gain re-derives the
alpha*(t) x gain LoRA scale before the base forward. New flags:
--live-edit-weather-guidance (2.5 default; snow needed 3.0 in sweeps)
and --live-edit-weather-corrector-gain (0 default = corrector off during
weather, the calibrated-safe behavior).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Rewrite the snow preset with front-loaded falling-precipitation cues
  (heavy snowfall, flakes streaking past the windshield, accumulation on
  the hood/roadsides) after the 2.5-guidance capture read as a dusting;
  A/B probes picked guidance 3.0 over 3.5 (3.5 added sky smear without
  more snow).
- Add an experimental hurricane-force 'storm' preset (appearance cues
  expected to land; dynamic wind effects documented as unlikely).
- Add --live-edit-weather-first to rotate the V-key cycle so any preset
  is one confirmed press away (no pass-through transition weather).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s-live-edit

Brings 671d9f99 (CUDA-graph-safe fused mode) and 39fe3af1
(DriftCorrectorDispatch + TextEditLoRA.release_targets) into the game
branch; library tests live on omnidreams-live-edit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
corrector_mode=fused (default; LIVE_EDIT_CORRECTOR_MODE /
--live-edit-corrector-mode fall back to unfused) deploys
DriftCorrectorDispatch with per-(base|skin|weather) pre-merged sets:
base keeps an optional photoreal corrector, the skin state composes the
released self-attn LoRA delta with the style corrector in one copy_
source, weather reuses the style (or a dedicated) corrector at its own
gain. compile_network + use_cuda_graph stay ON in fused mode; the
graph-free session rebuild now happens only for unfused. The K/V state
machine selects the dispatch state at chunk boundaries.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@greptile-apps

greptile-apps Bot commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR adds flag-gated live editing to Crazy Robotaxi and extends OmniDreams state management for runtime model edits.

  • Adds skin, weather, coin, and obstacle abilities across native and MJPEG presentation paths.
  • Adds fused drift-corrector dispatch compatible with compiled CUDA-graph execution.
  • Adds optional token authentication to the MJPEG server and supporting demo scripts.
  • Adds CPU coverage for live-edit behavior, stream authentication, and runtime integration.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename Overview
apps/crazy_robotaxi/crazy_robotaxi/app.py Wires live-edit configuration, abilities, runtime event handling, and presenter composition into the application lifecycle.
apps/crazy_robotaxi/crazy_robotaxi/live_edit/style_ability.py Implements the skin and weather state machine and installs model-side prompt and corrector hooks.
apps/crazy_robotaxi/crazy_robotaxi/live_edit/coin_ability.py Implements lane-based coin placement, spatial lookup, collection, and camera projection.
apps/crazy_robotaxi/crazy_robotaxi/live_edit/obstacle_ability.py Implements cloned-vehicle obstacle events and optional model guidance.
apps/crazy_robotaxi/crazy_robotaxi/streaming_presenter.py Adds browser live-edit controls and optional shared-token authentication across MJPEG endpoints.
integrations/omnidreams/omnidreams/_drift_corrector.py Adds per-state fused drift-corrector weight dispatch for CUDA-graph-compatible model execution.
flashdreams/flashdreams/core/attention/kvcache.py Updates KV-cache behavior used by the live-edit model execution path.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Input[Native or browser controls] --> Requests[LiveEditRequests]
  Requests --> Runtime[CrazyRobotaxiRuntime]
  Runtime --> Abilities[Skin / Weather / Coins / Obstacles]
  Abilities --> Model[OmniDreams model state]
  Abilities --> Presenter[LiveEditPresenter]
  Model --> Frames[Generated frames]
  Frames --> Presenter
  Presenter --> Output[Native HUD or MJPEG stream]
Loading

Reviews (7): Last reviewed commit: "Add traffic events: multi-clone obstacle..." | Re-trigger Greptile

@@ -589,7 +589,7 @@ def _print_port_conflict_help(host: str, port: int, exc: OSError) -> None:
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Browser hint omits controls

The changed browser control hint advertises K and C but omits the supported V weather-cycle and O obstacle-spawn bindings, leaving those two live-edit abilities undiscoverable to MJPEG users.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

@aidanfnv
aidanfnv force-pushed the dev/aidanf/game/crazy-robotaxi branch from be12871 to 8fd5362 Compare August 21, 2026 01:08
aidanfnv and others added 8 commits August 20, 2026 18:23
…edit

Signed-off-by: aidanfnv <aidanf@nvidia.com>
… off and guidance-window knobs

Fixes the three perf problems reported against PR NVIDIA#494 + NVIDIA#496:

- --live-edit-corrector-mode off disables every drift corrector even when
  checkpoints are configured: no DriftCorrectorDispatch is built, no
  weight sets are snapshotted, and the scheduler/finalize gate drivers are
  never installed (weights verified bit-identical in tests).
- The native-DIT guard is now precise and actionable: it fires only for
  the prompt-swap abilities that genuinely need the Python transformer
  forward (replace_text_embeddings raises NotImplementedError under the
  native executor, and TextEditLoRA's copy_ toggles never reach the
  native fp8 weight snapshot), names the exact flags to drop, and only
  mentions corrector flags when a corrector is actually enabled. Coins
  and other pixel-only abilities never construct the ability and run
  under native_dit_acceleration untouched.
- New knobs --live-edit-skin-guidance-chunks (default 6) and
  --live-edit-weather-guidance-chunks (default 20). The skin window is
  LoRA-realized single-branch (length is not a per-chunk cost); a GPU A/B
  on a cyberpunk swap showed 6 lands the style as fast and as strong as
  20. Weather has no LoRA: its guided window costs ~2x per chunk while
  open, and the 8-chunk re-swap refresh re-opens it - now documented in
  the README with the knob to shorten it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…sident

The presenter used to materialize the model frame to host numpy for PIL
compositing, which on the native Vulkan HUD fast path (lazy CUDA uint8 HWC
frames) forced a GPU->CPU->GPU round trip per frame (~10 fps observed).

LiveEditFrameCompositor keeps the frame on device: coin sprites, contact
shadows, and HUD chips are pre-rendered once (PIL) and cached as
premultiplied per-size textures; the per-frame work is one float32
conversion, a fused lerp ROI blend or two per coin, and a single
round/clamp/uint8 cast (plus an optional separable-Gaussian unsharp for
skins). Measured on GB300 at 1280x704: 0.09 ms/frame with 0 coins,
0.59 ms with 9 coins (~4.7 ms per 8-frame chunk), issued from the
presenter thread so it overlaps model compute.

The composited result is re-wrapped as a LazyCudaFrame whose CUDA event is
recorded after the blends, so both consumers stay ordered (Vulkan interop
copy stream, MJPEG host prefetch) and the single-materialization
numpy-XOR-cuda contract holds. Host/numpy sources keep the PIL path, as
does the obstacle box-outline annotation (debug aid). All compositor math
is device-agnostic and unit-tested on CPU tensors.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
obstacle_ability/test_runner import-block ordering (repo ruff 0.12.7
--select I), C408 dict literals, PLR0402/RUF059/RUF046/UP037 from the
current ruff default set on the live-edit files.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every HTTP endpoint (page, /stream, /bev_stream, /control, /state,
/scenes, /thumbnail, /scene/select, /taxi/name) now requires the shared
token via ?token= query param or X-Stream-Token header when the flag is
set; requests without a valid token get 403. The served page reads the
token from its own URL and appends it to all of its requests. Omitting
the flag keeps the historical open behavior.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Escalates storm along the cues that materialize (2026-08-21 A/B at
guidance 3.5/4.0 vs the shipped storm frames): visibility collapse,
walls of spray/mist, static debris lying on the flooded road, and a
black-green emergency-gloom sky. Dynamic wind wording (bending trees,
flying debris) is deliberately absent — it never materializes in the
history-anchored model. Shipped at weather guidance 4.0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@aidanfnv aidanfnv left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome!
The style transfers probably need a follow-up to clear up some issues with how they look in motion (see my message in Slack), but the weather effects look incredible, and the coins and obstacles work well. My new map format will break the obstacle ability, but that will of course be on me to fix.
Thank you for this!

wenqingw-nv and others added 2 commits August 21, 2026 22:18
Aidan reported coins still hurting native-path perf after the CUDA
compositor fix. Profiling at real course scale (4395 coins on the shipped
suburb scene) found two costs the 9-coin measurement missed:

- visible_sprites/advance_frames scanned all coins every frame
  (0.60 ms/frame CPU at 4400 coins). A static 32 m spatial hash with
  per-cell cached windows plus a vectorized projection filter makes both
  O(nearby): 0.20 ms/frame, pickup 0.10 -> 0.015 ms.
- The compositor is CPU-launch-bound, not bandwidth-bound (GPU drain adds
  ~0 in every measurement), and dense areas put 211 sprites inside the
  120 m render radius -> 11-14 ms/frame enqueue. Fixes: one blend per coin
  (shadow+coin+quantized fade pre-composited into one cached texture,
  2.6x fewer kernels: 1.14 -> 0.44 ms at 9 sprites, 5.20 -> 1.84 ms at
  48) and a nearest-N sprite cap (--live-edit-coin-max-visible,
  default 64) so dense courses cannot blow the budget.

Also adds the ROI-only uint8 blend path (LIVE_EDIT_COMPOSITOR=roi) that
never converts the full frame to float32 - exact parity with the float
path (max diff 0) and minimal GPU traffic, but slower wall-clock on
launch-bound machines, so the float canvas stays the default - and a
perf self-report (--live-edit-perf-log N / LIVE_EDIT_PERF_LOG) that logs
p50/p95 coin CPU ms, compositor enqueue CPU ms, and compositor GPU ms
(lazy CUDA event pairs, no added sync) every N composited frames so
remote users can send numbers instead of guesses.

Validated headless MJPEG at course scale: chunk cadence unchanged
(219-242 ms, matching the no-coins baseline), coin rendering parity vs
the previous compositor within 1.5 LSB mean on sprite pixels (fade
quantization + <=1 px shadow anchor). New tests cover the windowed
culling, the sprite cap, ROI/float parity, and the perf log.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One O press now spawns --live-edit-obstacle-count clones (default 1 =
unchanged single obstacle): distinct crossing/oncoming template tracks
(never pace-matched lead cars, which render at ghost strength), staggered
across a 16-40 m band ahead of the ego (spacing_m per slot) and
--live-edit-obstacle-stagger-chunks apart in time, each retimed+rigidly
shifted with ground-z correction and despawning after its own pass.

Guidance already strips every OBSTACLE_ENTITY_PREFIX actor from the
shadow branch, so the box-axis path handles simultaneous boxes unchanged.
Mask-anchored validation (RNG-matched baseline, per-clone camera-projected
boxes, 26-chunk headless runs): s=2.0 lands 2/3 clones at N=3 (52/61/18%
of real-car in-box benchmark) and 4/5 at N=5 (52/60/15/69/83%), out-box
clean (median 5.7-7.5); s=2.5 is stronger still at N=3 (63/79/22%) with
no breakup, so the traffic demo ships at s=2.5. Simultaneous spawns
(stagger 0) are stable; the 1-chunk default is for on-screen pacing.
The one weak clone is the fastest template (~11 m/s) which holds distance
until late. The presenter annotates every active clone and shows a
TRAFFIC xN chip.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants