fix(g1): add video stream and camera_info to G1SimConnection#1335
Merged
spomichter merged 8 commits intodevfrom Feb 21, 2026
Merged
fix(g1): add video stream and camera_info to G1SimConnection#1335spomichter merged 8 commits intodevfrom
spomichter merged 8 commits intodevfrom
Conversation
G1SimConnection was missing color_image and camera_info outputs, so MuJoCo-rendered frames never reached the rerun bridge. Mirrors the Go2 connection pattern: subscribes to video_stream(), publishes camera_info in a background thread, and adds camera_optical TF frame.
Contributor
Greptile SummaryAdded video streaming and camera info publication to Key changes:
Confidence Score: 3/5
Important Files Changed
Last reviewed commit: 6233235 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
dimos/robot/unitree/g1/sim.py
Outdated
Comment on lines
111
to
114
| def _publish_camera_info_loop(self) -> None: | ||
| while True: | ||
| self.camera_info.publish(_camera_info_static()) | ||
| time.sleep(1.0) |
Contributor
There was a problem hiding this comment.
infinite loop without exit condition
the while True loop has no mechanism to stop when stop() is called. the thread join with 1s timeout might not be sufficient if the loop is sleeping. add a stop event:
Suggested change
| def _publish_camera_info_loop(self) -> None: | |
| while True: | |
| self.camera_info.publish(_camera_info_static()) | |
| time.sleep(1.0) | |
| def _publish_camera_info_loop(self) -> None: | |
| while not self._stop_event.is_set(): | |
| self.camera_info.publish(_camera_info_static()) | |
| self._stop_event.wait(1.0) |
then add _stop_event: threading.Event to the class and initialize it in start()
Comment on lines
41
to
57
| def _camera_info_static() -> CameraInfo: | ||
| """Default camera intrinsics for G1 MuJoCo sim camera.""" | ||
| fx, fy, cx, cy = (819.553492, 820.646595, 625.284099, 336.808987) | ||
| width, height = (1280, 720) | ||
|
|
||
| return CameraInfo( | ||
| frame_id="camera_optical", | ||
| height=height, | ||
| width=width, | ||
| distortion_model="plumb_bob", | ||
| D=[0.0, 0.0, 0.0, 0.0, 0.0], | ||
| K=[fx, 0.0, cx, 0.0, fy, cy, 0.0, 0.0, 1.0], | ||
| R=[1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0], | ||
| P=[fx, 0.0, cx, 0.0, 0.0, fy, cy, 0.0, 0.0, 0.0, 1.0, 0.0], | ||
| binning_x=0, | ||
| binning_y=0, | ||
| ) |
Contributor
There was a problem hiding this comment.
hardcoded camera intrinsics don't match actual video dimensions
the function uses width=1280, height=720 and specific focal lengths, but MujocoConnection actually outputs 320x240 video (per dimos/simulation/mujoco/constants.py:18-19). this mismatch will cause incorrect pinhole projection in rerun.
either:
- use
MujocoConnection.camera_info_static(already computed correctly) - or match the constants:
width=320, height=240and computefx/fyfrom FOV=45°
see mujoco_connection.py:91-114 for the correct implementation
The rerun visual_overrides from #1334 used world/camera_info and world/color_image (matching Go2), but the G1 transports used /g1/ prefixed topics. This mismatch meant the rerun bridge never applied the camera_info pinhole overlay. Remove the /g1/ prefix from color_image and camera_info transports in the primitive blueprint and the SHM blueprint. Now matches Go2 convention and the rerun overrides work correctly.
In sim, G1SimConnection now provides video from MuJoCo. The webcam camera_module in the primitive was also publishing to color_image, causing interleaved webcam + MuJoCo frames. Conditionally skip camera_module when global_config.simulation is True.
…dition - Replace hardcoded 1280x720 Go2 intrinsics with computed values from MuJoCo constants (320x240, FOV=45°) matching MujocoConnection pattern - Add _stop_event to camera_info loop for clean shutdown
MuJoCo framebuffer can't exceed 640 width without XML config. Reverted render resolution to 320x240. Use the same 1280x720 camera intrinsics as Go2 for rerun display scaling (matches Go2 sim behavior exactly).
Set offscreen framebuffer size in model XML to support the higher resolution. Previously failed because MuJoCo's default offscreen buffer is only 640 wide.
This reverts commit f21d37b.
Contributor
Author
|
@greptile |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
G1 sim shows no video in rerun —
G1SimConnectiononly hadlidarandodomoutputs, missingcolor_imageandcamera_info. MuJoCo-rendered frames were never published.Solution
Mirror the Go2 connection pattern:
color_image: Out[Image]andcamera_info: Out[CameraInfo]MujocoConnection.video_stream()→ publish oncolor_imagecamera_infoin background thread (1Hz, matching Go2)camera_opticalTF frame for rerun pinhole projectionHow to Test
dimos --simulation run unitree-g1-sim # Verify video stream appears in rerun viewer