webRTC ClientWindow - #491
Conversation
Provide v2 browser input buffering and video delivery without depending on the existing serving stack.
Keep transport callbacks internal while preserving the polling interface used by the session runner.
5231021 to
49de20e
Compare
Wire application lifecycle and WebRTC presentation end to end, including browser controls and responsive red-screen output.
Keep browser markup and behavior independently editable while packaging and serving both assets with the runtime.
Centralize presentation mode selection so integrations reuse one runtime-owned factory.
| <video id="video" autoplay playsinline></video> | ||
| <button id="activate" type="button">Activate</button> | ||
| <button id="reset" type="button">Reset</button> | ||
| <script src="/app.js"></script> |
There was a problem hiding this comment.
This is meant to demo the functionality. Can be cleaned up. In the production, only the video track and <script src="/app.js"></script> for registering browser event handlers are needed
|
|
||
|
|
||
| def _result_to_rgb_frames( | ||
| result: StepResult, session_desc: SessionDesc |
There was a problem hiding this comment.
This can be simplified once we have proper shared util classes
Greptile SummaryThe PR adds a v2 application runner and WebRTC client-window implementation, packages its browser resources, and extends the red-screen integration into a browser-accessible demonstration.
Confidence Score: 1/5The PR is not yet safe to merge because buffered controls are still dropped and both remote-input and video-frame queues can grow without bound. The current code still selects only the last event from each buffered batch, accepts unlimited remote control events into a SimpleQueue, and places every generated frame into an unbounded paced-playback queue; these previously reported failures remain reachable. Files Needing Attention: flashdreams/flashdreams/runtime_v2/webrtc_client_window.py, flashdreams/flashdreams/runtime_v2/serving/webrtc_server.py, integrations_v2/red_screen/red_screen/app.py Important Files Changed
Sequence DiagramsequenceDiagram
participant Browser
participant Server as WebRTCServer
participant Window as WebRTCClientWindow
participant Runner as ApplicationRunner
participant Session as RedScreenSession
Browser->>Server: keyboard/button message
Server->>Window: timestamped UserInputEvent
Window-->>Runner: buffered UserInputEvents
Runner->>Session: step(events)
Session-->>Runner: StepResult
Runner->>Window: write(result)
Window->>Server: enqueue video frames
Server-->>Browser: paced WebRTC video
Reviews (2): Last reviewed commit: "Document WebRTC client-window protocol m..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| def _apply_events(self, events: UserInputEvents) -> None: | ||
| # Events are edges, not levels: a key stays held across steps that carry | ||
| # no events for it, so only the last edge per step changes the state. | ||
| for event in events.get_events(): | ||
| data = event.get_event_data() | ||
| if ( | ||
| isinstance(data, KeyboardUserInputEventData) | ||
| and data.key == self._config.activation_key | ||
| ): | ||
| self._key_held = data.pressed | ||
| received_events = events.get_events() | ||
| if not received_events: | ||
| return | ||
| data = received_events[-1].get_event_data() | ||
| if not isinstance(data, KeyboardUserInputEventData): | ||
| return | ||
| if data.key == self._config.activation_key: | ||
| self._key_held = data.pressed | ||
| elif data.pressed and data.key.lower() == "w": | ||
| self._color_intensity = min(1.0, self._color_intensity + 0.1) | ||
| elif data.pressed and data.key.lower() == "s": | ||
| self._color_intensity = max(0.0, self._color_intensity - 0.1) |
There was a problem hiding this comment.
When multiple input edges arrive during one polling interval, _apply_events processes only the final event, causing quick activation taps and repeated intensity changes to be lost.
| def _apply_events(self, events: UserInputEvents) -> None: | |
| # Events are edges, not levels: a key stays held across steps that carry | |
| # no events for it, so only the last edge per step changes the state. | |
| for event in events.get_events(): | |
| data = event.get_event_data() | |
| if ( | |
| isinstance(data, KeyboardUserInputEventData) | |
| and data.key == self._config.activation_key | |
| ): | |
| self._key_held = data.pressed | |
| received_events = events.get_events() | |
| if not received_events: | |
| return | |
| data = received_events[-1].get_event_data() | |
| if not isinstance(data, KeyboardUserInputEventData): | |
| return | |
| if data.key == self._config.activation_key: | |
| self._key_held = data.pressed | |
| elif data.pressed and data.key.lower() == "w": | |
| self._color_intensity = min(1.0, self._color_intensity + 0.1) | |
| elif data.pressed and data.key.lower() == "s": | |
| self._color_intensity = max(0.0, self._color_intensity - 0.1) | |
| def _apply_events(self, events: UserInputEvents) -> None: | |
| for event in events.get_events(): | |
| data = event.get_event_data() | |
| if not isinstance(data, KeyboardUserInputEventData): | |
| continue | |
| if data.key == self._config.activation_key: | |
| self._key_held = data.pressed | |
| elif data.pressed and data.key.lower() == "w": | |
| self._color_intensity = min(1.0, self._color_intensity + 0.1) | |
| elif data.pressed and data.key.lower() == "s": | |
| self._color_intensity = max(0.0, self._color_intensity - 0.1) |
| port: Listening port. Zero asks the operating system to choose one. | ||
| startup_timeout_seconds: Maximum time to wait for server startup. | ||
| """ | ||
| self._input_events: queue.SimpleQueue[UserInputEvent] = queue.SimpleQueue() |
There was a problem hiding this comment.
When a connected client sends valid control messages faster than the UI loop drains them, every event is retained in an unbounded SimpleQueue, causing process memory exhaustion and denial of service. How this was verified: The data-channel callback reaches an unconditional SimpleQueue.put, and the changed path contains no capacity, rate, or coalescing guard before the UI-rate drain.
Knowledge Base Used: WebRTC Serving Flow
| self._frames: asyncio.Queue[np.ndarray[Any, np.dtype[np.uint8]] | None] = ( | ||
| asyncio.Queue() | ||
| ) |
There was a problem hiding this comment.
When a session supplies frames faster than WebRTC playback consumes them, _VideoTrack.enqueue retains every full-resolution CPU frame in an unbounded queue while recv removes only one paced frame at a time, causing memory exhaustion during sustained high-throughput sessions.
Knowledge Base Used: WebRTC Serving Flow
| parser.add_argument( | ||
| "application_args", | ||
| nargs=argparse.REMAINDER, | ||
| help="Arguments after -- are passed to the red-screen application.", |
There was a problem hiding this comment.
all of these should be parsed by flashdreams-run
Make each implementation method's InputSource or OutputSink responsibility explicit.
1f8fb87 to
cc7ff22
Compare
|
/ok to test cc7ff22 |
| @@ -0,0 +1,27 @@ | |||
| # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | |||
There was a problem hiding this comment.
Just leaving a note that I did not use this for the Mp4ClientWindow, as I assume it will be superseded by the new flashdreams-run, so we can remove this file when that happens. Is my understanding right?
2 participants
Summary
ApplicationRunnerand shared client-window factory.Workflow