Skip to content

Add t2v API and integrations and benchmark support - #493

Merged
jarcherNV merged 20 commits into
mainfrom
dev/jarcher/v2-t2v
Aug 21, 2026
Merged

Add t2v API and integrations and benchmark support#493
jarcherNV merged 20 commits into
mainfrom
dev/jarcher/v2-t2v

Conversation

@jarcherNV

@jarcherNV jarcherNV commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

The first model behind the v2 protocol and the first user of the batch runner and Mp4OutputSink. Output is MP4 only, so there is no input side yet. The check it runs is shared, in flashdreams.testing_v2.t2v_conformance, since every t2v model wants the same coverage. CPU tests use a stand-in model; the real one carries ci_gpu and skips unless T2V_SELF_FORCING_REAL_MODEL_RUN is set, because pytest-manual-marker fails manual tests at setup and they never run.

@copy-pr-bot

copy-pr-bot Bot commented Aug 20, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@jarcherNV
jarcherNV force-pushed the dev/jarcher/v2-t2v branch 3 times, most recently from 4a058ac to 7c4b0d3 Compare August 20, 2026 07:21
@jarcherNV jarcherNV changed the title Add a Self-Forcing text-to-video application on the v2 API Add t2v API and integrations and benchmarks for them Aug 20, 2026
@jarcherNV jarcherNV changed the title Add t2v API and integrations and benchmarks for them Add t2v API and integrations and benchmark support Aug 20, 2026
@jarcherNV
jarcherNV force-pushed the dev/jarcher/v2-mp4-output branch 2 times, most recently from d8e49b8 to fc7be28 Compare August 20, 2026 19:02
Comment thread flashdreams/flashdreams/runtime_v2/cli.py Outdated
Comment thread flashdreams/flashdreams/runtime_v2/cli.py Outdated
Comment thread flashdreams/flashdreams/runtime_v2/cli.py Outdated
Comment thread flashdreams/flashdreams/t2v_v2/application.py Outdated
Base automatically changed from dev/jarcher/v2-mp4-output to main August 20, 2026 20:42
The first model behind the v2 protocol and the first user of the batch
runner and Mp4OutputSink. Output is MP4 only, so there is no input side
yet. The check it runs is shared, in
flashdreams.testing_v2.t2v_conformance, since every t2v model wants the
same coverage. CPU tests use a stand-in model; the real one carries
ci_gpu and skips unless T2V_SELF_FORCING_REAL_MODEL_RUN is set, because
pytest-manual-marker xfails manual tests at setup and they never run.
The v2 sink writes the measurements a step reports, and a benchmark is
only one thing that reads them, so MetricsOutputSink says what it is
where BenchmarkStatsOutputSink said who asked for it. The v1 sink keeps
its name, and the artifact type both write is unchanged, so a report can
still hold runs of either API.
The base branch removed run_batch, so the two places here that drove it
now drive run_session: the command line against an Mp4ClientWindow, and
the shared t2v check against a window of its own that wraps its frame
inspector.

A benchmark still wants the clip and the measurements from one run, and a
window owns what it writes, so Mp4ClientWindow takes the stats path and
adds a MetricsOutputSink beside the MP4 writer. That is the last caller
CompositeOutputSink had, so it goes.
@@ -0,0 +1,414 @@
{

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file is just the benchmark config stuff for running the new integrations.

Comment thread flashdreams/flashdreams/runtime_v2/cli.py Outdated
Comment thread flashdreams/flashdreams/runtime_v2/cli.py Outdated
Comment thread flashdreams/flashdreams/runtime_v2/cli.py
Comment thread flashdreams/flashdreams/runtime_v2/cli.py Outdated
@jarcherNV
jarcherNV marked this pull request as ready for review August 20, 2026 22:14
@greptile-apps

greptile-apps Bot commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds the v2 text-to-video application protocol, model integrations, MP4/metrics output, CLI discovery, and benchmark scenarios. It also extends lifecycle ownership so windows are closed when application initialization or session creation fails.

  • Adds shared T2V application/session defaults and conformance coverage.
  • Adds five independently packaged v2 model integrations.
  • Adds MP4 benchmark output, runtime metrics, strict deterministic execution, and benchmark scenarios.
  • Adds application discovery and MP4/WebRTC client-window modes to the v2 CLI.

Confidence Score: 4/5

The PR is not yet safe to merge because a timed-out WebRTC construction can leave an unreachable server thread and listening socket alive.

WebRTCServer starts its thread before waiting for readiness, but a timeout raises through mode.create without stopping that thread or returning an object the CLI can close.

Files Needing Attention: flashdreams/flashdreams/runtime_v2/cli.py and flashdreams/flashdreams/runtime_v2/serving/webrtc_server.py

Important Files Changed

Filename Overview
flashdreams/flashdreams/runtime_v2/cli.py Adds the v2 application CLI and mode lifecycle, but WebRTC construction failure can leave startup resources unowned.
flashdreams/flashdreams/runtime_v2/application_runner.py Correctly closes the client window when application initialization or session creation fails before run_session takes ownership.
flashdreams/flashdreams/runtime_v2/client_window_factory.py Centralizes MP4 and WebRTC mode creation, arguments, and user-facing run reports.
flashdreams/flashdreams/t2v_v2/application.py Implements the shared v2 text-to-video application configuration and session factory.
flashdreams/flashdreams/t2v_v2/session.py Implements finite T2V generation sessions and model-output adaptation.
configs/v2_model_benchmarks.json Adds seeded quality and performance scenarios for the five v2 T2V integrations.

Sequence Diagram

sequenceDiagram
  participant User
  participant CLI as flashdreams-run-v2
  participant Mode as ClientWindowMode
  participant Server as WebRTCServer
  participant Runner as ApplicationRunner
  participant App as T2VApplication
  participant Session as T2VSession
  User->>CLI: select application and mode
  CLI->>Mode: create(parsed)
  Mode->>Server: construct and start thread
  alt startup succeeds
    Server-->>Mode: live window
    Mode-->>CLI: window
    CLI->>Runner: run(session_desc, args)
    Runner->>App: init and create_session
    Runner->>Session: run_session
    Runner-->>CLI: cleanup complete
  else startup times out
    Server--xMode: TimeoutError
    Note over Server: Thread may continue without a cleanup owner
  end
Loading

Reviews (2): Last reviewed commit: "Simplified readmes" | Re-trigger Greptile

Comment thread flashdreams/flashdreams/runtime_v2/cli.py Outdated
Comment thread flashdreams/flashdreams/runtime_v2/cli.py

@gtong-nv gtong-nv left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@jarcherNV
jarcherNV added this pull request to the merge queue Aug 20, 2026
Merged via the queue into main with commit d5c4ce0 Aug 21, 2026
7 checks passed
@jarcherNV
jarcherNV deleted the dev/jarcher/v2-t2v branch August 21, 2026 00:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants