Add waypoint-1.5-1B support

[lpy]

This patch adds support for waypoint-1.5-1B, a real time interactive video world model, which is designed to generate on consumer hardware.

This patch:

1. extends the existing taehv to support encoder;
2. implements the necessary bits, e.g. DiT, adaptive RMSNorm, etc;
3. supports using JSON file to pass in controls conditions.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[greptile-apps]

Greptile Summary

The PR adds the Waypoint 1.5 1B integration and extends TAEHV with Hunyuan Video 1.5 encoding support.

• Adds the Waypoint DiT, control conditioning, scheduler, checkpoint validation, cache handling, pipeline, runner, and plugin registration.
• Adds JSON control timelines, example data, documentation, and integration tests.
• Updates seed initialization so transformer and decoder causal histories each advance once.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
integrations/waypoint/waypoint/pipeline.py	Initializes aligned transformer and decoder histories from a single encoded seed action.
integrations/waypoint/waypoint/transformer/network.py	Implements the Waypoint DiT topology, conditioning paths, attention, and checkpoint-compatible parameter layout.
integrations/waypoint/waypoint/transformer/cache.py	Implements local and pinned-global Waypoint KV-cache visibility and replacement behavior.
flashdreams/flashdreams/recipes/taehv/impl.py	Extends TAEHV with a causal streaming encoder and Hunyuan Video 1.5 model topology.
integrations/waypoint/waypoint/runner.py	Adds seed preparation, control timeline execution, rollout sequencing, and video output.
uv.lock	Registers the new workspace package and its image/video dependencies without changing the pre-existing advised dependency versions.

Sequence Diagram

sequenceDiagram
  participant Runner
  participant Pipeline
  participant Encoder as TAEHV Encoder
  participant Transformer as Waypoint DiT
  participant Decoder as TAEHV Decoder
  Runner->>Pipeline: initialize_cache(seed_pixels)
  Pipeline->>Encoder: encode four seed frames
  Encoder-->>Pipeline: one seed latent
  Pipeline->>Transformer: commit seed as action 0
  Pipeline->>Decoder: decode seed once into causal history
  loop Actions 1..N
    Runner->>Pipeline: generate(action, controls)
    Pipeline->>Transformer: denoise next latent
    Transformer-->>Pipeline: generated latent
    Pipeline->>Decoder: decode four RGB frames
    Decoder-->>Runner: output frames
  end

Loading

_{Reviews (2): Last reviewed commit: "Add waypoint-1.5-1B support" | Re-trigger Greptile}

[natduca]

Thanks for the patch! What's the perf you're seeing on this so far?

[natduca]

@jarcherNV @csyonghe do we have a normative way to update the public landing page on which models we support and what perf we're getting so that as we get patches the world knows where we stand in terms of model support and leading perf?

[lpy]

@natduca On an RTX5090, I enabled --pipeline.enable-sync-and-profile True and if I take step index 20 and 40 as examples, below are some perf data, note that for each step the model generates 4 frames.

  {
    "step_index": 20,
    "frames": 4,
    "encode_ms": 0.033663999289274216,
    "diffuse_ms": 232.82838439941406,
    "decode_ms": 4.548480033874512,
    "finalize_ms": 48.62908935546875,
    "total_ms": 286.0396177880466,
    "total_ms_wo_finalize": 237.41052843257785,
    "mem_alloc_gib": 5.735983371734619,
    "mem_reserved_gib": 6.326171875,
    "mem_peak_gib": 5.996957778930664
  },

  {
    "step_index": 40,
    "frames": 4,
    "encode_ms": 0.03190400078892708,
    "diffuse_ms": 183.836669921875,
    "decode_ms": 4.573440074920654,
    "finalize_ms": 46.94508743286133,
    "total_ms": 235.3871014304459,
    "total_ms_wo_finalize": 188.44201399758458,
    "mem_alloc_gib": 5.735983371734619,
    "mem_reserved_gib": 6.326171875,
    "mem_peak_gib": 5.996957778930664
  },

Full command I ran was uv run flashdreams-run waypoint-1.5-1b --example-data True --pipeline.enable-sync-and-profile True --output-dir .\outputs\waypoint_perf.

[jarcherNV]

@natduca I don't believe we have such a process yet. I have created a github issue for it here for someone to take on depending on who has cycles: #475

[jarcherNV]

@aidanfnv Is that issue related to the documentation stuff you are working on?