From 4488cd8b3480b34c8f4272f7e8d55efea2389574 Mon Sep 17 00:00:00 2001 From: William Lew Date: Tue, 18 Aug 2026 11:53:25 -0700 Subject: [PATCH 1/3] Clarify agent coding and architecture guidance Define simple engineering principles and canonical execution paths so agents extend one coherent architecture instead of copying legacy bypasses. Signed-off-by: William Lew --- AGENTS.md | 322 +++++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 259 insertions(+), 63 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 1c4624c1b..3920744f1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,35 +1,260 @@ -# FlashDreams Agent Guide +# FlashDreams — AI Agent Context -FlashDreams is a GPU-heavy inference and serving library for autoregressive video and world models. Default to inspection, docs, config checks, and CPU tests unless the user explicitly asks to run generation or GPU workflows. +## Purpose -Start here, then use the narrower docs for the task in front of you: +FlashDreams is a GPU-heavy inference and serving library for autoregressive +video and world models. Researchers will modify the code for new models, +hardware, and experiments, so optimize for code that is easy to understand and +change rather than code that anticipates every possible use. -- `skills/` contains repo-authored Agent Skills. Use `skills/README.md` for opt-in setup and skill authoring rules. -- `README.md` covers user-facing setup, requirements, supported models, and first-run commands. -- `CONTRIBUTING.md` covers PR process, DCO sign-off, coding conventions, test markers, and dependency rules. -- `docs/README.md` covers Sphinx docs build and hosting. `tests/README.md` covers local, Docker, and CI-oriented test entry points. +These rules apply to all newly written or modified code, even when nearby code +uses older patterns. -## Repo Map +## Coding Principles -- `flashdreams/flashdreams/core/`: reusable numerical primitives, checkpoint loading, distributed helpers, attention, and I/O. Keep it model-agnostic. -- `flashdreams/flashdreams/infra/`: framework contracts and orchestration for configs, pipelines, encoders, decoders, diffusion models, schedulers, runners, profiling, and CUDA graph wrapping. -- `flashdreams/flashdreams/recipes/`: built-in reusable recipe code such as WAN, Cosmos, TAEHV, and template wiring. -- `flashdreams/flashdreams/configs/`, `plugins/`, and `scripts/`: runner registry, plugin discovery, and CLI entry points. -- `integrations//`: workspace-member model/plugin packages with their own configs, runners, tests, README files, and `pyproject.toml` entry points. -- `docs/source/`: Sphinx sources for quickstart, models, developer guides, API, and community docs. -- `tests/`: root test helpers plus package/integration tests. Ignore `.claude/worktrees/` when scanning the source tree; those are nested worktree artifacts, not the repo's current source. +- **Readability over flexibility.** Add an abstraction only when it reduces net + complexity. Prefer direct code over frameworks, registries, wrappers, and + extension points created for hypothetical reuse. +- **Solve the present problem.** Implement the smallest complete behavior the + request needs. Do not bundle adjacent features, compatibility layers, or + speculative generality. +- **One way to do it.** Each concept should have one representation and one + execution path. Remove superseded paths instead of keeping aliases, fallback + branches, or parallel APIs. +- **Locality of behavior.** Keep related state and logic together. Every + indirection must improve readability or provide genuine reuse across several + call sites. +- **Configuration has one owner.** Do not add a CLI flag, environment variable, + constant default, and config field for the same setting. Extend the existing + authoritative configuration path. +- **Fail fast.** Reject malformed or unsupported input at the boundary. Do not + silently substitute defaults, swallow exceptions, or continue through a + partially initialized state. +- **Preserve dependency direction.** Generic runtime code must not know about a + model integration. Model-specific behavior belongs with that integration. +- **Comments describe current code.** Never leave migration history, “temporary” + notes without an issue, or commentary about what code used to do. +- **Google Python Style Guide** is the general baseline. -## Skill Map +## Simplicity Rules -- `skills/flashdreams-integrations`: read before changing recipe/integration architecture, runner registration, pipeline wiring, or model boundaries. -- `skills/integrate-a-model`: read before porting a new external model or reproducing an integration workflow. -- `skills/profile-model-performance`: read before starting performance work on an existing model integration, demo, runner, or serving path; use it to map execution, add stage timings, and identify decode/model/cache/transfer/presentation bottlenecks. -- `skills/apply-inference-optimizations`: read before porting runtime speedups such as bounded K/V caches, overlap, compile, CUDA graphs, decoder layout changes, or presentation queue tuning into an integration. -- `skills/validate-performance-quality`: read before adding benchmark sweeps, quality comparisons, profiler probes, performance summaries, or docs for a performance change. -- `skills/flashdreams-postprocessing`: read before adding or modifying video post-processors, postprocess presets, `VideoPostprocessStream`, buffering/layout behavior, or runner postprocess wiring. -- `skills/python-docstring-style`: read before adding or polishing Python docstrings, field docstrings, module comments, or SPDX headers. -- `skills/maintaining-oss-state`: read before dependency, license, NOTICE, REUSE, or OSS-release collateral changes. -- When adding a new `skills//SKILL.md`, update this section so agents can discover when to use it. +### Minimize abstraction layers + +Add a class or wrapper only when it removes meaningful duplication across at +least three call sites or represents a real domain object. + +```python +# Discouraged: one-use delegation. +class SessionStarter: + def start(self, runtime, inputs): + return runtime.start_session(inputs) + +# Preferred. +session = runtime.start_session(inputs) +``` + +Use dataclasses for domain objects, protocol payloads, persisted artifacts, and +configuration schemas. Do not create dataclasses that merely bundle arguments +for one local call. + +### Avoid defensive attribute probing + +Do not use `getattr(...)` chains to guess which interface an object implements. +Define the expected interface and call it directly. Let missing attributes fail +at the actual programming error. + +### Avoid unnecessary constants + +Inline file-local literals by default. Introduce a named constant when multiple +components must agree on the value and drift would cause a subtle failure, such +as a wire-protocol key or environment variable name. + +### No dead-code compatibility + +FlashDreams is evolving quickly. Unless the user explicitly requires a staged +migration, delete old constructors, parameters, aliases, adapters, and branches +when replacing them. A second path is a maintenance cost, not free safety. + +### Explicit signatures and useful types + +Prefer explicit keyword parameters over argument bags. Add local annotations +when a boundary returns an untyped value and naming the concrete type improves +understanding. Do not add annotations that force casts or repeat obvious types. + +## Error Handling + +- Use normal exceptions and preserve their traceback. +- Validate external data, user input, checkpoint metadata, and transport + messages at their boundary. +- Do not catch an exception unless the code can recover, add actionable context, + or complete required cleanup. +- Never use a broad fallback to hide a missing dependency, unsupported backend, + or failed optimization. +- “Auto” behavior must report what it selected and why; it must not silently + change model semantics or output quality. + +## Documentation and Comments + +- Every public method needs a clear docstring describing behavior, arguments, + return value, and important failure modes. +- Private methods need concise docstrings when their purpose is not obvious. +- Prefer plain language over architecture jargon. +- Explain invariants and non-obvious constraints, not line-by-line mechanics. +- Keep docs, examples, and launch commands synchronized with real entry points. +- Use the repository SPDX header on new source files. + +## Tests Prove Behavior + +Write a small number of high-signal tests for: + +- user-visible behavior and regressions; +- numerical or state-machine correctness; +- validation at external boundaries; +- parity between two paths that claim identical semantics; +- failure and cleanup behavior that local reasoning cannot prove. + +Do not write tests that: + +- restate constants, config defaults, package metadata, or function signatures; +- verify that a mock was called without asserting the resulting behavior; +- snapshot implementation details instead of a public outcome; +- duplicate checks already guaranteed by static typing; +- require a GPU when a CPU/meta-tensor test proves the same contract. + +Every pytest test must carry exactly one marker: `ci_cpu`, `ci_gpu`, or +`manual`. Use module-level `pytestmark = pytest.mark.ci_cpu` for CPU-safe test +modules. Plain `pytest` includes manual tests; normal validation should use a +marker expression. + +## Project Boundaries + +- `flashdreams/flashdreams/core/`: model-agnostic numerical primitives, + checkpoint loading, attention, distributed helpers, and I/O. +- `flashdreams/flashdreams/infra/`: framework contracts and orchestration. +- `flashdreams/flashdreams/recipes/`: built-in reusable model building blocks. +- `integrations//`: model-specific packages, configs, runners, and tests. +- `apps//`: user-facing application behavior composed with integrations. +- `docs/source/`: Sphinx user and developer documentation. + +Dependency direction is: + +```text +core -> infra -> recipes/integrations -> apps +``` + +`core` and `infra` must never import from `integrations` or `apps`. Do not add a +model-name conditional to shared code. If behavior is truly reusable, define a +small generic contract; otherwise keep it in the integration. + +## Critical Paths + +These are the intended end-to-end routes. New code should extend them rather +than create a second orchestration, input, inference, or presentation path. + +### Application execution + +```text +CLI +-> Application +-> InputHandler +-> CanonicalInputWindow +-> ApplicationSession +-> StepResult +-> OutputSink +``` + +The application owns user-facing behavior and session orchestration. +`InputHandler` owns device/transport capture and produces canonical, +device-independent input. `ApplicationSession` owns sequential application +state. `OutputSink` consumes results; it does not drive generation. + +### Realtime and WebRTC input + +```text +browser/device +-> UserInputEvent +-> UserInputWindow +-> DeviceConverter +-> CanonicalInputWindow +-> ModelInputProvider +-> InferenceSession +-> video transport +``` + +Transport code captures, timestamps, validates, and buffers raw input. +`DeviceConverter` owns device-to-canonical semantics. `ModelInputProvider` owns +canonical-to-model conditioning. Do not translate controller input into +keyboard input, interpret devices inside a model provider, or let a presenter +mutate model state. + +### Ownership + +```text +app -> orchestration and user-visible behavior +integration -> checkpoint loading, model setup, and inference semantics +core runtime -> session lifecycle, step execution, and shared contracts +presenter -> output delivery only +``` + +Code that bypasses these routes may be unmigrated legacy code; it is not a +pattern to copy. When the requested change touches such a bypass, prefer moving +that behavior onto the critical path and removing the duplicate route. Do not +expand an unrelated task into a migration solely to satisfy this guideline. + +## Working Process + +1. Read the relevant implementation, tests, config, and local documentation + before proposing a change. +2. Identify the current owner of the behavior and remove duplicate ownership. +3. State the smallest end-to-end behavior that will count as done. +4. Implement one coherent path; delete the path it replaces. +5. Validate proportionally: focused CPU tests first, then broader checks only + when the change warrants them. +6. Review the diff for unnecessary wrappers, comments, compatibility code, + unrelated formatting, and generated artifacts. + +Keep changes reviewable. Do not turn a bug fix into a subsystem rewrite or a +feature into unrelated cleanup. + +## GPU and Performance Work + +- Default to inspection, config resolution, CPU tests, and meta tensors unless + the user explicitly requests GPU execution. +- Use `flashdreams-run --no-instantiate` before constructing models. +- Do not run generation, `torchrun`, large downloads, or GPU test suites merely + to “be thorough.” +- Performance claims require reproducible measurements and unchanged quality + semantics. +- Separate warmup/compile/autotune time from steady-state measurements. +- Never report asynchronous CUDA timings as end-to-end latency without the + required synchronization. + +## Dependencies and OSS State + +- Add dependencies with the owning package manager and the narrowest package + scope. +- Do not add a dependency for behavior that is simpler to implement directly. +- Never hand-edit generated lockfile contents. +- Check license, NOTICE, REUSE, and package-data implications before vendoring + code or assets. +- Never commit credentials, internal hostnames, private URLs, or local cache + paths. + +## Repository Skills + +Read the relevant skill before changing its area: + +- `skills/flashdreams-integrations` +- `skills/integrate-a-model` +- `skills/profile-model-performance` +- `skills/apply-inference-optimizations` +- `skills/validate-performance-quality` +- `skills/flashdreams-postprocessing` +- `skills/python-docstring-style` +- `skills/maintaining-oss-state` + +When adding or moving a skill, update this list and run the skill frontmatter +tests. ## Common Commands @@ -37,49 +262,20 @@ Start here, then use the narrower docs for the task in front of you: uv sync --extra dev --extra runners uv sync --package --extra dev uv run flashdreams-run --help -uv run flashdreams-run --help uv run flashdreams-run --no-instantiate -uv run --group lint pre-commit run -a uv run pytest -m ci_cpu uv run pytest -m "not manual" -./tests/run_tests_local.sh [target] -./tests/run_tests_docker.sh [target] +uv run --group lint pre-commit run -a uv run --group docs sphinx-build -b html docs/source docs/_build/html -uv run --group docs sphinx-autobuild -E docs/source docs/_build/html --port 8000 ``` -Use `--no-instantiate` before GPU work to inspect the resolved runner config without constructing models or loading checkpoints. - -## No-GPU Workflow - -- Inspect available runners with `uv run flashdreams-run --help`, then inspect a specific runner with `uv run flashdreams-run --no-instantiate `. -- Prefer CPU checks first: config imports, checkpoint key-remap shape/bijection tests on CPU or meta tensors, docs builds, `pytest -m ci_cpu`, and static assertions about runner names and pipeline wiring. -- Avoid `ci_gpu`, generation, `torchrun`, Docker GPU tests, large Hugging Face downloads, rollout parity, CUDA graph, WebRTC runtime, and quality-regression tests on CPU-only hosts unless the user requests them or the test explicitly skips cleanly. - -## Testing Guidance - -Every pytest test must carry exactly one of `ci_cpu`, `ci_gpu`, or `manual`; `CONTRIBUTING.md` has the exact rules. Use module-level `pytestmark = pytest.mark.ci_cpu` for pure Python/metadata tests. Keep GPU, `libGL`/`cv2`, large-checkpoint, credential, and download-heavy checks out of `ci_cpu`. - -## Boundaries - -Keep dependency direction strict: `core` -> `infra` -> recipes/integrations. `core` and `infra` must not import from `integrations/`; expose a generic config slot or override hook instead of adding model-specific branches. Built-in reusable model pieces belong in `flashdreams/flashdreams/recipes/`; standalone plugin packages belong in `integrations//`. - -## Known Pitfalls - -- FlashDreams targets large NVIDIA GPUs; many real rollouts need 80 GB VRAM and CUDA-capable dependencies. -- Full `uv sync`/`uv run` can build CUDA packages such as Transformer Engine. For narrow work, use `uv sync --package --extra dev` when possible. -- `flashdreams-run --no-instantiate` resolves config only; it does not prove weights, CUDA execution, or parity. -- First GPU runs include compile/autotune warmup, so benchmark harnesses should discard warmup chunks. -- Docs CI mocks heavy GPU packages; local docs work should use the commands in `docs/README.md`. -- Plain `pytest` includes manual tests. Use `pytest -m ci_cpu` or `pytest -m "not manual"` for normal local validation. -- New or moved skills must pass the Agent Skills frontmatter checks in `tests/test_agent_skills.py`. +Prefer package-scoped sync and tests. Full workspace operations can build CUDA +packages such as Transformer Engine and should be used only when necessary. -## Troubleshooting Links +## Source of Truth -- Setup and requirements: `README.md`, `docs/source/quickstart/installation.rst` -- CLI details: `docs/source/api/cli.rst` -- Integration/plugin layout: `docs/source/api/integrations.rst` -- New integrations: `docs/source/developer_guides/new_integration.rst` -- Docs and CPU autodoc: `docs/README.md` -- Tests and quality regressions: `tests/README.md` -- Security reports: `SECURITY.md` +- `README.md`: installation, supported models, and user workflows. +- `CONTRIBUTING.md`: DCO, PR process, coding conventions, and test markers. +- `docs/README.md`: documentation builds. +- `tests/README.md`: test entry points and CI behavior. +- `SECURITY.md`: vulnerability reporting. From 4f5b60704b14687e4d3689ecf5502b1a289e7181 Mon Sep 17 00:00:00 2001 From: William Lew Date: Wed, 19 Aug 2026 09:34:51 -0700 Subject: [PATCH 2/3] Trim outdated content --- AGENTS.md | 150 ------------------------------------------------------ 1 file changed, 150 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 3920744f1..b94a39114 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -126,156 +126,6 @@ Every pytest test must carry exactly one marker: `ci_cpu`, `ci_gpu`, or modules. Plain `pytest` includes manual tests; normal validation should use a marker expression. -## Project Boundaries - -- `flashdreams/flashdreams/core/`: model-agnostic numerical primitives, - checkpoint loading, attention, distributed helpers, and I/O. -- `flashdreams/flashdreams/infra/`: framework contracts and orchestration. -- `flashdreams/flashdreams/recipes/`: built-in reusable model building blocks. -- `integrations//`: model-specific packages, configs, runners, and tests. -- `apps//`: user-facing application behavior composed with integrations. -- `docs/source/`: Sphinx user and developer documentation. - -Dependency direction is: - -```text -core -> infra -> recipes/integrations -> apps -``` - -`core` and `infra` must never import from `integrations` or `apps`. Do not add a -model-name conditional to shared code. If behavior is truly reusable, define a -small generic contract; otherwise keep it in the integration. - -## Critical Paths - -These are the intended end-to-end routes. New code should extend them rather -than create a second orchestration, input, inference, or presentation path. - -### Application execution - -```text -CLI --> Application --> InputHandler --> CanonicalInputWindow --> ApplicationSession --> StepResult --> OutputSink -``` - -The application owns user-facing behavior and session orchestration. -`InputHandler` owns device/transport capture and produces canonical, -device-independent input. `ApplicationSession` owns sequential application -state. `OutputSink` consumes results; it does not drive generation. - -### Realtime and WebRTC input - -```text -browser/device --> UserInputEvent --> UserInputWindow --> DeviceConverter --> CanonicalInputWindow --> ModelInputProvider --> InferenceSession --> video transport -``` - -Transport code captures, timestamps, validates, and buffers raw input. -`DeviceConverter` owns device-to-canonical semantics. `ModelInputProvider` owns -canonical-to-model conditioning. Do not translate controller input into -keyboard input, interpret devices inside a model provider, or let a presenter -mutate model state. - -### Ownership - -```text -app -> orchestration and user-visible behavior -integration -> checkpoint loading, model setup, and inference semantics -core runtime -> session lifecycle, step execution, and shared contracts -presenter -> output delivery only -``` - -Code that bypasses these routes may be unmigrated legacy code; it is not a -pattern to copy. When the requested change touches such a bypass, prefer moving -that behavior onto the critical path and removing the duplicate route. Do not -expand an unrelated task into a migration solely to satisfy this guideline. - -## Working Process - -1. Read the relevant implementation, tests, config, and local documentation - before proposing a change. -2. Identify the current owner of the behavior and remove duplicate ownership. -3. State the smallest end-to-end behavior that will count as done. -4. Implement one coherent path; delete the path it replaces. -5. Validate proportionally: focused CPU tests first, then broader checks only - when the change warrants them. -6. Review the diff for unnecessary wrappers, comments, compatibility code, - unrelated formatting, and generated artifacts. - -Keep changes reviewable. Do not turn a bug fix into a subsystem rewrite or a -feature into unrelated cleanup. - -## GPU and Performance Work - -- Default to inspection, config resolution, CPU tests, and meta tensors unless - the user explicitly requests GPU execution. -- Use `flashdreams-run --no-instantiate` before constructing models. -- Do not run generation, `torchrun`, large downloads, or GPU test suites merely - to “be thorough.” -- Performance claims require reproducible measurements and unchanged quality - semantics. -- Separate warmup/compile/autotune time from steady-state measurements. -- Never report asynchronous CUDA timings as end-to-end latency without the - required synchronization. - -## Dependencies and OSS State - -- Add dependencies with the owning package manager and the narrowest package - scope. -- Do not add a dependency for behavior that is simpler to implement directly. -- Never hand-edit generated lockfile contents. -- Check license, NOTICE, REUSE, and package-data implications before vendoring - code or assets. -- Never commit credentials, internal hostnames, private URLs, or local cache - paths. - -## Repository Skills - -Read the relevant skill before changing its area: - -- `skills/flashdreams-integrations` -- `skills/integrate-a-model` -- `skills/profile-model-performance` -- `skills/apply-inference-optimizations` -- `skills/validate-performance-quality` -- `skills/flashdreams-postprocessing` -- `skills/python-docstring-style` -- `skills/maintaining-oss-state` - -When adding or moving a skill, update this list and run the skill frontmatter -tests. - -## Common Commands - -```bash -uv sync --extra dev --extra runners -uv sync --package --extra dev -uv run flashdreams-run --help -uv run flashdreams-run --no-instantiate -uv run pytest -m ci_cpu -uv run pytest -m "not manual" -uv run --group lint pre-commit run -a -uv run --group docs sphinx-build -b html docs/source docs/_build/html -``` - -Prefer package-scoped sync and tests. Full workspace operations can build CUDA -packages such as Transformer Engine and should be used only when necessary. - ## Source of Truth -- `README.md`: installation, supported models, and user workflows. -- `CONTRIBUTING.md`: DCO, PR process, coding conventions, and test markers. -- `docs/README.md`: documentation builds. -- `tests/README.md`: test entry points and CI behavior. - `SECURITY.md`: vulnerability reporting. From dca0d92cafb03d7c7cd33287105720d953c01c15 Mon Sep 17 00:00:00 2001 From: William Lew Date: Wed, 19 Aug 2026 09:54:43 -0700 Subject: [PATCH 3/3] address comments --- AGENTS.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index b94a39114..e2c9fd489 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -100,8 +100,6 @@ understanding. Do not add annotations that force casts or repeat obvious types. - Private methods need concise docstrings when their purpose is not obvious. - Prefer plain language over architecture jargon. - Explain invariants and non-obvious constraints, not line-by-line mechanics. -- Keep docs, examples, and launch commands synchronized with real entry points. -- Use the repository SPDX header on new source files. ## Tests Prove Behavior