Establish a public model-support and performance reporting process
Summary
Define a repeatable process for keeping FlashDreams' public documentation current
as new model integrations land. The process should make it clear which models are
supported and, where applicable, what performance users can expect.
Motivation
Model-support information and performance results currently exist across PR
discussions, model documentation, the README, and benchmark data. A contributor
adding a new integration should have a clear, consistent path for publishing this
information so users can understand the project's current capabilities without
having to inspect individual pull requests.
Desired outcome
Document and adopt a lightweight contribution process that specifies:
- Where the public list of supported models is maintained.
- When a new or materially updated integration should be added to that list.
- Where and how performance results should be published.
- The minimum context needed for published results (for example: model/config,
hardware, command or workload, measurement definitions, and relevant quality
or runtime settings).
- How to distinguish preliminary or contributor-reported measurements from
maintained/comparable benchmark results.
- Who reviews or owns updates to these public-facing surfaces.
Scope
This issue is about defining the policy, ownership, and public presentation. It
does not prescribe the documentation structure, benchmark harness, automation,
or CI implementation; those decisions can be made by the owner as part of the
proposal or follow-up work.
Acceptance criteria
- Contributors can identify the required public documentation updates for a new
supported model.
- Published performance data has enough provenance for users to interpret and
compare it responsibly.
- The process identifies the canonical public locations for model support and
benchmark/performance information.
- The process is documented in a contributor-facing location and can be applied
to future integration PRs.
Initial example
The Waypoint 1.5 1B integration in PR #464 provides a useful first example:
it introduced a newly supported model and included RTX 5090 profiling results in
the PR discussion. The established process should make clear whether and how
those results move into the public model-support and performance surfaces.
Establish a public model-support and performance reporting process
Summary
Define a repeatable process for keeping FlashDreams' public documentation current
as new model integrations land. The process should make it clear which models are
supported and, where applicable, what performance users can expect.
Motivation
Model-support information and performance results currently exist across PR
discussions, model documentation, the README, and benchmark data. A contributor
adding a new integration should have a clear, consistent path for publishing this
information so users can understand the project's current capabilities without
having to inspect individual pull requests.
Desired outcome
Document and adopt a lightweight contribution process that specifies:
hardware, command or workload, measurement definitions, and relevant quality
or runtime settings).
maintained/comparable benchmark results.
Scope
This issue is about defining the policy, ownership, and public presentation. It
does not prescribe the documentation structure, benchmark harness, automation,
or CI implementation; those decisions can be made by the owner as part of the
proposal or follow-up work.
Acceptance criteria
supported model.
compare it responsibly.
benchmark/performance information.
to future integration PRs.
Initial example
The Waypoint 1.5 1B integration in PR #464 provides a useful first example:
it introduced a newly supported model and included RTX 5090 profiling results in
the PR discussion. The established process should make clear whether and how
those results move into the public model-support and performance surfaces.