Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
4e9a80e
Preserve reasoning and replay tool history for multi-turn tool calling
gary149 Jul 30, 2026
f8865aa
Measure streaming TTFT and throughput in the replay harness
gary149 Jul 30, 2026
ce6265a
Harden tool history replay: provider-safe ids, honest interrupts, siz…
gary149 Jul 30, 2026
56382cf
Extend reasoning_content replay to the plain completion flow
gary149 Jul 30, 2026
c04f2e9
Clarify why replayed reasoning attaches to the final message
gary149 Jul 31, 2026
2f773dd
Make replay budget degradation monotonic
gary149 Jul 31, 2026
d7fcbfb
Attribute replayed reasoning to its tool round; honor user reasoning …
gary149 Jul 31, 2026
24ec859
Accept reasoning_text as a third incoming reasoning field
gary149 Jul 31, 2026
2ba08f8
Flag reasoning support for DeepSeek-V4-Flash and Qwen3.6 models
gary149 Jul 31, 2026
e1c5f1d
Teach sync-models that supportsReasoning also gates reasoning replay
gary149 Jul 31, 2026
2040703
Fix reasoning byte fidelity, preamble attribution, and budget-fallbac…
gary149 Jul 31, 2026
120ffdb
Fix preamble duplication on replay and whitespace loss in reasoning s…
gary149 Jul 31, 2026
2530e4f
Strip Gemma-flow think leak, gate cross-producer reasoning, persist r…
gary149 Jul 31, 2026
64425f0
Fix leading-whitespace reasoning loss, stale-producer reasoning after…
gary149 Jul 31, 2026
d8a9700
Omit empty replayed final messages; fail the harness when the nonce r…
gary149 Jul 31, 2026
05aa616
Scope round-reasoning dedup to a positional match; omit phantom empty…
gary149 Jul 31, 2026
7e0c7a2
Prefix-only preamble dedup, empty-flat guard, per-rep coherence gate
gary149 Jul 31, 2026
bc8e231
Merge remote-tracking branch 'origin/main' into preserve-reasoning-to…
gary149 Aug 5, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 14 additions & 2 deletions .claude/skills/sync-models/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,19 @@ Both directions share the same source of truth: whatever `https://router.hugging
- **Whether it's a reasoning model** (see step 5)

5. **Decide if the model is reasoning-capable**
A model is "reasoning-capable" for chat-ui purposes if it accepts the OpenAI-style `reasoning_effort: low|medium|high` parameter via the HF router and _meaningfully changes its chain-of-thought depth_ in response. Whether that holds depends on **both the model and the providers serving it** — the router is a transparent proxy, so behavior comes from each provider's implementation. Don't decide from the name alone.
The `supportsReasoning` flag controls TWO behaviors, and both must be appropriate before flagging:

1. chat-ui renders the thinking-effort dropdown and forwards `reasoning_effort` to the router;
2. chat-ui echoes the model's prior reasoning back as `reasoning_content` on past assistant messages (cross-turn "preserved thinking" — see `prepareFiles.ts` / `endpointOai.ts`).

A model qualifies for (1) if it accepts the OpenAI-style `reasoning_effort: low|medium|high` parameter via the HF router and _meaningfully changes its chain-of-thought depth_ in response. Whether that holds depends on **both the model and the providers serving it** — the router is a transparent proxy, so behavior comes from each provider's implementation. Don't decide from the name alone.

For (2), check the vendor's preserved-thinking / multi-turn guidance, because it can point in either direction:

- **Flag-strengthening**: the vendor documents that prior `reasoning_content` must or should be passed back in multi-turn or tool-calling flows. Known examples: Moonshot `thinking.keep` (Kimi K2.6+/K3), MiniMax "Interleaved Thinking" ("must preserve the model's thinking content completely"), DeepSeek V4 thinking mode (hard 400 if `reasoning_content` is missing on tool-call turns), Z.ai "Preserved Thinking" (`clear_thinking: false`), Qwen3.6 `preserve_thinking`.
- **Flag-blocking**: the vendor documents that historical thoughts must be STRIPPED across completed turns. Known example: the Gemma family — Google requires removing thoughts from previous turns ("historical model output must only include the final response") while preserving them only inside a single turn's tool loop, which chat-ui handles automatically without the flag. **Do not flag such models even though they emit reasoning and may accept an effort knob** — flagging would make chat-ui echo reasoning the vendor says to strip.

Vendor doc entry points for the preserved-thinking check: <https://platform.kimi.ai/docs/guide/use-kimi-k2-thinking-model>, <https://platform.minimax.io/docs/guides/text-m3-function-call>, <https://api-docs.deepseek.com/guides/thinking_mode/>, <https://docs.z.ai/guides/capabilities/thinking-mode>, <https://ai.google.dev/gemma/docs/capabilities/thinking>, plus the model card's own multi-turn/best-practices section (Qwen cards document what their chat template does with historical `<think>` blocks).

**Heuristic shortlist (candidates worth verifying):**

Expand Down Expand Up @@ -141,7 +153,7 @@ Both directions share the same source of truth: whatever `https://router.hugging
}
```

`supportsReasoning` is what makes chat-ui render the Thinking-effort dropdown in the chat footer for that model and forward `reasoning_effort` to the router. `supportsArtifacts` enables the artifacts side panel for the model.
`supportsReasoning` makes chat-ui render the Thinking-effort dropdown in the chat footer, forward `reasoning_effort` to the router, AND echo the model's prior reasoning back as `reasoning_content` on past assistant messages (preserved thinking). Models whose vendor requires stripping historical thoughts (Gemma family) must stay unflagged — see step 5. `supportsArtifacts` enables the artifacts side panel for the model.

9. **Remove deprecated models from both files**
Delete the full entry line for every id in the deprecated set (step 3) from **both** `chart/env/prod.yaml` and `chart/env/dev.yaml`. Match on the exact `"id"` value so near-duplicate ids aren't removed by accident, and keep the removal symmetric — the two files must end with the same model set.
Expand Down
6 changes: 3 additions & 3 deletions chart/env/dev.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -111,16 +111,16 @@ envVars:
{ "id": "moonshotai/Kimi-K2.7-Code", "description": "1T MoE coding agent with persistent thinking across long tool workflows." , "supportsReasoning": true, "supportsArtifacts": true, "parameters": { "max_tokens": 98304 } },
{ "id": "MiniMaxAI/MiniMax-M3", "description": "Natively multimodal 428B MoE with 1M context for frontier coding and agents." , "supportsReasoning": true, "supportsArtifacts": true, "parameters": { "max_tokens": 98304 } },
{ "id": "nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4", "description": "Hybrid Mamba-Transformer 550B MoE with 55B active params for efficient agentic reasoning.", "parameters": { "max_tokens": 98304 }, "supportsReasoning": true, "supportsArtifacts": true },
{ "id": "Qwen/Qwen3.6-27B", "description": "Dense 27B hybrid with DeltaNet attention and 1M context.", "supportsArtifacts": true },
{ "id": "Qwen/Qwen3.6-27B", "description": "Dense 27B hybrid with DeltaNet attention and 1M context.", "supportsReasoning": true, "supportsArtifacts": true },
{ "id": "meta-llama/Llama-4-Scout-17B-16E-Instruct", "description": "Native multimodal 17B MoE with 16 experts and very long context."},
{ "id": "CohereLabs/c4ai-command-r7b-12-2024", "description": "Compact 7B Command R for multilingual RAG, tool use, and agents." },
{ "id": "pearl-ai/Gemma-4-31B-it-pearl", "description": "Community Gemma-4-31B variant integrated with Pearl mining for chain-validated inference."},
{ "id": "inclusionAI/Ling-2.6-1T", "description": "1T MoE with 50B active params, hybrid MLA-Linear attention, and fast thinking.", "supportsArtifacts": true },
{ "id": "deepseek-ai/DeepSeek-V4-Pro", "description": "Frontier 1.6T MoE with 49B active params, hybrid attention, and 1M context.", "parameters": { "max_tokens": 98304 }, "supportsReasoning": true, "supportsArtifacts": true },
{ "id": "deepseek-ai/DeepSeek-V4-Flash", "description": "Compact 284B MoE with 13B active params, hybrid attention, and 1M context.", "parameters": { "max_tokens": 49152 }, "supportsArtifacts": true },
{ "id": "deepseek-ai/DeepSeek-V4-Flash", "description": "Compact 284B MoE with 13B active params, hybrid attention, and 1M context.", "parameters": { "max_tokens": 49152 }, "supportsReasoning": true, "supportsArtifacts": true },
{ "id": "deepseek-ai/DeepSeek-V3.2", "description": "Stable 671B MoE with sparse attention for agentic reasoning and long contexts.", "supportsArtifacts": true },
{ "id": "stepfun-ai/Step-3.5-Flash", "description": "Sparse 197B MoE agent with multi-token prediction and 256K context.", "supportsArtifacts": true },
{ "id": "Qwen/Qwen3.6-35B-A3B", "description": "Hybrid 35B MoE with 3B active params, DeltaNet attention, and 1M context.", "supportsArtifacts": true },
{ "id": "Qwen/Qwen3.6-35B-A3B", "description": "Hybrid 35B MoE with 3B active params, DeltaNet attention, and 1M context.", "supportsReasoning": true, "supportsArtifacts": true },
{ "id": "zai-org/GLM-4.7-Flash", "description": "Compact 30B MoE for agentic coding with 128K context and tool use.", "supportsReasoning": true, "supportsArtifacts": true },
{ "id": "moonshotai/Kimi-K2.6", "description": "Native multimodal 1T MoE for long-horizon coding and 300-sub-agent swarms.", "supportsArtifacts": true },
{ "id": "MiniMaxAI/MiniMax-M2.7", "description": "Self-evolving 230B MoE agent for frontier coding, reasoning, and tool use." , "supportsReasoning": true, "supportsArtifacts": true },
Expand Down
6 changes: 3 additions & 3 deletions chart/env/prod.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -121,16 +121,16 @@ envVars:
{ "id": "moonshotai/Kimi-K2.7-Code", "description": "1T MoE coding agent with persistent thinking across long tool workflows." , "supportsReasoning": true, "supportsArtifacts": true, "parameters": { "max_tokens": 98304 } },
{ "id": "MiniMaxAI/MiniMax-M3", "description": "Natively multimodal 428B MoE with 1M context for frontier coding and agents." , "supportsReasoning": true, "supportsArtifacts": true, "parameters": { "max_tokens": 98304 } },
{ "id": "nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4", "description": "Hybrid Mamba-Transformer 550B MoE with 55B active params for efficient agentic reasoning.", "parameters": { "max_tokens": 98304 }, "supportsReasoning": true, "supportsArtifacts": true },
{ "id": "Qwen/Qwen3.6-27B", "description": "Dense 27B hybrid with DeltaNet attention and 1M context.", "supportsArtifacts": true, "parameters": { "max_tokens": 49152 } },
{ "id": "Qwen/Qwen3.6-27B", "description": "Dense 27B hybrid with DeltaNet attention and 1M context.", "supportsReasoning": true, "supportsArtifacts": true, "parameters": { "max_tokens": 49152 } },

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Separate preserved-thinking support from reasoning effort

supportsReasoning currently enables both historical reasoning_content replay and the low/medium/high effort control. Qwen3.6 documents preserved thinking through preserve_thinking, while the current Qwen/DashScope interface exposes only Boolean enable_thinking, not effort tiers. Marking these models supportsReasoning: true therefore exposes a misleading control and can send an unsupported or ineffective reasoning_effort value. Please split replay/preservation capability from effort-tier capability and wire Qwen’s provider-specific preservation option separately.

{ "id": "meta-llama/Llama-4-Scout-17B-16E-Instruct", "description": "Native multimodal 17B MoE with 16 experts and very long context."},
{ "id": "CohereLabs/c4ai-command-r7b-12-2024", "description": "Compact 7B Command R for multilingual RAG, tool use, and agents." },
{ "id": "pearl-ai/Gemma-4-31B-it-pearl", "description": "Community Gemma-4-31B variant integrated with Pearl mining for chain-validated inference."},
{ "id": "inclusionAI/Ling-2.6-1T", "description": "1T MoE with 50B active params, hybrid MLA-Linear attention, and fast thinking.", "supportsArtifacts": true, "parameters": { "max_tokens": 32768 } },
{ "id": "deepseek-ai/DeepSeek-V4-Pro", "description": "Frontier 1.6T MoE with 49B active params, hybrid attention, and 1M context.", "parameters": { "max_tokens": 98304 }, "supportsReasoning": true, "supportsArtifacts": true },
{ "id": "deepseek-ai/DeepSeek-V4-Flash", "description": "Compact 284B MoE with 13B active params, hybrid attention, and 1M context.", "parameters": { "max_tokens": 49152 }, "supportsArtifacts": true },
{ "id": "deepseek-ai/DeepSeek-V4-Flash", "description": "Compact 284B MoE with 13B active params, hybrid attention, and 1M context.", "parameters": { "max_tokens": 49152 }, "supportsReasoning": true, "supportsArtifacts": true },
{ "id": "deepseek-ai/DeepSeek-V3.2", "description": "Stable 671B MoE with sparse attention for agentic reasoning and long contexts.", "supportsArtifacts": true, "parameters": { "max_tokens": 49152 } },
{ "id": "stepfun-ai/Step-3.5-Flash", "description": "Sparse 197B MoE agent with multi-token prediction and 256K context.", "supportsArtifacts": true, "parameters": { "max_tokens": 49152 } },
{ "id": "Qwen/Qwen3.6-35B-A3B", "description": "Hybrid 35B MoE with 3B active params, DeltaNet attention, and 1M context.", "supportsArtifacts": true, "parameters": { "max_tokens": 49152 } },
{ "id": "Qwen/Qwen3.6-35B-A3B", "description": "Hybrid 35B MoE with 3B active params, DeltaNet attention, and 1M context.", "supportsReasoning": true, "supportsArtifacts": true, "parameters": { "max_tokens": 49152 } },
{ "id": "zai-org/GLM-4.7-Flash", "description": "Compact 30B MoE for agentic coding with 128K context and tool use.", "supportsReasoning": true, "supportsArtifacts": true, "parameters": { "max_tokens": 98304 } },
{ "id": "moonshotai/Kimi-K2.6", "description": "Native multimodal 1T MoE for long-horizon coding and 300-sub-agent swarms.", "supportsArtifacts": true, "parameters": { "max_tokens": 65536 } },
{ "id": "MiniMaxAI/MiniMax-M2.7", "description": "Self-evolving 230B MoE agent for frontier coding, reasoning, and tool use." , "supportsReasoning": true, "supportsArtifacts": true, "parameters": { "max_tokens": 98304 } },
Expand Down
Loading
Loading