fix: reduce reasoning-model truncation and stop saving cut-off responses as complete - #2383
fix: reduce reasoning-model truncation and stop saving cut-off responses as complete#2383gary149 wants to merge 4 commits into
Conversation
When a model hit its max_tokens budget mid-generation, the provider returns finish_reason: "length". The stream adapters treated "length" identically to a clean "stop" (marking the final token special), so generate.ts computed interrupted=false and the truncated message was persisted as if it were complete. No log, no interrupted flag, no UI signal. This was especially visible with reasoning models: e.g. GLM-5.2 (max_tokens 49152) could spend its entire budget on <think> reasoning and never emit a final answer, leaving the user with a 160KB think-dump cut off mid-code and no way to tell it was truncated or to continue it. - Stream adapters (openAIChatToTextGenerationStream, openAICompletionToTextGenerationStream, and the non-streaming single adapter) now mark the final output truncated:true when finish_reason === "length". - generate.ts forces interrupted=true on a truncated output and logs a warning. - The MCP flow tracks finish_reason and emits its FinalAnswer with interrupted set accordingly, plus a warning. - Adds a regression test covering the length vs stop cases.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cb66522e70
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // (e.g. exhausted the budget while still reasoning, never reaching the answer). The | ||
| // provider marks the final token as a normal stop, so flag it explicitly as interrupted | ||
| // and log it instead of silently saving a truncated answer as if it were complete. | ||
| if ((output as { truncated?: boolean }).truncated) { |
There was a problem hiding this comment.
The OpenAI adapters added in this change can yield a terminal truncated output with generated_text: "" when the provider returns finish_reason: "length" before emitting any visible content or reasoning, such as with hidden-reasoning models or a very small token budget. This new truncation handling only runs inside the existing truthy if (output.generated_text) branch, so that final output is skipped, the special token is ignored, and the request falls through to the generic “No output generated” error instead of persisting an interrupted final answer. Check for generated_text != null before applying this block so empty truncated completions are finalized.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Good catch on the skipped block, fixed in e56aaca by gating on generated_text != null instead of truthiness, so empty finish_reason: "length" completions now hit the truncation handling (flag + warning log) instead of being dropped.
One correction on the suggested outcome though: this does not end up "persisting an interrupted final answer" for a genuinely empty body. +server.ts independently emits the "No output generated" error whenever the final content equals the initial (empty) content (it keys off content, not finalAnswerReceived), so an empty completion still surfaces as an error, which is the right UX since there's nothing to render. The concrete win here is observability (the truncation now logs) and a consistent interrupted flag, not a behavior change for the empty case. Added an adapter test for the empty length-terminated stream.
Reasoning models share a single max_tokens budget between the <think> phase and the answer. With 49152, a hard prompt could spend the entire budget on reasoning and hit finish_reason=length before emitting any answer (observed with GLM-5.2 on a complex coding task). OpenAI-compatible APIs expose no separate reasoning budget, so the lever is a larger shared cap. Raise every supportsReasoning model with an explicit cap below 98304 (2x the common 49152) to 98304, giving room for both reasoning and the answer. Only reasoning lines are touched; non-reasoning models keep their caps. These are 1M / 256K-context models, so the larger output cap fits the context window. Note: reasoning models without an explicit cap (e.g. GLM-4.7, MiniMax-M2, Kimi-K2-Thinking) still fall back to the provider default and are left as-is.
gary149
changed the title
Gate the completion handling on `generated_text != null` instead of truthiness so a terminal output with empty text is still finalized. A model can hit finish_reason "length" before emitting any visible content (hidden-reasoning models, very small budgets); previously that output was skipped (the special token was ignored) and the truncation warning never fired. Now the truncation is flagged + logged for the empty case too. Note: a genuinely empty body still surfaces as a "no output" error downstream (content equals the initial empty content), which is the appropriate UX; this change is about observability and a consistent interrupted flag, not changing that outcome. Adds an adapter test for the empty length-terminated stream.
Live verification against the router showed all bumped reasoning models accept max_tokens=98304, but gpt-oss-120b is the one outlier with only a 131072-token total context (every provider reports context_length 131072), versus 256K-1M for the others. At 98304 that leaves only ~33k tokens for the prompt, risking context_length_exceeded on long conversations. 65536 balances reasoning+answer budget against prompt headroom (~65k each) for a 128k-context model.
|
Split into two focused PRs:
Closing this combined PR in favor of those. |
1 participant
The problem (user-facing)
A user reported a conversation that just stops mid-answer (e.g.
conversation/6a3bcad5a78f58a9cd8f8def): GLM-5.2 produced a ~160KB<think>reasoning dump that cut off mid-code, with no actual answer and no indication anything went wrong.Root cause: reasoning models share a single
max_tokensbudget between the<think>phase and the answer. On a hard prompt, GLM-5.2 (max_tokens: 49152) spent the entire budget reasoning and hitfinish_reason: "length"before emitting a single answer token. Worse, the pipeline treated"length"exactly like a clean"stop", so the truncated message was saved withinterrupted: false, as if complete. No log, no flag, no signal.Reproduced live against the router (GLM-5.2 with a low
max_tokens): the model streamed onlyreasoning_content, emitted zerocontent, and terminated withfinish_reason: "length".The fix (two parts)
1. Raise the budget so it rarely happens (primary, user-facing).
OpenAI-compatible APIs expose no separate reasoning budget, so the lever is a larger shared cap. Every
supportsReasoningmodel with an explicit cap below 98304 (2x the common 49152) is raised to 98304, giving room for both the reasoning and the answer. Only reasoning lines inchart/env/{prod,dev}.yamlare touched; non-reasoning models keep their caps. These are 1M / 256K-context models, so the larger output cap fits the context window.Not covered: reasoning models with no explicit cap (GLM-4.7, MiniMax-M2/M2.5, Kimi-K2-Thinking, gpt-oss-20b, gpt-oss-safeguard-20b) still fall back to the provider default and are left as-is.
2. Stop silently saving the truncated ones as complete (correctness + logging).
When a model still hits the cap, surface it instead of masking it:
openAIChatToTextGenerationStream,openAICompletionToTextGenerationStream, and the non-streaming single adapter) mark the final outputtruncated: truewhenfinish_reason === "length".generate.tsforcesinterrupted = trueon a truncated output and logs awarn.runMcpFlow.ts) tracksfinish_reasonand setsinterruptedon itsFinalAnsweraccordingly, with a warning.This makes the persisted
interruptedflag and the logs truthful. Note: chat-ui has no dedicated "response was cut off" banner today, so part 2 is mostly backend correctness + ops visibility; a visible banner / continue affordance gated on this signal is a sensible follow-up.Test plan
npx vitest run src/lib/server/endpoints/openai/openAIChatToTextGenerationStream.spec.ts(2 passing: length → truncated, stop → not truncated)npm run check(0 errors)chart/env/{prod,dev}.yamlMODELS JSON re-validated after the bump (parses; only reasoning models changed; no non-reasoning model altered)