fix(textGeneration): stop saving max_tokens-truncated responses as complete - #2385
fix(textGeneration): stop saving max_tokens-truncated responses as complete#2385gary149 wants to merge 2 commits into
Conversation
When a model hit its max_tokens budget mid-generation, the provider returns finish_reason: "length". The stream adapters treated "length" identically to a clean "stop" (marking the final token special), so generate.ts computed interrupted=false and the truncated message was persisted as if it were complete. No log, no interrupted flag, no UI signal. This was especially visible with reasoning models: e.g. GLM-5.2 (max_tokens 49152) could spend its entire budget on <think> reasoning and never emit a final answer, leaving the user with a 160KB think-dump cut off mid-code and no way to tell it was truncated or to continue it. - Stream adapters (openAIChatToTextGenerationStream, openAICompletionToTextGenerationStream, and the non-streaming single adapter) now mark the final output truncated:true when finish_reason === "length". - generate.ts forces interrupted=true on a truncated output and logs a warning. - The MCP flow tracks finish_reason and emits its FinalAnswer with interrupted set accordingly, plus a warning. - Adds a regression test covering the length vs stop cases.
Gate the completion handling on `generated_text != null` instead of truthiness so a terminal output with empty text is still finalized. A model can hit finish_reason "length" before emitting any visible content (hidden-reasoning models, very small budgets); previously that output was skipped (the special token was ignored) and the truncation warning never fired. Now the truncation is flagged + logged for the empty case too. Note: a genuinely empty body still surfaces as a "no output" error downstream (content equals the initial empty content), which is the appropriate UX; this change is about observability and a consistent interrupted flag, not changing that outcome. Adds an adapter test for the empty length-terminated stream.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 287b8b2771
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // with empty text is still finalized: a model can hit finish_reason "length" before | ||
| // emitting any visible content (hidden-reasoning models, tiny budgets), and that | ||
| // truncation must still be flagged + logged rather than silently dropped. | ||
| if (output.generated_text != null) { |
There was a problem hiding this comment.
When a finish_reason: "length" response has no surfaced tokens, this new != null branch emits a FinalAnswer with text: "" and interrupted: true, but the conversation route still checks only messageToWriteTo.content === initialMessageContent in src/routes/conversation/[id]/+server.ts:713 and then appends the generic “No output was generated” error status. This affects the hidden-reasoning/tiny-budget case called out in the comment: the truncation is finalized, but live users still see an error and the persisted update stream contains both an interrupted final answer and an error. Consider teaching that guard about a received truncated/interrupted final answer, or otherwise avoiding the empty finalization path until it can suppress the no-output error.
Useful? React with 👍 / 👎.
1 participant
Problem
When a model hits its
max_tokensbudget, the provider returnsfinish_reason: "length". The OpenAI stream adapters treated"length"identically to a clean"stop"(marking the final tokenspecial: true), sogenerate.tscomputedinterrupted = falseand the truncated message was persisted as if complete. No log, no flag, no signal.Most visible with reasoning models: GLM-5.2 spent its whole budget inside a
<think>block and hit the limit before any answer, leaving the user with a ~160KB reasoning dump cut off mid-code, saved as a "finished" reply (e.g.conversation/6a3bcad5a78f58a9cd8f8def). Reproduced live: the model streamed onlyreasoning_content, emitted zerocontent, and terminated withfinish_reason: "length".Fix
openAIChatToTextGenerationStream,openAICompletionToTextGenerationStream, non-streaming single adapter) mark the final outputtruncated: truewhenfinish_reason === "length".generate.tsforcesinterrupted = trueon a truncated output and logs awarn. It also gates the completion handling ongenerated_text != null(not truthiness) so empty truncated completions (hidden-reasoning models, tiny budgets) are still flagged + logged rather than silently dropped.runMcpFlow.ts) tracksfinish_reasonand setsinterruptedon itsFinalAnsweraccordingly, with a warning.This makes the persisted
interruptedflag and the logs truthful. chat-ui has no dedicated "response was cut off" banner today, so this is backend correctness + ops visibility; a visible banner/continue affordance gated on this signal is a sensible follow-up.Test plan
npx vitest run src/lib/server/endpoints/openai/openAIChatToTextGenerationStream.spec.ts(3 passing)npm run check(0 errors); ESLint + Prettier cleanSplit from #2383 (this is the code half).