feat: add max-tokens field selector for custom OpenAI-compatible LLMs#505
Open
Lingxi-Li wants to merge 1 commit into
Open
feat: add max-tokens field selector for custom OpenAI-compatible LLMs#505Lingxi-Li wants to merge 1 commit into
Lingxi-Li wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
For custom OpenAI-compatible LLMs, a heuristic is in place to choose between
max_tokensandmax_completion_tokens.However, it fails for certain models provided by GitHub Marketplace.
E.g. with GPT-5, we have
https://models.github.ai/inferenceopenai/gpt-5The heuristic chooses
max_tokensthat leads to the error below.Until now there is no way for a user to override the field name.
This PR adds a settings dropdown that lets the user force the field name for the main LLM on the custom OpenAI-compatible path.
What changed
maxTokensParamwith three values:auto(default),max_tokens, andmax_completion_tokens, declared once as the exportedMaxTokensParamunion in the store.LlmPresettemplate, the settings draft, and the global Save path so it persists and cannot be clobbered by a Save triggered from another tab.custompresets inchat_completionsmode, with label, option, and hint strings added to both the English and Chinese locales.applyMaxTokensParamOverride, that forces the chosen field name on the request body.Behavior and backward compatibility
autois the default and reproduces today's output exactly, so existing configurations behave unchanged.The field is optional and missing values resolve to
auto, so no persistence migration is required and older saved configs keep working.A manual choice can still be rejected by strict endpoints, so the in-app hint recommends leaving the setting on auto unless the endpoint requires a specific field.
Implementation notes
The override runs as the final body transform, so it overrides the token-field renaming done by the strict-OpenAI/Azure and MiMo adapters.
buildOpenAiCompatibleBodywas refactored from three separatereturn bodypoints into a single exit point so the override also applies on the DeepSeek and Ollama paths, which previously returned early.The override reads whichever of
max_tokensormax_completion_tokensis present, deletes both, and writes the value under the chosen field; it is a no-op when neither is present or when the choice isauto, and it preserves a value of0.Testing
npx vitest run llm-providerspasses with 111 tests, including 5 new cases.npm run typecheckis clean.The new test cases cover forcing
max_completion_tokenson a plain custom endpoint, the override winning over an adapter rename on a Xiaomi MiMo endpoint,autoand omitted being identical to current behavior, a no-op when no token budget is passed, and the override still applying on the DeepSeek early-return path.