Skip to content

feat: add max-tokens field selector for custom OpenAI-compatible LLMs#505

Open
Lingxi-Li wants to merge 1 commit into
nashsu:mainfrom
Lingxi-Li:expose-param
Open

feat: add max-tokens field selector for custom OpenAI-compatible LLMs#505
Lingxi-Li wants to merge 1 commit into
nashsu:mainfrom
Lingxi-Li:expose-param

Conversation

@Lingxi-Li

Copy link
Copy Markdown

Summary

For custom OpenAI-compatible LLMs, a heuristic is in place to choose between max_tokens and max_completion_tokens.
However, it fails for certain models provided by GitHub Marketplace.
E.g. with GPT-5, we have

  • Endpoint: https://models.github.ai/inference
  • Model: openai/gpt-5

The heuristic chooses max_tokens that leads to the error below.

HTTP 400: Bad Request — { "error": { "message": "Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead.", "type": "invalid_request_error", "param": "max_tokens", "code": "unsupported_parameter" } }

Until now there is no way for a user to override the field name.
This PR adds a settings dropdown that lets the user force the field name for the main LLM on the custom OpenAI-compatible path.

What changed

  1. New optional config field maxTokensParam with three values: auto (default), max_tokens, and max_completion_tokens, declared once as the exported MaxTokensParam union in the store.
  2. The field is threaded through the preset resolver, the LlmPreset template, the settings draft, and the global Save path so it persists and cannot be clobbered by a Save triggered from another tab.
  3. A new dropdown in the LLM provider section, gated to custom presets in chat_completions mode, with label, option, and hint strings added to both the English and Chinese locales.
  4. A final reconciliation step, applyMaxTokensParamOverride, that forces the chosen field name on the request body.

Behavior and backward compatibility

auto is the default and reproduces today's output exactly, so existing configurations behave unchanged.
The field is optional and missing values resolve to auto, so no persistence migration is required and older saved configs keep working.
A manual choice can still be rejected by strict endpoints, so the in-app hint recommends leaving the setting on auto unless the endpoint requires a specific field.

Implementation notes

The override runs as the final body transform, so it overrides the token-field renaming done by the strict-OpenAI/Azure and MiMo adapters.
buildOpenAiCompatibleBody was refactored from three separate return body points into a single exit point so the override also applies on the DeepSeek and Ollama paths, which previously returned early.
The override reads whichever of max_tokens or max_completion_tokens is present, deletes both, and writes the value under the chosen field; it is a no-op when neither is present or when the choice is auto, and it preserves a value of 0.

Testing

  1. npx vitest run llm-providers passes with 111 tests, including 5 new cases.
  2. npm run typecheck is clean.

The new test cases cover forcing max_completion_tokens on a plain custom endpoint, the override winning over an adapter rename on a Xiaomi MiMo endpoint, auto and omitted being identical to current behavior, a no-op when no token budget is passed, and the override still applying on the DeepSeek early-return path.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant