-
Notifications
You must be signed in to change notification settings - Fork 6.3k
feat(llm): add first-class llmtr/ provider prefix for the LLMTR gateway #1148
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -466,6 +466,11 @@ def _resolve_prefixed_model( | |
| ) | ||
| if prefix == "ollama" and stripped_model_name: | ||
| return self._get_fallback_provider("litellm"), f"ollama_chat/{stripped_model_name}" | ||
| if prefix == "llmtr" and stripped_model_name: | ||
| # LLMTR is an OpenAI-compatible gateway, so route llmtr/<model> | ||
| # through LiteLLM's openai/ provider. The gateway base URL and | ||
| # attribution headers are configured in _configure_llmtr_routing. | ||
| return self._get_fallback_provider("litellm"), f"openai/{stripped_model_name}" | ||
| return self._get_fallback_provider("litellm"), original_model_name | ||
|
|
||
| def get_model(self, model_name: str | None) -> Model: | ||
|
|
@@ -560,6 +565,7 @@ def configure_sdk_model_defaults(settings: Settings) -> None: | |
| return | ||
| _configure_litellm_compatibility() | ||
| _configure_openrouter_attribution(llm.model) | ||
| _configure_llmtr_routing(llm.model, llm.api_base) | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
When the main and deduplication models use different providers and exactly one uses Knowledge Base Used: Configuration and Telemetry Prompt To Fix With AIThis is a comment left during a code review.
Path: strix/config/models.py
Line: 568
Comment:
**Global routing ignores dedupe model**
When the main and deduplication models use different providers and exactly one uses `llmtr/`, `_configure_llmtr_routing` configures LiteLLM's process-global endpoint from only the main model before the independently selected deduplication model is resolved, causing deduplication requests to reach the wrong gateway and fail authentication or model lookup.
**Knowledge Base Used:** [Configuration and Telemetry](https://app.greptile.com/strix-org-3/-/custom-context/knowledge-base/usestrix/strix/-/docs/telemetry-and-config.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly. |
||
| if llm.api_key: | ||
| set_default_openai_key(llm.api_key, use_for_tracing=False) | ||
| _configure_litellm_default("api_key", llm.api_key) | ||
|
|
@@ -679,6 +685,39 @@ def _configure_openrouter_attribution(model_name: str | None) -> None: | |
| litellm.headers = {**existing, **OPENROUTER_ATTRIBUTION_HEADERS} # type: ignore[assignment] | ||
|
|
||
|
|
||
| LLMTR_API_BASE = "https://llmtr.com/v1" | ||
|
|
||
| LLMTR_ATTRIBUTION_HEADERS = { | ||
| "HTTP-Referer": "https://strix.ai", | ||
| "X-Title": "Strix", | ||
| } | ||
|
|
||
|
|
||
| def is_llmtr_model(model_name: str | None) -> bool: | ||
| return bool(model_name) and (model_name or "").strip().lower().startswith("llmtr/") | ||
|
|
||
|
|
||
| def _configure_llmtr_routing(model_name: str | None, api_base: str | None) -> None: | ||
| """Make ``llmtr/`` a first-class prefix for the LLMTR gateway. | ||
|
|
||
| LLMTR (https://llmtr.com) is a Turkey-hosted OpenAI-compatible gateway, so | ||
| ``StrixProvider`` resolves ``llmtr/<model>`` to LiteLLM's ``openai/<model>`` | ||
| route. That route needs the gateway base URL, which is supplied here (an | ||
| explicit ``LLM_API_BASE`` still wins, for a proxy in front of LLMTR), along | ||
| with Strix attribution headers so scans are identifiable in the LLMTR | ||
| dashboard. Users set only ``STRIX_LLM`` and ``LLM_API_KEY``. | ||
| """ | ||
| if not is_llmtr_model(model_name): | ||
| return | ||
| import litellm | ||
|
|
||
| if not api_base: | ||
| _configure_litellm_default("api_base", LLMTR_API_BASE) | ||
| current: object = litellm.headers | ||
| existing: dict[str, str] = current if isinstance(current, dict) else {} | ||
| litellm.headers = {**existing, **LLMTR_ATTRIBUTION_HEADERS} # type: ignore[assignment] | ||
|
|
||
|
|
||
| def _configure_extra_headers(llm: LlmSettings) -> None: | ||
| """Send user-provided default headers on every LLM request. | ||
|
|
||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When an LLMTR-hosted reasoning model is used with
STRIX_REASONING_EFFORT, capability detection examines the originalllmtr/<provider>/<model>name without removing the new prefix, somake_model_settingssilently omits the configured reasoning effort.Knowledge Base Used: Configuration and Telemetry
Prompt To Fix With AI