Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ ENV=dev # options: dev|s

# Active LLM provider. Selects which provider answers credentials,
# metadata, and default-model lookups. Leave unset to default to nv_build.
# Options: openai | anthropic | anthropic_proxy | nv_build
# Options: openai | anthropic | anthropic_proxy | orcarouter | nv_build
SKILLSPECTOR_PROVIDER=

# Provider credentials — set the one matching SKILLSPECTOR_PROVIDER (or
Expand All @@ -24,6 +24,12 @@ SKILLSPECTOR_REASONING_EFFORT=
# For SKILLSPECTOR_PROVIDER=anthropic.
ANTHROPIC_API_KEY=

# For SKILLSPECTOR_PROVIDER=orcarouter (OrcaRouter model-routing gateway,
# OpenAI-compatible). Leave ORCAROUTER_BASE_URL unset to use the default
# api.orcarouter.ai endpoint.
ORCAROUTER_API_KEY=
# ORCAROUTER_BASE_URL=https://api.orcarouter.ai/v1 # optional override

# For SKILLSPECTOR_PROVIDER=anthropic_proxy (Vertex-style raw-predict proxy).
# Supports corporate API gateways, GCP Vertex AI, and self-hosted proxies.
ANTHROPIC_PROXY_ENDPOINT_URL=
Expand Down
10 changes: 9 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -234,6 +234,7 @@ inference gateways.
| `anthropic_proxy` | `ANTHROPIC_PROXY_API_KEY` + `ANTHROPIC_PROXY_ENDPOINT_URL` | Any Vertex-style raw-predict proxy | `claude-sonnet-4-6` |
| `bedrock` | `AWS_PROFILE` (optional) + `AWS_REGION` — SigV4 via boto3 | AWS Bedrock Runtime | `us.anthropic.claude-sonnet-4-6-20250915-v1:0` |
| `nv_build` | `NVIDIA_INFERENCE_KEY` | build.nvidia.com | `deepseek-ai/deepseek-v4-flash` |
| `orcarouter` | `ORCAROUTER_API_KEY` | api.orcarouter.ai (OpenAI-compatible model-routing gateway) | `orcarouter/auto` |
| `claude_cli` | _(none — uses local CLI auth)_ | local `claude` binary | local Claude runtime fallback, or `SKILLSPECTOR_MODEL` |
| `codex_cli` | _(none — uses local CLI auth)_ | local `codex` binary | local Codex runtime fallback, or `SKILLSPECTOR_MODEL` |

Expand Down Expand Up @@ -272,6 +273,11 @@ export SKILLSPECTOR_PROVIDER=nv_build
export NVIDIA_INFERENCE_KEY=nvapi-...
skillspector scan ./my-skill/

# OrcaRouter — model-routing gateway (OpenAI-compatible)
export SKILLSPECTOR_PROVIDER=orcarouter
export ORCAROUTER_API_KEY=sk-orca-...
skillspector scan ./my-skill/

# Local Claude CLI — no API key; uses your existing `claude auth login` session
# Requires: claude CLI installed and authenticated (claude auth login)
export SKILLSPECTOR_PROVIDER=claude_cli
Expand Down Expand Up @@ -574,13 +580,15 @@ Issues (2)

| Variable | Description | Required |
|----------|-------------|----------|
| `SKILLSPECTOR_PROVIDER` | Active LLM provider: `openai`, `anthropic`, `anthropic_proxy`, `bedrock`, `nv_build`, `claude_cli`, `codex_cli`, or `gemini_cli`. Hosted providers use bundled `model_registry.yaml` defaults; `claude_cli` and `codex_cli` fall back to the local CLI runtime's default model unless `SKILLSPECTOR_MODEL` is set. Defaults to `nv_build`. | Optional |
| `SKILLSPECTOR_PROVIDER` | Active LLM provider: `openai`, `anthropic`, `anthropic_proxy`, `bedrock`, `nv_build`, `orcarouter`, `claude_cli`, `codex_cli`, or `gemini_cli`. Hosted providers use bundled `model_registry.yaml` defaults; `claude_cli` and `codex_cli` fall back to the local CLI runtime's default model unless `SKILLSPECTOR_MODEL` is set. Defaults to `nv_build`. | Optional |
| `NVIDIA_INFERENCE_KEY` | Credential for the `nv_build` provider (build.nvidia.com). | Required for LLM analysis when `SKILLSPECTOR_PROVIDER=nv_build` |
| `OPENAI_API_KEY` | Credential for the OpenAI provider (`SKILLSPECTOR_PROVIDER=openai`). Also serves as the tier-2 fallback in the credential waterfall when the active provider returns no credentials. | Required for LLM analysis when `SKILLSPECTOR_PROVIDER=openai` |
| `OPENAI_BASE_URL` | Override the OpenAI endpoint (e.g. point at Ollama). | Optional |
| `SKILLSPECTOR_REASONING_EFFORT` | Optional provider- and model-dependent reasoning-effort setting. Non-empty values are trimmed and passed through unchanged; unset or blank preserves provider-default behavior. | Optional |
| `ANTHROPIC_API_KEY` | Credential for the Anthropic provider (`SKILLSPECTOR_PROVIDER=anthropic`). | Required for LLM analysis when `SKILLSPECTOR_PROVIDER=anthropic` |
| `ANTHROPIC_BASE_URL` | Override the native Anthropic endpoint (default: `https://api.anthropic.com`). | Optional |
| `ORCAROUTER_API_KEY` | Credential for the OrcaRouter provider (`SKILLSPECTOR_PROVIDER=orcarouter`). | Required when `SKILLSPECTOR_PROVIDER=orcarouter` |
| `ORCAROUTER_BASE_URL` | Override the OrcaRouter endpoint (default: `https://api.orcarouter.ai/v1`). | Optional |
| `ANTHROPIC_PROXY_ENDPOINT_URL` | Full endpoint URL for the Anthropic proxy provider (Vertex-style raw-predict). | Required when `SKILLSPECTOR_PROVIDER=anthropic_proxy` |
| `ANTHROPIC_PROXY_API_KEY` | Bearer token for the Anthropic proxy provider. | Required when `SKILLSPECTOR_PROVIDER=anthropic_proxy` |
| `ANTHROPIC_PROXY_API_VERSION` | `anthropic_version` value sent in the request body (default: `vertex-2023-10-16`). | Optional |
Expand Down
7 changes: 6 additions & 1 deletion src/skillspector/providers/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
anthropic_proxy → AnthropicProxyProvider (Vertex-style raw-predict proxy)
bedrock → BedrockProvider (AWS Bedrock Runtime, SigV4)
nv_build → NvBuildProvider (build.nvidia.com)
orcarouter → OrcaRouterProvider (api.orcarouter.ai)
claude_cli → ClaudeCLIProvider (local ``claude`` binary, no API key)
codex_cli → CodexCLIProvider (local ``codex`` binary, no API key)
gemini_cli → GeminiCLIProvider (local ``gemini`` binary, no API key)
Expand Down Expand Up @@ -120,6 +121,10 @@ def _select_active_provider() -> LLMProvider:
return BedrockProvider()
if name == "nv_build":
return NvBuildProvider()
if name == "orcarouter":
from .orcarouter import OrcaRouterProvider

return OrcaRouterProvider()
if name == "claude_cli":
from .claude_cli import ClaudeCLIProvider

Expand Down Expand Up @@ -149,7 +154,7 @@ def _select_active_provider() -> LLMProvider:
raise ValueError(
f"Unknown SKILLSPECTOR_PROVIDER: {name!r}. "
"Expected one of: openai, anthropic, anthropic_proxy, bedrock, nv_build, "
"claude_cli, codex_cli, gemini_cli, antigravity_cli (or unset)."
"orcarouter, claude_cli, codex_cli, gemini_cli, antigravity_cli (or unset)."
)


Expand Down
20 changes: 20 additions & 0 deletions src/skillspector/providers/orcarouter/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

"""OrcaRouter provider package (api.orcarouter.ai OpenAI-compatible gateway)."""

from .provider import ORCAROUTER_BASE_URL, REGISTRY_PATH, OrcaRouterProvider

__all__ = ["ORCAROUTER_BASE_URL", "REGISTRY_PATH", "OrcaRouterProvider"]
18 changes: 18 additions & 0 deletions src/skillspector/providers/orcarouter/model_registry.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Token-budget metadata for the OrcaRouter provider (api.orcarouter.ai).
#
# ``orcarouter/auto`` routes each request to the best available frontier
# model for the task, so the exact backend context window varies. The
# budgets below mirror the conservative Claude-class defaults already
# shipped in this package; tune with SKILLSPECTOR_MODEL_REGISTRY if your
# routing target needs more headroom.
#
# Format:
# models:
# "<model-label>":
# context_length: <int> # total context window in tokens (required)
# max_output_tokens: <int> # model's max output cap (optional)

models:
"orcarouter/auto":
context_length: 200000

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] A fixed 200k/64k budget cannot describe orcarouter/auto, whose candidate backend changes per request and whose default allowed-model pattern matches every chat model the account can access. These values make SkillSpector accept roughly 150k input tokens and request up to 50k output tokens, which can exceed a selected backend context or output cap and turn larger scans into provider errors. Please use limits OrcaRouter contractually guarantees for every auto-router candidate, or avoid publishing metadata for this dynamic alias and require a concrete/custom-router registry entry. Reference: auto router.

max_output_tokens: 64000
79 changes: 79 additions & 0 deletions src/skillspector/providers/orcarouter/provider.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

"""OrcaRouter provider — model-routing gateway via api.orcarouter.ai.

Reads ``ORCAROUTER_API_KEY`` for credentials and serves the OpenAI-compatible
``https://api.orcarouter.ai/v1`` endpoint (optionally overridden with
``ORCAROUTER_BASE_URL``). The ``orcarouter/auto`` model routes each request
to the best available frontier model for the task, so no per-model API key
or model id is required.
"""

from __future__ import annotations

import os
from pathlib import Path

from langchain_core.language_models.chat_models import BaseChatModel

from skillspector.providers import registry
from skillspector.providers.chat_models import create_openai_compatible_chat_model

# Default endpoint; overridden by ``ORCAROUTER_BASE_URL`` when set.
ORCAROUTER_BASE_URL = "https://api.orcarouter.ai/v1"

REGISTRY_PATH = str(Path(__file__).with_name("model_registry.yaml"))


class OrcaRouterProvider:
"""OrcaRouter credentials + bundled-YAML metadata provider."""

DEFAULT_MODEL = "orcarouter/auto"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] orcarouter/auto is not a safe default for the SkillSpector structured-output path. LLMAnalyzerBase calls ChatOpenAI.with_structured_output(...), whose current default uses response_format: json_schema, but OrcaRouter documents that this is unsupported when the upstream is Anthropic and only works where the selected upstream can honor it. The seeded auto router matches every accessible chat model, so it may select such an upstream and make core analyzer calls fail even though the raw-chat smoke test succeeds. Please either use a concrete default model that guarantees JSON-schema output or a capability-constrained router, and add an OrcaRouter live structured-output test alongside tests/provider/test_provider_endpoint.py. References: structured outputs and auto router.

SLOT_DEFAULTS: dict[str, str] = {}

def resolve_credentials(self) -> tuple[str, str | None] | None:
"""Return ``(api_key, base_url)`` from ``ORCAROUTER_API_KEY`` / ``ORCAROUTER_BASE_URL``."""
api_key = os.environ.get("ORCAROUTER_API_KEY", "").strip()
if not api_key:
return None
base_url = os.environ.get("ORCAROUTER_BASE_URL", "").strip() or ORCAROUTER_BASE_URL
return api_key, base_url

def create_chat_model(
self,
model: str,
*,
max_tokens: int,
timeout: float | None = 120,
) -> BaseChatModel | None:
"""Create ``ChatOpenAI`` for the OrcaRouter OpenAI-compatible endpoint."""
return create_openai_compatible_chat_model(
model=model,
credentials=self.resolve_credentials(),
max_tokens=max_tokens,
timeout=timeout,
)

def get_context_length(self, model: str) -> int | None:
return registry.lookup_context_length(REGISTRY_PATH, model)

def get_max_output_tokens(self, model: str) -> int | None:
return registry.lookup_max_output_tokens(REGISTRY_PATH, model)

def resolve_model(self, slot: str = "default") -> str:
"""Resolve model: ``SKILLSPECTOR_MODEL`` env > slot default > ``DEFAULT_MODEL``."""
user_input = os.environ.get("SKILLSPECTOR_MODEL", "").strip()
return user_input or self.SLOT_DEFAULTS.get(slot, "") or self.DEFAULT_MODEL
45 changes: 45 additions & 0 deletions tests/unit/test_providers.py
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@
from skillspector.providers.gemini_cli import GeminiCLIProvider
from skillspector.providers.nv_build import BUILD_BASE_URL, NvBuildProvider
from skillspector.providers.openai import OpenAIProvider
from skillspector.providers.orcarouter import ORCAROUTER_BASE_URL, OrcaRouterProvider

try:
from skillspector.providers.nv_inference import (
Expand Down Expand Up @@ -116,6 +117,8 @@ def _clean_provider_env(monkeypatch: pytest.MonkeyPatch):
monkeypatch.delenv("OPENAI_API_KEY", raising=False)
monkeypatch.delenv("OPENAI_BASE_URL", raising=False)
monkeypatch.delenv("OPENAI_PROJECT_ID", raising=False)
monkeypatch.delenv("ORCAROUTER_API_KEY", raising=False)
monkeypatch.delenv("ORCAROUTER_BASE_URL", raising=False)
monkeypatch.delenv("SKILLSPECTOR_REASONING_EFFORT", raising=False)
monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)
monkeypatch.delenv("ANTHROPIC_BASE_URL", raising=False)
Expand Down Expand Up @@ -309,6 +312,41 @@ def test_metadata_known_model(self) -> None:
assert provider.get_max_output_tokens("gpt-5.4") == 128_000


class TestOrcaRouterProvider:
"""OrcaRouter provider — model-routing gateway credentials + YAML metadata."""

def test_returns_none_without_env_var(self) -> None:
assert OrcaRouterProvider().resolve_credentials() is None

def test_resolves_to_orcarouter_default_base_url(self, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("ORCAROUTER_API_KEY", "sk-orca-test")
creds = OrcaRouterProvider().resolve_credentials()
assert creds == ("sk-orca-test", ORCAROUTER_BASE_URL)

def test_honors_orcarouter_base_url_override(self, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("ORCAROUTER_API_KEY", "sk-orca-test")
monkeypatch.setenv("ORCAROUTER_BASE_URL", "https://router.example.com/v1")
creds = OrcaRouterProvider().resolve_credentials()
assert creds == ("sk-orca-test", "https://router.example.com/v1")

def test_creates_chat_openai(self, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("ORCAROUTER_API_KEY", "sk-orca-test")
llm = OrcaRouterProvider().create_chat_model("orcarouter/auto", max_tokens=123)
assert isinstance(llm, ChatOpenAI)
assert llm.model_name == "orcarouter/auto"
assert llm.max_tokens == 123

def test_default_model(self) -> None:
assert OrcaRouterProvider().resolve_model() == "orcarouter/auto"
# All slots inherit DEFAULT_MODEL — orcarouter/auto everywhere.
assert OrcaRouterProvider().resolve_model("meta_analyzer") == "orcarouter/auto"

def test_metadata_known_model(self) -> None:
provider = OrcaRouterProvider()
assert provider.get_context_length("orcarouter/auto") == 200_000
assert provider.get_max_output_tokens("orcarouter/auto") == 64_000


class TestAnthropicProvider:
"""Anthropic provider — Claude credentials + bundled YAML metadata."""

Expand Down Expand Up @@ -570,6 +608,13 @@ def test_select_anthropic(self, monkeypatch: pytest.MonkeyPatch) -> None:
assert creds == ("sk-ant-x", None)
assert isinstance(get_metadata_provider(), AnthropicProvider)

def test_select_orcarouter(self, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("SKILLSPECTOR_PROVIDER", "orcarouter")
monkeypatch.setenv("ORCAROUTER_API_KEY", "sk-orca-x")
creds = resolve_provider_credentials()
assert creds == ("sk-orca-x", ORCAROUTER_BASE_URL)
assert isinstance(get_metadata_provider(), OrcaRouterProvider)

def test_create_chat_model_uses_native_anthropic_when_configured(
self, monkeypatch: pytest.MonkeyPatch
) -> None:
Expand Down
Loading