Claude Arena

A dependency-free Claude router and local model arena for engineering work. It routes by complexity and budget, calls Anthropic's Messages API, records actual token usage and cost, and offers both a CLI and a blind A/B browser experience.

Highlights

• Cost-aware routing across Haiku, Sonnet, and Opus tiers.
• Real Anthropic Messages API adapter with timeouts and structured errors.
• Explicit budget enforcement before paid requests.
• Parallel model battles, anonymous browser voting, and a leaderboard.
• SQLite audit history with prompt, response, latency, token usage, and cost.
• Local HTTP API and responsive web UI; API keys stay on the server.
• Zero runtime dependencies; Python 3.11+ is enough.

Install

From a release artifact:

python3 -m pip install claude_arena-0.2.0-py3-none-any.whl

Or from a checkout:

python3 -m pip install .

Set the API key only in the environment:

export ANTHROPIC_API_KEY="your-key"

Use

Routing is offline and free:

arena route "review this Terraform change for production risk" --pretty
arena models

Run the automatically selected model or select one explicitly:

arena run "Explain the failure mode in this stack trace"
arena run - --model claude-sonnet-5 --max-tokens 1500 < prompt.txt

Compare models in parallel:

arena battle "Design a safe Kubernetes rollout" \
  --models claude-sonnet-5 claude-opus-4-8 \
  --max-tokens 1200 --allow-over-budget

Open the local arena:

arena serve
# http://127.0.0.1:8765

The browser battle uses the two strongest configured models, randomizes their positions, hides their identities until preferences are recorded, and stores votes locally. The server binds to localhost by default.

Inspect local usage:

arena history --limit 10
arena stats

Budget behavior

Every request is estimated before it is sent. The router downgrades within the configured model tiers when allowed. If even the cheapest valid choice exceeds the ceiling, execution stops unless --allow-over-budget is passed explicitly. Provider-reported usage is used for the final cost, including prompt-cache read and write multipliers.

The default model catalog and prices reflect Anthropic's published catalog when v0.2.0 was released. They are configuration, not hard-coded provider logic, so teams can update them without changing code.

Configuration

Pass --config path.json before the subcommand. See examples/router-config.json for a complete file.

{
  "router": {
    "max_cost_usd": 0.08,
    "prefer_capability": true,
    "allow_downgrade": true
  },
  "runtime": {
    "base_url": "https://api.anthropic.com",
    "timeout_seconds": 120,
    "history_path": "~/.claude-arena/arena.db"
  }
}

prefer_capability: false selects one tier below the classifier recommendation when possible. Model names, prices, context sizes, and cache multipliers can all be overridden in the models array.

Local data and security

The API key is read from ANTHROPIC_API_KEY and is never written to SQLite or sent to the browser. By default, completed prompts and responses are stored in ~/.claude-arena/arena.db for history and voting. Use --no-store on run or battle for sensitive requests. Do not expose arena serve to an untrusted network without adding authentication and TLS in front of it.

Development

python3 -m unittest discover -s tests
ruff check .
mypy arena
python3 -m build

See docs/architecture.md for component and API details.