Skip to content

fix(pricing): price OpenRouter turns from the lake and declared rates - #6691

Merged
Hmbown merged 2 commits into
integrate/0.10.1from
fix/6690-openrouter-session-cost
Sep 28, 2026
Merged

Hmbown merged 2 commits into
integrate/0.10.1from
fix/6690-openrouter-session-cost

Conversation

@Hmbown

@Hmbown Hmbown commented Sep 28, 2026 •

Copy link
Copy Markdown
Owner

Fixes #6690

Cause

  • The OpenRouter /v1/models refresh failed as a whole on any single bad row, so the OpenRouter lake scope stayed failed/invalid_response and fresh_provider_live_pricing_quote_at never returned a rate. Every OpenRouter turn showed "rate unavailable". The live list has two kinds of row that did this:
    • ~-prefixed "latest" aliases (~deepseek/deepseek-pro-latest, 18 in the live list), which fail valid_catalog_model_id.
    • six routers (openrouter/auto, auto-beta, fusion, pareto-code, bodybuilder, typesafe/jev-router) that publish "prompt":"-1","completion":"-1". parse_price rejected negatives and the conversion collected with ?.
  • The main interactive turn (turn_loop dispatch boundary) froze its quote only from the lake, never from the operator-declared [[custom_models]] rate that effective_route_envelope (background/review) already honors.

Fix

  • parse_openrouter_models_response still requires a {"data": [...]} list but decodes each row on its own. It drops ~ alias rows, and it skips and counts (with a tracing::warn!) rows that do not decode or whose id the catalog cannot hold. fetch_catalog_delta also skips and counts rows with an unreadable or implausible price. A structurally invalid response, or one where no row survives, is still InvalidResponse. The secret-in-id guard still rejects the whole list.
  • A negative OpenRouter price means "no fixed rate": the row is kept with cost: None, never with a partial rate.
  • New CodewhaleClient::configured_pricing_quote_at and client::main_turn_pricing_quote_at. The turn loop now prefers a declared rate, scoped to the installed client's endpoint fingerprint, before the lake quote.
  • New provider_catalog_live::declared_or_catalog_quote, used at all three dispatch boundaries (main turn, effective_route_envelope, EffectiveRouteEnvelope::capture). A declared rate wins. A declaration with no rates yields to the same endpoint's catalog price, and stays frozen rate-less only when the catalog has none, so a same-named bundled price still cannot fill it.
  • EffectiveRouteEnvelope::audit: a pinned openrouter_vendor still blocks the aggregate catalog price (RoutingDependentPrice), but no longer blocks an explicit declared (UserOverride) rate for the exact route.

Against the saved live list (458 rows), the refresh now succeeds with 440 rows, 434 of them priced. deepseek/deepseek-v4-pro prices at 0.95526 / 1.91052 USD per M (cache read 0.079605).

Still open: a server-tag-suffixed route the catalog does not list (the reporter's deepseek/deepseek-v4.1-flash:nitro) still needs an exact catalog row or a declared [[custom_models]] rate, which now prices it on the main turn. Tag prices can differ from the base id, so they are not stripped.

Tests

Each new or strengthened test fails with its fix reverted and passes with it. They assert the actual rates and cost, not just that a quote exists:

  • client::tests::fetch_catalog_delta_skips_openrouter_latest_aliases_and_keeps_prices: live-shaped fixture (~ alias, -1 router, undecodable row, unreadable price, bad id, real deepseek/deepseek-v4-pro row). It also checks that structurally invalid bodies are still rejected.
  • client::tests::provider_live_price_parsers_reject_present_bad_rates_but_keep_zero_and_omission: updated so a negative OpenRouter price gives cost: None.
  • cost_status::tests::main_turn_rateless_declaration_yields_to_the_endpoint_catalog_price
  • cost_status::tests::openrouter_vendor_pin_is_priced_by_an_explicit_declared_rate
  • core::engine::tests::main_turn_dispatch_freezes_declared_custom_model_rate: now asserts the frozen rates.
fixed:    test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 13793 filtered out
reverted (neg price, rate-less fallthrough, vendor pin):
          test result: FAILED. 0 passed; 3 failed; 0 ignored; 0 measured; 13800 filtered out
reverted (per-row decode skip):
          test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 13802 filtered out
related (fetch_catalog openrouter configured_model effective_route provider_live_pricing
         exact_turn_snapshot main_turn cost_status provider_catalog_live):
          test result: ok. 120 passed; 0 failed; 0 ignored; 0 measured; 13683 filtered out

Only focused cargo test -p codewhale-tui --lib runs happened locally. The full suite is left to CI.

🤖 Generated with Claude Code

https://claude.ai/code/session_014ZwqatxgVFxHvovngywnks

Cause: OpenRouter's /v1/models now lists `~`-prefixed "latest" alias ids
(`~deepseek/deepseek-pro-latest`). They fail `valid_catalog_model_id`, and
one invalid row fails the whole roster closed, so the OpenRouter lake scope
was permanently `failed/invalid_response` and
`fresh_provider_live_pricing_quote_at` never returned a rate: every
OpenRouter turn showed "rate unavailable". Separately, the main interactive
turn froze its dispatch quote only from the lake and never consulted the
operator-declared `[[custom_models]]` rate that background/review
envelopes (`effective_route_envelope`) already honor.

Fix: `parse_openrouter_models_response` drops `~` alias rows (moving
aliases, not billing identities); the fail-closed validator is unchanged
for every other id. A new `CodewhaleClient::configured_pricing_quote_at`
is shared by `effective_route_envelope` and the turn-loop dispatch
boundary, which now prefers a declared rate (scoped to the installed
client's endpoint fingerprint) before the lake quote.

Not covered: server-tag suffixed ids (`:nitro`, `:free`) still need an
exact catalog or declared row; their price can differ from the base id,
so they are not stripped.

Tests (each fails with its fix hunk reverted, passes with it):
- client::tests::fetch_catalog_delta_skips_openrouter_latest_aliases_and_keeps_prices
- core::engine::tests::main_turn_dispatch_freezes_declared_custom_model_rate
test result: ok. 2 passed; 0 failed (reverted: 0 passed; 2 failed)
Related filters (fetch_catalog, openrouter, configured_model_client_tests,
exact_turn_snapshot, effective_route, provider_live_pricing):
test result: ok. 46 passed; 0 failed
cost_status, provider_catalog_live, core::engine::tests::exact:
test result: ok. 60 passed; 0 failed

Refs #6690

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014ZwqatxgVFxHvovngywnks
Copilot AI lite review requested due to automatic review settings September 28, 2026 03:02
@Hmbown Hmbown added this to the v0.10.1 milestone Sep 28, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-28T03:06:31.894112Z 2412a6e PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@gitguardian

gitguardian Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

️✅ There are no secrets present in this pull request anymore.

If these secrets were true positive and are still valid, we highly recommend you to revoke them.
While these secrets were previously flagged, we no longer have a reference to the
specific commits where they were detected. Once a secret has been leaked into a git
repository, you should consider it compromised, even if it was deleted immediately.
Find here more information about risks.


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

@Hmbown
Hmbown changed the base branch from main to integrate/0.10.1 September 28, 2026 03:15
Review follow-up for #6691. The live OpenRouter /v1/models lists six
routers (openrouter/auto, auto-beta, fusion, pareto-code, bodybuilder,
typesafe/jev-router) at "prompt":"-1","completion":"-1". parse_price
rejected negatives and the conversion collected with `?`, so the whole
refresh still failed and every turn stayed "rate unavailable".

- A negative OpenRouter price now means "no fixed rate": the row is kept
  with cost None. Rows that do not decode, carry an id the catalog cannot
  hold, or an unreadable/implausible price are skipped and counted with a
  warning. A response that is not a `{"data": [...]}` list, or whose rows
  all fail, is still rejected.
- declared_or_catalog_quote: a [[custom_models]] row with no rates yields
  to the exact endpoint's catalog price at every dispatch boundary (main
  turn, background envelope, EffectiveRouteEnvelope::capture); it stays
  frozen rate-less only when the catalog has none. The turn-loop logic
  moved into client::main_turn_pricing_quote_at so it is testable.
- A pinned OpenRouter vendor no longer blocks an explicit declared rate
  for the exact route; the aggregate catalog price stays blocked.

Against the saved live list (458 rows): refresh succeeds with 440 rows,
434 priced; deepseek/deepseek-v4-pro prices at 0.95526/1.91052 per M.

Focused (cargo test -p codewhale-tui --lib):
fixed:    test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 13793 filtered out
reverted (neg price, rate-less fallthrough, vendor pin):
          test result: FAILED. 0 passed; 3 failed; 0 ignored; 0 measured; 13800 filtered out
reverted (per-row decode skip):
          test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 13802 filtered out
related (fetch_catalog openrouter configured_model effective_route
provider_live_pricing exact_turn_snapshot main_turn cost_status
provider_catalog_live):
          test result: ok. 120 passed; 0 failed; 0 ignored; 0 measured; 13683 filtered out

Fixes #6690

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014ZwqatxgVFxHvovngywnks
@Hmbown
Hmbown merged commit 3a4b961 into integrate/0.10.1 Sep 28, 2026
8 checks passed
@Hmbown
Hmbown deleted the fix/6690-openrouter-session-cost branch September 28, 2026 03:40
Hmbown pushed a commit that referenced this pull request Sep 28, 2026
…6693

Hosted CI on 776e3fe (PR #6672) failed two codewhale-tui tests.

1. client::catalog_tests::raw_rows_keep_duplicate_known_fields_invalid_in_existing_parsers
   Cause: #6691 (79d9815) made OpenRouter parsing per-row by decoding
   `data` as Vec<serde_json::Value>. A Value map keeps the last of a
   duplicated key, so `{"id":"second","id":"replacement"}` was silently
   accepted as "replacement" (and a duplicated pricing.prompt took the
   last rate) instead of being rejected.
   Fix (product): keep rows as Box<RawValue> and decode each with
   from_str, so serde's duplicate-field error still fires and the row is
   skipped and counted as malformed like any other undecodable row. The
   test now asserts the per-row contract: the ambiguous row is dropped,
   "first" survives; Baseten still rejects the whole response.

2. tui::ui::tests::first_run_ollama_choice_survives_restart_from_canonical_config
   Cause: #6693 (1394c29, eb48db7) moves a root default_text_model
   onto the leaf of the outgoing route that was resolving it. The test
   seeded default_text_model = "deepseek-v4-pro" with DeepSeek active and
   no leaf, then asserted the root key survived the switch to Ollama
   ("index not found"). That encoded superseded behavior.
   Fix (test): assert the root alias is gone and
   [providers.deepseek].model = "deepseek-v4-pro", which is the #6693
   contract (switching back keeps the choice; Ollama never inherits it).

Tests (CARGO_BUILD_JOBS=4, cargo test -p codewhale-tui --lib):
- raw_rows_keep_duplicate_known_fields_invalid_in_existing_parsers:
  test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 13814 filtered out
- first_run_ollama_choice_survives_restart_from_canonical_config:
  test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 13814 filtered out
- filters catalog openrouter fetch_catalog first_run_ provider_picker local_ollama:
  test result: ok. 450 passed; 0 failed; 2 ignored; 0 measured; 13363 filtered out
  (a first run of the same filters had 1 failure in
  provider_picker::tests::credential_draft_is_masked_and_escape_drops_it_without_persistence,
  which passed 3/3 alone and in the rerun; it counts files in a home dir
  and is untouched by this change)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014ZwqatxgVFxHvovngywnks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants