fix(zen): OpenAI cache-write tokens are part of input_tokens, not extra

[xyzs996]

Issue for this PR

Refs #42910 (comment: #42910 (comment)). Independent of #44223, which fixes the local-side experimentalOver200K fallback.

Type of change

• Bug fix
• New feature
• Refactor / code improvement
• Documentation

What does this PR do?

normalizeUsage in the Zen OpenAI helper subtracts cached_tokens from input_tokens but leaves cache_write_tokens in. Both are subsets of input_tokens, per OpenAI's own sample in the prompt-caching guide:

usage.input_tokens = 2600
usage.input_tokens_details.cached_tokens = 2000
usage.input_tokens_details.cache_write_tokens = 400

In this example, 2,000 tokens were read from the cache and 400 additional tokens were written.
The remaining 200 input tokens were neither read nor written.

2,000 + 400 + 200 = 2,600 — https://developers.openai.com/api/docs/guides/prompt-caching

So the written tokens stayed inside inputTokens and were reported as cacheWrite5mTokens, and handler.ts uses those buckets two ways:

1. Long-context threshold trips early. calculateCost sums all four buckets to pick the tier. Substituting, that sum is (input_tokens − cached) + cached + written = input_tokens + written — over the real prompt by exactly cache_write_tokens, every request. On a well-cached turn written ≈ inputTokens, so the sum runs near 2× the prompt and the gate trips at roughly half the real context. This matches what @Zaczero measured on a 302-request Luna burst in #42910: inputTokens and cacheWrite5mTokens differing by 3 on almost every row (that 3 is the "neither read nor written" remainder), 94 requests in the 200–272k band paying the high rate, +$2.53 on one afternoon.

2. The same tokens are billed twice. inputCost = modelCost.input * inputTokens charges the written tokens at the input rate, then cacheWrite5mCost = modelCost.cacheWrite5m * cacheWrite5mTokens charges them again. OpenAI rules that out explicitly: "Tokens written to the cache are billed at 1.25× the uncached input token rate." … "It is not an additional charge on top of another full input-token charge." On gpt-5.6-luna ($0.20/M input) a written token should cost $0.25/M and is billed $0.45/M — 1.8×, on every cached request, regardless of any threshold.

One line fixes both.

session.ts already does this correctly on the local side (it subtracts both, with a comment explaining why), so this brings Zen in line with it.

Anthropic is deliberately untouched and must stay that way — Anthropic's input_tokens genuinely excludes both cache buckets, so anthropic.ts is right to pass it through and the four-bucket sum is already correct there. Google has no cache-write bucket. That asymmetry is why the fix belongs in openai.ts rather than in calculateCost.

How did you verify your code works?

packages/console/app/test/providerUsage.test.ts:

• The existing "parses OpenAI stream cache write usage" case (input_tokens: 10, cached_tokens: 4, cache_write_tokens: 3) expected inputTokens: 6; it now expects 3. That existing expectation was pinning the bug, so it changes with the fix rather than being worked around.
• New case using OpenAI's documented sample verbatim: 2600 / 2000 / 400 → inputTokens: 200, plus an assertion that the four buckets sum back to exactly 2600, which is the invariant handler.ts relies on for the tier.

Screenshots / recordings

N/A

Checklist

• I have tested my changes locally
• I have not included unrelated changes in this PR

[github-actions]

Thanks for your contribution!

This PR doesn't have a linked issue. All PRs must reference an existing issue.

Please:

1. Open an issue describing the bug/feature (if one doesn't exist)
2. Add Fixes #<number> or Closes #<number> to this PR description

See CONTRIBUTING.md for details.

[Enough1122]

AI code review — automated review for reference, author can ignore or act on any point.

The accounting model (four disjoint buckets partitioning input_tokens) is internally consistent, and the new partition test is exactly the right regression guard (packages/console/app/test/providerUsage.test.ts:84-105) — summing the buckets back to 2600 pins the invariant, not just the arithmetic. Two things to tighten:

1. Verify the cited source before merge. The comment at packages/console/app/src/routes/zen/util/provider/openai.ts:54-58 attributes the 2600 = 2000-read + 400-written + 200-neither breakdown to "OpenAI's own sample", and the code reads input_tokens_details.cache_write_tokens. To my knowledge OpenAI's published usage schema documents only cached_tokens as a subset of input tokens; a first-class cache_write_tokens field is characteristic of Anthropic-style accounting. If the field actually arrives via a zen-side gateway/upstream rather than OpenAI proper, please point the comment (and PR body) at the real producer — otherwise future maintainers may "fix" this back based on OpenAI's official docs. The ?? undefined fallback means unpopulated fields stay safe either way.

2. Extend the partition invariant to the other helpers. handler.ts sums the four buckets for long-context tiering regardless of provider; the same sum-to-prompt assertion added here for OpenAI would be cheap to replicate for the Anthropic/other helpers in providerUsage.test.ts, whose input_tokens semantics differ (Anthropic excludes cache tokens from input_tokens natively), guarding against someone "harmonizing" the wrong direction later.

Otherwise clean, minimal, and well-tested.

[xyzs996]

Thanks — both points were worth raising. Taking them in order.

1. The field is OpenAI's, and the sample is from their guide.

I re-checked https://developers.openai.com/api/docs/guides/prompt-caching before answering. cache_write_tokens is documented there as a first-class member of input_tokens_details on the Responses API (and of prompt_tokens_details on Chat Completions), and the 2600/2000/400 breakdown is their example verbatim:

"input_tokens": 2600,
"input_tokens_details": {
  "cached_tokens": 2000,
  "cache_write_tokens": 400
}

with the accompanying text stating that 2,000 tokens were read from cache, 400 were newly written, and the remaining 200 were "neither read nor written". The billing side is equally explicit: "Tokens written to the cache are billed at 1.25× the uncached input token rate.", "The 1.25× cache-write rate is the total rate for written tokens.", and "It is not an additional charge on top of another full input-token charge."

So the comment's attribution is accurate as written, and no zen-side gateway is inventing the field. Your underlying worry is the right one though — someone reading an older copy of the usage schema would see only cached_tokens and could well "correct" this back. The new partition assertion is what stops that silently: revert the fix and the sum comes out 3000 instead of 2600, so the test names the invariant rather than the arithmetic.

2. Extended the invariant to the other helpers — 73c2363.

Rather than duplicating the assertion per provider, the new case runs the same 2,600-token prompt through all three helpers in each one's own wire shape and asserts both the normalized inputTokens and the four-bucket sum:

• OpenAI — input_tokens: 2600 with the cache buckets nested inside it → inputTokens: 200
• Anthropic — input_tokens: 200 with cache_read_input_tokens: 2000 and cache_creation.ephemeral_5m_input_tokens: 400 alongside it → inputTokens: 200
• Google — promptTokenCount: 2600, cachedContentTokenCount: 2000, no cache-write bucket → inputTokens: 600

All three normalize to a partition summing to 2600. That is exactly the "harmonizing in the wrong direction" guard you described: the wire formats disagree about whether the cache buckets live inside the prompt count, and the assertion pins that they must agree after normalization, which is the only place handler.ts looks.

3. Unrelated heads-up while I was in that file.

Two pre-existing cases in providerUsage.test.ts fail on dev as of today, with or without this PR:

(fail) provider usage extraction > extracts Google non-stream usage metadata
(fail) provider usage extraction > parses Google stream usage metadata
       expected outputTokens: 3, received 5

Both assert outputTokens: 3 while google.ts:68 returns candidatesTokenCount + thoughtsTokenCount = 3 + 2 = 5. Reproduced on a clean up/dev checkout with bun test test/providerUsage.test.ts, so it is not something this branch introduced — the expectations look stale relative to the helper. Happy to open a separate PR for it; I have deliberately left it alone here so this diff stays on one subject.

[xyzs996]

Thanks — both points are answerable, one with a citation and one with a commit.

1. The field is OpenAI's own, documented, with that exact example

Not a zen-side gateway invention. From OpenAI's prompt-caching guide (https://developers.openai.com/api/docs/guides/prompt-caching):

For the Responses API, both fields appear in usage.input_tokens_details

and the worked example the code comment is pointing at:

"input_tokens": 2600,
"input_tokens_details": { "cached_tokens": 2000, "cache_write_tokens": 400 }

In this example, 2,000 tokens were read from the cache and 400 additional tokens were written. The remaining 200 input tokens were neither read nor written.

So cache_write_tokens is a first-class documented field, it lives in input_tokens_details alongside cached_tokens, and both are subsets of input_tokens — which is precisely what the current code got wrong by adding the write bucket on top. 2600 − 2000 − 400 = 200 uncached, not 2600 − 2000 = 600.

You're right that this reads as Anthropic-shaped accounting; the difference is that Anthropic puts its cache buckets outside input_tokens and OpenAI puts them inside. Same three numbers, opposite containment — which is exactly the trap this PR is about.

The same page also independently supports the double-billing half of the fix:

A longer cache write does not bill the already cached 2,000 tokens again.

No comment or PR-body change needed, but if you'd prefer the doc URL inlined at openai.ts:54-58 rather than the prose "OpenAI's own sample", say the word and I'll add it.

2. Already extended — 73c2363

Pushed after your review: packages/console/app/test/providerUsage.test.ts:113, test("every helper partitions the prompt into the same four buckets"). Same 2,600-token prompt expressed three ways, asserting both the uncached remainder and the four-bucket sum:

helper	wire shape	expected inputTokens	four buckets sum to
openai	cache buckets inside input_tokens	200	2600
anthropic	cache buckets outside input_tokens	200	2600
google	no cache-write bucket at all	600	2600

Google lands at 600 legitimately — cachedContentTokenCount has no write counterpart, so the 400 has nowhere to go but the uncached bucket. That asymmetry is the reason the guard is written against the normalized values rather than per-helper: handler.ts sums the same four fields for every provider when it picks the long-context tier, so normalized is the only place all three are required to agree.

One helper I deliberately left out: openai-compatible.ts. It subtracts cacheReadTokens from inputTokens but not cacheWriteTokens, so whether the partition holds depends on whether the upstream provider counts cache_creation_input_tokens inside prompt_tokens — and that varies per provider behind that adapter. Its adjustCacheUsage path also synthesizes a 90% cache-read share when the field is absent, which would make a sum-to-prompt assertion vacuous. Pinning a single answer there felt like a behaviour claim I can't source, unlike the three above. Happy to add it as a documented-expectation test if you'd rather have the current behaviour locked down.