bench: tb21 harness carry + usage accounting, now synced with main (merge, no rebase) by tlongwell-block · Pull Request #5145 · block/buzz

tlongwell-block · 2026-08-07T03:10:22Z

What this branch is

benchmark/harness-accounting-and-solo is the branch both tb21 A/B benchmark boxes run (solo i-0f04fcd287e19da4e, team i-05e08e50c1c714f47, deepseek-v4-flash via OpenRouter). Until now it lived as an orphan branch — never in a PR — which is how it silently drifted 128 commits behind main and missed agent-level fixes like #4896. This PR gives it a review surface and a drift banner.

The 6 bench-only commits (what main doesn't have)

c6d7b356 bench: carry the harbor-buzz-orchestra harness onto latest main
90b56ba2 bench: log what a tool call touched, and pin the effort-wave endpoints
10f09396 bench: pin the OpenRouter upstream so a cell is one condition
131891fc bench: add the OpenRouter wave — kimi-k3 and deepseek-v4-flash cells
b915d567 fix: settle usage on the timeout path, not only after DONE
1cf841af fix: report agent usage per provider round, not once per turn

(Author of the bench/accounting commits: @atish — flagging for review of which of these should graduate to main.)

The new merge commit (`9cafe8f1`)

Merges current origin/main in as a merge commit — no rebase, no history rewrite, no force push. This picks up, among 128 commits:

fix(agent): recover from unsupported image input instead of poisoning the turn #4896 — recover from unsupported image input instead of poisoning the turn (the vision-404 poison loop that tainted 15 tb21 trials: 10 solo fails, 5 team)
fix(bench): mention the orchestrator by pubkey when posting the task #5136 — mention the orchestrator by pubkey when posting the task (the @a Vim-macro mention-resolver crash that killed large-scale-text-editing in both conditions)

One conflict: container_runtime.py — the bench branch's solo-mode _wait_for_done changes vs #5136's _send(mention=) plumbing. Resolved by keeping both; identical to the resolution already e2e-validated on the scratch box (branch eva/rg-fix-on-bench-branch 2944a88, rerun job rg-fix-1, reward 1.0, zero RuntimeLaunchError).

Validation

benchmarks/harbor-buzz-orchestra: full pytest suite at merge head 9cafe8f — 273 passed.
ruff: 4 pre-existing F401s in tests/test_container_runtime.py (unused FORWARDER import) — present at bench head 1cf841a before the merge, not introduced here.
git merge-base --is-ancestor confirms fix(agent): recover from unsupported image input instead of poisoning the turn #4896 (8a7eb8d), fix(bench): mention the orchestrator by pubkey when posting the task #5136 merge commit (f53bbd1), and full origin/main are all ancestors of 9cafe8f.

Intent / merge posture

Primarily a visibility PR: review what's bench-only and decide what graduates to main. Safe to leave open as the bench branch's standing PR. The two in-flight A/B boxes are still pinned to 1cf841a and are unaffected; post-run reruns of tainted trials will build from this merge head after a re-smoke on the scratch box.

Originating Buzz thread: buzz://message?channel=c3252dd2-0142-4e01-88c7-a2183c3960a5&id=74a65a0990fd2197882b66b5ea2707169d4a3dbd2020d1610c45150fb99f140b

…8 pin (run 2) Commits the tb21 Meli-solo condition that until now lived only as untracked files on the runner box, plus the take-2 variant Tyler asked for (thread 74a65a09, event f7457e0b): - tb-meli-solo.yaml + personas/bench/meli-solo.md: the exact manifest and persona (sha256 de633019...) that produced run tb21-solo-1 on the gmicloud/fp8 pin. Recorded for provenance. - tb-meli-solo-baseten.yaml: identical cell, condition tb-meli-solo-baseten, priced at the baseten/fp8 listed rates (prompt $0.13/M, completion $0.26/M — OpenRouter endpoints API, 2026-08-07). - testbed/endpoints/openrouter-baseten.json: the provider pin, OPENROUTER_PROVIDER_ORDER=baseten/fp8. A separate endpoint file so openrouter-live.json (which run 1's provenance points at) stays byte-identical. Run 2 executes on the post-merge harness (9cafe8f, PR #5145), which carries #4896 (vision-input recovery) and #5136 (explicit-mention task post) that run 1 predated. Originating Buzz thread: buzz://message?channel=c3252dd2-0142-4e01-88c7-a2183c3960a5&id=74a65a0990fd2197882b66b5ea2707169d4a3dbd2020d1610c45150fb99f140b Co-authored-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>

Brings the benchmark harness (accounting, container runtime, manifests, personas, summarize/leaderboard tooling) forward onto main. main does not touch benchmarks/, so this is the branch tree verbatim. Signed-off-by: Atish Patel <atish@squareup.com> Co-authored-by: Claude Code <noreply@anthropic.com>

Two harness-facing deltas main does not already carry: - acp.rs: `tool_call: shell (other)` records that a shell ran but not what it ran, and the command is not persisted anywhere else in a session bundle. Without it a teammate that read the repo and one that rewrote it leave identical logs, which is the whole content of a read-only role. - config.rs: tests pinning that the benchmark's endpoint spellings survive the effort clamp. An unrecognised model silently clamps down, which turns a graded sweep into a null that reads as a finding. Everything else this branch carried in crates/ (cache_control breakpoints, cached-token accounting, provider_extra/thoughtSignature, turn-start logging) has landed on main independently, so main's version is taken. Signed-off-by: Atish Patel <atish@squareup.com> Co-authored-by: Claude Code <noreply@anthropic.com>

An OpenRouter model id is a family of upstream deployments, not one deployment. `deepseek-v4-flash-0731` is served by nine upstreams spanning fp4 to fp8 quantization, 262K to 1M context and a 1.6x price spread; `kimi-k3` by ten spanning 1.5x. Unpinned, consecutive requests land on different ones, so a benchmark cell measures a mixture whose composition moves with provider load rather than a fixed condition. It also decides whether prompt caching happens at all. Measured today against a fixed 21.8K-token prefix, three calls each: deepseek, unpinned cached 0, 0, 21760 (GMICloud, Cloudflare x2) deepseek gmicloud/fp8 cached 21760 x3 $0.00059/call deepseek siliconflow/fp8 cached 0 x3 $0.00306/call kimi, unpinned cached 0, 0, 20480 (Fireworks, Moonshot x2) kimi moonshotai/mxfp4 cached 20480 x3 $0.00715 vs $0.06245 That is a 5-9x swing in input cost for identical work, decided by routing luck. The `/endpoints` metadata is no guide: it advertises `supports_implicit_caching: false` for every endpoint measured caching above, and `true` only for one that is not routable on this account. So: `OPENROUTER_PROVIDER_ORDER` (comma-separated, accepting both the bare `moonshotai` slug and the `gmicloud/fp8` slug/quantization form) sets `provider.order`. It is strictly opt-in -- unset leaves the body byte identical, which is what keeps the existing "no body shape adds a provider routing filter" assertions honest. Paired with `allow_fallbacks: false`, because `order` alone is only a preference: OpenRouter still serves from elsewhere when the named upstream is busy, which is exactly the silent mid-run condition change this exists to prevent. A pin that quietly falls back is not a pin. This is not `require_parameters` -- that filter screens on advertised parameters and 404s otherwise-valid model ids, which is why the reasoning block nearby stays away from it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Atish Patel <atish@squareup.com>

Three cells, all at reasoning effort `high`, all on pinned OpenRouter upstreams: tb-solo-kimi-k3-high TB 2.1 solo, moonshotai/kimi-k3 tb-solo-deepseek-v4-flash-high TB 2.1 solo, deepseek-v4-flash-0731 lhtb-or-kimi-lead-2deepseek-high LHTB-46 team: kimi lead, deepseek scout, deepseek worker The first cells in the study on open-weight models and on a route other than direct-OpenAI or Databricks. The solo pair spans a 22x input price ratio ($0.133 vs $3.00/Mtok), making it the open-weight analogue of the luna -> sol ladder; the team cell is the open-weight restaging of tb-gt-sol-luna-terra-high, which is currently the study's only clean long-horizon team win. NO MEDIUM-EFFORT CELLS. The original plan had a high/medium pair per model. Neither model supports it: both advertise supported_efforts ["max","high","low"], and while OpenRouter accepts `medium` without error, measured reasoning length on deepseek did not separate low from medium from high (medians 1097 / 1102 / 1067, within-level spread wider than between-level). A cell labelled `medium` would name a condition the provider does not implement, so the effort axis is dropped rather than faked. Effort is still pinned on all three cells even though `high` matches deepseek's provider default -- an unpinned cell records no effort in its manifest, so a later provider-side default change would silently redefine the condition without moving the hash. kimi defaults to `max`, so there `high` is a deliberate step down. Prices are the PINNED endpoints' own rates, not the model ids' headline rates -- gmicloud/fp8 bills $0.133/$0.266 where the model id advertises $0.14/$0.28, and two of kimi's ten upstreams bill 1.5x the row here. The pin is what makes these rows the correct ones, so dropping it would require restating cost as well as re-establishing the condition. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Atish Patel <atish@squareup.com>

`_settle_usage` waited for the agent to report its tokens before teardown killed it, but only on the success path. The timeout path fell straight through to the kill, on the stated reasoning that "a turn that never completed has no usage to flush". That was true while buzz-agent reported once per turn. It is not true now that it reports after every provider round, and it was always the wrong path to skip: under `continue_until_timeout` every phase but the last ends here, so the skip applied to essentially the whole run. It is the harness half of the 97%-zero-rows / 5.9x-undercount measured on the OpenRouter LHTB-46 cell; the agent half is the per-round reporting. Moving the call into the `finally` covers both paths and keeps the ordering that matters — settle, then kill. It stays cheap: it returns on the first poll once a usage line exists, which after the first round it does, so only a phase that never completed a single round can spend the full budget, and that phase has nothing to report anyway. Depends on the per-round reporting in buzz-agent to be worth much on its own. Signed-off-by: Atish Patel <atish@squareup.com> Co-authored-by: Claude Code <noreply@anthropic.com>

…8 pin (run 2) Commits the tb21 Meli-solo condition that until now lived only as untracked files on the runner box, plus the take-2 variant Tyler asked for (thread 74a65a09, event f7457e0b): - tb-meli-solo.yaml + personas/bench/meli-solo.md: the exact manifest and persona (sha256 de633019...) that produced run tb21-solo-1 on the gmicloud/fp8 pin. Recorded for provenance. - tb-meli-solo-baseten.yaml: identical cell, condition tb-meli-solo-baseten, priced at the baseten/fp8 listed rates (prompt $0.13/M, completion $0.26/M — OpenRouter endpoints API, 2026-08-07). - testbed/endpoints/openrouter-baseten.json: the provider pin, OPENROUTER_PROVIDER_ORDER=baseten/fp8. A separate endpoint file so openrouter-live.json (which run 1's provenance points at) stays byte-identical. Run 2 executes on the post-merge harness (9cafe8f, PR #5145), which carries #4896 (vision-input recovery) and #5136 (explicit-mention task post) that run 1 predated. Originating Buzz thread: buzz://message?channel=c3252dd2-0142-4e01-88c7-a2183c3960a5&id=74a65a0990fd2197882b66b5ea2707169d4a3dbd2020d1610c45150fb99f140b Co-authored-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>

Same Meli-solo condition as tb-meli-solo-baseten.yaml with the pin moved to cloudflare/fp8 (Tyler, thread 74a65a09, event 857619b8), after the baseten/fp8 shared pool rate-limited the smoke gate (~22 upstream 429s per trial). Priced at cloudflare's listed rates (prompt $0.14/M, completion $0.28/M -- OpenRouter endpoints API, 2026-08-07; no cached rate listed, cached priced as input). Co-authored-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>

Two-seat twin roster smoked green on hello-world (reward 1.0) and now running the full TB2.1 89-task bench as tb21-twins-1. Mia holds the conn (orchestrator seat — the harness watches that pubkey for DONE:), Mie is the delegable worker twin. Both personas are the Meli solo prompt plus twin team sections, byte-identical to each other except the identity line and the conn/worker paragraph. Contract: thread-of-record comms (--reply-to the task post), both keep the todo tool loaded, Mia never ends her turn while the task is incomplete (sleep-30 poll loop; requires unbounded agent rounds from #5318), Mie ends every turn with a report mentioning Mia, only Mia sends DONE:. Model: deepseek-v4-flash via OpenRouter pinned cloudflare/fp8 — the existing openrouter-cloudflare.json endpoint config, unchanged. Manifest persona sha256 pins match the committed files. Co-authored-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>

@name

…ble-first Amendments from the tb21-twins-1 fail triage (all 24 zero-reward trials): - Delegation paragraph replaced: four wake triggers (parallel exploration, rival hypotheses, test-as-I-build, independent verification) with the burden inverted — an idle Mie is a decision to defend. Outstanding Mie lane doubles as Mia's heartbeat (fixes the dead-turn trial). - Todo paragraph: task event id written at minute 0 + mandatory fetch-by-id reread of the full task text as the last item before DONE: (fixes grading-against-memory failures). - Deliverable-first clause: rough deliverable at the exact task path ASAP (12 of 24 fails were timeouts with nothing at the required path; 4 other timeouts scored 1.0 purely on disk artifacts). - Mention-mechanics clause: silent lane => treat wake as unrouted, recheck exact @name, resend once (filter-js-from-html lost its verify lane to a missing @ prefix). mie.md unchanged. Manifest mia sha256 re-pinned; loader validates. Co-authored-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>

Add matched Databricks Luna xhigh conditions for a pinned solo persona and a pinned driver/navigator pair. Both keep the existing route, rates, trial budget, and container resources while exposing prompt bytes as the optimization axis. Co-authored-by: Atish Patel <atish@squareup.com> Signed-off-by: Atish Patel <atish@squareup.com>

Co-authored-by: Atish Patel <atish@squareup.com> Signed-off-by: Atish Patel <atish@squareup.com>

Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: Atish Patel <atishpatel2012@gmail.com>

tlongwell-block requested a review from a team as a code owner August 7, 2026 03:10

atishpatel force-pushed the benchmark/harness-accounting-and-solo branch from 4444d6d to 477902e Compare August 14, 2026 17:49

atishpatel and others added 10 commits August 14, 2026 12:49

atishpatel force-pushed the benchmark/harness-accounting-and-solo branch from 477902e to 817cccf Compare August 14, 2026 17:52

Brainy Bumble and others added 2 commits August 14, 2026 13:19

bench: add persona v0 max baselines

bac61f2

Co-authored-by: Atish Patel <atish@squareup.com> Signed-off-by: Atish Patel <atish@squareup.com>

moar benchmarks

90e3d9d

Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: Atish Patel <atishpatel2012@gmail.com>

Navigation Menu

Uh oh!

bench: tb21 harness carry + usage accounting, now synced with main (merge, no rebase) - #5145

bench: tb21 harness carry + usage accounting, now synced with main (merge, no rebase)#5145
tlongwell-block wants to merge 12 commits into
mainfrom
benchmark/harness-accounting-and-solo

tlongwell-block commented Aug 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

tlongwell-block commented Aug 7, 2026

What this branch is

The 6 bench-only commits (what main doesn't have)

The new merge commit (9cafe8f1)

Validation

Intent / merge posture

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

The new merge commit (`9cafe8f1`)