Plugin authoring costs ~2x the tokens it needs to: measurements and proposed fixes

[t1mdurden]

Summary

Writing one plugin against the bundled bb-plugin-authoring skill cost 43.5M input tokens over 197 assistant turns and 122 tool calls. I kept the transcript and measured where it went. Roughly half is recoverable, and almost none of it is model behaviour — it is the plugin toolchain making an agent go and look things up that it could have been handed.

Filing this as a task because the concrete work below is a set of refactors and one doc restructure, and CONTRIBUTING asks for an issue before a PR on anything that is not a plain bug. The three plain bugs I found on the way are filed separately.

What was measured

Session transcript, one plugin (a kanban board: backend + frontend + CLI + agent tools + tests), taken to a working installed state:

input tokens	43,491,913
output tokens	199,882
assistant turns	197
tool calls	122
context at turn 1 → last turn	48,396 → 304,159

Cost is Σ(context at each turn), so a turn near the end costs ~300k, not "one request". That makes turn count the dominant lever, and it makes anything permanently resident in context the second one.

Where it went

1. The skill is 150,648 bytes (~37k tokens) and resident for every turn — ~7.3M tokens, 17% of the session.

bb-plugin-authoring/SKILL.md is 2,777 lines, the largest builtin skill by 2.4× (bb-cli is 63,216 bytes). It has no references/ directory — while bb-cli already uses exactly that pattern (references/app-settings.md, references/theming.md). I consulted maybe ten sections of it and paid for all of it, 197 times.

2. 23 of 122 tool calls were grepping bb-plugin-sdk.d.ts for signatures — before a single line of the plugin was written.

The skill is prose. It says "read the bundled declarations" for exact signatures, so that is what an agent does: grep -n "interface ThreadsArea", sed -n '15700,15760p', twelve more like it, over a 700KB file, two of them dead ends. Sample from the transcript:

grep -n "PluginThreadEventMap\|interface PluginThreadEvents\|thread.idle" $D
sed -n '15700,15760p' $D
grep -n "threadResponseSchema" $D
sed -n '10957,11010p' $D
grep -n "interface ThreadsArea" $D; awk '/^interface ThreadsArea/,/^}/' $D
for t in ThreadListArgs ThreadListResult ThreadGetArgs …; do … done

This is also where the doc bug bit: the skill's threads.list example is wrong, and the only way to find that out is the file I was already grepping.

3. Four calls to find valid icon names. bb.branding.icon and every slot's icon take "a BB icon name", and the only way to learn the vocabulary is to grep the vendored components/ui/icon-registry.ts and icon.tsx in the scaffold.

4. Four calls to make one plugin RPC call. There is no bb plugin rpc. Reaching the RPC route the frontend uses meant finding the server port in ~/.bb/bb-app-runtime.json, then curl with a hand-written Origin header, after two dead-end calls looking for a CLI verb that does not exist.

5. Eight calls of pure environment friction, including the three separately-filed bugs and bb thread spawn requiring --project when invoked from inside a thread that already has one.

Counterfactual

Removing the 54 turns classified as friction-or-lookup — and their contribution to every later turn's context — gives 28.9M instead of 43.5M, −33%. Shrinking the resident skill from ~37k to a ~3k router takes off roughly another 15%. Together: about half.

For honesty about what is not bb's fault: ~15 further calls went to learning an unrelated browser tool's syntax, and three rebuild cycles were my own UI polish. Those are excluded from the 54.

Proposed work

Ordered by measured impact per unit of change.

A. Split the authoring skill into a router plus references/. Keep a short entry document — the factory, the surface list, the gotchas — and move the SDK area map, frontend slots, testing, provider bridges and distribution into references/*.md loaded on demand. bb-cli already does this, so the loader supports it. Acceptance: the entry document is under ~400 lines; every section currently in SKILL.md is reachable from it; plugin-authoring-docs.test.ts passes against the union of the router and its references.

B. bb plugin api <symbol> — print the declaration of a type or method from the bundled .d.ts the running bb ships. bb plugin api ThreadListArgs, bb plugin api threads.list. Would have collapsed 23 calls to about 5, and would make "read the bundled declarations" an instruction an agent can follow in one step instead of six. Acceptance: exact-name lookup and a substring search mode; output is the declaration plus its doc comment; works with no server running, like bb plugin types.

C. bb plugin rpc <id> <method> [json] — call a loaded plugin's RPC method from the CLI. Four calls to one, and it removes the need to know the port or the Origin rule. Acceptance: resolves the server the way other bb commands do; --json; the same error envelope the wire returns.

D. bb plugin icons — list the icon names a manifest and the slots accept. Acceptance: names only, --json for the machine-readable form.

E. Ship a working test setup in bb plugin new. After the cron-parser bug is fixed the harness imports, but the scaffold still has no vitest, no scripts, and no example test — so "write a test" starts with three decisions that have one right answer each. Acceptance: npm test passes in a fresh scaffold and runs one real createFakePluginHost test against the generated server.ts.

F. bb thread spawn defaults --project to the invoking thread's project the way bb kanban-style plugin commands get ctx.projectId. Acceptance: bb thread spawn --prompt … inside a thread succeeds; outside one, the existing error stands.

A and B are the two that matter; C, D, E and F are each an hour. Happy to take any subset — say which, and whether A should land as one PR or as a router-first change with the references moved incrementally.

Base commit: ad79bbb5ec909524f8f281e62d860c588a86f332.

Related bugs found in the same session, filed separately: the scaffold's missing cron-parser, the wrong threads.list example, and bb plugin new swallowing npm's failure reason.

AGENT GENERATED