Add token-efficiency comparison script (native vs trajectory vs Harbor ATIF) - #23
Add token-efficiency comparison script (native vs trajectory vs Harbor ATIF)#23sarahwooders wants to merge 4 commits into
Conversation
Takes a Claude Code session or Codex rollout path and reports token counts (via the Anthropic count-tokens API) for the native file, the normalized trajectory JSONL, and a content-matched Harbor ATIF projection, with reduction factors vs native. Source is auto-detected; --untruncated adds a row with tool-result truncation disabled.
Harbor has no standalone conversion CLI, so scripts/harbor_atif_convert.py drives its exact ClaudeCode/Codex _convert_events_to_trajectory code from a harbor checkout with the harness-only imports stubbed (only pydantic needed, run via uv). token-efficiency.ts now reports Harbor-produced ATIF (minified and as-persisted) instead of the content-matched projection, auto-cloning harbor to ~/.cache/trajectory/harbor-repo on first use (HARBOR_REPO overrides).
Outputs native/trajectory/atif files under token-efficiency-out/<session-stem>/ (override with --out-dir) and prints the path per row; directory gitignored.
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
cpacker
left a comment
cpacker
left a comment
There was a problem hiding this comment.
Please make the reference converter version reproducible. ensureHarborCheckout() shallow-clones whatever the upstream default branch contains on first use, then indefinitely reuses whatever checkout happens to be cached. There is no pinned revision, refresh behavior, or recorded SHA, so two runs labeled as the same comparison can use different converter implementations.
Please pin or explicitly accept a reference revision and print/store that SHA with the results. Recording the trajectory revision and counting model alongside it would make the generated comparison auditable.
👾 Generated with Letta Code
2 participants
Summary
Adds
scripts/token-efficiency/— a reproducible way to measure how many tokens a real agent session costs in its native format vs this repo's normalized trajectory format vs Harbor's ATIF (RFC 0001).--sourceoverrides)claude-opus-4-8by default, chunked at 500K chars)token-efficiency-out/<session-stem>/and prints per-row paths--untruncatedadds a trajectory row with tool-result truncation disabledATIF via Harbor's own converters
Harbor has no standalone conversion CLI, so
harbor_atif_convert.pydrives the exact upstreamClaudeCode/Codex._convert_events_to_trajectorycode from a harbor checkout (auto-cloned to~/.cache/trajectory/harbor-repo,HARBOR_REPOoverrides) with harness-only imports stubbed — onlyuv+pydanticneeded. Output validates clean against harbor's own ATIF validator (harbor-atif2otel) and the RFC's normative rules.Example (small codex rollout)
On large real sessions (~18.8MB Claude Code / ~18.5MB Codex), trajectory lands at 7.0–8.5× vs native while Harbor ATIF lands at 1.4–2.4× — ATIF intentionally keeps untruncated tool results, structured result payloads, and per-step token metrics, so the gap reflects content policy, not format syntax (details in the folder README).
Test plan
bun run typecheckclean