benchmarks

jcodemunch-mcp — Token Efficiency Benchmark

Result: 99.6% average token reduction · tiktoken cl100k_base · 15 task-runs · 3 repos

What this measures

How many tokens a code-retrieval tool consumes versus an agent that reads every source file before acting.

Baseline: concatenate all indexed source files and count tokens. This is the minimum cost for a "read everything first" agent — real agents typically read files multiple times, so production savings are higher.

jcodemunch workflow: search_symbols (top 5 results) + get_symbol_source × 3 hits per query. Total = search response tokens + 3 × symbol source tokens.

Tokenizer: tiktoken cl100k_base — the GPT-4 / Claude family encoding. Consistent across runs regardless of model.

Reproducing the results

pip install jcodemunch-mcp tiktoken

# Index the three canonical repos
jcodemunch index_repo expressjs/express
jcodemunch index_repo fastapi/fastapi
jcodemunch index_repo gin-gonic/gin

# Run the benchmark (prints markdown table + grand summary)
python benchmarks/harness/run_benchmark.py

# Optional: write results to file
python benchmarks/harness/run_benchmark.py --out benchmarks/results/my_run.md

Task corpus

Tasks are defined in tasks.json — 5 queries × 3 repos = 15 measurements.

ID	Query	Description
router-route-handler	router route handler	Core route registration / dispatch logic
middleware	middleware	Middleware chaining and execution
error-exception	error exception	Error handling and exception propagation
request-response	request response	Request/response object definitions
context-bind	context bind	Context creation and parameter binding

Repos: expressjs/express, fastapi/fastapi, gin-gonic/gin

Canonical results

Full per-task tables are in results.md.

Repo	Files	Baseline tokens	Avg reduction
expressjs/express	165	137,978	99.4%
fastapi/fastapi	951	699,425	99.8%
gin-gonic/gin	98	187,018	99.4%
Grand total	—	5,122,105	99.6%

99.6% average token reduction across 15 task-runs · 263.9x ratio · tiktoken cl100k_base.

To regenerate:

python benchmarks/harness/run_benchmark.py --out benchmarks/results.md

Benchmarking a different tool

The task corpus in tasks.json is tool-agnostic. To evaluate another tool:

1. Use the same 3 repos and 5 queries.
2. Use the same baseline: all indexed source files concatenated, tokenized with tiktoken cl100k_base.
3. Measure total tokens consumed by your retrieval workflow per query (tool calls + responses).
4. Report per-task rows and the grand average using the same formula: (1 - tool_tokens / baseline_tokens) * 100.

If you publish results against this corpus, open an issue or PR and we'll link them here.

Methodology notes

• The baseline is a lower bound. Agents that re-read files mid-task spend more.
• The jcodemunch workflow counts search_symbols + get_symbol_source responses only — it does not count system prompt or tool description tokens, which are identical for both approaches.
• Token counts are from serialized JSON responses, not raw source, so they include field names and structure overhead. This slightly understates the reduction.

Related harnesses (v1.74.0+)

• benchmarks/replay/ — replayable retrieval-quality benchmark. Fixtures pin (query, expected_top_k_ids) tuples; the harness runs each query through search_symbols and reports nDCG@k, MRR@k, and Recall@k. Wired into CI as the Replay workflow (.github/workflows/replay.yml): every push to main and every PR indexes the repo and runs run_replay.py --fixture … --repo <indexed-id> --baseline-file results/self_v1_75_0-golden.json --gate 0.02, which exits non-zero if any aggregate metric drops more than 2% (relative) below the committed golden baseline. This is the regression gate that lets ranking-affecting changes (fusion weights, BM25 normalization, parser extraction) land with a proof they did not degrade retrieval. The self_v1_75_0 fixture is locked at 1.0 across all metrics; update self_v1_75_0-golden.json (via --write-result) only on a deliberate, reviewed ranking change. Pass --repo to override the fixture's machine-specific repo id; --baseline X.Y.Z still gates against a version-pinned results/{fixture}-v{X.Y.Z}.json snapshot.
• benchmarks/token_baselines/ — per-release token-savings + latency snapshots. capture_token_baseline.py reads the live session's get_session_stats + latency_stats and writes benchmarks/token_baselines/v{VERSION}.json. The analyze_perf tool consumes these via compare_release="X.Y.Z".