A benchmark that answers one question: can AI coding agents build correct, private-by-construction Aztec applications from public knowledge?
Instead of asking "are our docs good?", run the bench and ask "can Claude/Codex/Gemini ship a correct Aztec app?". Every failure is tagged to a concrete failure mode (pre-v5 syntax, privacy anti-pattern, unclear compiler error, missing doc), so the output is a ranked list of what to fix, not a vibe.
The bench is not a model leaderboard; it measures Aztec's legibility to human-agent teams, and it is meant to run as a loop:
real developer / agent failure
-> convert into a benchmark task (tasks/TASK_INTAKE.md)
-> run the task against agents (aabench run)
-> identify the failure mode (AP/BM codes in the report)
-> patch docs, examples, context, APIs
-> re-run and measure the delta (aabench run with a new run-id, compare reports)
Every failure is classified by fix owner (docs, llms.txt, error messages, examples, protocol), which separates DevRel work from protocol work. Tasks are sliced by capability: build (correct private apps from scratch), fix (migrate/debug), reason (privacy analysis and proofs of non-exposure), and use-context (does Aztec-provided context measurably help; varied via run modes).
Docs: docs/USAGE.md (operator guide: running evaluations, reading results, troubleshooting), PLAN.md (design), tasks/AUTHORING.md (building tasks), tasks/TASK_INTAKE.md (support failure -> task pipeline).
- Aztec CLI 5.0.0-rc.2 (
aztec compile,aztec testmust work):VERSION=5.0.0-rc.2 bash -i <(curl -s https://install.aztec.network) - Node 24, yarn 1.22
- Agent CLIs you want to evaluate:
claude,codex,gemini
No local network or docker needed: all grading is compile + TXE.
yarn install
# Prove the bench catches what it claims (solutions pass, planted-bad submissions get caught)
yarn aabench validate
# Evaluate agents
yarn aabench run --agents claude-sonnet,claude-opus --tasks T04-private-public-counter --run-id smoke
yarn aabench run --agents claude-sonnet,codex # full suite
# Re-grade without re-running agents (e.g. after grader fixes)
yarn aabench grade --run runs/<run-id>
# Regenerate the report
yarn aabench report --run runs/<run-id>Reports land in reports/<run-id>.md: a per-agent capability matrix (compile / functional / privacy / quality per task) and a failure-mode frequency table with evidence.
| ID | Task | Category |
|---|---|---|
| T01-private-token | Build a private token | build |
| T02-selective-disclosure | Add selective disclosure to a payment app | extend |
| T03-private-voting | Create private voting | build |
| T04-private-public-counter | Private/public counter with correct execution boundaries | build |
| T05-agent-permissions | Add account permissioning for an AI agent | build/extend |
| T06-migrate-old-example | Migrate an old aztec.nr example to the current version | migrate |
| T07-fix-compiler-errors | Fix common compiler errors | debug |
| T08-find-privacy-leak | Identify privacy leaks in a sample app | audit |
| T09-prove-non-exposure | Write tests proving a value is not publicly exposed | test-writing |
| T10-explain-information-flow | Explain what information each participant learns | comprehension |
Every task ships: starting repo, agent prompt, reference solution, compile/test requirements, machine-checked privacy invariants, known failure modes, and a grading rubric. yarn aabench validate enforces that reference solutions pass all graders and that calibration (known-bad) submissions are caught by exactly the checks that claim to catch them.
- Compile (
aztec compile) on the submission. Static failure classifiers run on source even when compile fails, so "failed with pre-v5 syntax" is signal, not silence. - Functional: held-out TXE tests in a separate crate that deploys the submission via
@crate/Contract; the agent never sees them. - Privacy: artifact checks (function kinds, ABI, storage names from the compiled artifact) + source checks (anti-pattern regexes) + privacy-focused TXE tests. For T09, mutation testing: the agent's tests must kill leak-injected mutants.
- Quality: LLM judge against the task rubric (low weight), or against a must-find answer key for audit/comprehension tasks.
Failure modes use AP codes from aztec-packages/aztec-anti-patterns.md plus bench-specific BM codes (BM-SYNTAX-V4, BM-COMPILE-FAIL, BM-WEAK-TESTS, ...).