-
Notifications
You must be signed in to change notification settings - Fork 0
Results
These come straight from the bundled examples
(examples/*.py)
and are fully deterministic:
| Demo | Base LLM | ACE | Δ |
|---|---|---|---|
| Quickstart (offline → held-out eval) | 44.4% | 83.3% | +38.9 pts |
| Context-collapse benchmark (online) | 41.7% | 88.3% | +46.6 pts |
| Offline warmup + online | 34.5% | 96.6% | +62.1 pts |
In the context-collapse demo, the monolithic-rewrite baseline collapses its
context 7× and stalls at 60.0%, while ACE never collapses. Adaptation token
ingestion for ACE is −94.9% vs. full re-ingestion (deltas are tiny).
Generate the visual report with ace demo --html report.html.
| Benchmark | Baseline | + ACE |
|---|---|---|
| AppWorld (agent, avg) | 42.4% (ReAct) | 59.5% (+17.1) |
| FiNER (financial NER) | 70.7% | 78.3% |
| Formula (financial reasoning) | 67.5% | 85.5% |
| Adaptation latency (offline AppWorld) | — | −86.9% |
| Token cost (online FiNER) | — | −83.6% |
On the AppWorld leaderboard, ReAct+ACE with an open-source model matches the top-ranked production GPT-4.1 agent and surpasses it on the harder test-challenge split. Numbers above are from the paper; this repo reproduces the mechanism and its qualitative behavior offline.
The paper reports 91.8% of input tokens served from KV cache during evaluation, cutting billed input cost ~82.6% vs. counting raw tokens — so a longer playbook does not mean linearly higher serving cost.
See How It Works for why incremental deltas keep adaptation cheap.
ACE — Agentic Context Engineering · MIT · Independent open-source reproduction of ICLR 2026 arXiv:2510.04618. All credit for the method belongs to the original authors.