txn2 · cjimti · Jul 15, 2026 · Jul 15, 2026
@@ -1077,7 +1077,7 @@ An opt-in pprof listener supports the harness: setting the `PPROF_ADDR` environm
 
 ## Agent-effectiveness benchmark
 
-A second measurement harness lives in `bench/` (also a separate Go module): the agent-effectiveness benchmark (issue #930). Where the load harness answers "how much" (throughput, latency, memory), the benchmark answers "how well": it holds the model, prompt scaffold, seed data, and task set constant and ablates only the platform configuration (arms as config profiles: `a0` raw toolkit tools vs `a2` the full semantic-first platform), measuring answer accuracy, pass^k reliability, and tool-call efficiency. Ground truth is generated from a fixed-seed dataset (Trino tables, DataHub metadata, knowledge pages) with the disambiguating facts for its knowledge-trap tasks (units-in-cents, gross-vs-net revenue policy) living only in the metadata and knowledge layers. Efficiency metrics are read back from the platform's own admin audit API (`audit.delivery: sync`), and a run fails loudly when a session's audit rows are incomplete. A deterministic scripted adapter validates the whole pipeline against a live stack with no model API key (`make bench-up`, `make bench-smoke`, `make bench-down`); real runs use a pinned Anthropic model. Like load testing, it is deliberately NOT part of `make verify`. See `bench/README.md`. The S5 suite adds multi-episode memory-insight-knowledge lifecycle protocols (`make bench-lifecycle`, on the `a3` arm): each protocol teaches a novel definition in one session and, across fresh sessions, measures personal recall, cross-identity transfer after the reviewer promotes the insight via `apply_knowledge` (to an entity description or a knowledge page), recall-first supersede when the fact is later corrected, and abstention on facts never taught — every lifecycle state transition verified through the admin insights and changesets APIs, not inferred from transcripts. Published results live in `docs/reference/benchmarks.md` with raw run JSON under `bench/results/`. The page covers both platform halves. Semantic layer (S1-S3, four arms a0/a1/a2/a3, claude-sonnet-5, k=3, 261 graded attempts per arm, zero harness failures): S3 knowledge-trap accuracy climbs 42.7% (raw tools) -> 57.3% (enrichment) -> 98.7% (full platform), a +56-point gain for the platform over bare tools (95% CI +44 to +67), while S1 discovery and S2 analytical accuracy are near ceiling for every arm; median knowledge-trap tool calls drop 16 -> 10. Arm a3 (platform + memory lifecycle) ties a2 (platform) on S1-S3 because these tasks are single-session and never exercise the memory layer, NOT because the memory layer lacks value; the memory layer's value is cross-session and is measured separately by S5. Memory layer (S5 lifecycle protocols, a3, k=3, 15 multi-session protocols): a capture/personal-recall/unprompted-surface/cross-identity-transfer/update-correctness/duplicate/abstention scorecard plus full-lifecycle pass^3, measuring whether the teach-once-answer-forever mechanism works reliably across fresh sessions rather than an accuracy delta over a (trivially-zero) no-memory baseline, with every state transition verified through the admin insights and changesets APIs. The S5 lifecycle is reported as three runs: a shared-store run (45 attempts, one accumulating knowledge store, k-repeats coupled) and two isolated replicates (each three independent k=1 passes with a full clean platform reset between passes, merged to k=3). The two isolated replicates, identical in configuration, disagree materially (duplicate rate 0% vs 42.9%, pass^3 26.7% vs 20.0%, capture/recall a few points each), which shows most metric-by-metric differences at this scale (15 protocols, small supersede denominators) are run-to-run sampling noise rather than the shared-store confound; the apparent isolation "lift" seen in the first replicate is not reproduced by the second. What is stable across all three runs and is read as the firm result: unprompted surface 100%, update correctness 100%, abstention 96-100%, and cross-identity transfer stuck at ~42-47% (the clearest reproducible reliability ceiling). All three runs recorded zero harness failures and zero episode errors, so transient Claude API outages (which surface as excluded harness failures, never mis-graded answers) did not affect the data. Spend is published as token counts (not dollars), computed from committed per-attempt records: ~158.6M tokens total across all runs (25,182 fresh input, 2,664,630 output, 147,183,943 cache-read, 8,676,898 cache-write), of which cache reads are ~93% -- apply current claude-sonnet-5 per-token pricing to these counts for a cost. S1-S3 and the shared-store S5 run used development build v1.102.0-9-gadfb9d90-dirty; the isolated runs used v1.102.0-10-g32d61254-dirty (platform code byte-identical between the two build commits, verified by diff; only benchmark-harness commits differ); each is pinned in its manifest. A `benchrun -baseline <results.json>` regression gate exits nonzero when a fresh run falls below a committed baseline beyond per-suite thresholds; an earlier phase-1 pilot and phase-3 S5 pilot remain in the result history.
+A second measurement harness lives in `bench/` (also a separate Go module): the agent-effectiveness benchmark (issue #930). Where the load harness answers "how much" (throughput, latency, memory), the benchmark answers "how well": it holds the model, prompt scaffold, seed data, and task set constant and ablates only the platform configuration (arms as config profiles: `a0` raw toolkit tools vs `a2` the full semantic-first platform), measuring answer accuracy, pass^k reliability, and tool-call efficiency. Ground truth is generated from a fixed-seed dataset (Trino tables, DataHub metadata, knowledge pages) with the disambiguating facts for its knowledge-trap tasks (units-in-cents, gross-vs-net revenue policy) living only in the metadata and knowledge layers. Efficiency metrics are read back from the platform's own admin audit API (`audit.delivery: sync`), and a run fails loudly when a session's audit rows are incomplete. A deterministic scripted adapter validates the whole pipeline against a live stack with no model API key (`make bench-up`, `make bench-smoke`, `make bench-down`); real runs use a pinned Anthropic model. Like load testing, it is deliberately NOT part of `make verify`. See `bench/README.md`. The S5 suite adds multi-episode memory-insight-knowledge lifecycle protocols (`make bench-lifecycle`, on the `a3` arm): each protocol teaches a novel definition in one session and, across fresh sessions, measures personal recall, cross-identity transfer after the reviewer promotes the insight via `apply_knowledge` (to an entity description or a knowledge page), recall-first supersede when the fact is later corrected, and abstention on facts never taught — every lifecycle state transition verified through the admin insights and changesets APIs, not inferred from transcripts. Published results live in `docs/reference/benchmarks.md` with raw run JSON under `bench/results/`. The page measures one knowledge system from two coupled angles: a surfacing half (cross-enrichment and search deliver curated knowledge from its sinks) and a capture-and-propagation half (memory -> insight -> apply_knowledge promotes facts into those sinks -- DataHub entity/column descriptions or knowledge pages); the lifecycle populates what enrichment and search deliver, so they are two ends of one pipe, not independent features. Surfacing half, validated by S1-S3 (four arms a0/a1/a2/a3, claude-sonnet-5, k=3, 261 graded attempts per arm, zero harness failures): S3 knowledge-trap accuracy climbs 42.7% (raw tools) -> 57.3% (enrichment) -> 98.7% (full platform), a +56-point gain for the platform over bare tools (95% CI +44 to +67), while S1 discovery and S2 analytical accuracy are near ceiling for every arm; median knowledge-trap tool calls drop 16 -> 10. Arm a3 (platform + memory lifecycle) ties a2 (platform) on S1-S3 because these single-session tasks exercise only the surfacing half, NOT because capture lacks value; capture-and-propagation is what produces the sink contents S1-S3 shows are effective once present, and it is measured separately by S5. Capture-and-propagation half (S5 lifecycle protocols, a3, k=3, 15 multi-session protocols): a capture/personal-recall/unprompted-surface/cross-identity-transfer/update-correctness/duplicate/abstention scorecard plus full-lifecycle pass^3, measuring whether the teach-once-answer-forever mechanism works reliably across fresh sessions rather than an accuracy delta over a (trivially-zero) no-memory baseline, with every state transition verified through the admin insights and changesets APIs. The S5 lifecycle is reported as three runs: a shared-store run (45 attempts, one accumulating knowledge store, k-repeats coupled) and two isolated replicates (each three independent k=1 passes with a full clean platform reset between passes, merged to k=3). The two isolated replicates, identical in configuration, disagree materially (duplicate rate 0% vs 42.9%, pass^3 26.7% vs 20.0%, capture/recall a few points each), which shows most metric-by-metric differences at this scale (15 protocols, small supersede denominators) are run-to-run sampling noise rather than the shared-store confound; the apparent isolation "lift" seen in the first replicate is not reproduced by the second. What is stable across all three runs and is read as the firm result: unprompted surface 100%, update correctness 100%, abstention 96-100%, and cross-identity transfer stuck at ~42-47% (the clearest reproducible reliability ceiling). Because S1-S3 already validates surfacing, the transfer gap is a capture/propagation limit upstream (the apply_knowledge promotion landing in an aspect enrichment reads -- mcp-datahub GetEntity fully populates only datasets/dashboards -- or the second identity surfacing the fact but not using it), not a delivery failure. All three runs recorded zero harness failures and zero episode errors, so transient Claude API outages (which surface as excluded harness failures, never mis-graded answers) did not affect the data. Spend is published as token counts (not dollars), computed from committed per-attempt records: ~158.6M tokens total across all runs (25,182 fresh input, 2,664,630 output, 147,183,943 cache-read, 8,676,898 cache-write), of which cache reads are ~93% -- apply current claude-sonnet-5 per-token pricing to these counts for a cost. S1-S3 and the shared-store S5 run used development build v1.102.0-9-gadfb9d90-dirty; the isolated runs used v1.102.0-10-g32d61254-dirty (platform code byte-identical between the two build commits, verified by diff; only benchmark-harness commits differ); each is pinned in its manifest. A `benchrun -baseline <results.json>` regression gate exits nonzero when a fresh run falls below a committed baseline beyond per-suite thresholds; an earlier phase-1 pilot and phase-3 S5 pilot remain in the result history.
 
 Environment-only configuration: `OTEL_TRACES_ENABLED` (default false), `OTEL_EXPORTER_OTLP_ENDPOINT` (OTLP/gRPC `host:port`, default `localhost:4317`), `OTEL_EXPORTER_OTLP_INSECURE` (default true — the common in-cluster topology; set false for a TLS remote collector), `OTEL_TRACES_SAMPLER_ARG` (head-based sampling ratio in [0,1] applied to root spans, default 0.1), and `OTEL_SERVICE_NAME` (service.name resource attribute, default `mcp-data-platform`). The OTLP exporter connects lazily: an unreachable or unconfigured collector never blocks or fails startup; spans are batched and dropped if undeliverable.
 

@@ -91,7 +91,7 @@ mcp-data-platform is the orchestration layer for the txn2 MCP ecosystem. DataHub
 - [Configuration Reference](https://mcp-data-platform.txn2.com/reference/configuration/): Redirect page; the full YAML schema lives at [Configuration](https://mcp-data-platform.txn2.com/server/configuration/)
 - [Providers](https://mcp-data-platform.txn2.com/reference/providers/): Semantic, query, and storage provider interfaces
 - [Middleware](https://mcp-data-platform.txn2.com/reference/middleware/): Request processing chain: tool visibility, description overrides, the search-first gate, icon enrichment, client logging, progress notifications
-- [Benchmarks](https://mcp-data-platform.txn2.com/reference/benchmarks/): Agent-effectiveness benchmark results (bench/) covering both platform halves: the semantic layer via the S1-S3 four-arm ablation (arm-ablated accuracy with bootstrap CIs, pass^k, the S3 knowledge-trap breakdown, and audit-derived tool-call efficiency) and the memory layer via the S5 memory-insight-knowledge lifecycle protocols (capture, personal recall, cross-identity transfer, supersede, and abstention, verified through the admin insights and changesets APIs); a2 ties a3 on S1-S3 because those tasks are single-session, so the memory layer is measured separately by S5; costs computed from committed per-attempt cache tokens, with a benchrun -baseline regression gate
+- [Benchmarks](https://mcp-data-platform.txn2.com/reference/benchmarks/): Agent-effectiveness benchmark results (bench/) measuring one knowledge system from two angles -- the surfacing half (cross-enrichment + search delivering curated knowledge from its sinks) and the capture-and-propagation half (memory -> insight -> apply_knowledge promoting facts into DataHub descriptions or knowledge pages); the lifecycle populates the sinks the enrichment/search layer delivers from, so they are two ends of one pipe, not independent features. S1-S3 (four-arm ablation) validates surfacing: accuracy with bootstrap CIs, pass^k, the S3 knowledge-trap breakdown showing each channel carrying its own facts (descriptions via enrichment, pages via search), and audit-derived tool-call efficiency. S5 (memory-insight-knowledge lifecycle protocols) tests capture-and-propagation: capture, personal recall, cross-identity transfer, supersede, abstention, verified through the admin insights and changesets APIs. a2 ties a3 on S1-S3 because those tasks exercise only surfacing; the transfer gap (~45%) is a capture/propagation limit upstream of the already-validated surfacing. Token spend published from committed per-attempt cache tokens; benchrun -baseline regression gate
 - [Tuning and Scaling](https://mcp-data-platform.txn2.com/reference/tuning-and-scaling/): Resource limits, Go runtime tuning, horizontal scaling characteristics, connection pool sizing, autoscaling guidance, and measured single-replica throughput/latency limits from the load harness (test/load); the sibling agent-effectiveness benchmark harness (bench/) measures arm-ablated accuracy and efficiency with audit-derived metrics
 
 ## Key Capabilities