Skip to content

Commit 70f7053

Browse files
committed
add comp figure
1 parent 6be1442 commit 70f7053

3 files changed

Lines changed: 10 additions & 0 deletions

File tree

docs/assets/images/comp-3a-1.png

581 KB
Loading

docs/assets/images/comp-3a-2.png

858 KB
Loading

docs/index.md

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,16 @@ The table below compares traditional agent benchmarking with our proposed AAA pa
3737
| **Realism** | Prone to test-production mismatch; mainly used for reference | Directly reflects production-level performance |
3838
| **Multi-agent assessment support** | Difficult, requiring bespoke integrations | Natively supported through standardized interfaces and platform-level coordination |
3939

40+
## Comparison with Traditional Benchmarks
41+
42+
![Agent Assessment Paradigms](./assets/images/comp-3a-1.png)
43+
44+
Traditional benchmarks often focus on evaluating LLMs within fixed harnesses. The evaluation target is usually either the LLM itself, or a few preset agent harnesses. In contrast, AAA allows any agent that conforms to the A2A protocol and MCP to be evaluated, regardless of its internal architecture or control flow.
45+
46+
![Traditional vs AAA](./assets/images/comp-3a-2.png)
47+
48+
When testing a new agent in a benchmark that comes with its own built-in harness, developers often face significant integration work because the harness is tightly coupled with the rest of the benchmark logic. In contrast, AAA removes this coupling: as long as an agent implements the A2A protocol, it can be evaluated directly—without any custom integration or benchmark-specific adaptation.
49+
4050
## Practicing AAA: The AgentBeats Platform
4151

4252
Despite growing recognition of the importance of agent evaluation, creating effective and impactful assessments remains challenging for both researchers and practitioners. Even with a clear and innovative benchmark concept, two major obstacles often hinder progress:

0 commit comments

Comments
 (0)