Agent Eval Lab

Agent Eval Lab is a Claude Code skill and TypeScript CLI for evaluating prompts, agents, transcripts, tool behavior, and model regressions. It gives agent work a repeatable reliability loop: define scenarios, run real models, grade transcripts, store results, and inspect failures in a local dashboard.

It supports deterministic graders for fast local checks and includes schema fields for LLM-judged rubrics and pairwise preference workflows.

What It Does

• Loads YAML or JSON eval suites and scenarios.
• Runs scenarios against real OpenAI or Anthropic models.
• Grades transcripts with deterministic graders.
• Calculates pass rate, mean score, pass@k, pass^k, and per-grader breakdowns.
• Serializes run results as JSON in results/.
• Builds a local static dashboard for reviewing runs and failures.
• Includes both a Claude skill (SKILL.md) and a CLI (ael).

How the Skill Works

Claude uses SKILL.md to operate Agent Eval Lab as a reliability workflow:

1. Turn the behavior under test into one or more scenarios.
2. Group scenarios into a suite.
3. Validate the suite before spending provider tokens.
4. Run the suite against a model reference like openai/gpt-5.4-mini or anthropic/claude-sonnet-4-5.
5. Review the report or build the dashboard.
6. Use failures to revise prompts, agent instructions, tools, or model choices.

The same suite files can be used by Claude, a developer in a terminal, or CI. That keeps behavior checks versioned with the project.

Repository Layout

.
├── SKILL.md                 # Claude skill instructions
├── Workflows/               # Eval workflow notes
├── scenarios/               # Reusable scenario definitions
├── suites/                  # Suite files that reference scenarios
├── graders/                 # Grader interface notes
├── src/                     # TypeScript CLI and library code
├── tests/                   # Bun test suite
├── results/                 # Runtime result JSON, kept empty in git
├── dashboard/               # Static dashboard docs and generated output path
└── package.json

Installation

bun install
bun run build

For local development, you can run the TypeScript source directly with Bun. After bun run build, the compiled CLI is available at dist/cli.js and the package binary is ael.

Configuration

Create a local .env from .env.example and set the provider key you want to use.

cp .env.example .env

Provider keys:

• OPENAI_API_KEY for OpenAI Responses API runs.
• ANTHROPIC_API_KEY for Anthropic Messages API runs.

Optional defaults:

• AEL_DEFAULT_MODEL
• AEL_JUDGE_MODEL

CLI Usage

Validate a suite:

bun src/cli.ts validate --suite suites/prompt-regression.yaml

Inspect a scenario:

bun src/cli.ts scenario --file scenarios/hallucinated-path.yaml

Run a suite:

bun src/cli.ts run \
  --suite suites/prompt-regression.yaml \
  --model openai/gpt-5.4-mini \
  --trials 3

Compare models:

bun src/cli.ts compare \
  --suite suites/research-answer-quality.yaml \
  --models openai/gpt-5.4-mini,anthropic/claude-sonnet-4-5 \
  --trials 2

Print a saved run report:

bun src/cli.ts report --run <run-id>

Build the local dashboard:

bun src/cli.ts dashboard

The dashboard is written to dashboard/generated/index.html.

Suite Format

Suites can inline scenarios or reference scenario files:

id: prompt-regression
name: Prompt Regression
description: Starter suite for output contract and instruction drift.
scenarioFiles:
  - ../scenarios/prompt-regression-basic.yaml

Scenarios define the model input and graders:

id: concise-json
name: Concise JSON Output
input: Return a JSON object with status "ok".
graders:
  - id: schema
    type: json_schema
    schema:
      type: object
      required: [status]
      properties:
        status:
          const: ok

Included Starter Suites

• prompt-regression: output contract and instruction drift checks.
• coding-agent-discipline: coding-agent process and honesty checks.
• hallucinated-path-detection: local path and uncertainty discipline.
• research-answer-quality: time-sensitive and source-aware answer checks.
• tool-call-ordering: transcript tool sequence validation.

Graders

Supported grader interfaces:

• string_match: passes when output contains an expected string.
• regex_match: passes when output matches a regular expression.
• json_schema: validates JSON output with AJV.
• tool_call_sequence: checks exact ordered tool-call names in a transcript.
• static_artifact_check: checks required local artifacts and optional content.
• llm_rubric: schema support for judged qualitative scoring.
• pairwise_preference: schema support for A/B model preference judging.

The deterministic graders run locally and are covered by tests. LLM-backed grader interfaces are included so suites can represent rubric and preference workflows.

Metrics

Every run stores:

• passRate: share of trials that passed all graders.
• meanScore: average trial score.
• passAtK: probability at least one of k trials passes.
• passPowerK: probability all k trials pass.
• perGrader: pass rate, mean score, and count for each grader.

Results and Dashboard

Run results are JSON files in results/. The dashboard reads those result files and renders:

• Results overview.
• Run detail summaries.
• Trial transcript excerpts.
• Model comparison tables.
• Failure categories by scenario and grader.

Testing

bun run typecheck
bun run test
bun run build

The test suite covers suite parsing, deterministic graders, pass@k and pass^k math, result serialization, dashboard generation, and provider smoke tests. Smoke tests are skipped automatically unless OPENAI_API_KEY or ANTHROPIC_API_KEY is present.

License

MIT