Gemini CLI Behavioral Evaluation & DevEx Experiments

Note: The prototype implementations are developed across dedicated feature branches. Please explore the repository branches for detailed implementations and related commits.

Explorations and prototype implementations focused on improving the behavioral evaluations, debugging infrastructure and developer experience for agentic CLI systems.

---

Overview

This repository documents a collection of my experiments and prototypes around behavioral evaluation systems, agent tooling and developer infrastructure to improve the Gemini CLI.

The prototypes in this repo explore approaches for:

• Behavioral eval infrastructure
• LLM-as-a-Judge systems
• Tool routing and boundary testing
• Iterative self-correction loops
• Dynamic workspace simulation
• Telemetry driven debugging
• Trace logging and replay systems
• Eval generation pipelines
• Contributor focused DevEx tooling

---

Key Areas Explored

1. Dynamic Workspace Scaffolding

One major issue with traditional eval systems is that they operate on static and unrealistic workspaces.

To address this, I designed a config-driven workspace scaffolder capable of generating realistic multi-file debugging environments.

Architecture

The scaffolding system follows a 3-layer structure:

Core Scenario Layer

Contains the actual problem the agent must solve:

• buggy logic
• incomplete implementations
• missing features
• broken tests

Context Layer

Adds realistic software structure:

• service layers
• imports across modules
• config files (package.json, tsconfig.json)
• dependency chains

Noise Layer

Introduces irrelevant files to evaluate:

• signal vs noise filtering
• retrieval efficiency
• context resilience

Features

• Config-driven workspace generation
• Multi-file dependency simulation
• Iterative debugging loops
• Integrated Vitest execution
• Mock mode vs real execution mode
• Tool usage tracing
• Failure convergence detection

This enabled evaluation scenarios where agents had to:

1. inspect failing tests
2. diagnose issues
3. edit files
4. rerun test suites
5. iteratively converge on a fix

---

2. Iterative Self-Correction Evals

Most existing eval systems only test “happy path” behaviors.

I explored a more realistic evaluation loop where the agent must:

• detect failures
• reason about the cause
• modify code
• retry execution
• validate correctness

Focus Areas

• Multi-turn repair loops
• Recovery-oriented evaluation
• Convergence detection
• Infinite loop prevention
• Tool failure handling
• Negative behavioral testing

---

3. LLM-as-a-Judge Evaluation Framework

Traditional regex-based assertions become brittle when evaluating things like summaries, plans, explanations, commit messages, etc.

To address this, I explored a semantic evaluation framework using an LLM as a grading layer.

Judge Architecture

The judge model evaluates outputs against structured rubrics.

Reliability Strategies

To reduce judge flakiness:

• Chain-of-thought grading
• Structured scoring schemas
• Majority voting
• Hybrid deterministic + semantic validation
• Model tiering
• Fallback assertion systems

Hybrid Validation Flow

Compile Check
    ↓
Lint Validation
    ↓
Deterministic Assertions
    ↓
Semantic Judge Evaluation

This allows semantic flexibility while still maintaining deterministic safeguards.

---

4. Tool Selection Boundary Evals

The CLI exposes multiple overlapping tools like web search, web fetch, grep, glob, edit, write-file etc.

A major challenge is not whether tools work but whether the agent chooses the correct tool.

I explored a dedicated boundary eval suite focused on:

• routing correctness
• tool misuse detection
• cost-aware tool selection
• context efficiency

Example Scenarios

Web Search vs Web Fetch

Prompt:
"What is the latest React version?"

Expected:
google_web_search

Prompt:
"Summarize this GitHub issue"

Expected:
web_fetch

Frugal Mutation Evals

Testing whether the model:

• minimally edits files
• avoids rewriting unnecessarily
• chooses efficient edit operations

---

5. Contributor-Focused DevEx Subagents

A major area of exploration was how agentic systems can help contributors build and improve the agent system itself.

Prototype Concepts

Subagent Scaffolder

CLI utility for generating:

• subagent templates
• TOML configs
• eval boilerplate
• registration files

Prompt Diff Utility

A side-by-side behavioral comparison system for prompts.

Features explored:

• dual eval execution
• telemetry diffing
• tool sequence comparison
• latency comparisons
• behavioral drift detection
• regression reports

---

6. Eval Development Toolkit

I also explored a complete developer toolkit for managing eval infrastructure.

Proposed CLI Commands

gemini evals:list
gemini evals:gaps
gemini evals:validate
gemini evals:improve

Features

• AST-based eval discovery
• Tool coverage mapping
• Gap analysis
• Stability validation
• Flakiness scoring
• Semantic assertion recommendations

Validation Pipeline

Static Analysis
    ↓
Smoke Test
    ↓
Semantic Validation
    ↓
Multi-run Stability Testing

---

Technical concepts learnt

Agentic Systems

• orchestration loops
• tool routing
• delegated subagents
• conversational execution systems

Evaluation Infrastructure

• behavioral testing
• semantic assertions
• deterministic + probabilistic validation
• eval stabilization

Retrieval & Context Systems

• signal vs noise filtering
• large-context resilience
• retrieval workflows
• context-aware debugging

DevEx Tooling

• trace pipelines
• replay systems
• debugging workflows
• contributor automation

Reliability Engineering for LLMs

• flaky behavior mitigation
• iterative repair loops
• convergence detection
• tool boundary enforcement

---