Skip to content

Add comprehensive Istio list evaluation test - #30

Open
josunect wants to merge 1 commit into
masterfrom
add-comprehensive-istio-eval
Open

Add comprehensive Istio list evaluation test#30
josunect wants to merge 1 commit into
masterfrom
add-comprehensive-istio-eval

Conversation

@josunect

@josunect josunect commented Jul 2, 2026

Copy link
Copy Markdown
Owner

Summary

Adds a new MCP evaluation test istio-list-comprehensive that creates a larger set of Istio resources (31 total) to validate token efficiency at scale.

Motivation

Current evaluation tests (istio-list, istio-list-destination-rules) create only 1-3 Istio resources, which is insufficient to demonstrate the token savings of grouped output formats when listing many resources.

This test establishes a baseline that will show the true benefit of the namespace→GVK→{valid,invalid} grouped format vs the flat array format.

What's Created

The test creates in the bookinfo namespace:

  • 10 VirtualServices (service-1-vs through service-10-vs)
  • 10 DestinationRules (service-1-dr through service-10-dr) with v1/v2 subsets
  • 5 Gateways (gateway-1 through gateway-5)
  • 5 ServiceEntries (external-service-1 through external-service-5)
  • 1 Invalid VirtualService (invalid-vs) to test validation error detection

Total: 31 Istio configuration objects

Expected Impact

With 31 resources, the grouped format should show:

  • ~60-75% token reduction in the istio list output compared to flat array
  • Demonstrates scalability benefits of deduplicating namespace/group/version/kind metadata

Test Details

  • File: tests/evals/tasks/istio-list-comprehensive/istio-list-comprehensive.yaml
  • Setup: Creates resources via kubectl apply
  • Cleanup: Deletes all resources with label gevals.kiali.io/test=gevals-testing
  • Verification: Checks that VirtualService, DestinationRule, and Gateway are mentioned in response

Next Steps

After merge:

  1. CI will run mcpchecker and establish new baseline
  2. PR [AI] Proposal for token reduction kiali/kiali#9945 (Istio list optimization) will show significant token savings on this test
  3. Validates that grouped format scales better with realistic resource counts

🤖 Generated with Claude Code

This test creates a larger set of Istio resources to validate token
efficiency of the grouped output format at scale:
- 10 VirtualServices
- 10 DestinationRules
- 5 Gateways
- 5 ServiceEntries
- 1 VirtualService with validation errors (for testing invalid detection)

Total: 31 Istio resources vs 1-3 in existing tests.

This will establish a baseline that demonstrates the token savings
of the grouped namespace→GVK→{valid,invalid} format compared to the
flat array format when listing many resources.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant