[codex] Add observability evaluation harness - #390
Merged
Conversation
|
🚅 Deployed to the qyl-pr-390 environment in qyl
|
There was a problem hiding this comment.
Pull request overview
Moderate risk: new deterministic evaluation logic is straightforward, but a couple of comparison/correlation edge cases can produce incorrect pass/fail results.
Purpose: Introduce a deterministic observability evaluation harness (evals/Qyl.Observability.Evaluation) to validate incident-triage agent behavior (tool use, evidence citation, trace linkage, and telemetry attribute safety) from JSONL fixtures without requiring an LLM judge.
Changes:
- Added a new executable evaluation harness project with JSONL scenario loading and per-scenario metric evaluation.
- Implemented initial evaluators for evidence citation/forbidden claims, tool-call argument subset matching, trace parent correlation, and telemetry attribute safety.
- Wired the new project into
qyl.slnxand added central package versioning forMicrosoft.Extensions.AI.Evaluation.
Blockers
- Tool-call argument comparison is not semantically reliable (
ToolCallAccuracyEvaluator.JsonValuesEqualcomparesJsonElement.ToString()), which can create false mismatches for equivalent JSON values. - Trace correlation can incorrectly pass when
TraceIdis missing (keys collapse to"/{spanId}"), allowing accidental collisions and invalid parent resolution.
Important issues
- None found.
Minor issues
- None found.
Reviewed changes
Copilot reviewed 22 out of 22 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| qyl.slnx | Adds the new /evals/ folder and evaluation project to the solution. |
| Directory.Packages.props | Adds central package version entry for Microsoft.Extensions.AI.Evaluation. |
| evals/Qyl.Observability.Evaluation/Qyl.Observability.Evaluation.csproj | New executable project referencing Microsoft.Extensions.AI + .Evaluation and copying JSONL data to output. |
| evals/Qyl.Observability.Evaluation/Program.cs | CLI entrypoint: loads JSONL packs and runs deterministic evaluation, returning non-zero on failures. |
| evals/Qyl.Observability.Evaluation/ScenarioLoader.cs | JSONL loader for ObservabilityEvaluationRecord fixtures. |
| evals/Qyl.Observability.Evaluation/ScenarioRunResult.cs | Aggregates evaluator metrics and checks expected failed metric names. |
| evals/Qyl.Observability.Evaluation/EvaluationRunner.cs | Runs evaluators per scenario and collects metrics into ScenarioRunResult. |
| evals/Qyl.Observability.Evaluation/Evaluators/AnalysisResult.cs | Small pass/fail + reason helper record. |
| evals/Qyl.Observability.Evaluation/Evaluators/EvaluationMetricFactory.cs | Creates BooleanMetric with an interpretation suitable for pass/fail gating. |
| evals/Qyl.Observability.Evaluation/Evaluators/ObservabilityMetricNames.cs | Defines metric name constants for deterministic harness outputs. |
| evals/Qyl.Observability.Evaluation/Evaluators/TelemetryEvidenceEvaluator.cs | Validates required evidence exists + is cited; blocks forbidden claims. |
| evals/Qyl.Observability.Evaluation/Evaluators/ToolCallAccuracyEvaluator.cs | Validates required tool calls exist with expected argument subsets. |
| evals/Qyl.Observability.Evaluation/Evaluators/TraceCorrelationEvaluator.cs | Validates parent span references resolve within trace. |
| evals/Qyl.Observability.Evaluation/Evaluators/CardinalitySafetyEvaluator.cs | Flags blocked/sensitive/high-cardinality attribute keys/values in telemetry. |
| evals/Qyl.Observability.Evaluation/Models/ObservabilityEvaluationRecord.cs | Scenario record schema (agent, tool calls, telemetry, expectations). |
| evals/Qyl.Observability.Evaluation/Models/AgentInfo.cs | Agent metadata record for fixtures. |
| evals/Qyl.Observability.Evaluation/Models/ToolCallRecord.cs | Captures actual tool calls and argument JSON. |
| evals/Qyl.Observability.Evaluation/Models/ExpectedToolCallRecord.cs | Captures expected tool calls and argument subset JSON. |
| evals/Qyl.Observability.Evaluation/Models/TelemetryEvidenceRecord.cs | Captures telemetry evidence items and attributes for evaluators. |
| evals/Qyl.Observability.Evaluation/README.md | Docs for running the harness and adding scenario packs. |
| evals/Qyl.Observability.Evaluation/Data/README.md | Documents JSONL schema/field meanings for scenario packs. |
| evals/Qyl.Observability.Evaluation/Data/incident-triage.jsonl | Adds initial incident-triage scenario pack with pass/fail examples. |
Comment on lines
+90
to
+91
| private static bool JsonValuesEqual(JsonElement actual, JsonElement expected) | ||
| => actual.ValueKind == expected.ValueKind && actual.ToString().Equals(expected.ToString(), StringComparison.Ordinal); |
Comment on lines
+25
to
+41
| List<TelemetryEvidenceRecord> spans = [.. record.Telemetry.Where(static telemetry => IsSpan(telemetry) && telemetry.SpanId is not null)]; | ||
| HashSet<string> spanKeys = [.. spans.Select(static span => BuildSpanKey(span.TraceId, span.SpanId))]; | ||
| List<string> missingParents = []; | ||
|
|
||
| foreach (TelemetryEvidenceRecord span in spans) | ||
| { | ||
| if (span.ParentSpanId is null) | ||
| { | ||
| continue; | ||
| } | ||
|
|
||
| string parentKey = BuildSpanKey(span.TraceId, span.ParentSpanId); | ||
| if (!spanKeys.Contains(parentKey)) | ||
| { | ||
| missingParents.Add($"{span.Id}:{span.ParentSpanId}"); | ||
| } | ||
| } |
Owner
Author
|
@claude[agent] what differantes https://github.com/ANcpLua/ai-only-eval-scaffold/commit/c8d99f7d86c9b1decbbd3c9b04689e8e6c7d58fc with this codex draft |
ANcpLua
marked this pull request as ready for review
June 10, 2026 14:48
Copilot stopped work on behalf of
ANcpLua due to an error
June 10, 2026 14:49
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
evals/Qyl.Observability.Evaluation, an executable deterministic observability evaluation harness for qylIEvaluatoradaptersqyl.slnxand central package management viaMicrosoft.Extensions.AI.EvaluationWhy
qyl has instrumentation and telemetry contracts, but it did not have a domain-specific evaluation layer that can fail an observability agent for unsupported root-cause claims, bad tool filters, broken trace linkage, or unsafe telemetry attributes. This PR adds that first layer without requiring an LLM judge or external API key.
Validation
dotnet restore evals/Qyl.Observability.Evaluation/Qyl.Observability.Evaluation.csprojdotnet build evals/Qyl.Observability.Evaluation/Qyl.Observability.Evaluation.csproj --configuration Release --no-restore -v:minimaldotnet run --project evals/Qyl.Observability.Evaluation/Qyl.Observability.Evaluation.csproj --configuration Release --no-builddotnet restore qyl.slnxdotnet build qyl.slnx --configuration Release --no-restore -v:minimal