Skip to content

Add JSON response support to 11 evaluators - #5320

Open
Salma El-Shafey (salma-elshafey) wants to merge 2 commits into
mainfrom
selshafey/json_response
Open

Add JSON response support to 11 evaluators#5320
Salma El-Shafey (salma-elshafey) wants to merge 2 commits into
mainfrom
selshafey/json_response

Conversation

@salma-elshafey

@salma-elshafey Salma El-Shafey (salma-elshafey) commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

11 evaluators updated with JSON-response support:

LLM evals (3):

  • fluency
  • similarity
  • response_completeness

Deterministic evals (8):

  • bbeh
  • bleu_score
  • gleu_score
  • meteor_score
  • rouge_score
  • f1_score
  • regex_match
  • ifeval

Response can now be a plain string or a JSON-encoded list of chat messages. Only the final plain-text assistant response is extracted for scoring; tool calls and tool results are never included. spec.yaml response schemas updated to anyOf: string | array, and versions bumped. Unit tests added covering plain-string and JSON message-list inputs, including verification that tool call/result content does not leak into the scored response.

@salma-elshafey
Add JSON response support to fluency, similarity, response_completene…
22142d3 
…ss, bbeh, bleu_score, gleu_score, meteor_score, rouge_score, f1_score, regex_match, ifeval

Response can now be a plain string or a JSON-encoded list of chat messages.
Only the final plain-text assistant response is extracted for scoring;
tool calls and tool results are never included. Spec.yaml response schemas
updated to anyOf: string | array, and versions bumped. Unit tests added
covering plain-string and JSON message-list inputs, including verification
that tool call/result content does not leak into the scored response.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: bdbf9b07-e361-470c-aa6c-894304289020
@salma-elshafey
Salma El-Shafey (salma-elshafey) marked this pull request as ready for review September 7, 2026 15:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@salma-elshafey