Skip to content

feat(testing): add declarative scenario runner for agent evaluation#1287

Open
diiviikk5 wants to merge 1 commit intomofa-org:mainfrom
diiviikk5:feat/testing-scenario-runner
Open

feat(testing): add declarative scenario runner for agent evaluation#1287
diiviikk5 wants to merge 1 commit intomofa-org:mainfrom
diiviikk5:feat/testing-scenario-runner

Conversation

@diiviikk5
Copy link
Copy Markdown
Contributor

Summary

closes #1279

Add a declarative ScenarioRunner to mofa-testing that loads YAML/JSON scenario specs, configures the existing mocks automatically, runs an async scenario body, and validates structured expectations into a TestReport.

Builds On my previous prs

What This Adds

  • ScenarioSpec::from_yaml_str and ScenarioSpec::from_json_str
  • a new scenario module in mofa-testing
  • ScenarioContext for configured mock backend, bus, clock, and tool access
  • expectation evaluation for:
    • total infer calls
    • prompt substring counts
    • tool call counts
    • bus messages by sender
  • end-to-end tests for parsing, successful execution, expectation failures, and injected execution failures

Why This Matters

mofa-testing already provides strong low-level primitives. This PR adds a higher-level declarative layer that makes end-to-end agent evaluation reproducible and easier to adopt across contributors. It is a direct step toward a real testing and evaluation platform for MoFA agents.

How I Tested

  • cargo test -p mofa-testing --test scenario_runner_tests
  • cargo test -p mofa-testing --test integration

@diiviikk5
Copy link
Copy Markdown
Contributor Author

@lijingrs @yangrudan builds towards the idea 6 / testing framework also towards the agent testing framework in the open task and 3 of my previous prs , starting from #486 , #888 , #895 , furthermore #1288 extends this

@diiviikk5
Copy link
Copy Markdown
Contributor Author

Can someone review/ merge this , i am building on top of these and #1441 and 3-4 other open prs since weeks
so to progress further if i could know the changes in these or complete merge it , would be helpfull @lijingrs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(testing) Declarative Scenario Runner for mofa-testing

1 participant