feat(agentos): Agent Simulation Engine — multi-judge evals + overview dashboard

[patel-lyzr]

Summary

Adds an Agent Simulation Engine to the AgentOS Console — an eval framework that grades registered agents against test suites, plus an observability-style overview dashboard.

Server (packages/agentos-server)

• eval-runner — runs each case against the harness /run, consumes the SSE stream server-side, and captures final output + full trace (thinking/text/tool calls/results) + policy denials + cost/latency. Scores incrementally and persists per-case so the UI polls live progress.
• 4 scorers (eval-scorers):
  • Golden match (exact / contains / regex)
  • Tool & policy compliance (expected/forbidden tools; a policy-denied tool counts as compliant)
  • NFR (cost / latency thresholds)
  • Trace-aware LLM-as-a-judge — judges the whole trace, not just final output. Multiple judges per suite, each scored independently against its own rubric (OpenAI score_model style: 0–1 score + pass threshold; template vars {{prompt}} {{criteria}} {{output}} {{trace}} {{tools}} {{golden}}). A case passes only when every enabled scorer + judge passes.
• eval-generate — synthesizes test cases from the agent's own identity files (agent.yaml/SOUL.md/RULES.md/…) plus a live tool probe.
• routes/evals — suite CRUD, run trigger, run readback, case generation. New eval_suites + eval_runs Mongo collections; router mounted under auth.

SPA (agentos)

• EvalsPage — suite editor with per-suite named judges (add/remove), suite detail, and a tabular run view where each case expands to its scores (with judge reasons), output, tool calls, policy denials, and a full trace/log panel.
• SimDashboard — overview: pass-rate KPIs, pass-rate-over-time trend, per-scorer / per-judge and per-suite breakdowns, recent-runs table.

Testing

Built both images and verified end-to-end against the live stack:

• 2-judge suite (Correctness + Safety) on guard-demo: the IMDS-credential case scored Correctness ✗ (refused → task not completed) but Safety ✓ (refusal is the safe outcome) — same trace, judged two ways, case fails because not all judges pass.
• Dashboard aggregation validated against live Mongo (5 runs / 13 cases), judges broken out separately.

🤖 Generated with Claude Code