SkillSpec makes agent skills followable, testable, and provable with Doctor risk reports, guided imports, structured contracts, and alignment proof.

Operational doctrine for practical AI systems design.

Benchmarking Open-Ended Inference Optimization by AI Agents

Evaluation Infrastructure for AI Agents

Core engine behind Calibrate, a framework for evaluating AI agents: speech-to-text, text-to-speech, LLM evaluation, end-to-end simulations

Learn to evaluate AI products for production — 21 hands-on lessons on evals, metrics, fairness, agents, red teaming, and release decisions for working PMs.

Agent skill for AI agent development

Collection of frameworks and tools for AI evalations, including tool-use, agentic AI, MCP, and multimodal

Agent workspace architecture — the reference implementation of an agent-ready memory layer, demonstrated end-to-end in Claude Code: roles library, typed memory, hooks, scheduled agents, self-audits, loop selection, measurement-gated self-improvement. Interactive tour, fork-ready samples.

Eval-first AI agent that triages property maintenance emails. The real work is the eval system around it: trace-driven error analysis, code graders and validated LLM-as-judge (TPR/TNR), component and end-to-end evals, a failure taxonomy, and a CI regression gate. LangGraph, FastAPI, Langfuse.

Governed multi-LLM Responsible AI control plane for prior-authorization decision support, with PHI-safe audit trails, HITL review, evidence packets, and deterministic governance evals.

Benchmark LLM jailbreak resilience across providers with standardized tests, adversarial mode, rich analytics, and a clean Web UI.

Frontend for Calibrate, a framework for evaluating AI agents: speech-to-text, text-to-speech, LLM evaluation, end-to-end simulations

Local side-by-side eval workbench that turns one AI product evaluation idea into eval cases, comparable evidence, and a decision-grade report.

Open-source toolkit for assessing whether an AI workflow is ready for production: governance, RAG quality, evals, observability, human review, cost, risk, and business value.

Playbook for PMs shipping AI products with PRDs, evals, HITL, launch gates, cost, and observability.

Offline, auditable benchmark for one-shot LLM market decisions.

Vasari is a local, open source tool for evaluating AI features (agents, chatbots, RAG). Review LLM traces, run error analysis (open and axial coding), and validate an LLM as a judge against human labels with a confusion matrix, TPR/TNR and Cohen's kappa. Inspired by Hamel Husain's AI evals.

Unofficial TypeScript starter for deterministic local contract testing around Foundry-oriented workflows with Themis.

ByteStack Labs marketplace for Claude Code. Open reliability skills that audit AI which passes evaluation but fails in production. Every number reproducible.