agent-assure

Output equivalence is not process equivalence.

Catch agent process regressions that final-answer evals miss with local-first release evidence for agentic AI pipelines.

Run the offline demo · For AI/ML leaders · For engineers · How it works · Claim boundary

agent-assure checks whether a candidate agent preserves declared, observable process expectations—not only whether it preserves the visible answer. It turns privacy-filtered run evidence into reproducible comparisons, reviewer-facing artifacts, portable evidence packets, and ordinary CI gate signals.

Local-first · offline flagship demo · versioned artifacts · CI-native · no hosted control plane required

approve → approve · claim-duration: linked → missing · classification: new_failure · configured gate: blocked

The visible approval stayed stable. A declared material-evidence invariant did not.

Note

This is a bundled deterministic fixture demonstration, not a model benchmark, live-model result, or customer outcome.

Read the flagship demo

Quickstart

Requires Python 3.11 or newer.

pip install agent-assure
agent-assure demo flagship --out .tmp/demo/flagship --clean

The flagship demo uses bundled deterministic fixtures—no provider API key, network call, or token spend.

output equivalence: preserved
missing evidence link: claim-duration
reason code: MATERIAL_CLAIM_MISSING_EVIDENCE
classification: new_failure
CI gate: blocked as expected

The demo wrapper exits 0 only when it verifies that the expected regression was caught. The underlying candidate evaluation, comparison, and CI commands remain strict and exit nonzero for the blocking finding.

Actual reviewer output

_{Screenshot of the reviewer-facing evidence-diff.html produced from the same bundled fixture. Open the image to inspect it at full resolution.}

Key artifacts are written under .tmp/demo/flagship:

Artifact	Review purpose
demo-summary.json	Machine-readable demonstration result
baseline-report/evaluation-summary.json	Baseline behavior against declared expectations
comparison-report/comparison-summary.json	Controlled baseline-to-candidate classification
ci-report/evidence-packet.json	Portable machine-readable review handoff
evidence-diff.html	Self-contained reviewer-facing evidence diff

How this README proof is verified against the fixtures

Flagship regression at a glance

This diagram is checked in CI against the bundled flagship fixtures, keeping README claims aligned with the evidence produced by the project itself.

flowchart LR
    subgraph OutputCheck["Ordinary visible-output check"]
        BOut["Baseline output<br/>recommendation=approve<br/>outcome=approve"]
        COut["Candidate output<br/>recommendation=approve<br/>outcome=approve"]
        Same["Visible answer unchanged"]
        BOut --> Same
        COut --> Same
    end

    subgraph InvariantCheck["agent-assure invariant check"]
        BEv["Baseline evidence<br/>claim-duration linked"]
        CEv["Candidate evidence<br/>claim-duration missing link"]
        Pass["Baseline evaluation: pass"]
        Fail["Candidate evaluation: fail<br/>MATERIAL_CLAIM_MISSING_EVIDENCE"]
        BEv --> Pass
        CEv --> Fail
    end

    Same --> Tension["Output unchanged<br/>but governance invariant regressed"]
    Equiv["Fixture equivalence: pass"] --> Compare["Baseline-to-candidate comparison"]
    Pass --> Compare
    Fail --> Compare
    Tension --> Compare

    Compare --> NewFailure["Classification: new_failure"]

Loading

Choose your path

Your role	Fastest path
AI/ML leaders and release owners	See how unchanged decisions can conceal control drift and how local evidence supports repeatable release review. Read the leader brief.
AI/ML engineers	Declare expectations, project run evidence, compare releases, and return an ordinary CI decision. Follow the engineering guide.
Risk, security, and governance reviewers	Inspect evaluated controls, evidence lineage, limitations, and the reason behind a gate decision. Review the claim boundary.

Tip

For release owners: Evidence packets can carry evaluation and comparison summaries, artifact digests, limitations, dependency and environment context, and human-readable reports. They remain in the team workspace for local review, while the configured gate returns an ordinary CI signal—no hosted control plane required.

Integrate your agent

The integration contract has three parts:

1. Declare observable process expectations in YAML.
2. Produce versioned run records from fixtures or privacy-filtered observations.
3. Evaluate the candidate, compare equivalent baseline evidence when available, and gate the resulting evidence packet.

A real expectation from the bundled flagship suite:

cases:
  - case_id: shared-source-multi-claim
    fixture_id: shared-source-multi-claim
    expectation:
      expected_recommendation: approve
      required_evidence_refs:
        - ref-shared-clinical-note
      material_claim_ids:
        - claim-eligibility
        - claim-duration

agent-assure does not infer material claims from rationale text. Authors declare the oracle, and run-record producers emit explicit claim-to-evidence links for the material claims they intend to satisfy.

Author expectations · Understand the CLI contract · Review the public API surface

GitHub Actions example using the bundled fixture

Pin both the package and composite action in release workflows. Replace the example suite and variant paths with your own controlled materials.

name: agent-assure
on: [pull_request]

jobs:
  assure:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: python -m pip install agent-assure==0.5.0
      - uses: acblabs/agent-assure/.github/actions/agent-assure@v0.5.0
        with:
          suite: examples/prior_auth_synthetic/suite.yaml
          baseline-variant: examples/prior_auth_synthetic/variants/baseline.yaml
          candidate-variant: examples/prior_auth_synthetic/variants/candidate_evidence_normalization.yaml
          report-mode: full

full produces the complete review artifacts; fail-fast gives shorter blocking feedback. The configured gate follows declared expectations and policies, the selected gate profile, and explicit strictness flags.

What it can surface

• Evidence and RAG support: a required source or material claim-to-evidence link disappears, or declared corpus and retrieval identity changes. Flagship example: MATERIAL_CLAIM_MISSING_EVIDENCE → new_failure.
• Human review: a required route or performed-review record is missing.
• Provider, tool, and privacy boundaries: a forbidden provider or tool appears, or declared route, redaction state, or detector identity changes.
• Usage and reliability: retries, tool calls, tokens, latency, rate-limit events, or declared estimated cost change materially.
• Streaming integrity: events are replayed, duplicated, conflicting, or outside the declared sequence contract.
• Stochastic live behavior: repeated observations drift outside a declared protocol or comparison boundary.

A surfaced difference may be blocking, review-only, or informational. Usage and reliability deltas block only when a suite or policy declares that behavior.

How it works

Declared controls and canonical run evidence remain distinct inputs. Evaluation checks the candidate against expectations; controlled comparison adds baseline context only after its equivalence prerequisites pass. The resulting evidence packet supports both ordinary CI enforcement and human release review.

The assurance model is deliberately bounded:

• Reproducible: fixed fixtures, controlled comparison inputs, canonical serialization, versioned schemas, and digest-bound manifests.
• Traceable: expectations connect to run records, findings, comparisons, evidence packets, and gate state.
• Fail-closed: malformed, conflicting, incompatible, ambiguous, or unbound evidence does not silently become a passing review.
• Statistically bounded: live conclusions remain tied to a declared protocol, data boundary, provider/model configuration, execution window, and explicit limitations.

Where it fits

agent-assure complements output evaluation, observability, runtime guardrails, and governance systems. Its focused role is release-time evidence for declared process expectations: did the controlled path around an agent decision regress when the implementation changed?

• Output and agent evals ask whether answers, trajectories, tools, or components meet quality targets.
• Observability and tracing capture and query runtime behavior.
• Runtime guardrails enforce policy while requests and actions execute.
• Governance and GRC systems manage organizational policy, inventory, approvals, and accountability.
• agent-assure checks declared release controls under controlled evidence, produces a portable packet, and returns an ordinary CI signal.

It is a particularly strong fit when release review must be local, reproducible, CI-enforceable, and traceable without a required hosted control plane.

Integrations and maturity

Current maturity: Release Candidate (RC, v0.5.0).

The CLI, YAML authoring format, persisted versioned JSON artifacts, and AgentRunRecord producer contract are the primary integration surface. Framework adapters, streaming, and live execution remain experimental.

If you have…	Start with…	Maturity
YAML suites or versioned JSON artifacts	CLI contract	Primary supported surface
A GitHub release workflow	Composite action	Packaged and documented
RAG retrieval evidence	RAG provenance demo	Reference implementation
JSONL or multi-agent events	Streaming example	Experimental
LangGraph or Google ADK events	LangGraph · Google ADK	Experimental
Live provider or external-script subjects	Adapter contract	Experimental, time-bound evidence
OpenTelemetry context or export	OpenTelemetry alignment	Optional alignment only

Framework adapters consume only privacy-filtered agent_assure metadata and project it into the shared framework-neutral run-record model. They ignore raw prompts, messages, completions, tool arguments, token chunks, and unredacted summaries.

Governance crosswalks

Packet-resident evidence can be mapped to selected concepts in the NIST AI RMF, OWASP Top 10 for LLM Applications 2025, ISO/IEC 42001, and MITRE ATLAS 2026.06.

These crosswalks are planning and review aids. They do not establish framework conformance, complete coverage, third-party assurance, or endorsement.

Claim boundary

Measured evidence, not a blanket trust claim.

This project is not a compliance attestation.

agent-assure supports human release review. It does not determine safety or replace domain, legal, regulatory, clinical, security, provider-quality, model-quality, or business-impact review.

agent-assure is	agent-assure is not
Release-review evidence for declared process expectations	A legal or regulatory determination
A deterministic and protocol-bound measurement toolkit	A safety determination
A way to surface evidence, routing, privacy, boundary, provenance, usage, and stream-integrity regressions	A general model-quality benchmark
A local artifact and CI-gate workflow	A production observability backend or enterprise governance system of record
An engineering evidence source for human and governance review	A replacement for organizational accountability

Pattern redaction is a guardrail, not comprehensive DLP or de-identification. Live conclusions remain bounded by the declared protocol, data boundary, provider/model configuration, and execution window. Review the claim boundary, limitations, threat model, privacy model, and security guidance.

Learn more

• Start: Documentation · For AI leaders · For engineers
• Demos: Flagship · RAG provenance · Expense approval
• Integrations: LangGraph · Google ADK · Adapter contract
• Assurance: What this measures · Evidence packets · Live calibration
• Security and governance: Claim boundary · Threat model · Governance crosswalks
• Project: Contributing · Changelog · License

Development from a repository checkout

pip install -e ".[dev]"
git config core.hooksPath .githooks
python scripts/check_docs_alignment.py
ruff check .
mypy src scripts
pytest
python -m build

Citing

This project ships a CITATION.cff. Use GitHub’s Cite this repository control for generated citation formats.