Skip to content

feat(release): canary agent releases with sticky arm assignment - #75

Merged
gibbsie merged 3 commits into
mainfrom
feat/agent-canary-releases
Aug 18, 2026
Merged

feat(release): canary agent releases with sticky arm assignment#75
gibbsie merged 3 commits into
mainfrom
feat/agent-canary-releases

Conversation

@gibbsie

@gibbsie gibbsie commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Summary

Agent releases promote between environments as immutable bundles via per-env pointers, but cutover is all-or-nothing: the pointer serves
100% of traffic from one release. There is no way to expose a candidate to a fraction of traffic and compare arms before committing. Weighted Lambda aliases don't apply - agents are data (configs), not code - so the split happens at release-pointer resolution.

Scope: attribution-first

This PR ships the full split, stickiness, and per-arm attribution machinery. Release resolution today is gate/attribution-level, arms run identical live config until release›runtime-config binding lands (an explicit follow-up). The runbook states this plainly and warns against presenting canaries as behavioral protection in the interim. What this PR proves deterministic assignment, no mid-flow arm mixing, per-arm data, one-mutation promote/abort, and the governance rails around all of it.

What changed

  • Canary spec on the env pointer: optional canary { candidateReleaseId, percentBasisPoints, stickiness, salt, startedAt, startedBy }; absent → 100% stable - existing pointers and behavior are byte-identical.
  • Deterministic sticky assignment: pure sha256(salt:key) % 10000 < percentBasisPoints, mirrored in TypeScript and Python with one shared parity fixture plus a mutant test proving the parity check bites. Salt is minted at canary start and preserved across reweight, so buckets never reshuffle - reweight moves only the delta band. The arm is assigned once at flow entry and propagated in server-minted fields; external arm claims are stripped at the boundary. Interim stickiness unit is the orchestration/execution (conversation-level stickiness arrives with the binding follow-up).
  • Four mutations:
    startCanary / reweightCanary / promoteCanary / abortCanary - each a single atomic, version-gated write (pointer + history row in one TransactWriteItems). Promote-to-100 re-runs the full ladder + gate + approval chain at the cutover moment; canary start obeys the same Ladder adjacency as promotion (a canary is prod exposure, not an exemption).
  • Authorization: new release: canary permission covers start/reweight/abort; "promoteCanary' additionally requires 'release:promote, so canary authority cannot escalate to full promotion.
  • Blast radius: canaryMaxBasisPoints joins the promotion policy chain (default 2500 = 25%, org-overridable; prod ceiling ≤ staging via the tightening-fields rule; fail-closed on unreadable policy) •
  • Per-arm attribution: "releaseId'/ releaseArm" cost ledger, and findings; Cloudwatch EMF gains only a low-cardinality ReleaseArm dimension (release ids stay out of metric dimensions).
  • Build hygiene (separate commit): arbiter-stack CDK template tests no longer perform real Docker/pip asset bundling - they were resolving from live PyPI on every run and flaking; the suites now run in seconds. boto3 is pinned exactly in arbiter requirements for deterministic deploy bundling.
  • Trip-wire robustness: the cross-runtime parity test hashes whitespace-normalized source, so formatters can't false-positive it; demonstrated that real logic edits still trip it.

Testing

  • tsc --noEmit clean; backend Jest and the arbiter supervisor/stepRunner pytest sets green on the final committed tree.
  • Property-based tests: assignment distribution (~10% at 1000 bp), determinism, stickiness across reweight.
  • Non-stubbed stateful race tests on canary transitions (exactly one of two concurrent transitions wins).
  • Named acceptance tests: backward compatibility (canary-absent pointer), re-gate at promote, ceiling + monotonicity enforcement, authorization split, start adjacency.

Deployment notes

No new tables - canary state lives on the existing pointer record and history rows. Additive GraphQL schema/resolver and IAM changes. Deploys are manual; a pre-merge branch deploy to dev is recommended as usual.

@gibbsie
gibbsie merged commit 35866f2 into main Aug 18, 2026
14 checks passed
@gibbsie
gibbsie deleted the feat/agent-canary-releases branch August 18, 2026 12:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant