Skip to content

feat(tools): idempotency keys and execution ledger for tool calls - #80

Merged
gibbsie merged 3 commits into
mainfrom
feat/tool-call-idempotency
Aug 20, 2026
Merged

feat(tools): idempotency keys and execution ledger for tool calls#80
gibbsie merged 3 commits into
mainfrom
feat/tool-call-idempotency

Conversation

@gibbsie

@gibbsie gibbsie commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Summary

Event redelivery is already deduplicated, but nothing protects tool side effects. A retried node or a redelivered dispatch can create the same Jira ticket twice, for example, write the same row twice, or call the same HTTP integration twice. The governed tool handler intercepts every worker tool call, so it is the natural place to make side effects idempotent. This is the first of two PRs for this capability. Scope is deliberate and stated below.

What changed

  • Deterministic idempotency key per tool call: execution id, node id, call index, tool name, and a canonical hash of the arguments. Canonicalization is pure and property-tested, including two hazards that silently break determinism: non-string dictionary keys (which raise under sorted serialization) and integral-float / negative-zero collapse.
  • Tool-execution ledger: a new org-scoped DynamoDB table keyed orgId#executionId / nodeId#calLIndex#toolName#argsHash, with a 48-hour TTL. It is an operational dedupe record, explicitly not an audit artifact - the governance ledger remains the audit trail.
  • Reserve → execute → finalize at one atomic seam: a single
    'selected_tool' swap in the governed handler, all three steps inside one coroutine. There is no pre/post-hook window in which a side effect could run outside the protocol. A completed key returns the recorded result instead of re-executing.
  • Concurrent-caller safety: the reservation is a conditional write. The loser never executes - it polls for the holder's result within a hard bound, then returns a retryable no-execution error. A crashed holder is reclaimed via a conditional compare-and-set so a key cannot wedge forever.
    Fail-safe failure semantics: an un-tokened call whose outcome is unknown is recorded as indeterminate, refused for re-execution, non-retryable, and surfaced - never silently retried (which would guarantee a duplicate) and never swallowed.
  • Protected by default: a tool with no classification flag is treated as side-effecting and goes through the ledger. Bypass requires an explicit flag, and a bypass flag on a demonstrably side-effecting tool blocks under strict enforcement rather than warning.
  • Context threading: execution id, node id, and org id now reach the handler. Org id is read server-side from the execution row and never trusted from a subprocess payload.

Guarantee (stated precisely)

Exactly-once within an attempt, plus safety under concurrent callers of the same key. This is not once-across-re-dispatch: if the watchdog re-dispatches a stalled node and the agent body replays nondeterministically, the keys differ and the ledger cannot absorb the duplicate. Closing that requires the dispatch-generation fence, which lands in future work, next - the code and docs say so explicitly so this PR isn't mistaken for the complete guarantee.

Testing

  • Forced double-delivery of a ticket-creating call (stubbed adapter, real conditional write): exactly one execution and one recorded result.
  • Hypothesis property test driving the execution path: for arbitrary arguments, two calls with the same key produce exactly one adapter invocation and completed row.
  • Concurrent-reservation race with a recorded red proof - a non-conditional reserve lets both callers execute; the conditional reserve lets exactly one.
  • Canonicalization properties, dead-holder reclaim, TTL and org-scope isolation, protected-by-default classification, strict-mode block on a mis-flagged tool.
  • Arbiter pytest and full backend suite green (6,806 tests); tsc clean; synth clean; ledger grant scoped to a single table ARN.

Deployment notes

Adds one DynamoDB table with TTL and a scoped grant. No schema changes. Work follows in this same capability: dispatch-generation fence, S3 offload for oversized results, and client-token passthrough for targets that support it.

Oliver Gibbs added 3 commits August 20, 2026 02:23
Introduce attempt-scoped exactly-once for governed tool calls:

- tool_idempotency.py: pure canonical args-hash + key derivation
  (orgId#executionId / nodeId#callIndex#toolName#argsHash). Rejects
  non-string dict keys deterministically and collapses integral-float
  and -0.0 so 2.0 == 2; preserves null != missing; rejects NaN/Inf.
  Bypass classification defaults to ledger (fail-safe) and blocks a
  demonstrably-writing bypass tool in strict enforcement mode.
- tool_execution_ledger.py: org-scoped TTL'd (48h, server-write-time)
  reserve/get/finalize over a conditional first-write-wins. Concurrent
  loser bounded-polls then returns a retryable no-execution error and
  never executes; dead holders reclaimed via conditional CAS. Failure
  matrix: terminal recorded, not-sent released and re-executable,
  unknown outcome fail-safe outcomeIndeterminate (never re-executed).
  Inline results only; oversized records a deterministic marker.
- tool_idempotency_hook.py: single atomic seam for strands-agents 1.30.0
  via a BeforeToolCallEvent HookProvider that wraps the selected tool so
  reserve->execute->finalize share one coroutine with no pre/post window.
- Thread executionId/nodeId/orgId to the worker subprocess; orgId is
  resolved server-side from the execution row, never from a payload.

Guarantee is exactly-once WITHIN an attempt plus reservation-race safety;
exactly-once across nondeterministic re-dispatch needs the dispatch
generation fence, which is deferred to PR2 and required for the complete
guarantee.
…nt (PR1)

- arbiter-stack.ts: new citadel-tool-execution-ledger-{env} DynamoDB table
  (PK orgId#executionId, SK nodeId#callIndex#toolName#argsHash, TTL attr
  ttl, PITR, AWS-managed SSE, PAY_PER_REQUEST). Worker env wires
  TOOL_EXECUTION_LEDGER_TABLE; worker IAM grants only PutItem/GetItem/
  UpdateItem scoped to this table ARN — no DeleteItem, no Scan/Query.
- Align the ledger release path to that grant: release is now an
  in_flight->released status transition (re-reservable via conditional
  CAS), not a delete, so the worker needs no dynamodb:DeleteItem.
- Add CDK assertions for the table schema/TTL/PITR/SSE, the env wiring,
  and the least-privilege grant (Put/Get/Update only, no Delete/Scan on
  the ledger). Update the executions-table IAM test: PutItem is now
  present but only in the ledger statement; executions stays UpdateItem-
  only and Delete/BatchWrite remain forbidden everywhere.
- docs/TOOL_IDEMPOTENCY.md states the guarantee precisely (exactly-once
  within an attempt + reservation-race safety) and that the
  dispatch-generation fence, S3 offload, and client-token passthrough are
  deferred to PR2 and required for the complete guarantee.
@gibbsie
gibbsie merged commit 11695b8 into main Aug 20, 2026
14 checks passed
@gibbsie
gibbsie deleted the feat/tool-call-idempotency branch August 20, 2026 12:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant