feat(tools): idempotency keys and execution ledger for tool calls - #80
Merged
Conversation
added 3 commits
August 20, 2026 02:23
Introduce attempt-scoped exactly-once for governed tool calls: - tool_idempotency.py: pure canonical args-hash + key derivation (orgId#executionId / nodeId#callIndex#toolName#argsHash). Rejects non-string dict keys deterministically and collapses integral-float and -0.0 so 2.0 == 2; preserves null != missing; rejects NaN/Inf. Bypass classification defaults to ledger (fail-safe) and blocks a demonstrably-writing bypass tool in strict enforcement mode. - tool_execution_ledger.py: org-scoped TTL'd (48h, server-write-time) reserve/get/finalize over a conditional first-write-wins. Concurrent loser bounded-polls then returns a retryable no-execution error and never executes; dead holders reclaimed via conditional CAS. Failure matrix: terminal recorded, not-sent released and re-executable, unknown outcome fail-safe outcomeIndeterminate (never re-executed). Inline results only; oversized records a deterministic marker. - tool_idempotency_hook.py: single atomic seam for strands-agents 1.30.0 via a BeforeToolCallEvent HookProvider that wraps the selected tool so reserve->execute->finalize share one coroutine with no pre/post window. - Thread executionId/nodeId/orgId to the worker subprocess; orgId is resolved server-side from the execution row, never from a payload. Guarantee is exactly-once WITHIN an attempt plus reservation-race safety; exactly-once across nondeterministic re-dispatch needs the dispatch generation fence, which is deferred to PR2 and required for the complete guarantee.
…nt (PR1)
- arbiter-stack.ts: new citadel-tool-execution-ledger-{env} DynamoDB table
(PK orgId#executionId, SK nodeId#callIndex#toolName#argsHash, TTL attr
ttl, PITR, AWS-managed SSE, PAY_PER_REQUEST). Worker env wires
TOOL_EXECUTION_LEDGER_TABLE; worker IAM grants only PutItem/GetItem/
UpdateItem scoped to this table ARN — no DeleteItem, no Scan/Query.
- Align the ledger release path to that grant: release is now an
in_flight->released status transition (re-reservable via conditional
CAS), not a delete, so the worker needs no dynamodb:DeleteItem.
- Add CDK assertions for the table schema/TTL/PITR/SSE, the env wiring,
and the least-privilege grant (Put/Get/Update only, no Delete/Scan on
the ledger). Update the executions-table IAM test: PutItem is now
present but only in the ledger statement; executions stays UpdateItem-
only and Delete/BatchWrite remain forbidden everywhere.
- docs/TOOL_IDEMPOTENCY.md states the guarantee precisely (exactly-once
within an attempt + reservation-race safety) and that the
dispatch-generation fence, S3 offload, and client-token passthrough are
deferred to PR2 and required for the complete guarantee.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Event redelivery is already deduplicated, but nothing protects tool side effects. A retried node or a redelivered dispatch can create the same Jira ticket twice, for example, write the same row twice, or call the same HTTP integration twice. The governed tool handler intercepts every worker tool call, so it is the natural place to make side effects idempotent. This is the first of two PRs for this capability. Scope is deliberate and stated below.
What changed
orgId#executionId/nodeId#calLIndex#toolName#argsHash, with a 48-hour TTL. It is an operational dedupe record, explicitly not an audit artifact - the governance ledger remains the audit trail.'selected_tool' swap in the governed handler, all three steps inside one coroutine. There is no pre/post-hook window in which a side effect could run outside the protocol. A completed key returns the recorded result instead of re-executing.
Fail-safe failure semantics: an un-tokened call whose outcome is unknown is recorded as indeterminate, refused for re-execution, non-retryable, and surfaced - never silently retried (which would guarantee a duplicate) and never swallowed.
Guarantee (stated precisely)
Exactly-once within an attempt, plus safety under concurrent callers of the same key. This is not once-across-re-dispatch: if the watchdog re-dispatches a stalled node and the agent body replays nondeterministically, the keys differ and the ledger cannot absorb the duplicate. Closing that requires the dispatch-generation fence, which lands in future work, next - the code and docs say so explicitly so this PR isn't mistaken for the complete guarantee.
Testing
tscclean; synth clean; ledger grant scoped to a single table ARN.Deployment notes
Adds one DynamoDB table with TTL and a scoped grant. No schema changes. Work follows in this same capability: dispatch-generation fence, S3 offload for oversized results, and client-token passthrough for targets that support it.