feat(execution): durable checkpoints and resume-not-rerun - #78
Merged
Conversation
added 4 commits
August 19, 2026 11:27
…y-once node execution Replace the unconditional pending->running node dispatch write with a conditional write (ConditionExpression status = pending) so that when two concurrent predecessor completions both find a convergence node ready, only one dispatcher wins the flip and sends the worker message; the other is a no-op. Make the execution running->completed finalize conditional too, so a single advancement emits the terminal workflow.completed under concurrent tail completions. Add a MUST-BITE differential test proving the unconditional write double- dispatches a convergence node while the conditional write dispatches it once, plus hypothesis properties for exactly-once dispatch and finalize under N racing callers. Extend the concurrency and end-to-end fake tables to honor ConditionExpression so the guards are genuinely arbitrated.
…e-only resume The worker now persists a node's completed result to the executions table (conditional first-write-wins, UpdateItem-only + FGAC attribute scope) BEFORE emitting workflow.node.completed, so a lost event leaves a durable, reconcilable checkpoint instead of a stuck-running node. The step runner becomes advance-only: extract a shared schedule_frontier re-entry primitive that re-derives the ready set from persisted status and dispatches/finalizes through the conditional guards, replacing the completed-status early-return. Add resumeExecution: a GraphQL mutation + execution.resume.requested event + StepRunnerResumeRule that re-derive the frontier server-side from persisted state (never a caller node list), reject terminal states, stay idempotent on a running execution, and never re-dispatch a running node. Enforce org-scope and an IDOR check before emit, with parity to startExecution. Document the recorded-state exactly-once guarantee and its agent-side limit in the worker docstring, the EventBridge catalog, and the tracing runbook. Tests: kill-resume simulation, resume-contract, worker write-then-signal ordering/idempotency, resolver emit/reject/IDOR, and CDK assertions for the resume rule and the UpdateItem-only FGAC worker grant.
The timeout watchdog now gives every stuck running execution a definite disposition instead of only failing on the execution-level timeout. Ordered per sweep: reconcile a lost-event frontier first (re-derive and dispatch via the executor's shared schedule_frontier, or finalize a run whose terminal signal was lost) so a lost node-completed event recovers within one cycle; then reconcile-or-fail a stalled node (re-dispatch if retries remain via a conditional running->pending flip, else drive it and the execution to terminal failure); then the execution-level backstop, preserved unchanged. Stall threshold is NODE_STALL_TIMEOUT_SECONDS * NODE_STALL_FACTOR (default 900s * 2). The watchdog's IAM widens to resource-scoped grants only: executions Scan/GetItem/Query/UpdateItem, workflows GetItem/Query read-only, SendMessage to the worker queue, PutEvents on the bus — no table-wide write, distinct from the worker's UpdateItem-only role, schedule-triggered only. Tests: lost-event reconcile within one sweep, lost-finalize, healthy-node untouched, stalled-node retry re-dispatch vs terminal fail, execution backstop, a detection property, and CDK assertions for the resource-scoped grants + env.
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| import sys | ||
| from contextlib import contextmanager | ||
|
|
||
| import pytest |
|
|
||
| sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..')) | ||
|
|
||
| import dag |
|
|
||
| sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..')) | ||
|
|
||
| import dag |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Workflow executions persist per-node state and outputs, but a crashed restarted execution has no resume entry point - it re-runs from the start. Two latent windows made partial failure worse: a node-completed event could be emitted before the output was persisted (a lost event meant lost work), and the convergence path could double-dispatch a node. This PR makes execution durable: checkpoint is derived from what's already persisted, resume dispatches only what never finished, and the watchdog reconciles instead of staying silent.
What changed (four separable commits)
pending-runningdispatch guard (this also fixes a latent double-dispatch in the convergence path), first-write-wins node completion, and arunning›completedfinalize guard. A differential test proves the guard bites - it was first run against a deliberately non-conditional write and observed to double-dispatch.node.completed. The signaled-but-unpersisted crash window is eliminated by construction; persisted-but-unsignaled becomes the benign, reconcilable mode. The worker's new table access isUpdateItem-only with an IAM attribute condition restricting writes to the node-results fields - not a general write grant.resumeExecution: Org-scoped mutation with RBAC parity tostartExecutionand an ownership check before any event is emitted. It accepts only an execution id: the server re-derives the ready frontier from persisted state and dispatches only non-terminal nodes. Advance-only - running nodes are never re-dispatched (stall recovery belongs to the watchdog); failed/completed/cancelled are rejected; resume of a running execution is safely idempotent. The re-entry scheduler is extracted once and shared by normal advancement, resume, and the watchdog - and is the substrate the future pause/approve flow will reuse.Honest limit
The guarantee is exactly-once recorded state. Agent-side side effects can still duplicate if a stalled-but-alive worker is re-dispatched. This is stated plainly in the code, the event catalog, and the runbook; the follow-up idempotency-key work planned addresses it at the tool boundary.
Testing
tscclean; both affected stack synths clean.Deployment notes
IAM deltas (worker: attribute-restricted
UpdateItem; watchdog: scoped reads plus SQS send to the worker queue), one new EventBridge rule for resume, and a GraphQL schema addition. No new tables. The usual pre-merge branch deploy to dev applies; after deploy, the watchdog's new behavior is observable on any running execution.