Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/lazy-hook-resume-durable-abort.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@workflow/core': patch
---

Keep the eager `hook_received` write for the internal resume that records a step-issued abort, so it stays committed before the step completes.
5 changes: 5 additions & 0 deletions .changeset/lazy-hook-resume-vitest.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@workflow/vitest': patch
---

`waitForHook()` accepts `notHookId` to skip a hook the caller already resumed, whose `hook_received` may not be written yet.
5 changes: 5 additions & 0 deletions .changeset/lazy-only-hook-resume-world.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@workflow/world': patch
---

Document that `hookInput` and the `(runId, resumeId)` dedup contract now cover repeated deliveries of one resume rather than a producer/consumer race, and accept `lazy` as a hook-resume strategy.
5 changes: 5 additions & 0 deletions .changeset/lazy-only-hook-resume.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@workflow/core': patch
---

`resumeHook()` no longer writes the `hook_received` event itself on the lazy path: the queue consumer creates it from the queue message, so a resume costs one round trip. A resume against an already-ended run now resolves instead of throwing `HookNotFoundError`.
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,9 @@ related:

Resumes a workflow run by sending a payload to a hook identified by its token.

It creates a `hook_received` event and re-triggers the workflow to continue execution.
It publishes a workflow invocation carrying the payload; the runtime creates the `hook_received` event and continues execution from it.

A Hook kept by `experimental_minRetention` after its workflow ends cannot be resumed. `resumeHook()` throws `HookNotFoundError` in that case.
`resumeHook()` throws `HookNotFoundError` when no hook holds the token. A run that has already ended cannot be resumed, including one whose Hook is kept by `experimental_minRetention`, but whether the call reports that depends on the path it takes: a resume dispatched without reading the run resolves and the ended state is only detected once the payload arrives, while one that reads the run, or that falls back to writing the event up front, throws `HookNotFoundError`. See [lazy hook resume](/docs/changelog/lazy-hook-resume).

<Callout type="warn">
`resumeHook` is a runtime function that must be called from outside a workflow function.
Expand Down Expand Up @@ -50,7 +50,7 @@ showSections={["parameters"]}

### Returns

Returns a `Promise<ResumedHook>`, a `Hook` extended with an optional `resilientResume` flag. Resolving means the resume was accepted and the workflow will continue, whether the `hook_received` event was written directly or, on the parallel fast path, delivered through the workflow queue for the runtime to materialize (see the [lazy hook resume changelog](/docs/changelog/resilient-resume)). `resilientResume` is `true` only when the direct event write failed transiently and the resume was recovered through the queue; on the happy path it is absent. The resolved hook:
Returns a `Promise<ResumedHook>`, a `Hook` extended with an optional `resilientResume` flag. Resolving means the resume was accepted for delivery: the payload rides the workflow queue message and the runtime materializes the `hook_received` event from it before replaying (see the [lazy hook resume changelog](/docs/changelog/lazy-hook-resume)). `resilientResume` is retained for source compatibility and is no longer set by any path. The resolved hook:

<TSDoc
definition={`
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ related:

Resumes a workflow run by sending an HTTP `Request` to a webhook identified by its token.

This function creates a `hook_received` event and re-triggers the workflow to continue execution. It's designed to be called from API routes or server actions that receive external HTTP requests.
This function publishes a workflow invocation carrying the request; the runtime creates the `hook_received` event from it and continues execution. It's designed to be called from API routes or server actions that receive external HTTP requests.

<Callout type="warn">
`resumeWebhook` is a runtime function that must be called from outside a workflow function.
Expand Down
1 change: 1 addition & 0 deletions docs/content/docs/v5/changelog/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ Stay up to date with the latest changes to Workflow SDK.

## 2026

- [Lazy hook resume](/docs/changelog/lazy-hook-resume) (August 2026)
- [Resilient hook resume](/docs/changelog/resilient-resume) (July 2026)
- [Eager processing of steps and incremental event replay](/docs/changelog/eager-processing) (March 2026)
- Serializable AbortController and AbortSignal (March 12, 2026)
51 changes: 51 additions & 0 deletions docs/content/docs/v5/changelog/lazy-hook-resume.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
---
title: Lazy hook resume
description: resumeHook() no longer writes hook_received itself. The queue consumer materializes the event from the message, so a resume costs one round trip.
---

# Lazy hook resume

## Motivation

[Resilient hook resume](/docs/changelog/resilient-resume) made `resumeHook()` write the `hook_received` event and publish the workflow queue message concurrently, with the queue consumer re-ensuring the event from the message's `hookInput` before replay. Both sides then wrote the same event, and a `(runId, resumeId)` constraint collapsed them onto one.

Running the two concurrently removed the second round trip from the critical path, but the write itself stayed: every resume still spent a request on an event the consumer was about to write anyway, and the producer still had to classify its outcome (conflict, throttle, terminal run) to decide whether the resume had survived.

This change drops the producer's write entirely. On the lazy path `resumeHook()` publishes the queue message and nothing else.

## Design

- `resumeHook()` publishes one message carrying `hookInput`: the dehydrated payload, a client-minted `resumeId`, the hook token, and a payload digest. It writes no event.
- The queue consumer materializes `hook_received` from `hookInput` before replay, keyed by `resumeId`. This is the same write it already performed; it is now the only one.
- The `(runId, resumeId)` constraint still matters: a queue redelivery, or a delivery re-routed for deployment affinity, repeats the write with the same key, and the backend collapses those onto exactly one committed event.
- **A failed publish fails the resume.** The message carries both the trigger and the only copy of the payload, so `resumeHook()` throws and nothing is persisted for a later delivery to pick up. This replaces the previous rule where a failed event write could still be recovered through the queue.
- `ResumedHook.resilientResume` is retained on the type but is never set: with a single writer there is no partial outcome to report. The `workflow.hook.resilient_resume` span attribute is likewise no longer emitted.
- The resume span reports `workflow.hook.resume_strategy: lazy` (previously `parallel`).

## Behavior change: the event is not visible when `resumeHook()` returns

`resumeHook()` used to await its own `hook_received` write, so by the time it resolved the event was in the log. It no longer writes, so **resolving means the message was published, not that the event exists**. The event appears when the run picks the resume up.

Code that reads the run back immediately after resuming now races. The pattern that breaks is a loop that resumes and then looks for the next thing to resume, keying off "this hook has no `hook_received` yet": it can be handed back the hook it just resumed and deliver a second payload to it. Wait for something that implies the run made progress instead. `waitForHook()` in `@workflow/vitest` takes a `notHookId` option for exactly this.

The runtime has one caller that needs the old guarantee. A step that aborts a shared `AbortController` resumes a hook to record the abort in the event log, and that write is an ordering barrier: it must land before the step completes, or the continuation `step_completed` enqueues can dispatch the next step with a stale, non-aborted signal. That path uses an internal durable resume which keeps the eager write and reports `resume_fallback_reason: durable_required`.

Nothing about delivery changes. The payload is on the queue message and reaches the workflow exactly once.

## Behavior change: resumes against an ended run

The hook lookup is unchanged: `resumeHook(token, ...)` still resolves the token through `hooks.getByToken()`, which throws `HookNotFoundError` when no hook holds it. Hook existence, and the token's binding to a run, are still validated before anything is published.

What the lookup does not carry is the run's *mutable* status. `HookResumeContext` is deliberately an immutable slice of the run, so a resume that runs off it never learns whether the run is still live. That used to be caught by the `hook_received` write being rejected. With no write, **a resume against an ended run resolves instead of throwing `HookNotFoundError`**.

This is only reachable when the hook record outlives its run, since otherwise the lookup itself fails: a hook kept by `experimental_minRetention`, or one whose token has not been released yet. Resumes that fall back to reading the run keep their terminal pre-check, as does the sequential path, so a resume on either of those still fails loudly.

Nothing resumes either way. The consumer's write is rejected the same way and the delivery is consumed, so the ended run is untouched. Only the producer's report changes: an accepted publish means the resume was dispatched, not that the run was still live when it arrived. A [webhook](/docs/api-reference/workflow-api/resume-webhook) whose run has ended can answer `202` rather than surfacing an error.

Callers that need the distinction have to read the run.

## Compatibility

The gating is unchanged: the lazy path activates only when the target run's queue consumer and the live backend both attest support, re-checked on every resume. Oversized payloads, legacy runs, non-CBOR transports, and `WORKFLOW_DISABLE_LAZY_HOOK_RESUME=1` fall back to the sequential write-then-publish path.

Consumers still accept a message from an older producer that wrote the event itself: such a message reports `strategy: parallel`, and the consumer's write converges on the producer's committed event exactly as before. No coordinated deploy is needed in either direction.
1 change: 1 addition & 0 deletions docs/content/docs/v5/changelog/meta.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
"title": "Changelog",
"pages": [
"index",
"lazy-hook-resume",
"eager-processing",
"resilient-resume",
"resilient-start",
Expand Down
8 changes: 8 additions & 0 deletions docs/content/docs/v5/changelog/resilient-resume.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,14 @@ description: resumeHook() now tolerates transient event storage failures when th

# Resilient `resumeHook()`

<Callout type="info">
Superseded by [lazy hook resume](/docs/changelog/lazy-hook-resume):
`resumeHook()` no longer writes the `hook_received` event at all on the fast
path, so the two-writer design and the `resilientResume` flag described below
are historical. The `(runId, resumeId)` constraint and the queue-carried
payload remain.
</Callout>

## Motivation

`resumeHook()` used to write the `hook_received` event and dispatch the workflow queue message strictly one after the other, so every resume paid two sequential round trips and a transient event-storage failure failed the whole resume even when the queue was healthy. This change runs both writes **concurrently** (cutting a round trip off resume latency) and, on the same path, brings `resumeHook()` to parity with [resilient `start()`](/docs/changelog/resilient-start): a transient event-write failure no longer fails the resume when the payload can still be delivered through the queue.
Expand Down
5 changes: 3 additions & 2 deletions docs/content/docs/v5/configuration/runtime-tuning.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -84,8 +84,9 @@ For example, a workflow can run a 10-minute inline step even with `WORKFLOW_REPL
### `WORKFLOW_DISABLE_LAZY_HOOK_RESUME`

- Default: enabled (lazy hook resume on)
- Resuming a hook persists the `hook_received` event and publishes the workflow invocation concurrently, cutting a round trip off resume latency. On this parallel path, the queue message also carries the payload, so a transient event-write failure still resumes the run. The queue consumer re-ensures the `hook_received` event before replay. A backend `(runId, resumeId)` constraint keeps the two writers converging on exactly one event.
- The runtime falls back to the sequential path when the consumer or backend does not attest dedup support (or the payload is too large to inline on the queue message). On the sequential path, the event is written *before* dispatch, and its failure fails the resume. The fallback trades away that resilience to stay safe when dedup is not enforced; it does not preserve it.
- Resuming a hook publishes the workflow invocation and writes no event, so the resume costs one round trip. The queue message carries the payload, and the consumer creates the `hook_received` event from it before replay. A backend `(runId, resumeId)` constraint keeps redeliveries of that message converging on exactly one event. Because the message is the only copy of the payload, a failed publish fails the resume.
- The runtime falls back to the sequential path when the consumer or backend does not attest dedup support (or the payload is too large to inline on the queue message). On the sequential path, the event is written *before* dispatch, and its failure fails the resume.
- The two paths differ on ended runs: the sequential write is rejected and surfaces as `HookNotFoundError`, while the lazy path resolves because it never writes. Neither resumes the run.
- Set `1` to force the sequential path as a kill switch. The chosen strategy is reported on the resume span as `workflow.hook.resume_strategy`.

### `WORKFLOW_DEPLOYMENT_MISMATCH_MAX_RETRIES`
Expand Down
24 changes: 21 additions & 3 deletions packages/core/e2e/e2e.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2389,9 +2389,15 @@ describe.concurrent('e2e', () => {

const hook = await getHookByToken(token);
expect(hook.runId).toBe(owner.runId);
await expect(resumeHook(hook, { duplicate: true })).rejects.toSatisfy(
(error: unknown) => HookNotFoundError.is(error)
);
// Whether the call itself rejects depends on which dispatch path
// `resumeHook` takes, so this only asserts the invariant both share: the
// ended run is never resumed. The sequential path writes `hook_received`
// and surfaces the server's rejection as HookNotFoundError; the lazy
// path writes nothing, so it resolves and the same rejection lands on
// the queue consumer, which consumes the delivery.
await resumeHook(hook, { duplicate: true }).catch((error: unknown) => {
if (!HookNotFoundError.is(error)) throw error;
});

const duplicate = await start(await e2e('hookMinRetentionWorkflow'), [
token,
Expand All @@ -2402,6 +2408,18 @@ describe.concurrent('e2e', () => {
conflictRunId: owner.runId,
conflictStatus: 'completed',
});

// Nothing was appended to the terminal run: no path may materialize a
// `hook_received` for it. Checked after the duplicate run so a lazy
// resume's consumer has had time to attempt (and be refused) its write.
const world = await getWorld();
const { data: ownerEvents } = await world.events.list({
runId: owner.runId,
});
expect(
ownerEvents.some((e) => e.eventType === 'hook_received'),
'a resume against an ended run must not append hook_received'
).toBe(false);
}
);

Expand Down
28 changes: 24 additions & 4 deletions packages/core/src/abort-controller-step.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,14 @@ const mockStreamReads = vi.hoisted(() => ({
}));

const mockResumeHook = vi.hoisted(() => vi.fn().mockResolvedValue(undefined));
// The abort path must use the DURABLE entry point: a plain `resumeHook` only
// guarantees the resume was published (the lazy path defers the event write to
// the queue consumer), which would satisfy preCompletionOps while leaving the
// stale-signal race it exists to prevent. Mocked separately so a regression to
// `resumeHook` shows up as this staying uncalled.
const mockResumeHookDurable = vi.hoisted(() =>
vi.fn().mockResolvedValue(undefined)
);

// Mock version module
vi.mock('./version.js', () => ({ version: '0.0.0-test' }));
Expand Down Expand Up @@ -84,6 +92,7 @@ vi.mock('./runtime/get-world-lazy.js', () => ({
// Mock resume-hook
vi.mock('./runtime/resume-hook.js', () => ({
resumeHook: mockResumeHook,
resumeHookDurable: mockResumeHookDurable,
}));

// ============================================================================
Expand Down Expand Up @@ -164,7 +173,7 @@ function reviveAbortController(opts: {
if (opts.hookToken) {
ctx.ops.push(
(async () => {
await mockResumeHook(opts.hookToken, {
await mockResumeHookDurable(opts.hookToken, {
aborted: true,
reason,
});
Expand Down Expand Up @@ -422,7 +431,7 @@ describe('AbortSignal deserialized in step context', () => {

await Promise.allSettled(stepCtx.ops);

expect(mockResumeHook).toHaveBeenCalledWith('abrt_test9', {
expect(mockResumeHookDurable).toHaveBeenCalledWith('abrt_test9', {
aborted: true,
reason: 'hook-resume-test',
});
Expand Down Expand Up @@ -615,10 +624,17 @@ describe('AbortSignal deserialized in step context', () => {
* flake). The hook resume must land in `ctx.preCompletionOps`, which the step
* handler awaits before writing `step_completed`. The real-time stream write
* (which reaches an in-flight sibling) stays in the background `ctx.ops`.
*
* Routing alone stopped being sufficient once `resumeHook()` went lazy: it
* resolves when the resume is PUBLISHED, leaving the event write to the queue
* consumer, so awaiting it in `preCompletionOps` would re-open the same race.
* The abort path therefore calls `resumeHookDurable()`, which forces the eager
* write, and this suite pins both the routing and the entry point.
*/
describe('step-initiated abort: durable hook resume is committed before completion', () => {
beforeEach(() => {
mockResumeHook.mockClear();
mockResumeHookDurable.mockClear();
mockStreamReads.readResults.clear();
mockStreamReads.writeLog = [];
mockStreamReads.closeLog = [];
Expand Down Expand Up @@ -708,10 +724,14 @@ describe('step-initiated abort: durable hook resume is committed before completi
// Draining preCompletionOps (what the step executor awaits before
// step_completed) actually fires the resume with the correct payload.
await Promise.all(preCompletionOps);
expect(mockResumeHook).toHaveBeenCalledTimes(1);
expect(mockResumeHook).toHaveBeenCalledWith('abrt_pre_completion', {
expect(mockResumeHookDurable).toHaveBeenCalledTimes(1);
expect(mockResumeHookDurable).toHaveBeenCalledWith('abrt_pre_completion', {
aborted: true,
reason: 'aborted from step',
});
// Routing is only half of it: the plain entry point would resolve as soon
// as the resume was published, so draining preCompletionOps would prove
// nothing about the event existing.
expect(mockResumeHook).not.toHaveBeenCalled();
});
});
6 changes: 3 additions & 3 deletions packages/core/src/capabilities.ts
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,8 @@
* - Lazy hook resume ("consumer re-ensures `hook_received` from `hookInput`"):
* deliberately NOT tracked here. Rather than predict a release cutoff, the
* run's creating deployment stamps an explicit `hookResumeInputVersion`
* execution-context marker; `resumeHook()` gates the parallel fast path on
* that marker (mirrored onto the hook's resumeContext by the server).
* execution-context marker; `resumeHook()` gates the lazy path on that
* marker (mirrored onto the hook's resumeContext by the server).
*/

import semver from 'semver';
Expand Down Expand Up @@ -120,7 +120,7 @@ const CAPABILITY_VERSION_TABLE: ReadonlyArray<{
// version-compare against a predicted release cutoff is a guess; instead the
// run's creating deployment stamps an explicit `hookResumeInputVersion`
// marker into its execution context, which the server mirrors onto the hook's
// resumeContext. `resumeHook()` gates the parallel fast path on that marker.
// resumeContext. `resumeHook()` gates the lazy path on that marker.
];

/**
Expand Down
Loading
Loading