From 9a6ef262ee69b80c68d3a40c8285934954d59dff Mon Sep 17 00:00:00 2001 From: Nathan Rajlich Date: Wed, 19 Aug 2026 15:00:43 -0700 Subject: [PATCH 1/4] feat(core): user-registerable workflow lifecycle hooks MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds registerLifecycleHooks (exported from workflow/api) so apps can observe run terminal transitions from one central place — e.g. report every failed run to Sentry from instrumentation.ts — without wrapping each workflow body. - onRunCompleted/onRunFailed handlers receive the lazily-hydrated Run instance; onRunFailed additionally receives a WorkflowRunFailedError whose errorCode carries the classification and whose cause is the hydrated thrown value (round-tripped through the run-error serialization pipeline, so VM-realm throws surface as host-realm Errors with class identity preserved — same shape run.returnValue rejects with). - Registry lives on globalThis under Symbol.for so every bundled copy of @workflow/core shares one list; multiple registrations allowed, handlers run in registration order, and registration returns an unregister function. - Dispatch is fire-and-forget via safeWaitUntil: handlers can't delay or change the run outcome, failures are logged and swallowed, and serverless invocations stay alive while handlers finish. - Wired into every terminal writer that lands a run_completed or run_failed event: the happy-path completion, the terminal catch, the suspension-commit failure, recordFatalRunError, the max-deliveries gate, replay-budget exhaustion, the deployment guard, and both QuickJS entrypoint writers — and only on the invocation whose write actually succeeded (never on EntityConflict/RunExpired). - e2e coverage registers handlers in the Next.js workbenches' instrumentation.ts and reports observations by resuming a durable hook, so the channel works across serverless instances. - Docs: observability guide with a Sentry example + workflow/api reference page. --- .changeset/workflow-lifecycle-hooks.md | 6 + .../v5/api-reference/workflow-api/index.mdx | 3 + .../workflow-api/register-lifecycle-hooks.mdx | 75 ++++++ .../docs/v5/observability/lifecycle-hooks.mdx | 84 +++++++ docs/content/docs/v5/observability/meta.json | 2 +- packages/core/e2e/e2e.test.ts | 72 ++++++ packages/core/package.json | 4 + packages/core/src/runtime.ts | 17 ++ packages/core/src/runtime/deployment-guard.ts | 6 + .../core/src/runtime/lifecycle-hooks.test.ts | 195 ++++++++++++++++ packages/core/src/runtime/lifecycle-hooks.ts | 218 ++++++++++++++++++ .../core/src/runtime/quickjs-entrypoint.ts | 12 + .../core/src/runtime/replay-budget.test.ts | 73 +++++- packages/core/src/runtime/replay-budget.ts | 2 + packages/workflow/src/api-workflow.ts | 7 + packages/workflow/src/api.ts | 6 + workbench/example/workflows/99_e2e.ts | 41 ++++ workbench/nextjs-turbopack/instrumentation.ts | 6 + .../nextjs-turbopack/lifecycle-hooks-e2e.ts | 55 +++++ workbench/nextjs-webpack/instrumentation.ts | 6 + .../nextjs-webpack/lifecycle-hooks-e2e.ts | 1 + 21 files changed, 889 insertions(+), 2 deletions(-) create mode 100644 .changeset/workflow-lifecycle-hooks.md create mode 100644 docs/content/docs/v5/api-reference/workflow-api/register-lifecycle-hooks.mdx create mode 100644 docs/content/docs/v5/observability/lifecycle-hooks.mdx create mode 100644 packages/core/src/runtime/lifecycle-hooks.test.ts create mode 100644 packages/core/src/runtime/lifecycle-hooks.ts create mode 100644 workbench/nextjs-turbopack/lifecycle-hooks-e2e.ts create mode 120000 workbench/nextjs-webpack/lifecycle-hooks-e2e.ts diff --git a/.changeset/workflow-lifecycle-hooks.md b/.changeset/workflow-lifecycle-hooks.md new file mode 100644 index 0000000000..04cbe419d1 --- /dev/null +++ b/.changeset/workflow-lifecycle-hooks.md @@ -0,0 +1,6 @@ +--- +'@workflow/core': minor +'workflow': minor +--- + +Add `registerLifecycleHooks` (exported from `workflow/api`) for registering global `onRunCompleted`/`onRunFailed` handlers that receive the lazily-hydrated `Run` instance — and, for failures, a `WorkflowRunFailedError` with the hydrated cause and error code — enabling centralized reporting (e.g. to Sentry) from `instrumentation.ts`. diff --git a/docs/content/docs/v5/api-reference/workflow-api/index.mdx b/docs/content/docs/v5/api-reference/workflow-api/index.mdx index 82d80d4bb6..214d249ec5 100644 --- a/docs/content/docs/v5/api-reference/workflow-api/index.mdx +++ b/docs/content/docs/v5/api-reference/workflow-api/index.mdx @@ -27,6 +27,9 @@ The API package is for access and introspection of workflow data to inspect runs Get workflow run status and metadata without waiting for completion. + + Observe run completions and failures with global handlers. + diff --git a/docs/content/docs/v5/api-reference/workflow-api/register-lifecycle-hooks.mdx b/docs/content/docs/v5/api-reference/workflow-api/register-lifecycle-hooks.mdx new file mode 100644 index 0000000000..29cd4e2a33 --- /dev/null +++ b/docs/content/docs/v5/api-reference/workflow-api/register-lifecycle-hooks.mdx @@ -0,0 +1,75 @@ +--- +title: registerLifecycleHooks +description: Register global handlers that observe workflow runs completing or failing. +type: reference +summary: Use registerLifecycleHooks to observe run completions and failures from one central place. +prerequisites: + - /docs/foundations/workflows-and-steps +related: + - /docs/observability/lifecycle-hooks + - /docs/api-reference/workflow-api/get-run +--- + +Registers global workflow lifecycle handlers, invoked by the runtime on the compute that records a run's terminal transition. Useful for centralized reporting — for example forwarding every failed run to Sentry — without wrapping each workflow body. + +Register early in the process lifecycle (in Next.js, `instrumentation.ts`) so handlers exist before the first run finishes. See the [lifecycle hooks guide](/docs/observability/lifecycle-hooks) for semantics and a full Sentry example. + +```typescript title="instrumentation.ts" lineNumbers +import { registerLifecycleHooks } from "workflow/api"; + +export function register() { + if (process.env.NEXT_RUNTIME === "nodejs") { + registerLifecycleHooks({ + async onRunCompleted({ run }) { + console.log(`Run ${run.runId} completed`); + }, + async onRunFailed({ run, error }) { + console.error(`Run ${run.runId} failed (${error.errorCode})`, error.cause); + }, + }); + } +} +``` + +## API Signature + +### Parameters + + + +### Returns + +Returns a function that unregisters these hooks. + +## Handlers + +Both handlers receive the run as a lazily-hydrated [`Run`](/docs/api-reference/workflow-api/get-run) instance — accessors like `run.workflowName` and `run.returnValue` only fetch from the backend when used. + +### `onRunCompleted` + +Invoked when a workflow run completes successfully. + +| Parameter | Type | Description | +| --- | --- | --- | +| `params.run` | `Run` | The completed run. | + +### `onRunFailed` + +Invoked when a workflow run fails terminally (after any retries). + +| Parameter | Type | Description | +| --- | --- | --- | +| `params.run` | `Run` | The failed run. | +| `params.error` | `WorkflowRunFailedError` | The failure, in the same shape `run.returnValue` rejects with: `error.errorCode` carries the classification (e.g. `USER_ERROR`) and `error.cause` is the hydrated thrown value. | + +## Behavior + +- Handlers run on the host (full Node.js), never inside the workflow VM. Calling `registerLifecycleHooks` from workflow code throws. +- Handlers are fire-and-forget: they cannot delay or change the run's outcome, and a throwing handler is logged and swallowed. On serverless platforms the invocation is kept alive via `waitUntil`. +- Handlers fire only on the invocation that wrote the terminal event. Transitions recorded outside your app's compute (e.g. a run cancelled from the CLI or dashboard) do not fire handlers. +- Multiple registrations are allowed; handlers run in registration order. diff --git a/docs/content/docs/v5/observability/lifecycle-hooks.mdx b/docs/content/docs/v5/observability/lifecycle-hooks.mdx new file mode 100644 index 0000000000..64c25406af --- /dev/null +++ b/docs/content/docs/v5/observability/lifecycle-hooks.mdx @@ -0,0 +1,84 @@ +--- +title: Lifecycle Hooks +description: Register global handlers that observe workflow runs completing or failing, for centralized reporting to services like Sentry. +type: guide +summary: Observe run completions and failures from a single place with registerLifecycleHooks. +prerequisites: + - /docs/foundations/workflows-and-steps +related: + - /docs/observability + - /docs/observability/tracing + - /docs/errors +--- + +Some failures never reach a `try/catch` in your workflow code — the run can fail in the runtime itself, after your workflow function has already suspended (for example when a replay times out, or a run exhausts its queue deliveries). Lifecycle hooks give you one place to observe every terminal transition, whatever its cause: register global handlers once, and the runtime invokes them whenever it records a run completing or failing. + +The most common use is centralized error reporting — forwarding every failed run to a service like Sentry without wrapping each workflow body. + +## Registering hooks + +Call `registerLifecycleHooks` from `workflow/api` early in your application's lifecycle, so the handlers exist before the first run finishes. In Next.js, [`instrumentation.ts`](https://nextjs.org/docs/app/building-your-application/optimizing/instrumentation) is the natural place; in any other app, any module that loads at startup works. + +```typescript title="instrumentation.ts" lineNumbers +import { registerLifecycleHooks } from "workflow/api" + +export function register() { + if (process.env.NEXT_RUNTIME === "nodejs") { + registerLifecycleHooks({ + async onRunCompleted({ run }) { + console.log(`Run ${run.runId} completed`) + }, + async onRunFailed({ run, error }) { + console.error( + `Run ${run.runId} failed with ${error.errorCode}:`, + error.cause + ) + }, + }) + } +} +``` + +`registerLifecycleHooks` returns an unregister function, and multiple registrations are allowed — handlers run in registration order. + +## Handler parameters + +Both handlers receive the [`Run`](/docs/api-reference/workflow-api/get-run) instance for the transitioned run. The instance hydrates lazily: accessors like `run.workflowName` or `run.returnValue` only fetch from the backend when the handler actually uses them, so a handler that filters on cheap metadata pays nothing for the runs it ignores. + +`onRunFailed` additionally receives the failure as a `WorkflowRunFailedError` — the same shape `run.returnValue` rejects with: + +- `error.errorCode` — the failure classification (`USER_ERROR`, `RUNTIME_ERROR`, `MAX_DELIVERIES_EXCEEDED`, …). See [error codes](/docs/errors) for the full list. +- `error.cause` — the hydrated thrown value, with Error subclass identity, message, stack, and cause chain preserved. Any JavaScript value can be thrown, so this is typed `unknown`. + +## Reporting failed runs to Sentry + +```typescript title="instrumentation.ts" lineNumbers +import * as Sentry from "@sentry/nextjs" +import { registerLifecycleHooks } from "workflow/api" + +export function register() { + if (process.env.NEXT_RUNTIME === "nodejs") { + Sentry.init({ dsn: process.env.SENTRY_DSN }) + + registerLifecycleHooks({ + async onRunFailed({ run, error }) { + Sentry.captureException(error.cause ?? error, { + tags: { + workflowRunId: run.runId, + workflowName: await run.workflowName, + errorCode: error.errorCode, + }, + }) + await Sentry.flush(2000) + }, + }) + } +} +``` + +## Semantics + +- **Host-only.** Handlers run with full Node.js access, never inside the workflow's sandboxed VM. Calling `registerLifecycleHooks` from workflow code throws. +- **Fire-and-forget.** Handlers cannot delay or change the run's outcome. A throwing handler is logged and swallowed; the remaining handlers still run. On serverless platforms the invocation is kept alive via `waitUntil` while handlers finish. +- **Fires where the transition is recorded.** Handlers fire on the compute that actually wrote the terminal event — for a failure that means after any retries are exhausted, exactly once per run under normal operation. Terminal transitions recorded outside your app's compute do **not** fire handlers: cancelling a run from the CLI or the Vercel dashboard, for example, is written by the backend, so no handler runs. For a complete record of every transition, consume the [event log](/docs/how-it-works/event-sourcing) or set up alerts on the [observability](/docs/observability) surface instead. +- **Register everywhere your workflows run.** The terminal write can happen in any function invocation that processes the run's queue messages, so registration must run at startup in every instance of the app (which `instrumentation.ts` guarantees). diff --git a/docs/content/docs/v5/observability/meta.json b/docs/content/docs/v5/observability/meta.json index 617b3eeff4..c94b9f648d 100644 --- a/docs/content/docs/v5/observability/meta.json +++ b/docs/content/docs/v5/observability/meta.json @@ -1,4 +1,4 @@ { "title": "Observability", - "pages": ["tracing", "attributes"] + "pages": ["tracing", "attributes", "lifecycle-hooks"] } diff --git a/packages/core/e2e/e2e.test.ts b/packages/core/e2e/e2e.test.ts index 52867c4e4b..5471ed8323 100644 --- a/packages/core/e2e/e2e.test.ts +++ b/packages/core/e2e/e2e.test.ts @@ -3456,6 +3456,78 @@ describe('e2e', () => { } ); + // Lifecycle hooks (`registerLifecycleHooks`) are registered in the Next.js + // workbenches' instrumentation.ts (see lifecycle-hooks-e2e.ts there). The + // handlers report each lifecycleHookTarget* run's terminal transition by + // resuming the lifecycleHookObserver workflow's hook — a durable channel + // that works even when the terminal write happens on a different instance + // than the one serving these HTTP requests. + describe.skipIf(!isNextJsApp)('lifecycle hooks', () => { + test( + 'onRunCompleted receives the Run and can read its return value', + { timeout: 90_000 }, + async () => { + const token = `lifecycle-completed-${Math.random().toString(36).slice(2)}`; + + const observer = await start(await e2e('lifecycleHookObserver'), [ + token, + ]); + await waitForHook(token, { runId: observer.runId }); + + const target = await start(await e2e('lifecycleHookTargetCompleted'), [ + token, + ]); + await expect(target.returnValue).resolves.toMatchObject({ + outcome: 'completed', + }); + + // The onRunCompleted handler fetched the target's workflowName and + // returnValue off the lazily-hydrated Run instance, then resumed the + // observer's hook with what it saw. + const payload = await observer.returnValue; + expect(payload).toMatchObject({ + observed: 'completed', + runId: target.runId, + workflowName: expect.stringContaining('lifecycleHookTargetCompleted'), + returnedOutcome: 'completed', + }); + } + ); + + test( + 'onRunFailed receives the hydrated error with errorCode and cause', + { timeout: 90_000 }, + async () => { + const token = `lifecycle-failed-${Math.random().toString(36).slice(2)}`; + + const observer = await start(await e2e('lifecycleHookObserver'), [ + token, + ]); + await waitForHook(token, { runId: observer.runId }); + + const target = await start(await e2e('lifecycleHookTargetFailed'), [ + token, + ]); + const error = await target.returnValue.catch((e: unknown) => e); + expect(WorkflowRunFailedError.is(error)).toBe(true); + + // The onRunFailed handler received a WorkflowRunFailedError whose + // errorCode carries the classification and whose cause is the + // hydrated thrown FatalError (name + message preserved). + const payload = await observer.returnValue; + expect(payload).toMatchObject({ + observed: 'failed', + runId: target.runId, + errorCode: 'USER_ERROR', + causeName: 'FatalError', + causeMessage: expect.stringContaining( + `lifecycle-hook-target-failed:${token}` + ), + }); + } + ); + }); + test( 'hookWithSleepWorkflow - hook payloads delivered correctly with concurrent sleep', { timeout: 90_000 }, diff --git a/packages/core/package.json b/packages/core/package.json index 4c0ceca24e..6b1a597545 100644 --- a/packages/core/package.json +++ b/packages/core/package.json @@ -36,6 +36,10 @@ "types": "./dist/runtime/run.d.ts", "default": "./dist/runtime/run.js" }, + "./runtime/lifecycle-hooks": { + "types": "./dist/runtime/lifecycle-hooks.d.ts", + "default": "./dist/runtime/lifecycle-hooks.js" + }, "./runtime/start": { "types": "./dist/runtime/start.d.ts", "default": "./dist/runtime/start.js" diff --git a/packages/core/src/runtime.ts b/packages/core/src/runtime.ts index fa54bcd167..4a2cb9fd59 100644 --- a/packages/core/src/runtime.ts +++ b/packages/core/src/runtime.ts @@ -87,6 +87,10 @@ import { stepDispatchIdempotencyKey, withHealthCheck, } from './runtime/helpers.js'; +import { + dispatchRunCompletedHooks, + dispatchRunFailedHooks, +} from './runtime/lifecycle-hooks.js'; import { handleReplayBudgetExhausted, ReplayBudget, @@ -429,6 +433,7 @@ async function recordFatalRunError({ } throw failErr; } + dispatchRunFailedHooks(runId, err, errorCode); } function hasRecordedTerminalRunEvent(events: Event[], runId: string): boolean { @@ -761,6 +766,11 @@ export function workflowEntrypoint( }, { requestId } ); + dispatchRunFailedHooks( + runId, + err, + RUN_ERROR_CODES.MAX_DELIVERIES_EXCEEDED + ); } catch (err) { if (EntityConflictError.is(err) || RunExpiredError.is(err)) { // Run already finished, consume the message silently @@ -3025,6 +3035,7 @@ export function workflowEntrypoint( } throw err; } + dispatchRunCompletedHooks(runId); span?.setAttributes({ ...Attribute.WorkflowRunStatus('completed'), @@ -3233,6 +3244,11 @@ export function workflowEntrypoint( } throw failErr; } + dispatchRunFailedHooks( + runId, + suspensionError, + errorCode + ); span?.setAttributes({ ...Attribute.WorkflowRunStatus('failed'), ...Attribute.WorkflowErrorCode(errorCode), @@ -4559,6 +4575,7 @@ export function workflowEntrypoint( } throw failErr; } + dispatchRunFailedHooks(runId, terminalError, errorCode); span?.setAttributes({ ...Attribute.WorkflowRunStatus('failed'), diff --git a/packages/core/src/runtime/deployment-guard.ts b/packages/core/src/runtime/deployment-guard.ts index 99f6c059df..de813d5486 100644 --- a/packages/core/src/runtime/deployment-guard.ts +++ b/packages/core/src/runtime/deployment-guard.ts @@ -14,6 +14,7 @@ import { runtimeLogger } from '../logger.js'; import { dehydrateRunError } from '../serialization.js'; import * as Attribute from '../telemetry/semantic-conventions.js'; import { getDeploymentMismatchMaxRetries } from './constants.js'; +import { dispatchRunFailedHooks } from './lifecycle-hooks.js'; /** Cap on the re-route backoff, in seconds. */ const MAX_REROUTE_DELAY_SECONDS = 8; @@ -212,6 +213,11 @@ export async function guardDeploymentAffinity({ }, { requestId } ); + dispatchRunFailedHooks( + run.runId, + error, + RUN_ERROR_CODES.DEPLOYMENT_MISMATCH + ); } catch (failError) { // Run already reached a terminal state (a concurrent writer failed it, or // it was cancelled/expired) — still stop. Anything else is a transient diff --git a/packages/core/src/runtime/lifecycle-hooks.test.ts b/packages/core/src/runtime/lifecycle-hooks.test.ts new file mode 100644 index 0000000000..7ad095bb46 --- /dev/null +++ b/packages/core/src/runtime/lifecycle-hooks.test.ts @@ -0,0 +1,195 @@ +import { WorkflowRunFailedError } from '@workflow/errors'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { + dispatchRunCompletedHooks, + dispatchRunFailedHooks, + registerLifecycleHooks, + type WorkflowLifecycleHooks, +} from './lifecycle-hooks.js'; +import { Run } from './run.js'; + +vi.mock('../version.js', () => ({ version: '0.0.0-test' })); + +// Capture every promise handed to waitUntil so tests can await the +// fire-and-forget dispatch work deterministically. +const waitUntilPromises: Promise[] = []; +vi.mock('@vercel/functions', () => ({ + waitUntil: (promise: Promise) => { + waitUntilPromises.push(promise); + }, +})); + +/** Await everything the dispatcher scheduled through waitUntil. */ +async function flushDispatches(): Promise { + // The dispatcher resolves a dynamic import before handing the promise to + // waitUntil, so yield to the microtask queue until the capture settles. + for (let i = 0; i < 10 && waitUntilPromises.length === 0; i++) { + await new Promise((resolve) => setImmediate(resolve)); + } + await Promise.all(waitUntilPromises); +} + +describe('lifecycle hooks', () => { + const unregisters: Array<() => void> = []; + + const register = (hooks: WorkflowLifecycleHooks) => { + const unregister = registerLifecycleHooks(hooks); + unregisters.push(unregister); + return unregister; + }; + + beforeEach(() => { + waitUntilPromises.length = 0; + }); + + afterEach(() => { + for (const unregister of unregisters) { + unregister(); + } + unregisters.length = 0; + }); + + it('invokes onRunCompleted with a lazily-hydrated Run instance', async () => { + const onRunCompleted = vi.fn(); + register({ onRunCompleted }); + + dispatchRunCompletedHooks('wrun_completed_1'); + await flushDispatches(); + + expect(onRunCompleted).toHaveBeenCalledTimes(1); + const { run } = onRunCompleted.mock.calls[0][0]; + expect(run).toBeInstanceOf(Run); + expect(run.runId).toBe('wrun_completed_1'); + }); + + it('invokes onRunFailed with the Run and a WorkflowRunFailedError carrying errorCode and cause', async () => { + const onRunFailed = vi.fn(); + register({ onRunFailed }); + + const cause = new Error('workflow exploded'); + dispatchRunFailedHooks('wrun_failed_1', cause, 'USER_ERROR'); + await flushDispatches(); + + expect(onRunFailed).toHaveBeenCalledTimes(1); + const { run, error } = onRunFailed.mock.calls[0][0]; + expect(run).toBeInstanceOf(Run); + expect(run.runId).toBe('wrun_failed_1'); + expect(WorkflowRunFailedError.is(error)).toBe(true); + expect(error.runId).toBe('wrun_failed_1'); + expect(error.errorCode).toBe('USER_ERROR'); + // The cause is the round-tripped (dehydrate → hydrate) value, not the + // original reference: handlers always see the host-realm hydrated shape. + expect(error.cause).toBeInstanceOf(Error); + expect((error.cause as Error).message).toBe('workflow exploded'); + expect(error.message).toContain('workflow exploded'); + }); + + it('hydrates a VM-realm thrown error into a host-realm Error for handlers', async () => { + const onRunFailed = vi.fn(); + register({ onRunFailed }); + + // Simulate a workflow-VM thrown error: a real native error from another + // realm, for which host `instanceof Error` is false. + const { runInNewContext } = await import('node:vm'); + const vmError = runInNewContext( + 'const e = new Error("vm exploded"); e.name = "FatalError"; e' + ); + expect(vmError instanceof Error).toBe(false); + + dispatchRunFailedHooks('wrun_vm_realm', vmError, 'USER_ERROR'); + await flushDispatches(); + + const { error } = onRunFailed.mock.calls[0][0]; + expect(error.cause).toBeInstanceOf(Error); + expect((error.cause as Error).name).toBe('FatalError'); + expect((error.cause as Error).message).toBe('vm exploded'); + }); + + it('does not schedule any work when no hooks are registered', async () => { + dispatchRunCompletedHooks('wrun_none'); + dispatchRunFailedHooks('wrun_none', new Error('x'), 'USER_ERROR'); + // Give a potential (buggy) schedule a chance to land. + await new Promise((resolve) => setImmediate(resolve)); + expect(waitUntilPromises).toHaveLength(0); + }); + + it('invokes multiple registrations in registration order', async () => { + const order: string[] = []; + register({ onRunCompleted: () => void order.push('first') }); + register({ + onRunCompleted: async () => { + // Async handler: the next handler must still wait for it. + await new Promise((resolve) => setTimeout(resolve, 5)); + order.push('second'); + }, + }); + register({ onRunCompleted: () => void order.push('third') }); + + dispatchRunCompletedHooks('wrun_order'); + await flushDispatches(); + + expect(order).toEqual(['first', 'second', 'third']); + }); + + it('swallows a throwing handler and still runs later handlers', async () => { + const later = vi.fn(); + register({ + onRunFailed: () => { + throw new Error('sync handler boom'); + }, + }); + register({ + onRunFailed: async () => { + throw new Error('async handler boom'); + }, + }); + register({ onRunFailed: later }); + + dispatchRunFailedHooks('wrun_boom', new Error('cause'), 'USER_ERROR'); + // Must not reject (safeWaitUntil relies on the promise never rejecting). + await expect(Promise.all(waitUntilPromises)).resolves.toBeDefined(); + await flushDispatches(); + + expect(later).toHaveBeenCalledTimes(1); + }); + + it('unregister removes the hooks', async () => { + const onRunCompleted = vi.fn(); + const unregister = registerLifecycleHooks({ onRunCompleted }); + unregister(); + + dispatchRunCompletedHooks('wrun_unregistered'); + await new Promise((resolve) => setImmediate(resolve)); + + expect(onRunCompleted).not.toHaveBeenCalled(); + expect(waitUntilPromises).toHaveLength(0); + }); + + it('shares one registry across module copies via the Symbol.for global', async () => { + const onRunCompleted = vi.fn(); + register({ onRunCompleted }); + + const registry = (globalThis as Record)[ + Symbol.for('@workflow/core//lifecycleHooks') + ] as WorkflowLifecycleHooks[]; + expect(Array.isArray(registry)).toBe(true); + expect(registry.some((h) => h.onRunCompleted === onRunCompleted)).toBe( + true + ); + }); + + it('non-Error thrown values round-trip through WorkflowRunFailedError.cause', async () => { + const onRunFailed = vi.fn(); + register({ onRunFailed }); + + const thrown = { kind: 'business-rule-violation', code: 'LOCKED' }; + dispatchRunFailedHooks('wrun_nonerror', thrown, 'USER_ERROR'); + await flushDispatches(); + + const { error } = onRunFailed.mock.calls[0][0]; + // Structural clone via the serialization round-trip, not coerced to an + // Error. + expect(error.cause).not.toBeInstanceOf(Error); + expect(error.cause).toEqual(thrown); + }); +}); diff --git a/packages/core/src/runtime/lifecycle-hooks.ts b/packages/core/src/runtime/lifecycle-hooks.ts new file mode 100644 index 0000000000..ead41de769 --- /dev/null +++ b/packages/core/src/runtime/lifecycle-hooks.ts @@ -0,0 +1,218 @@ +import { WorkflowRunFailedError } from '@workflow/errors'; +import { runtimeLogger } from '../logger.js'; +import { dehydrateRunError, hydrateRunError } from '../serialization.js'; +import { Run } from './run.js'; +import { safeWaitUntil } from './wait-until.js'; + +/** + * Parameters passed to an {@link WorkflowLifecycleHooks.onRunCompleted} + * handler. + */ +export interface RunCompletedHookParams { + /** + * The completed run. The instance hydrates lazily, so reading + * `run.returnValue` (or any other accessor) fetches from the backend only + * when the handler actually uses it. + */ + run: Run; +} + +/** + * Parameters passed to an {@link WorkflowLifecycleHooks.onRunFailed} + * handler. + */ +export interface RunFailedHookParams { + /** + * The failed run. The instance hydrates lazily, so accessors fetch from + * the backend only when the handler actually uses them. + */ + run: Run; + /** + * The failure, in the same shape `run.returnValue` rejects with: a + * `WorkflowRunFailedError` whose `errorCode` carries the failure + * classification (e.g. `USER_ERROR`, `RUNTIME_ERROR`) and whose `cause` is + * the hydrated thrown value (original Error subclass identity preserved). + */ + error: WorkflowRunFailedError; +} + +/** + * Global handlers observing workflow run lifecycle transitions. Register via + * {@link registerLifecycleHooks}. + */ +export interface WorkflowLifecycleHooks { + /** Invoked when a workflow run completes successfully. */ + onRunCompleted?: (params: RunCompletedHookParams) => void | Promise; + /** Invoked when a workflow run fails terminally (after any retries). */ + onRunFailed?: (params: RunFailedHookParams) => void | Promise; +} + +/** + * The registry lives on `globalThis` under a `Symbol.for` key so that every + * copy of `@workflow/core` in the process (bundled + unbundled, ESM + CJS) + * shares one list — same pattern as the cross-realm error-class registry in + * `@workflow/errors` and the World cache in `get-world-lazy.ts`. The property + * is non-writable/non-configurable so accidental clobbering is loud; the + * array's contents stay mutable for register/unregister. + */ +const REGISTRY_KEY = Symbol.for('@workflow/core//lifecycleHooks'); + +function getRegistry(): WorkflowLifecycleHooks[] { + if (!Object.hasOwn(globalThis, REGISTRY_KEY)) { + Object.defineProperty(globalThis, REGISTRY_KEY, { + value: [], + writable: false, + enumerable: false, + configurable: false, + }); + } + return (globalThis as Record)[ + REGISTRY_KEY + ] as WorkflowLifecycleHooks[]; +} + +/** + * Registers global workflow lifecycle handlers, invoked by the runtime on + * the compute that records a run's terminal transition. Useful for + * centralized reporting — e.g. forwarding failed runs to Sentry — without + * wrapping every workflow body. + * + * Register early in the process lifecycle so handlers exist before the first + * run finishes: in Next.js, `instrumentation.ts` is the natural place; in any + * other app, any module that loads at startup works. + * + * Semantics: + * - Handlers run on the host (full Node.js), never inside the workflow VM. + * - Handlers fire only on the invocation that actually wrote the terminal + * event. Transitions recorded elsewhere — e.g. a run cancelled from the + * CLI or dashboard — do not fire handlers in the app. + * - Handlers are fire-and-forget: they cannot delay or change the run's + * outcome, and a throwing handler is logged and swallowed. On serverless + * platforms the invocation is kept alive via `waitUntil`. + * - Multiple registrations are allowed; handlers run in registration order. + * + * @returns A function that unregisters these hooks. + */ +export function registerLifecycleHooks( + hooks: WorkflowLifecycleHooks +): () => void { + const registry = getRegistry(); + registry.push(hooks); + return () => { + const index = registry.indexOf(hooks); + if (index !== -1) { + registry.splice(index, 1); + } + }; +} + +/** + * Runs every registered handler for one lifecycle transition without ever + * throwing into (or blocking) the runtime's terminal-write path: the work is + * scheduled through `safeWaitUntil`, the params are prepared at most once + * per transition, each handler's failure is logged and swallowed + * individually, and handlers run sequentially in registration order. + */ +function dispatch( + runId: string, + event: 'onRunCompleted' | 'onRunFailed', + prepare: () => Promise, + invoke: ( + hooks: WorkflowLifecycleHooks, + params: TParams + ) => void | Promise | undefined +): void { + // Snapshot so an unregister inside a handler cannot skew iteration. + const registered = [...getRegistry()]; + if (registered.length === 0) { + return; + } + safeWaitUntil( + (async () => { + const params = await prepare(); + for (const hooks of registered) { + try { + await invoke(hooks, params); + } catch (err) { + runtimeLogger.error(`Workflow lifecycle ${event} handler threw`, { + workflowRunId: runId, + error: err instanceof Error ? err.message : String(err), + }); + } + } + })(), + // Covers a `prepare()` rejection; handler failures are caught above. + (err) => { + runtimeLogger.error(`Workflow lifecycle ${event} dispatch failed`, { + workflowRunId: runId, + error: err instanceof Error ? err.message : String(err), + }); + } + ); +} + +/** + * Called by the runtime after it successfully wrote a `run_completed` event. + * Never throws. + */ +export function dispatchRunCompletedHooks(runId: string): void { + dispatch( + runId, + 'onRunCompleted', + async () => ({ run: new Run(runId) }), + (hooks, params) => hooks.onRunCompleted?.(params) + ); +} + +/** + * The thrown value a `run_failed` writer holds is often a VM-realm object + * (the workflow runs in a separate realm, so `instanceof Error` on it is + * `false` for handlers) and may carry VM-realm exotics in its cause chain. + * Round-trip it through the run-error serialization pipeline so handlers + * receive the same host-realm hydrated shape `run.returnValue` rejects with + * — real host Error instances with name/message/stack/cause preserved and + * registered classes (FatalError, custom serde classes) revived with their + * class identity. No encryption: the bytes never leave this process. + * + * Falls back to the original value when the round-trip fails — a degraded + * report beats no report. + */ +async function hydrateForHandlers( + error: unknown, + runId: string +): Promise { + try { + const bytes = await dehydrateRunError(error, runId, undefined); + return await hydrateRunError(bytes, runId, undefined); + } catch { + return error; + } +} + +/** + * Called by the runtime after it successfully wrote a `run_failed` event. + * Never throws. + * + * @param error - The thrown value the terminal write recorded (host-side + * object where available; QuickJS passes its rehydrated reconstruction). + * @param errorCode - The classification written to the event's `errorCode`. + */ +export function dispatchRunFailedHooks( + runId: string, + error: unknown, + errorCode: string +): void { + dispatch( + runId, + 'onRunFailed', + async () => ({ + run: new Run(runId), + error: new WorkflowRunFailedError( + runId, + await hydrateForHandlers(error, runId), + { errorCode } + ), + }), + (hooks, params) => hooks.onRunFailed?.(params) + ); +} diff --git a/packages/core/src/runtime/quickjs-entrypoint.ts b/packages/core/src/runtime/quickjs-entrypoint.ts index 238d34752d..fba5bf6465 100644 --- a/packages/core/src/runtime/quickjs-entrypoint.ts +++ b/packages/core/src/runtime/quickjs-entrypoint.ts @@ -57,6 +57,10 @@ import { queueMessage, stepDispatchIdempotencyKey, } from './helpers.js'; +import { + dispatchRunCompletedHooks, + dispatchRunFailedHooks, +} from './lifecycle-hooks.js'; import { BASELINE_BUNDLE_FILENAME, type PendingAttribute, @@ -1599,6 +1603,7 @@ export async function runWorkflowWithQuickJS(params: { }, }); wfdiag('exit_completed', { result: 'run_completed_written' }); + dispatchRunCompletedHooks(runId); } catch (err) { if (EntityConflictError.is(err) || RunExpiredError.is(err)) { runtimeLogger.warn( @@ -1837,6 +1842,11 @@ export async function runWorkflowWithQuickJS(params: { // `dehydrateRunError`. Used when valueBytes is absent (e.g. // extractError pseudo-failures from VM bootstrap). let dehydratedError: Uint8Array; + // The most faithful host-side error value available, handed to the + // lifecycle onRunFailed hooks after the terminal write lands: the + // hydrated VM value when the modern path succeeds, otherwise the + // reconstructed host Error. + let lifecycleError: unknown = reconstructed; if (result.failed.valueBytes) { // Hydrate the VM-side bytes, remap the error stack with the // host-side source map (the VM can't do this — it lacks both the @@ -1891,6 +1901,7 @@ export async function runWorkflowWithQuickJS(params: { runId, encryptionKey ); + lifecycleError = hydrated; } catch (rehydrateErr) { // If hydration / re-dehydration fails for any reason, fall // back to passing through the original VM bytes (just apply @@ -1957,6 +1968,7 @@ export async function runWorkflowWithQuickJS(params: { }); throw err; } + dispatchRunFailedHooks(runId, lifecycleError, errorCode); wfdiag('exit_failed', { result: 'run_failed_written' }); } } diff --git a/packages/core/src/runtime/replay-budget.test.ts b/packages/core/src/runtime/replay-budget.test.ts index ee0acf44f3..4bc33fa9a5 100644 --- a/packages/core/src/runtime/replay-budget.test.ts +++ b/packages/core/src/runtime/replay-budget.test.ts @@ -1,6 +1,7 @@ import type { World } from '@workflow/world'; import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import { runtimeLogger } from '../logger.js'; +import { registerLifecycleHooks } from './lifecycle-hooks.js'; import { handleReplayBudgetExhausted, ReplayBudget, @@ -12,7 +13,11 @@ vi.mock('./world.js', () => ({ getWorld: vi.fn(), })); -vi.mock('../serialization.js', () => ({ +// Partial mock: the lifecycle-hook registry pulls in `run.ts` (for the Run +// instance handed to handlers), whose import chain needs the real module's +// other exports (e.g. SerializationFormat). +vi.mock(import('../serialization.js'), async (importOriginal) => ({ + ...(await importOriginal()), dehydrateRunError: vi.fn(async () => new Uint8Array([1, 2, 3])), })); @@ -20,6 +25,27 @@ vi.mock('./helpers.js', () => ({ memoizeEncryptionKey: () => async () => undefined, })); +// Capture lifecycle-hook dispatch work (scheduled via waitUntil) so tests +// can await it deterministically. +const waitUntilPromises: Promise[] = []; +vi.mock('@vercel/functions', () => ({ + waitUntil: (promise: Promise) => { + waitUntilPromises.push(promise); + }, +})); + +/** + * Await everything the lifecycle dispatcher scheduled through waitUntil. + * The dispatcher resolves a dynamic import before handing the promise to + * waitUntil, so yield to the macrotask queue until the capture lands. + */ +async function flushLifecycleDispatches(): Promise { + for (let i = 0; i < 10 && waitUntilPromises.length === 0; i++) { + await new Promise((resolve) => setImmediate(resolve)); + } + await Promise.all(waitUntilPromises); +} + describe('ReplayBudget', () => { beforeEach(() => { vi.useFakeTimers(); @@ -238,4 +264,49 @@ describe('handleReplayBudgetExhausted', () => { expect(exitSpy).not.toHaveBeenCalled(); }); + + it('fires onRunFailed lifecycle hooks after the terminal write lands, and not on write failure', async () => { + const onRunFailed = vi.fn(); + const unregister = registerLifecycleHooks({ onRunFailed }); + try { + // Write failure: no dispatch. + mockEventsCreate.mockRejectedValueOnce(new Error('storage unavailable')); + vi.mocked(getWorld).mockResolvedValue(makeMockWorld()); + await expect( + handleReplayBudgetExhausted({ + runId: 'wrun_test', + workflowName: 'wf', + requestId: 'req_test', + attempt: 4, + limitMs: 240_000, + }) + ).rejects.toThrow('storage unavailable'); + // Give a (buggy) schedule a chance to land before asserting none did. + await new Promise((resolve) => setImmediate(resolve)); + expect(waitUntilPromises).toHaveLength(0); + expect(onRunFailed).not.toHaveBeenCalled(); + + // Successful write: dispatch with the Run and classified error. + await handleReplayBudgetExhausted({ + runId: 'wrun_test', + workflowName: 'wf', + requestId: 'req_test', + attempt: 4, + limitMs: 240_000, + }); + await flushLifecycleDispatches(); + + expect(onRunFailed).toHaveBeenCalledTimes(1); + const { run, error } = onRunFailed.mock.calls[0][0]; + expect(run.runId).toBe('wrun_test'); + expect(error.errorCode).toBe('REPLAY_TIMEOUT'); + expect(error.cause).toBeInstanceOf(Error); + expect((error.cause as Error).message).toContain( + 'exceeded maximum duration' + ); + } finally { + unregister(); + waitUntilPromises.length = 0; + } + }); }); diff --git a/packages/core/src/runtime/replay-budget.ts b/packages/core/src/runtime/replay-budget.ts index 53f70f3b1f..60c58e9222 100644 --- a/packages/core/src/runtime/replay-budget.ts +++ b/packages/core/src/runtime/replay-budget.ts @@ -5,6 +5,7 @@ import { runtimeLogger } from '../logger.js'; import { dehydrateRunError } from '../serialization.js'; import { getReplayTimeoutMaxRetries, getReplayTimeoutMs } from './constants.js'; import { memoizeEncryptionKey, type SlotSnapshotParams } from './helpers.js'; +import { dispatchRunFailedHooks } from './lifecycle-hooks.js'; import { getWorld } from './world.js'; /** @@ -180,4 +181,5 @@ export async function handleReplayBudgetExhausted(args: { }, { requestId, ...slotSnapshot } ); + dispatchRunFailedHooks(runId, timeoutErr, RUN_ERROR_CODES.REPLAY_TIMEOUT); } diff --git a/packages/workflow/src/api-workflow.ts b/packages/workflow/src/api-workflow.ts index a489fd929d..a1eec8a7b0 100644 --- a/packages/workflow/src/api-workflow.ts +++ b/packages/workflow/src/api-workflow.ts @@ -22,3 +22,10 @@ export const getHookByToken = () => workflowStub('getHookByToken'); export const resumeHook = () => workflowStub('resumeHook'); export const resumeWebhook = () => workflowStub('resumeWebhook'); export const runStep = () => workflowStub('runStep'); +export const registerLifecycleHooks = () => + workflowStub('registerLifecycleHooks'); +export type { + RunCompletedHookParams, + RunFailedHookParams, + WorkflowLifecycleHooks, +} from '@workflow/core/runtime/lifecycle-hooks'; diff --git a/packages/workflow/src/api.ts b/packages/workflow/src/api.ts index 31ca6ed146..12ab004256 100644 --- a/packages/workflow/src/api.ts +++ b/packages/workflow/src/api.ts @@ -14,6 +14,12 @@ export type { StopSleepResult, WorkflowRun, } from '@workflow/core/runtime'; +export { + type RunCompletedHookParams, + type RunFailedHookParams, + registerLifecycleHooks, + type WorkflowLifecycleHooks, +} from '@workflow/core/runtime/lifecycle-hooks'; export { getHookByToken, type ResumedHook, diff --git a/workbench/example/workflows/99_e2e.ts b/workbench/example/workflows/99_e2e.ts index 28a4ccb1cf..d84f1bad3c 100644 --- a/workbench/example/workflows/99_e2e.ts +++ b/workbench/example/workflows/99_e2e.ts @@ -3809,3 +3809,44 @@ export async function crossRegionStreamWorkflow(chunkCount: number) { await closeCrossRegionStream(writable); return 'done'; } + +// ============================================================ +// LIFECYCLE HOOK TESTS +// Exercised only by the Next.js workbenches, whose +// instrumentation.ts registers `registerLifecycleHooks` handlers +// (see workbench/nextjs-*/instrumentation.ts). The handlers +// observe these target runs' terminal transitions and report +// them by resuming the observer workflow's hook — a durable +// channel that works across serverless instances. +// ============================================================ + +/** + * Target: completes immediately. The `onRunCompleted` handler reads this + * run's return value (exercising the Run instance's lazy hydration) to + * discover the observer's hook token. + */ +export async function lifecycleHookTargetCompleted(token: string) { + 'use workflow'; + return { token, outcome: 'completed' }; +} + +/** + * Target: fails immediately. The token is embedded in the thrown error's + * message so the `onRunFailed` handler can find the observer without any + * backend reads (the hydrated cause is on the WorkflowRunFailedError it + * receives). + */ +export async function lifecycleHookTargetFailed(token: string) { + 'use workflow'; + throw new FatalError(`lifecycle-hook-target-failed:${token}`); +} + +/** + * Observer: parks on a hook until a lifecycle handler reports the target + * run's terminal transition, then returns the reported payload verbatim. + */ +export async function lifecycleHookObserver(token: string) { + 'use workflow'; + using hook = createHook>({ token }); + return await hook; +} diff --git a/workbench/nextjs-turbopack/instrumentation.ts b/workbench/nextjs-turbopack/instrumentation.ts index bd66985a97..72f71ab981 100644 --- a/workbench/nextjs-turbopack/instrumentation.ts +++ b/workbench/nextjs-turbopack/instrumentation.ts @@ -1,6 +1,12 @@ import { registerOTel } from '@vercel/otel'; +import { registerE2eLifecycleHooks } from './lifecycle-hooks-e2e'; export function register() { + if (process.env.NEXT_RUNTIME === 'nodejs') { + // Workflow lifecycle hooks are host-only; skip the edge runtime's + // instrumentation pass. + registerE2eLifecycleHooks(); + } registerOTel({ serviceName: 'nextjs-turbopack', instrumentationConfig: { diff --git a/workbench/nextjs-turbopack/lifecycle-hooks-e2e.ts b/workbench/nextjs-turbopack/lifecycle-hooks-e2e.ts new file mode 100644 index 0000000000..d7e997e766 --- /dev/null +++ b/workbench/nextjs-turbopack/lifecycle-hooks-e2e.ts @@ -0,0 +1,55 @@ +import { registerLifecycleHooks, resumeHook } from 'workflow/api'; + +/** + * E2E coverage for `registerLifecycleHooks` (see the "lifecycle hooks" + * describe in packages/core/e2e/e2e.test.ts and the fixtures in + * workflows/99_e2e.ts). + * + * The handlers observe the `lifecycleHookTarget*` workflows' terminal + * transitions and report them by resuming the `lifecycleHookObserver` + * workflow's hook. Resuming a durable hook is deliberately the observation + * channel: the handler runs on whichever instance wrote the terminal event, + * which on a deployed app is generally NOT the instance serving the e2e + * test's HTTP requests — an in-memory buffer would not travel. + */ +export function registerE2eLifecycleHooks(): void { + registerLifecycleHooks({ + async onRunCompleted({ run }) { + // Fires for every completed run in the app, so filter cheaply by + // workflow name (a metadata read) before touching the return value. + const workflowName = await run.workflowName; + if (!workflowName?.includes('lifecycleHookTargetCompleted')) { + return; + } + // Lazy hydration: the return value is only fetched for matching runs. + const returnValue = (await run.returnValue) as { + token: string; + outcome: string; + }; + await resumeHook(returnValue.token, { + observed: 'completed', + runId: run.runId, + workflowName, + returnedOutcome: returnValue.outcome, + }); + }, + async onRunFailed({ run, error }) { + // The hydrated thrown value is already on the error — no backend + // reads needed to filter. + const cause = error.cause; + const causeMessage = + cause instanceof Error ? cause.message : String(cause); + const match = causeMessage.match(/lifecycle-hook-target-failed:(\S+)/); + if (!match) { + return; + } + await resumeHook(match[1], { + observed: 'failed', + runId: run.runId, + errorCode: error.errorCode, + causeName: cause instanceof Error ? cause.name : typeof cause, + causeMessage, + }); + }, + }); +} diff --git a/workbench/nextjs-webpack/instrumentation.ts b/workbench/nextjs-webpack/instrumentation.ts index 007ff5f971..5cfdf5132c 100644 --- a/workbench/nextjs-webpack/instrumentation.ts +++ b/workbench/nextjs-webpack/instrumentation.ts @@ -1,6 +1,12 @@ import { registerOTel } from '@vercel/otel'; +import { registerE2eLifecycleHooks } from './lifecycle-hooks-e2e'; export function register() { + if (process.env.NEXT_RUNTIME === 'nodejs') { + // Workflow lifecycle hooks are host-only; skip the edge runtime's + // instrumentation pass. + registerE2eLifecycleHooks(); + } registerOTel({ serviceName: 'nextjs-webpack', instrumentationConfig: { diff --git a/workbench/nextjs-webpack/lifecycle-hooks-e2e.ts b/workbench/nextjs-webpack/lifecycle-hooks-e2e.ts new file mode 120000 index 0000000000..ed7314a288 --- /dev/null +++ b/workbench/nextjs-webpack/lifecycle-hooks-e2e.ts @@ -0,0 +1 @@ +../nextjs-turbopack/lifecycle-hooks-e2e.ts \ No newline at end of file From c262506aa23c0cde538ead91750e982af7b208b8 Mon Sep 17 00:00:00 2001 From: Nathan Rajlich Date: Wed, 19 Aug 2026 16:28:34 -0700 Subject: [PATCH 2/4] Load lifecycle-hooks-e2e via guarded dynamic import; comment fixes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The static top-level import in instrumentation.ts pulled workflow/api → world-init → @workflow/world-local → proper-lockfile → fs into every compile target of instrumentation.ts, and the non-node ones cannot resolve fs — 500ing every request in the nextjs-webpack lanes. The NEXT_RUNTIME guard only helps at runtime; the canonical Next.js pattern is a dynamic import inside the guard so each compile target dead-code- eliminates the branch. Verified: lifecycle + pages-router e2e slices green against local nextjs-webpack and nextjs-turbopack dev servers. Also: clarify the 'describe block' doc comment and correct the microtask/macrotask wording in the test flush helper. --- packages/core/src/runtime/lifecycle-hooks.test.ts | 3 ++- workbench/nextjs-turbopack/instrumentation.ts | 12 ++++++++---- workbench/nextjs-turbopack/lifecycle-hooks-e2e.ts | 2 +- workbench/nextjs-webpack/instrumentation.ts | 12 ++++++++---- 4 files changed, 19 insertions(+), 10 deletions(-) diff --git a/packages/core/src/runtime/lifecycle-hooks.test.ts b/packages/core/src/runtime/lifecycle-hooks.test.ts index 7ad095bb46..781cf64186 100644 --- a/packages/core/src/runtime/lifecycle-hooks.test.ts +++ b/packages/core/src/runtime/lifecycle-hooks.test.ts @@ -22,7 +22,8 @@ vi.mock('@vercel/functions', () => ({ /** Await everything the dispatcher scheduled through waitUntil. */ async function flushDispatches(): Promise { // The dispatcher resolves a dynamic import before handing the promise to - // waitUntil, so yield to the microtask queue until the capture settles. + // waitUntil, so yield macrotask (check-phase) turns via setImmediate — + // which drains the intervening microtasks too — until the capture lands. for (let i = 0; i < 10 && waitUntilPromises.length === 0; i++) { await new Promise((resolve) => setImmediate(resolve)); } diff --git a/workbench/nextjs-turbopack/instrumentation.ts b/workbench/nextjs-turbopack/instrumentation.ts index 72f71ab981..9bfa71bee3 100644 --- a/workbench/nextjs-turbopack/instrumentation.ts +++ b/workbench/nextjs-turbopack/instrumentation.ts @@ -1,10 +1,14 @@ import { registerOTel } from '@vercel/otel'; -import { registerE2eLifecycleHooks } from './lifecycle-hooks-e2e'; -export function register() { +export async function register() { if (process.env.NEXT_RUNTIME === 'nodejs') { - // Workflow lifecycle hooks are host-only; skip the edge runtime's - // instrumentation pass. + // Workflow lifecycle hooks are host-only. The import MUST be dynamic + // and inside the runtime guard (the canonical Next.js pattern for + // node-only instrumentation): a static top-level import would pull + // `workflow/api` → world-init → @workflow/world-local → fs into every + // compile target of instrumentation.ts, and the non-node ones cannot + // resolve `fs` (breaks the webpack workbench's build). + const { registerE2eLifecycleHooks } = await import('./lifecycle-hooks-e2e'); registerE2eLifecycleHooks(); } registerOTel({ diff --git a/workbench/nextjs-turbopack/lifecycle-hooks-e2e.ts b/workbench/nextjs-turbopack/lifecycle-hooks-e2e.ts index d7e997e766..6ff579a742 100644 --- a/workbench/nextjs-turbopack/lifecycle-hooks-e2e.ts +++ b/workbench/nextjs-turbopack/lifecycle-hooks-e2e.ts @@ -2,7 +2,7 @@ import { registerLifecycleHooks, resumeHook } from 'workflow/api'; /** * E2E coverage for `registerLifecycleHooks` (see the "lifecycle hooks" - * describe in packages/core/e2e/e2e.test.ts and the fixtures in + * describe block in packages/core/e2e/e2e.test.ts and the fixtures in * workflows/99_e2e.ts). * * The handlers observe the `lifecycleHookTarget*` workflows' terminal diff --git a/workbench/nextjs-webpack/instrumentation.ts b/workbench/nextjs-webpack/instrumentation.ts index 5cfdf5132c..e41114618d 100644 --- a/workbench/nextjs-webpack/instrumentation.ts +++ b/workbench/nextjs-webpack/instrumentation.ts @@ -1,10 +1,14 @@ import { registerOTel } from '@vercel/otel'; -import { registerE2eLifecycleHooks } from './lifecycle-hooks-e2e'; -export function register() { +export async function register() { if (process.env.NEXT_RUNTIME === 'nodejs') { - // Workflow lifecycle hooks are host-only; skip the edge runtime's - // instrumentation pass. + // Workflow lifecycle hooks are host-only. The import MUST be dynamic + // and inside the runtime guard (the canonical Next.js pattern for + // node-only instrumentation): a static top-level import would pull + // `workflow/api` → world-init → @workflow/world-local → fs into every + // compile target of instrumentation.ts, and the non-node ones cannot + // resolve `fs` (breaks this workbench's webpack build). + const { registerE2eLifecycleHooks } = await import('./lifecycle-hooks-e2e'); registerE2eLifecycleHooks(); } registerOTel({ From 177148f78b9f506e09cae297b3f37e13d3fa873e Mon Sep 17 00:00:00 2001 From: Nathan Rajlich Date: Thu, 20 Aug 2026 12:57:04 -0700 Subject: [PATCH 3/4] Remove em dashes from docs and comments added on this branch Per the Vercel technical writing guidelines: em dashes create ambiguity for agents parsing sentence boundaries. Replaced with periods, commas, parentheses, or colons across the new docs pages, the changeset, and the code comments this branch adds. Pre-existing em dashes elsewhere are left for the repo-wide docs audit. --- .changeset/workflow-lifecycle-hooks.md | 2 +- .../workflow-api/register-lifecycle-hooks.mdx | 4 ++-- .../docs/v5/observability/lifecycle-hooks.mdx | 14 ++++++------- packages/core/e2e/e2e.test.ts | 2 +- .../core/src/runtime/lifecycle-hooks.test.ts | 4 ++-- packages/core/src/runtime/lifecycle-hooks.ts | 20 +++++++++---------- workbench/example/workflows/99_e2e.ts | 2 +- .../nextjs-turbopack/lifecycle-hooks-e2e.ts | 6 +++--- 8 files changed, 27 insertions(+), 27 deletions(-) diff --git a/.changeset/workflow-lifecycle-hooks.md b/.changeset/workflow-lifecycle-hooks.md index 04cbe419d1..ddbe454c4c 100644 --- a/.changeset/workflow-lifecycle-hooks.md +++ b/.changeset/workflow-lifecycle-hooks.md @@ -3,4 +3,4 @@ 'workflow': minor --- -Add `registerLifecycleHooks` (exported from `workflow/api`) for registering global `onRunCompleted`/`onRunFailed` handlers that receive the lazily-hydrated `Run` instance — and, for failures, a `WorkflowRunFailedError` with the hydrated cause and error code — enabling centralized reporting (e.g. to Sentry) from `instrumentation.ts`. +Add `registerLifecycleHooks` (exported from `workflow/api`) for registering global `onRunCompleted`/`onRunFailed` handlers that receive the lazily-hydrated `Run` instance (and, for failures, a `WorkflowRunFailedError` with the hydrated cause and error code), enabling centralized reporting (e.g. to Sentry) from `instrumentation.ts`. diff --git a/docs/content/docs/v5/api-reference/workflow-api/register-lifecycle-hooks.mdx b/docs/content/docs/v5/api-reference/workflow-api/register-lifecycle-hooks.mdx index 29cd4e2a33..89115906fe 100644 --- a/docs/content/docs/v5/api-reference/workflow-api/register-lifecycle-hooks.mdx +++ b/docs/content/docs/v5/api-reference/workflow-api/register-lifecycle-hooks.mdx @@ -10,7 +10,7 @@ related: - /docs/api-reference/workflow-api/get-run --- -Registers global workflow lifecycle handlers, invoked by the runtime on the compute that records a run's terminal transition. Useful for centralized reporting — for example forwarding every failed run to Sentry — without wrapping each workflow body. +Registers global workflow lifecycle handlers, invoked by the runtime on the compute that records a run's terminal transition. Useful for centralized reporting (for example, forwarding every failed run to Sentry) without wrapping each workflow body. Register early in the process lifecycle (in Next.js, `instrumentation.ts`) so handlers exist before the first run finishes. See the [lifecycle hooks guide](/docs/observability/lifecycle-hooks) for semantics and a full Sentry example. @@ -48,7 +48,7 @@ Returns a function that unregisters these hooks. ## Handlers -Both handlers receive the run as a lazily-hydrated [`Run`](/docs/api-reference/workflow-api/get-run) instance — accessors like `run.workflowName` and `run.returnValue` only fetch from the backend when used. +Both handlers receive the run as a lazily-hydrated [`Run`](/docs/api-reference/workflow-api/get-run) instance. Accessors like `run.workflowName` and `run.returnValue` only fetch from the backend when used. ### `onRunCompleted` diff --git a/docs/content/docs/v5/observability/lifecycle-hooks.mdx b/docs/content/docs/v5/observability/lifecycle-hooks.mdx index 64c25406af..5ee2de361c 100644 --- a/docs/content/docs/v5/observability/lifecycle-hooks.mdx +++ b/docs/content/docs/v5/observability/lifecycle-hooks.mdx @@ -11,9 +11,9 @@ related: - /docs/errors --- -Some failures never reach a `try/catch` in your workflow code — the run can fail in the runtime itself, after your workflow function has already suspended (for example when a replay times out, or a run exhausts its queue deliveries). Lifecycle hooks give you one place to observe every terminal transition, whatever its cause: register global handlers once, and the runtime invokes them whenever it records a run completing or failing. +Some failures never reach a `try/catch` in your workflow code: the run can fail in the runtime itself, after your workflow function has already suspended (for example when a replay times out, or a run exhausts its queue deliveries). Lifecycle hooks give you one place to observe every terminal transition, whatever its cause. Register global handlers once, and the runtime invokes them whenever it records a run completing or failing. -The most common use is centralized error reporting — forwarding every failed run to a service like Sentry without wrapping each workflow body. +The most common use is centralized error reporting, such as forwarding every failed run to a service like Sentry without wrapping each workflow body. ## Registering hooks @@ -39,16 +39,16 @@ export function register() { } ``` -`registerLifecycleHooks` returns an unregister function, and multiple registrations are allowed — handlers run in registration order. +`registerLifecycleHooks` returns an unregister function, and multiple registrations are allowed. Handlers run in registration order. ## Handler parameters Both handlers receive the [`Run`](/docs/api-reference/workflow-api/get-run) instance for the transitioned run. The instance hydrates lazily: accessors like `run.workflowName` or `run.returnValue` only fetch from the backend when the handler actually uses them, so a handler that filters on cheap metadata pays nothing for the runs it ignores. -`onRunFailed` additionally receives the failure as a `WorkflowRunFailedError` — the same shape `run.returnValue` rejects with: +`onRunFailed` additionally receives the failure as a `WorkflowRunFailedError`, the same shape `run.returnValue` rejects with: -- `error.errorCode` — the failure classification (`USER_ERROR`, `RUNTIME_ERROR`, `MAX_DELIVERIES_EXCEEDED`, …). See [error codes](/docs/errors) for the full list. -- `error.cause` — the hydrated thrown value, with Error subclass identity, message, stack, and cause chain preserved. Any JavaScript value can be thrown, so this is typed `unknown`. +- `error.errorCode`: the failure classification (`USER_ERROR`, `RUNTIME_ERROR`, `MAX_DELIVERIES_EXCEEDED`, and more). See [error codes](/docs/errors) for the full list. +- `error.cause`: the hydrated thrown value, with Error subclass identity, message, stack, and cause chain preserved. Any JavaScript value can be thrown, so this is typed `unknown`. ## Reporting failed runs to Sentry @@ -80,5 +80,5 @@ export function register() { - **Host-only.** Handlers run with full Node.js access, never inside the workflow's sandboxed VM. Calling `registerLifecycleHooks` from workflow code throws. - **Fire-and-forget.** Handlers cannot delay or change the run's outcome. A throwing handler is logged and swallowed; the remaining handlers still run. On serverless platforms the invocation is kept alive via `waitUntil` while handlers finish. -- **Fires where the transition is recorded.** Handlers fire on the compute that actually wrote the terminal event — for a failure that means after any retries are exhausted, exactly once per run under normal operation. Terminal transitions recorded outside your app's compute do **not** fire handlers: cancelling a run from the CLI or the Vercel dashboard, for example, is written by the backend, so no handler runs. For a complete record of every transition, consume the [event log](/docs/how-it-works/event-sourcing) or set up alerts on the [observability](/docs/observability) surface instead. +- **Fires where the transition is recorded.** Handlers fire on the compute that actually wrote the terminal event. For a failure, that means after any retries are exhausted, exactly once per run under normal operation. Terminal transitions recorded outside your app's compute do **not** fire handlers: cancelling a run from the CLI or the Vercel dashboard, for example, is written by the backend, so no handler runs. For a complete record of every transition, consume the [event log](/docs/how-it-works/event-sourcing) or set up alerts on the [observability](/docs/observability) surface instead. - **Register everywhere your workflows run.** The terminal write can happen in any function invocation that processes the run's queue messages, so registration must run at startup in every instance of the app (which `instrumentation.ts` guarantees). diff --git a/packages/core/e2e/e2e.test.ts b/packages/core/e2e/e2e.test.ts index 851e11e528..b5b5b03f54 100644 --- a/packages/core/e2e/e2e.test.ts +++ b/packages/core/e2e/e2e.test.ts @@ -3669,7 +3669,7 @@ describe.concurrent('e2e', () => { // Lifecycle hooks (`registerLifecycleHooks`) are registered in the Next.js // workbenches' instrumentation.ts (see lifecycle-hooks-e2e.ts there). The // handlers report each lifecycleHookTarget* run's terminal transition by - // resuming the lifecycleHookObserver workflow's hook — a durable channel + // resuming the lifecycleHookObserver workflow's hook, a durable channel // that works even when the terminal write happens on a different instance // than the one serving these HTTP requests. describe.skipIf(!isNextJsApp)('lifecycle hooks', () => { diff --git a/packages/core/src/runtime/lifecycle-hooks.test.ts b/packages/core/src/runtime/lifecycle-hooks.test.ts index 781cf64186..21d0e73efa 100644 --- a/packages/core/src/runtime/lifecycle-hooks.test.ts +++ b/packages/core/src/runtime/lifecycle-hooks.test.ts @@ -22,8 +22,8 @@ vi.mock('@vercel/functions', () => ({ /** Await everything the dispatcher scheduled through waitUntil. */ async function flushDispatches(): Promise { // The dispatcher resolves a dynamic import before handing the promise to - // waitUntil, so yield macrotask (check-phase) turns via setImmediate — - // which drains the intervening microtasks too — until the capture lands. + // waitUntil, so yield macrotask (check-phase) turns via setImmediate + // (which drains the intervening microtasks too) until the capture lands. for (let i = 0; i < 10 && waitUntilPromises.length === 0; i++) { await new Promise((resolve) => setImmediate(resolve)); } diff --git a/packages/core/src/runtime/lifecycle-hooks.ts b/packages/core/src/runtime/lifecycle-hooks.ts index ead41de769..8ac94d66be 100644 --- a/packages/core/src/runtime/lifecycle-hooks.ts +++ b/packages/core/src/runtime/lifecycle-hooks.ts @@ -50,7 +50,7 @@ export interface WorkflowLifecycleHooks { /** * The registry lives on `globalThis` under a `Symbol.for` key so that every * copy of `@workflow/core` in the process (bundled + unbundled, ESM + CJS) - * shares one list — same pattern as the cross-realm error-class registry in + * shares one list, the same pattern as the cross-realm error-class registry in * `@workflow/errors` and the World cache in `get-world-lazy.ts`. The property * is non-writable/non-configurable so accidental clobbering is loud; the * array's contents stay mutable for register/unregister. @@ -74,7 +74,7 @@ function getRegistry(): WorkflowLifecycleHooks[] { /** * Registers global workflow lifecycle handlers, invoked by the runtime on * the compute that records a run's terminal transition. Useful for - * centralized reporting — e.g. forwarding failed runs to Sentry — without + * centralized reporting (e.g. forwarding failed runs to Sentry) without * wrapping every workflow body. * * Register early in the process lifecycle so handlers exist before the first @@ -84,8 +84,8 @@ function getRegistry(): WorkflowLifecycleHooks[] { * Semantics: * - Handlers run on the host (full Node.js), never inside the workflow VM. * - Handlers fire only on the invocation that actually wrote the terminal - * event. Transitions recorded elsewhere — e.g. a run cancelled from the - * CLI or dashboard — do not fire handlers in the app. + * event. Transitions recorded elsewhere (e.g. a run cancelled from the + * CLI or dashboard) do not fire handlers in the app. * - Handlers are fire-and-forget: they cannot delay or change the run's * outcome, and a throwing handler is logged and swallowed. On serverless * platforms the invocation is kept alive via `waitUntil`. @@ -169,13 +169,13 @@ export function dispatchRunCompletedHooks(runId: string): void { * (the workflow runs in a separate realm, so `instanceof Error` on it is * `false` for handlers) and may carry VM-realm exotics in its cause chain. * Round-trip it through the run-error serialization pipeline so handlers - * receive the same host-realm hydrated shape `run.returnValue` rejects with - * — real host Error instances with name/message/stack/cause preserved and - * registered classes (FatalError, custom serde classes) revived with their - * class identity. No encryption: the bytes never leave this process. + * receive the same host-realm hydrated shape `run.returnValue` rejects + * with: real host Error instances with name/message/stack/cause preserved + * and registered classes (FatalError, custom serde classes) revived with + * their class identity. No encryption: the bytes never leave this process. * - * Falls back to the original value when the round-trip fails — a degraded - * report beats no report. + * Falls back to the original value when the round-trip fails, since a + * degraded report beats no report. */ async function hydrateForHandlers( error: unknown, diff --git a/workbench/example/workflows/99_e2e.ts b/workbench/example/workflows/99_e2e.ts index 9eac410821..22ca31b1e2 100644 --- a/workbench/example/workflows/99_e2e.ts +++ b/workbench/example/workflows/99_e2e.ts @@ -3901,7 +3901,7 @@ export async function crossRegionStreamWorkflow(chunkCount: number) { // instrumentation.ts registers `registerLifecycleHooks` handlers // (see workbench/nextjs-*/instrumentation.ts). The handlers // observe these target runs' terminal transitions and report -// them by resuming the observer workflow's hook — a durable +// them by resuming the observer workflow's hook, a durable // channel that works across serverless instances. // ============================================================ diff --git a/workbench/nextjs-turbopack/lifecycle-hooks-e2e.ts b/workbench/nextjs-turbopack/lifecycle-hooks-e2e.ts index 6ff579a742..e0cdfa4511 100644 --- a/workbench/nextjs-turbopack/lifecycle-hooks-e2e.ts +++ b/workbench/nextjs-turbopack/lifecycle-hooks-e2e.ts @@ -10,7 +10,7 @@ import { registerLifecycleHooks, resumeHook } from 'workflow/api'; * workflow's hook. Resuming a durable hook is deliberately the observation * channel: the handler runs on whichever instance wrote the terminal event, * which on a deployed app is generally NOT the instance serving the e2e - * test's HTTP requests — an in-memory buffer would not travel. + * test's HTTP requests, so an in-memory buffer would not travel. */ export function registerE2eLifecycleHooks(): void { registerLifecycleHooks({ @@ -34,8 +34,8 @@ export function registerE2eLifecycleHooks(): void { }); }, async onRunFailed({ run, error }) { - // The hydrated thrown value is already on the error — no backend - // reads needed to filter. + // The hydrated thrown value is already on the error, so filtering + // needs no backend reads. const cause = error.cause; const causeMessage = cause instanceof Error ? cause.message : String(cause); From f635d02f811175f8384d224205a6670eb69ec7a3 Mon Sep 17 00:00:00 2001 From: Nathan Rajlich Date: Thu, 20 Aug 2026 13:04:57 -0700 Subject: [PATCH 4/4] Apply Vercel technical writing rules to the lifecycle hooks docs Beyond the em-dash pass: the guide's intro now leads with what lifecycle hooks let you do (the first sentence is parseable as the page's purpose) instead of opening with the failure mode; passive constructions are active ('the runtime keeps the invocation alive with waitUntil', 'the runtime logs and swallows a throwing handler', 'you can register multiple hook sets', 'the backend writes that transition'); the one-word 'Semantics' heading is now the standalone statement 'How handlers behave'; and the key lazy-hydration sentence names the Run instance instead of leading with a pronoun so it reads correctly when extracted alone. --- .../workflow-api/register-lifecycle-hooks.mdx | 6 +++--- .../docs/v5/observability/lifecycle-hooks.mdx | 14 ++++++-------- 2 files changed, 9 insertions(+), 11 deletions(-) diff --git a/docs/content/docs/v5/api-reference/workflow-api/register-lifecycle-hooks.mdx b/docs/content/docs/v5/api-reference/workflow-api/register-lifecycle-hooks.mdx index 89115906fe..078f6f7d57 100644 --- a/docs/content/docs/v5/api-reference/workflow-api/register-lifecycle-hooks.mdx +++ b/docs/content/docs/v5/api-reference/workflow-api/register-lifecycle-hooks.mdx @@ -10,7 +10,7 @@ related: - /docs/api-reference/workflow-api/get-run --- -Registers global workflow lifecycle handlers, invoked by the runtime on the compute that records a run's terminal transition. Useful for centralized reporting (for example, forwarding every failed run to Sentry) without wrapping each workflow body. +Registers global workflow lifecycle handlers, invoked by the runtime on the compute that records a run's terminal transition. Use it for centralized reporting, such as forwarding every failed run to Sentry, without wrapping each workflow body. Register early in the process lifecycle (in Next.js, `instrumentation.ts`) so handlers exist before the first run finishes. See the [lifecycle hooks guide](/docs/observability/lifecycle-hooks) for semantics and a full Sentry example. @@ -70,6 +70,6 @@ Invoked when a workflow run fails terminally (after any retries). ## Behavior - Handlers run on the host (full Node.js), never inside the workflow VM. Calling `registerLifecycleHooks` from workflow code throws. -- Handlers are fire-and-forget: they cannot delay or change the run's outcome, and a throwing handler is logged and swallowed. On serverless platforms the invocation is kept alive via `waitUntil`. +- Handlers are fire-and-forget: they cannot delay or change the run's outcome, and the runtime logs and swallows a throwing handler. On serverless platforms, the runtime keeps the invocation alive with `waitUntil`. - Handlers fire only on the invocation that wrote the terminal event. Transitions recorded outside your app's compute (e.g. a run cancelled from the CLI or dashboard) do not fire handlers. -- Multiple registrations are allowed; handlers run in registration order. +- You can register multiple hook sets, and handlers run in registration order. diff --git a/docs/content/docs/v5/observability/lifecycle-hooks.mdx b/docs/content/docs/v5/observability/lifecycle-hooks.mdx index 5ee2de361c..81599157e2 100644 --- a/docs/content/docs/v5/observability/lifecycle-hooks.mdx +++ b/docs/content/docs/v5/observability/lifecycle-hooks.mdx @@ -11,9 +11,7 @@ related: - /docs/errors --- -Some failures never reach a `try/catch` in your workflow code: the run can fail in the runtime itself, after your workflow function has already suspended (for example when a replay times out, or a run exhausts its queue deliveries). Lifecycle hooks give you one place to observe every terminal transition, whatever its cause. Register global handlers once, and the runtime invokes them whenever it records a run completing or failing. - -The most common use is centralized error reporting, such as forwarding every failed run to a service like Sentry without wrapping each workflow body. +Lifecycle hooks let you register global handlers that the runtime invokes whenever a workflow run completes or fails. They observe even the failures that never reach a `try/catch` in workflow code, such as a replay timing out or a run exhausting its queue deliveries. The most common use is centralized error reporting, such as forwarding every failed run to a service like Sentry without wrapping each workflow body. ## Registering hooks @@ -39,11 +37,11 @@ export function register() { } ``` -`registerLifecycleHooks` returns an unregister function, and multiple registrations are allowed. Handlers run in registration order. +`registerLifecycleHooks` returns an unregister function. You can register multiple hook sets, and handlers run in registration order. ## Handler parameters -Both handlers receive the [`Run`](/docs/api-reference/workflow-api/get-run) instance for the transitioned run. The instance hydrates lazily: accessors like `run.workflowName` or `run.returnValue` only fetch from the backend when the handler actually uses them, so a handler that filters on cheap metadata pays nothing for the runs it ignores. +Both handlers receive the [`Run`](/docs/api-reference/workflow-api/get-run) instance for the transitioned run. The `Run` instance hydrates lazily, meaning accessors like `run.workflowName` or `run.returnValue` only fetch from the backend when the handler uses them, so a handler that filters on cheap metadata pays nothing for the runs it ignores. `onRunFailed` additionally receives the failure as a `WorkflowRunFailedError`, the same shape `run.returnValue` rejects with: @@ -76,9 +74,9 @@ export function register() { } ``` -## Semantics +## How handlers behave - **Host-only.** Handlers run with full Node.js access, never inside the workflow's sandboxed VM. Calling `registerLifecycleHooks` from workflow code throws. -- **Fire-and-forget.** Handlers cannot delay or change the run's outcome. A throwing handler is logged and swallowed; the remaining handlers still run. On serverless platforms the invocation is kept alive via `waitUntil` while handlers finish. -- **Fires where the transition is recorded.** Handlers fire on the compute that actually wrote the terminal event. For a failure, that means after any retries are exhausted, exactly once per run under normal operation. Terminal transitions recorded outside your app's compute do **not** fire handlers: cancelling a run from the CLI or the Vercel dashboard, for example, is written by the backend, so no handler runs. For a complete record of every transition, consume the [event log](/docs/how-it-works/event-sourcing) or set up alerts on the [observability](/docs/observability) surface instead. +- **Fire-and-forget.** Handlers cannot delay or change the run's outcome. The runtime logs and swallows a throwing handler, and the remaining handlers still run. On serverless platforms, the runtime keeps the invocation alive with `waitUntil` while handlers finish. +- **Fires where the transition is recorded.** Handlers fire on the compute that wrote the terminal event. For a failure, that means after any retries are exhausted, exactly once per run under normal operation. Terminal transitions recorded outside your app's compute do **not** fire handlers. For example, when you cancel a run from the CLI or the Vercel dashboard, the backend writes that transition, so no handler runs. For a complete record of every transition, consume the [event log](/docs/how-it-works/event-sourcing) or set up alerts on the [observability](/docs/observability) surface instead. - **Register everywhere your workflows run.** The terminal write can happen in any function invocation that processes the run's queue messages, so registration must run at startup in every instance of the app (which `instrumentation.ts` guarantees).