handoffd: missed wake-up notification permanently stalls the pipeline (fire-and-forget notify, no inbox reconciliation)

[hgardneriv]

Summary

A single missed wake-up notification permanently stalls the handoff pipeline. handoffd delivers each handoff exactly once with an unverified tmux send-keys keystroke injection, never re-notifies, and the receive-side rules forbid agents from polling — so if the one wake-up doesn't land, the payload sits in inbox/new/ forever and the swarm deadlocks.

We hit this in production use today: a five-stage pipeline (specifier → coder → cleaner → architect → hardender → QA) ran cleanly for three hours, then froze at the last stage for 4+ hours because QA never saw its wake-up, even though the handoff file was correctly delivered to its inbox.

Environment

• macOS (darwin 25.5.0), tmux, Babashka v1.12.218
• Six Claude-backed agents in tmux windows, per-role git worktrees
• swarmforge/scripts/handoffd.bb identical to current main in this repo

Observed timeline (from .swarmforge/daemon/handoffd.log)

15:44:01 delivered ..._from_specifier_to_coder.handoff      → processed OK
16:37:40 delivered ..._from_coder_to_cleaner.handoff        → processed OK
17:01:16 delivered ..._from_cleaner_to_architect.handoff    → processed OK
17:11:24 delivered ..._from_architect_to_hardender.handoff  → processed OK
18:31:01 delivered ..._from_hardender_to_QA.handoff         → NEVER PROCESSED

At 22:37 the handoff file was still sitting in .worktrees/QA/.swarmforge/handoffs/inbox/new/, the QA agent idle at an empty prompt. Its transcript showed it had responded to a wake-up by checking for work, concluding (wrongly — it was in a subdirectory at the time) that its inbox was empty, and then, per the handoff rules ("If it prints NO_TASK, stop waiting for work"), stopping. Nothing ever woke it again. All processes (daemon, watchdog, tmux, all six agents) were alive and healthy the whole time.

Root cause

Two compounding design gaps in deliver! / poll-once! (swarmforge/scripts/handoffd.bb):

1. Fire-and-forget, unverified notification. notify! (lines 94–105) types a wake-up sentence into the recipient's TUI via tmux send-keys with fixed 150ms/50ms sleeps, and treats a zero exit code as success. But send-keys exiting 0 only means the keystrokes were injected into the pane — not that the agent's input box received and submitted them. A busy/rendering TUI, a popped-open autocomplete menu, or an agent that mishandles the prompt all silently drop the wake with no error anywhere.

2. No reconciliation. poll-once! (lines 164–178) scans every role's outbox/ once per second, forever — but never looks at inbox/new/. The evidence of a failed delivery (a file sitting unclaimed in inbox/new/ for hours) is durable and on disk, yet nothing acts on it. Delivery is exactly-once by construction, while the transport is at-most-once in practice.

Because the receiving rules explicitly (and sensibly) forbid agents from polling, the agent side cannot compensate. One lost wake-up = deadlocked pipeline, with every component "healthy."

Proposed fix

Make notification at-least-once by adding inbox reconciliation to the daemon poll loop: in poll-once!, also scan each role's inbox/new/; if any file's enqueued_at header (already written by add-delivery-headers) is older than a threshold (e.g. 60s), call notify! for that role again. The wake message is already idempotent ("If idle, run ready_for_next.sh") and the receive rules already ignore wake-ups mid-task, so re-notification is safe and would have self-healed our stall automatically.

Optional hardening: after notify!, use tmux capture-pane to verify the composer submitted (input line empty) rather than trusting send-keys exit codes.

Workaround

Manually type anything into the stalled agent's tmux window (e.g. "You have a handoff in your inbox — run ready_for_next.sh") and the pipeline resumes.