gate shells hang 240s+ on a single bash call mid review/ship

[usirin]

Reviewer and shipper gate shells stalled three times on 2026-08-22 (lanes on PRs #6994 and #7011): a single bash call stayed open 240+ seconds mid-gate until a driver nudge killed it, and the re-run then completed — so the underlying call blocks indefinitely instead of failing fast. Work was never lost, only stalled, but an unattended pipeline has nobody to nudge, so every headless run risks stranding here, and each stall taxes every lane passing through review/ship.

What triage verified

• Triage note: read at origin/main (eb7c6fa4b). ship gate itself (packages/fabrika-cli/src/ship/gate-verb.ts) is a read-only conjunction over comments, reviews and PR files — nothing interactive. No PAGER, timeout, or non-interactive handling exists anywhere under packages/fabrika-cli/src/ship or packages/fabrika-cli/src/review, so the CLI layer does not currently force headless-safe invocations of the gh calls underneath.
• The exact hung argv was never captured — the filer's own report defers it to the next occurrence, so the blocking mechanism (pager, interactive prompt, auth wait, network retry, closed stdin) is unknown.
• Dedup and attach search (both queries, gate shells hang 240s+ on a single bash call mid review/ship #7025 excluded; queue + board search over hang/stall/interactive/headless): no open issue or epic owns this surface. Nearest neighbours are different failure modes: Builder runner died mid-run on epic child #7006 (lane 5979): async runner exited before writing a result #7023 (async runner died mid-run), lane emit boots a pre-plan epic lane on the single-task coder machine, and lane migrate --check does not flag it #7024 (pre-plan epic lane boot), Clearing human:queue-stall buys one conclusive read and nothing grants more waits #6717 (queue-stall park vocabulary). Minted standalone because nothing absorbs it.

The investigation

The deliverable is knowledge, not code: name the blocking mechanism before anyone chooses a remedy.

1. Capture the next occurrence: full argv, cwd, elapsed time, tty/stdin state of the hung bash call, from whichever reviewer/shipper lane hits it next (or reproduce deliberately against the same command class).
2. Bisect the mechanism locally with a minimal reproduction that hangs without driver input — distinguish pager waiting on a tty, an interactive prompt (auth refresh, host confirmation), a network retry loop, and a command reading a never-closing stdin.
3. Report the diagnosis plus a recommendation split along the two candidate remedies the filer named — (a) verbs defaulting to non-interactive (no pager, explicit timeouts) versus (b) a documented headless env contract for gate shells — naming exactly one as the follow-on work. Implementation is out of scope for this ticket.

Acceptance criteria

• A hung invocation is recorded with its full argv, elapsed duration, and the lane/PR it occurred on.
• The blocking mechanism is identified and demonstrated by a minimal reproduction that stalls without driver input.
• A written recommendation commits to one primary remedy path (non-interactive defaults vs documented headless env contract) with evidence for setting the other aside.
• the client-side timeout bounds the response body read too, not only connect plus headers

---

Original report (verbatim)

Summary

Three times today, a reviewer or shipper shell had one bash call open for 240+ seconds mid-gate, needing a driver nudge to kill and re-run. The gate still completes after the nudge, so the underlying call looks like it blocks indefinitely instead of failing.

What I was doing

Running reviewer→shipper lanes on PRs #6994 and #7011 in kamp-us/phoenix. Each lane hung once; a fourth hang just occurred on the #6994 shipper resume.

What I observed

• Reviewer on chore: Wire @kampus/fabrika-pi into release-please and the publish workflow #6994: tool bash open 240s at step 1 (10 turns in).
• Shipper on ADR 0334: bootstrap merges into present files by key-merge or append-if-absent #7011: same signal, 240s open.
• Shipper on chore: Wire @kampus/fabrika-pi into release-please and the publish workflow #6994 (resume): same signal a third time.
• In each case the agent recovered after being told to kill the call and re-run non-interactively — the work was never lost, only stalled.
• The exact hung command string is being collected from the current shipper's next report.

Why it matters

A gate that silently blocks for minutes per call slows every lane through review/ship and can strand an unattended run: nobody nudges an unwatched pipeline. If the cause is a verb waiting on interactive input (a pager, a prompt), every headless caller hits it.

Pointers

• fabrika-cli verbs invoked by the review/ship skills; gh underneath
• Occurrences logged 2026-08-22 across PRs chore: Wire @kampus/fabrika-pi into release-please and the publish workflow #6994 and ADR 0334: bootstrap merges into present files by key-merge or append-if-absent #7011 lanes

Suggested next step (non-binding)

Capture the exact hung argv from the next occurrence, then decide: a verb defaulting to non-interactive (no pager, explicit timeouts) or a documented headless env contract for gate shells.

---

_{Filed by an agent · session 01a02ab8-f5ac-7647-9d28-963dc3ecf62a · branch umut/adr-0334-bootstrap-merge-arms · 2026-08-22T21:39:02Z}

[usirin]

[usirin]

Culprit identified. Transcript evidence from the ship-6894 lane (2026-08-23 00:14Z): the hung call is fabrika ship reconcile <pr> — it blocked 240s+ before the driver killed it. Preceding context: ship enqueue 6894 had just answered "mergeable_state is dirty (mergeable: false) — a definite read". So the likely mechanism is reconcile waiting/polling on a dirty-or-stale mergeable state without an internal timeout, in a headless shell where nothing ever unblocks it.

Occurrences now four: reviewers once, shippers three times (#6994 ×2, #7011, #6872, #6894). Every one recovered only after a driver nudge; no data was lost.

[usirin]

lane-claim: lane:01a02bf1-5ef0-7960-bc38-6af0f64be51b:d68d4f15-16d5-49c5-a64e-89d28b5f5b89 · 2026-08-23T06:57:47.963Z

[usirin]

Salvage: builder round 1 (reaped mid-investigation)

The round-1 builder shell (claim session 01a02d68-3129-7cf6-a15e-b03690124581, adopted + released by the driver) was reaped by the harness run timeout after ~30 min — one bash call blew past even its 300s tool cap, the very hang under investigation. Its transcript survived; driver-relayed partial findings, unverified by the driver, for the successor to confirm and carry to a diagnosis:

1. packages/fabrika-cli/src/io/gh-api.ts — HTTP path has no application-level timeout. HttpClient.execute(request) is unbounded; the only bounded leg found was the gh auth token subprocess (timeoutSeconds: 15). API_ROOT is hardcoded to https://api.github.com, so transport stalls cannot be pointed at a local repro directly.
2. OS TCP connect window measured ~75s per attempt against a black-hole listener (the subprocess stall class: git fetch/git push). Repeated attempts compound well past the observed 240s+.
3. Pager class: with stdout captured/piped, gh returns immediately even with GH_PAGER forced (no tty → no pager engages). Under an allocated tty the same command class hangs (probe killed by SIGALRM at 12s). So pager hangs require a pty; plain captured shells are safe on this axis.
4. Unfinished: the main undici-fetch stall measurement (local listener that accepts and never answers) — the shell was reaped before it landed.

Working theory left on the table: verb calls into gh-api.ts can block indefinitely (no client-side timeout/signal), which during tonight's GitHub/network stall presents exactly as gate/guard shells stuck 240s+ on one bash call mid review/ship.

Driver relay on behalf of the reaped shell — lane 7025, pass 2.

[usirin]

build-claim: build:01a02bf1-5ef0-7960-bc38-6af0f64be51b:b1e6b79e-8b4b-4060-a027-1d62a0f5687f · 2026-08-23T19:37:03.492Z