Builder-runner resilience: retry/resume affordance for epic-child builders killed by infra faults, and a timeout horizon sized to cold starts
Summary
Epic-child builder shells die on infrastructure faults — provider network_error, the 30-minute workflow timeout — before recording any terminal, and the harness then reaps their worktree, destroying uncommitted edits. Tonight this burned at least four builder rounds across two epics (lane 6767's tracer #7029 twice; sibling incidents #7023 and #7027). The lane machine never sees a verdict for these deaths, so every recovery is operator judgment instead of a mapped transition.
What changes
- A builder-runner-level retry affordance: when a child run dies without a terminal (network_error, timeout, process death), its replacement can resume from what survived — the claim marker, branch, and any committed work — rather than re-claiming from zero; where nothing survived (worktree reaped mid-edit), the retry is recorded as a mapped lane event so the fold shows the burn instead of silence.
- The 30-minute default workflow horizon is re-priced against a cold-start builder's real cost (large-repo read + multi-file edit) so healthy builds stop being reaped by the ceiling; if the verdict is "keep 30 minutes," that finding is recorded with the evidence.
- Salvage-first ordering: before a reaped worktree's edits are declared lost, the runner checks for any persisted artifact the harness already keeps (scratch dirs, staged patches) and hands it to the successor.
Acceptance criteria
Pointers
Original report (verbatim)
Summary
Two consecutive epic-child builder runs on lane 6767 / task issue_7029 died before recording a terminal: the first to a provider network_error ~48s in, the second to the harness's 30-minute workflow timeout mid-edit.
What I was doing
Operating lane 6767 (epic) after founder plan approval; dispatched #7029 (capture upsert) to a builder shell in an isolated worktree, twice.
What I observed
Run 1 (5897eea3): failed at ~48s with finish_reason network_error; zero changed patches. Run 2 (eb1ad578): claimed #7029 (comment 5383653096), cut branch build/7029-user-activity-day-c67e125c, edited apps/web/worker/features/pasaport/user-activity-day.ts and its unit test, then hit the 30-min workflow timeout mid-write ("Now the test file:"). The harness cleaned up its worktree, so its uncommitted edits are lost; only the empty branch and the claim marker survived. The stranded claim was released under ADR-style succession (session-owned release, exit 0).
Why it matters
Epic children keep dying on infrastructure faults (network errors, harness timeouts), which burns builder rounds and driver passes without any machine-visible verdict — the same class as the gate-shell hangs (#7025) and earlier builder-runner crashes (#7023/#7027).
Pointers
- Lane ledger:
.fabrika/lanes/6767/events.jsonl
- Dead run transcripts: subagent runs 9c8b6c9e-9ac4 (network_error) and eb1ad578-c3e1 (timeout)
Suggested next step (non-binding)
Consider a builder-runner-level retry/resume affordance for timeout-killed children whose worktree is reaped, and whether the 30-minute default workflow horizon is long enough for a cold-start builder that must read a large repo.
Filed by an agent · session 01a02bf1-5ef0-7960-bc38-6af0f64be51b · branch main · 2026-08-23T02:24:02Z
Original report (verbatim)
Summary
Two consecutive epic-child builder runs on lane 6767 / task issue_7029 died before recording a terminal: the first to a provider network_error ~48s in, the second to the harness's 30-minute workflow timeout mid-edit.
What I was doing
Operating lane 6767 (epic) after founder plan approval; dispatched #7029 (capture upsert) to a builder shell in an isolated worktree, twice.
What I observed
Run 1 (5897eea3): failed at ~48s with finish_reason network_error; zero changed patches. Run 2 (eb1ad578): claimed #7029 (comment 5383653096), cut branch build/7029-user-activity-day-c67e125c, edited apps/web/worker/features/pasaport/user-activity-day.ts and its unit test, then hit the 30-min workflow timeout mid-write ("Now the test file:"). The harness cleaned up its worktree, so its uncommitted edits are lost; only the empty branch and the claim marker survived. The stranded claim was released under ADR-style succession (session-owned release, exit 0).
Why it matters
Epic children keep dying on infrastructure faults (network errors, harness timeouts), which burns builder rounds and driver passes without any machine-visible verdict — the same class as the gate-shell hangs (#7025) and earlier builder-runner crashes (#7023/#7027).
Pointers
- Lane ledger:
.fabrika/lanes/6767/events.jsonl
- Dead run transcripts: subagent runs 9c8b6c9e-9ac4 (network_error) and eb1ad578-c3e1 (timeout)
Suggested next step (non-binding)
Consider a builder-runner-level retry/resume affordance for timeout-killed children whose worktree is reaped, and whether the 30-minute default workflow horizon is long enough for a cold-start builder that must read a large repo.
Filed by an agent · session 01a02bf1-5ef0-7960-bc38-6af0f64be51b · branch main · 2026-08-23T02:24:02Z
Builder-runner resilience: retry/resume affordance for epic-child builders killed by infra faults, and a timeout horizon sized to cold starts
Summary
Epic-child builder shells die on infrastructure faults — provider
network_error, the 30-minute workflow timeout — before recording any terminal, and the harness then reaps their worktree, destroying uncommitted edits. Tonight this burned at least four builder rounds across two epics (lane 6767's tracer #7029 twice; sibling incidents #7023 and #7027). The lane machine never sees a verdict for these deaths, so every recovery is operator judgment instead of a mapped transition.What changes
Acceptance criteria
.fabrika/lanes/<epic>/events.jsonl), not silencePointers
9c8b6c9e(network_error ~48s) andeb1ad578(30-min timeout mid-edit) on lane.fabrika/lanes/6767/events.jsonl; lost edits were inapps/web/worker/features/pasaport/user-activity-day.tsOriginal report (verbatim)
Summary
Two consecutive epic-child builder runs on lane 6767 / task issue_7029 died before recording a terminal: the first to a provider
network_error~48s in, the second to the harness's 30-minute workflow timeout mid-edit.What I was doing
Operating lane 6767 (epic) after founder plan approval; dispatched #7029 (capture upsert) to a builder shell in an isolated worktree, twice.
What I observed
Run 1 (5897eea3): failed at ~48s with finish_reason
network_error; zero changed patches. Run 2 (eb1ad578): claimed #7029 (comment 5383653096), cut branchbuild/7029-user-activity-day-c67e125c, edited apps/web/worker/features/pasaport/user-activity-day.ts and its unit test, then hit the 30-min workflow timeout mid-write ("Now the test file:"). The harness cleaned up its worktree, so its uncommitted edits are lost; only the empty branch and the claim marker survived. The stranded claim was released under ADR-style succession (session-owned release, exit 0).Why it matters
Epic children keep dying on infrastructure faults (network errors, harness timeouts), which burns builder rounds and driver passes without any machine-visible verdict — the same class as the gate-shell hangs (#7025) and earlier builder-runner crashes (#7023/#7027).
Pointers
.fabrika/lanes/6767/events.jsonlSuggested next step (non-binding)
Consider a builder-runner-level retry/resume affordance for timeout-killed children whose worktree is reaped, and whether the 30-minute default workflow horizon is long enough for a cold-start builder that must read a large repo.
Filed by an agent · session
01a02bf1-5ef0-7960-bc38-6af0f64be51b· branchmain· 2026-08-23T02:24:02ZOriginal report (verbatim)
Summary
Two consecutive epic-child builder runs on lane 6767 / task issue_7029 died before recording a terminal: the first to a provider
network_error~48s in, the second to the harness's 30-minute workflow timeout mid-edit.What I was doing
Operating lane 6767 (epic) after founder plan approval; dispatched #7029 (capture upsert) to a builder shell in an isolated worktree, twice.
What I observed
Run 1 (5897eea3): failed at ~48s with finish_reason
network_error; zero changed patches. Run 2 (eb1ad578): claimed #7029 (comment 5383653096), cut branchbuild/7029-user-activity-day-c67e125c, edited apps/web/worker/features/pasaport/user-activity-day.ts and its unit test, then hit the 30-min workflow timeout mid-write ("Now the test file:"). The harness cleaned up its worktree, so its uncommitted edits are lost; only the empty branch and the claim marker survived. The stranded claim was released under ADR-style succession (session-owned release, exit 0).Why it matters
Epic children keep dying on infrastructure faults (network errors, harness timeouts), which burns builder rounds and driver passes without any machine-visible verdict — the same class as the gate-shell hangs (#7025) and earlier builder-runner crashes (#7023/#7027).
Pointers
.fabrika/lanes/6767/events.jsonlSuggested next step (non-binding)
Consider a builder-runner-level retry/resume affordance for timeout-killed children whose worktree is reaped, and whether the 30-minute default workflow horizon is long enough for a cold-start builder that must read a large repo.
Filed by an agent · session
01a02bf1-5ef0-7960-bc38-6af0f64be51b· branchmain· 2026-08-23T02:24:02Z