v0.8.66: Release gate for multi sub-agent fanout freeze

[Hmbown]

Problem

A high-fanout parent turn can make the TUI appear frozen while launching or completing many sub-agents. The current repro is a user-triggered fanout of roughly 20 agent calls: sidebar state changes, input responsiveness stalls, and completions appear to roll in slowly or inconsistently.

This issue is the release-gate umbrella for the v0.8.66 freeze work. It should stay open until CodeWhale can launch and observe a large sub-agent fanout without freezing the UI or losing critical completion state.

Verified evidence from current tree

• agent inherits supports_parallel() == false from the default tool spec (crates/tui/src/tools/spec.rs), so each agent call is planned as a serial tool batch.
• Non-parallel tools take the write side of the global tool execution lock in crates/tui/src/core/engine/tool_execution.rs.
• turn_loop.rs already documents the user-visible freeze symptom for serial fanout: six agent calls resolving routes under the global tool lock can read as a hard TUI freeze.
• The engine op channel is bounded at 32 and the event channel at 256 (crates/tui/src/core/engine.rs). UI code awaits Op::ListSubAgents after event-drain batches.
• Sub-agent completion UI events use try_send, so under event-channel pressure they can be dropped from the UI stream even if parent completion signaling uses a separate path.
• ListSubAgents and sub-agent completions contend on the SubAgentManager write lock; manager update paths can persist JSON state synchronously while holding that lock.
• The async TUI loop still calls blocking std::sync::Mutex::lock() on the shell manager in sidebar/live-output refresh paths.

Critical framing

Do not treat this as the old v0.8.61 freeze unless fresh measurements prove it is the same failure mode. The earlier speculative “blocking I/O starves the worker pool” theory was disproven. This lane should fix the currently observed fanout/backpressure paths with targeted evidence, not a blanket spawn_blocking patch.

Required outcome

A parent turn that requests 20 sub-agents should remain interruptible and visibly alive:

• TUI input polling continues while launches are in progress.
• The sidebar/status view updates without blocking the event loop.
• Launching N agents should not take roughly N × route-resolution/spawn latency when the work is independent and within configured sub-agent limits.
• Completion events that matter to parent-turn correctness are not lost; UI refresh events can be coalesced but must converge to correct state.
• Cancelling the parent turn interrupts any remaining launch queue quickly and leaves well-formed tool results.

Acceptance criteria

• Add an automated or documented release-gate repro that launches a high-fanout sub-agent batch (target: 20) and records launch latency, input responsiveness, event backlog, and completion convergence.
• The repro passes with no multi-second TUI input freeze on a normal local checkout.
• The fix set includes targeted tests for tool dispatch batching, engine/op channel backpressure, ListSubAgents coalescing, and manager persistence/locking behavior.
• Runtime logs include enough bounded diagnostics to distinguish launch serialization, event-channel pressure, op-channel pressure, and manager-lock contention.
• The implementation does not remove sub-agent depth/concurrency configurability and does not reintroduce old lifecycle/delegate tool surfaces.

Child work

Create/track child issues for:

• parallel-safe agent launch dispatch,
• engine/TUI channel backpressure,
• nonblocking/coalesced sub-agent sidebar refresh,
• shell-manager lock removal from async UI hot paths,
• SubAgentManager lock/persistence hot-path cleanup.

Tracked child issues

• v0.8.66: Allow independent agent launches to fan out without global tool-lock serialization #3801 — Allow independent agent launches to fan out without global tool-lock serialization
• v0.8.66: Remove engine/TUI channel backpressure from sub-agent status storms #3802 — Remove engine/TUI channel backpressure from sub-agent status storms
• v0.8.66: Make sub-agent sidebar refresh read-only and coalesced #3803 — Make sub-agent sidebar refresh read-only and coalesced
• v0.8.66: Remove blocking shell-manager locks from async TUI refresh paths #3804 — Remove blocking shell-manager locks from async TUI refresh paths
• v0.8.66: Move sub-agent state persistence out of manager write-lock hot paths #3805 — Move sub-agent state persistence out of manager write-lock hot paths

Security / policy guardrails

Responsiveness fixes must not change the authority model. The implementation must preserve these constraints:

• Parallel agent launch must not bypass approval policy, sandbox policy, tool gates, sub-agent depth limits, sub-agent concurrency limits, token/budget limits, or cancellation.
• Parallelism should only reduce waiting to start independent children; it must not grant children more authority than a serial launch would.
• Critical events must remain durable or recoverable: approval prompts, tool results, fatal errors, user-input requests, cancellation, parent completion, and terminal sub-agent state. Only refresh/status events may be coalesced or dropped, and only if the next snapshot converges to correct state.
• Cleanup cannot disappear when sidebar listing becomes read-only. Stale child timeout/cancel enforcement must remain periodic or event-driven.
• UI snapshots may become stale under contention, but enforcement must never depend on a successful UI refresh.
• Persistence changes must keep atomic writes, symlink/path hardening, and recoverable terminal state.

[Hmbown]

Audit: fanout freeze (verified against main 174405f)

All 7 evidence claims verified (with two precisions):

1. agent is serial — but indirectly: AgentTool inherits supports_parallel()==false (tools/spec.rs:911), and it's RequiresApproval + not read-only (subagent/mod.rs:3526/3530), so dispatch.rs:478-486 excludes it from any parallel batch regardless of the flag.
2. Non-parallel tools take the global write lock — tool_execution.rs:330-334 on tool_exec_lock (engine.rs:592).
3. op channel 32 / event 256 — engine.rs:823-824.
4. try_send droppable — precision: parent-completion delivery is an unbounded, reliable channel (engine.rs:828); only best-effort UI progress/AgentSpawned/mailbox events drop under pressure (engine.rs:2462, mod.rs:2664). Completion correctness is preserved.
5. UI awaits Op::ListSubAgents after drain — ui.rs:3152 (coalesced per drain, ui.rs:1866).
6. SubAgentManager spawn persists JSON synchronously under write().await — mod.rs:3756 → 2695 → write_json_atomic 3326 (fs::write+rename), not debounced (unlike the checkpoint path).
7. Blocking std::sync::Mutex on the async UI loop — ui.rs:1432/1457 (SharedShellManager, shell.rs:1914).

⚠️ The tempting fix is wrong

Flipping AgentTool::supports_parallel() to true is ineffective (still excluded by RequiresApproval/not-read-only at dispatch.rs:479) and dangerous (the global write lock currently serializes spawns for read-then-write invariants like session-name uniqueness, mod.rs:2593-2616). Don't.

Child breakdown (the 5 you asked for) + tractability + in-flight mapping

1. Nonblocking/coalesced sidebar refresh [safe-now] — ui.rs:3152 send is still awaited; make Op::ListSubAgents a best-effort try_send (new EngineHandle helper). Idempotent + already coalesced + re-requested next drain → dropping under backpressure is harmless. This directly removes the freeze symptom.
2. Shell-manager lock removal from async UI hot paths [safe-now] — try_lock() (skip refresh if contended) at ui.rs:1432/1457.
3. Engine/TUI channel backpressure [medium] — Op::SendMessage runs the whole turn inline (engine.rs:1374) so rx_op(32) isn't drained during fanout. In flight: PR fix(subagent): preserve event headroom for progress #3783 (progress event headroom) + branch devin/…-fanout-backpressure appear to target this.
4. SubAgentManager lock/persistence cleanup [medium] — debounce/await-off the unconditional full-fleet spawn persist (mod.rs:2695).
5. Parallel-safe agent dispatch [risky] — let independent agent start overlap without weakening the spawn critical section; needs the lock-invariant work first. (branch issue/3289-fleet-fanout-regression may be in this space.)

Recommended safe release-gate fix

Children 1 + 2 together (ListSubAgents → try_send; shell lock() → try_lock()) remove the UI-freeze symptom with zero spawn/correctness risk. Release-gate repro: YOLO + a ~20 agent fanout; assert a queued keystroke echoes and Ctrl+C cancels < ~200ms during the launch burst (pre-fix hangs once rx_op(32) fills).

I held off implementing because this area is crowded with in-flight branches (#3783, devin/…-fanout-backpressure, issue/3289-fleet-fanout-regression) and I don't want to collide. Say the word and I'll PR children 1+2 (safe-now) with the repro test, and/or file standalone child issues for the uncovered ones (2, 4, 5) so they don't duplicate the in-flight work.