multi: standardize panic recovery at async boundaries

[ziggie1984]

Change Description

This PR introduces shared helpers for recovering from and reporting unexpected
panics in asynchronous processing paths.

The helpers are applied to selected subsystem boundaries, including state
machines, worker tasks, peer processing, channel links, funding, actors, and
message routing. Existing recovery code in the gossip and RPC subsystems is
also migrated to the shared helpers.

Each subsystem retains its existing lifecycle and error-handling semantics.
Recovered failures are reported through structured logging, followed by the
appropriate local cleanup or shutdown behavior.

[github-actions]

🔴 PR Severity: CRITICAL

gh pr view | 22 files | 2381 lines changed

🔴 Critical (3 files)

• funding/manager.go - channel funding workflow coordination
• htlcswitch/link.go - HTLC forwarding / link state logic
• peer/brontide.go - encrypted peer connection (Noise/brontide) handling

🟠 High (3 files)

• discovery/gossiper.go - gossip protocol message handling
• protofsm/state_machine.go - generic protocol state machine used by core FSMs
• rpcperms/interceptor.go - RPC permission/auth interceptor

🟡 Medium (7 files)

• actor/actor.go - actor framework core
• actor/interface.go - actor framework interfaces
• fn/stack.go - new generic stack utility
• log.go - logging subsystem registration
• msgmux/msg_router.go - message router utility
• pool/log.go - worker pool logging
• pool/worker.go - worker pool implementation

🟢 Low (9 files)

• actor/actor_test.go, fn/stack_test.go, funding/manager_test.go, htlcswitch/link_test.go, msgmux/msg_router_test.go, peer/brontide_test.go, pool/worker_test.go, protofsm/state_machine_test.go, rpcperms/interceptor_test.go - test-only changes

Analysis

This PR touches several critical subsystems directly: funding/manager.go (channel funding workflow), htlcswitch/link.go (HTLC forwarding), and peer/brontide.go (encrypted peer connection / Noise protocol handshake and message dispatch). Any of these alone would warrant CRITICAL classification, and together they represent three distinct critical packages being modified in the same PR.

The change is also broad: excluding test files, ~13 non-test files and ~896 lines are modified, which exceeds the 500-line threshold for a severity bump on its own. It also touches discovery/gossiper.go (gossip protocol) and rpcperms/interceptor.go (RPC auth), both HIGH-severity areas, plus supporting utility packages (actor, fn, msgmux, pool) that appear to back a new actor-model/message-routing abstraction being wired into the peer and link code paths.

Given the surface area (peer connection handling, HTLC link logic, and channel funding all changed together) this warrants careful expert review, particularly around message ordering/concurrency guarantees in the new actor/message-router plumbing and its integration into brontide.go and link.go.

---

_{To override, add a severity-override-{critical,high,medium,low} label.}

[TechLateef]

Any reason htlcManager uses a dedicated recoverFromPanic method with manual recover() / fn.PanicStack() calls rather than defer fn.RecoverPanic(...) like peer.readHandler and protofsm?

I noticed this handler also needs to protect the l.failf cleanup path from a secondary panic, so maybe that's the reason. If not, could we inline the callback or have the helper take a callback to keep the recovery style consistent across the PR?

[TechLateef]

%T on msg will always report msgmux.PeerMsg, rather than the actual wire message type. Should this use msg.Message like the log below?

log.Tracef("MsgRouter: sending msg %T to endpoint %s",
    msg.Message, endpoint.Name())

[TechLateef]

Good fix! LGTM

[gijswijs]

Made some remarks throughout. Concept is ack, but it needs some hardening, and wider implementation.

Consider renaming fn/stack.go to fn/panic.go (and fn/panic_test.go)

[gijswijs]

This example contradicts the PR's own goal of standardized structured reporting.

I would provide this example

defer fn.RecoverPanic(func(p fn.Panic) {
        fn.LogRecoveredPanic(ctx, s.log, p)
        // ...subsystem-specific cleanup...
})

[gijswijs]

Before this PR we recovered from panic in the gossiper and emitted the stack trace at Debug level.

A remote peer with a panic-inducing gossip message now generates ~8 KiB of ERROR logs per attempt; per-event size is bounded but repetition is not, and there is no rate limit.

Fix: keep the stack trace at Debug (or add a level parameter to LogRecoveredPanic), or rate-limit repeated panic logs per peer.

[gijswijs]

This recover now has a funny side-effect: There are several hook invocations on the remote commitment path with non-deferred Unlock. If a hook panics, recovery (in htlcManager) fires while the RWMutex is still write-locked: any goroutine with a stale link reference that calls a locking method hangs forever.

defer the unlocks at htlcswitch/link.go:2124-2126, 4496-4498, 4559-4563, 4664-4668 but check if those are all.

[gijswijs]

If this loop panics:

https://github.com/ziggie1984/lnd/blob/4080b75b1ec40036f5ba8adc6f0bc56747ffffdf/funding/manager.go#L1601-L1607

resMtx.RUnlock()will never be called, and we recover, but with a potential deadlock.

Defer resMtx.RUnlock() at L1607

[gijswijs]

Add the following for proper documentation:

// Where the defer is placed decides what happens after recovery:
//
//   - Deferred at the top of a goroutine's main function, recovery ends
//     the goroutine: the unit of work stops (fail-stop). Any cleanup the
//     rest of the system relies on must run in onPanic or in other
//     deferred calls.
//   - Deferred inside a per-item function invoked from a loop, recovery
//     returns from that function and the loop continues with the next
//     item (fail-and-continue). State shared between iterations may be
//     partially applied; only use this posture when items are
//     independent.
//
// Choose the placement deliberately: recovering a state machine driver
// with the per-item pattern resumes it on possibly corrupt state, while
// recovering a request handler with the goroutine-top pattern kills a
// service that could have kept serving.
//
// onPanic runs while the goroutine is unwinding, so treat it as fragile:
// prefer logging the panic's type (%T) over calling methods on values
// involved in the panic, and keep cleanup simple. A panic inside onPanic
// is not recovered and crashes the process.

[gijswijs]

The last line wouldn't be true anymore if you apply my suggestion from #11097 (comment)

[gijswijs]

Only the fundingMsgs select arm of reservationCoordinator is wrapped with recover logic. A
panic in handleInitFundingMsg (RPC-initiated) or the fundingRequests/zombie-sweeper branches on would still kill it.

[gijswijs]

There are still goroutines uncovered.

The spawned goroutines at:4202-4221 and :4275-4281 are unrecoverably, because defers don't propagate across goroutine boundaries.

Fix: add recovery inside each spawned goroutine.

[gijswijs]

We could add a tiny analyzer to check for this misuse.

The recommended fix: a tiny go/analysis analyzer in tools/linters/ _ right next to ll.go, registered the same way as a module plugin in
.golangci.yml, built by the existing make build-native-linter target. The rule: any call to fn.RecoverPanic whose parent node isn't
directly a defer statement is an error. That catches defer func() { fn.RecoverPanic(cb) }(), plain calls, and any future creative misuse
_ repo-wide, at CI time, with zero API churn.

The core is about 40 lines:

// tools/linters/deferrecover.go
func run(pass *analysis.Pass) (any, error) {
inspect := pass.ResultOf[inspect.Analyzer].(*inspector.Inspector)

inspect.Preorder([]ast.Node{(*ast.CallExpr)(nil)}, func(n ast.Node) {
        call := n.(*ast.CallExpr)

        // Resolve the callee by type, not name: catches
        // fn/v2.RecoverPanic regardless of import alias.
        sel, ok := call.Fun.(*ast.SelectorExpr)
        if !ok || pass.TypesInfo.Selections[sel] != nil {
        return
        }
        obj := pass.TypesInfo.Uses[sel.Sel]
        if obj == nil || obj.Pkg() == nil ||
        obj.Pkg().Path() != "github.com/lightningnetwork/lnd/fn/v2" ||
        obj.Name() != "RecoverPanic" {
        return
        }

        // The call is fine iff its parent is a defer of this call.
        // (track parentage via a stack or a second pass over DeferStmts)
        if !directlyDeferred {
        pass.Reportf(call.Pos(),
                "fn.RecoverPanic only works when deferred directly: "+
                "use `defer fn.RecoverPanic(...)` _ wrapping it "+
                "in a closure silently recovers nothing")
        }
})
return nil, nil
}

ymmv

[gijswijs]

No nil guard and no internal recovery around the callback: a nil onPanic, or a callback that itself panics, re-panics during unwinding and kills the
process. How about:

// RecoverPanic stops a panic from unwinding the calling goroutine any
// further and hands the details to onPanic. ...
func RecoverPanic(onPanic func(Panic)) {
r := recover()
if r == nil {
        return
}

// A nil callback would panic below, turning containment into a
// crash.
if onPanic == nil {
        return
}

p := Panic{
        Value: r,
        Stack: TruncatePanicStack(debug.Stack()),
}

// The original panic has been consumed at this point, so a panic
// inside onPanic is a NEW panic that nothing else will catch. Arm a
// second recover that stays pending while the callback runs. There
// is no logger available here, and the logging path itself may be
// the thing that failed, so fall back to stderr _ the same place
// the runtime reports unhandled panics.
defer func() {
        if r2 := recover(); r2 != nil {
        fmt.Fprintf(os.Stderr,
                "fn: panic handler panicked: %v (while "+
                "containing: %v)\n%s", r2, r,
                TruncatePanicStack(debug.Stack()),
        )
        }
}()

onPanic(p)
}

[gijswijs]

This keeps func handleLocalCloseReq uncovered by recovery.