feat(stream): runtime source failover when the preferred track source is unreachable

[benders]

FROM @claude: Proposal from a codebase + issue-backlog analysis (2026-07-08). Plan written for execution by a coding agent.

Problem

Source selection happens entirely at merge time: mergeLibraries() flags one track_sources row preferred = 1 (merge.ts, Step 5 ~line 775), and handleStream (hub/src/routes/subsonic.ts:1393) reads only that row (WHERE ts.unified_track_id = ? AND ts.preferred = 1 LIMIT 1, ~line 1426). If the preferred source's hub is offline, every track preferring it fails until the next merge, even when a healthy local or peer copy exists in track_sources. The ranked candidate list is already in the table — we just never consult it at request time.

The whole point of holding multiple sources per unified track is redundancy; today it provides none.

Related issues

• Refs Stale "Active" Streams #146 — "Stale Active Streams": peer liveness isn't consulted at request time.
• Refs UI: distinguish 'Last Seen' vs 'Last Sync' for peers #154 — "Last Seen vs Last Sync": same blind spot, admin-facing.
• Complements Offer different ways to view album library (All, Random, Local, by peer, etc.) #82 (view by source) — no behavior overlap.

Design

sequenceDiagram
    participant C as Client
    participant H as handleStream
    participant DB as track_sources
    participant P1 as Peer A (preferred)
    participant P2 as Local Navidrome (fallback)

    C->>H: GET /rest/stream?id=t<uuid>
    H->>DB: SELECT all sources ORDER BY rank (preferred first)
    H->>P1: signed GET /federation/stream (or proxy)
    P1--xH: connect timeout / 5xx
    Note over H: headers not yet sent → safe to fail over
    H->>P2: stream via SubsonicClient
    P2-->>H: 200 + audio bytes
    H-->>C: pipe response (records actual source in stream_operations)

Loading

Rules:

1. Candidate list: fetch all track_sources rows for the unified track, ordered preferred DESC then by the same ranking merge uses (format rank → bitrate → local tie-break). Extract that ranking into a shared helper so merge and stream time can never drift.
2. Failover window: only fail over while no response headers have been sent to the client. Once piping starts, abort the response (current behavior).
3. What counts as failure: connection error, timeout, or upstream 5xx. An upstream 404/4xx for the preferred source should also advance to the next candidate (the peer may have re-scanned) but must be logged at warn.
4. Liveness short-circuit: an in-memory, TTL-based circuit breaker per instance id (e.g. skip candidates whose instance failed a stream/connect within the last 60 s). In-memory only — no schema change, resets on restart.
5. Activity truthfulness: app.streamTracking.start() currently records sourceKind/sourcePeerId before the upstream fetch (subsonic.ts ~1473). Move the start() call (or update the record) after the winning source responds, so stream_operations reflects the source actually used. Honor the existing pitfall: use resolved params after buildStreamParams, not the raw request.

Implementation plan

1. hub/src/library/source-ranking.ts (new) — export the ranking as (a) an SQL ORDER BY fragment and (b) a comparator, with a unit test asserting both agree. Refactor merge.ts Step 5 (UPDATE track_sources SET preferred = 1 WHERE id IN (…), ~line 783) to use the fragment. Pure refactor of merge behavior — merge output must not change.
2. hub/src/federation/peer-health.ts (new) — minimal circuit breaker: recordFailure(instanceId), isSuspect(instanceId): boolean with a 60 s TTL (make it a constructor arg for tests). No timers held open (.unref() if any — see pitfalls "Concurrency"/"Federation").
3. hub/src/routes/subsonic.ts — in handleStream:
   • Replace the single-row preferred = 1 query with the ranked multi-row query (hoist as a prepared statement at plugin init, per Hoist prepared statements in star/unstar/getStarred2 out of request handlers #130 convention).
   • Wrap the existing local-vs-peer fetch branch (~line 1493 onward) in a for loop over candidates, skipping suspects, breaking on first success. Extract the per-source fetch into a helper fetchFromSource(source, streamParams, rangeHeader) so the loop body stays readable.
   • On total exhaustion: sendBinaryError(reply, 502, "No reachable source for track") (404 only when track_sources has zero rows). Never echo upstream error text (pitfalls: Federation).
4. hub/src/routes/federation.ts — no contract change; /federation/stream/:id untouched. Do not bump FEDERATION_API_VERSION.
5. Docs: update docs/system-architecture.md "Play flow" (step 2/3 become "iterate ranked sources"), add a pitfalls row ("source selection is merge-time preference, stream-time failover").

Testing plan

• hub/test/source-ranking.test.ts (new): SQL fragment vs comparator parity; local tie-break; format/bitrate ordering.
• hub/test/stream.test.ts (extend — existing file covers the stream path):
  • preferred peer connect-refused → falls back to local, response 200, stream_operations row records the local source;
  • preferred source 404 → next candidate wins;
  • all sources down → 502, generic message body;
  • circuit breaker: second request within TTL never contacts the dead peer (assert via mock fetch call counts);
  • transcode params (format, maxBitRate) survive failover — both attempts go through buildStreamParams (pitfall Audio transcoding #77/Player should show actually-streamed format/bitrate, not source file #91);
  • no failover after headers sent (mid-stream abort still aborts).
• hub/test/merge.test.ts: assert preferred flags identical before/after the ranking-helper refactor on a fixture library.
• Run pnpm verify AND pnpm test:federation — this touches /rest/stream; verify alone is insufficient (pitfalls: Subsonic API).

Acceptance criteria

• Kill hub-b in the federation compose stack; tracks whose preferred source is hub-b still play from another source without waiting for a merge.
• stream_operations records the source actually streamed from.
• Zero lint output; merge output byte-identical on unchanged libraries.

Agent execution notes

Read docs/system-architecture.md and docs/pitfalls.md first (per AGENTS.md). Feature branch + Draft PR workflow. Commit phases 1–3 separately.