feat(federation): peer lifecycle — disable, removal tombstones, rate limiting (umbrella for #150–#153)

[benders]

FROM @claude: Proposal from a codebase + issue-backlog analysis (2026-07-08). Plan written for execution by a coding agent.

Problem

Federation lifecycle is admission-only. A hub can join a mesh (signed invitation → handshake → gossip) but there is no way to disable, evict, or throttle a peer, and no way for a peer's removal to propagate. In a trust-transitive gossip mesh, removal is a security control, not a convenience: today a compromised or misbehaving peer is permanent, and gossip will keep re-introducing it.

Four open issues are fragments of this one design problem, plus one bug in the same code region. Doing them as a single design pass with one state machine is much cheaper than four independent PRs that would each touch gossip, schema, sync, and admin.

Related issues (this is the umbrella)

• Closes Tombstones & peer removal propagation #150 — Tombstones & peer removal propagation
• Closes Admin enable/disable for discovered peers #151 — Admin enable/disable for discovered peers
• Closes Peer record backup/restore tooling #152 — Peer record backup/restore tooling
• Closes Gossip rate limiting + transitive trust policies #153 — Gossip rate limiting + transitive trust policies
• Refs fix(sync): instances.track_count inflated by duplicate upserts #157 (track_count inflation — verify fixed or fix in passing while touching sync), UI: distinguish 'Last Seen' vs 'Last Sync' for peers #154 (Last Seen vs Last Sync UI — expose new status alongside)

Design: peer state machine

stateDiagram-v2
    [*] --> invited: invitation issued
    invited --> active: handshake verified
    active --> disabled: admin disable (local-only)
    disabled --> active: admin enable
    active --> tombstoned: admin remove (gossiped)
    disabled --> tombstoned: admin remove (gossiped)
    tombstoned --> [*]: never re-admitted without a NEW invitation

Loading

• disabled is local policy: this hub stops syncing from / proxying to / accepting requests from the peer, but does not tell anyone. Reversible.
• tombstoned is mesh-wide: a signed removal record propagates via gossip exactly like admission does. A tombstoned instance id is refused at handshake and ignored in gossip even if a valid-looking invitation chain is presented — re-admission requires a fresh invitation issued after the tombstone (compare timestamps).
• Tombstone records must be signed by the removing hub and carry (instance_id, removed_by, reason?, created_at, signature). Any active peer may issue one (small trusted networks, 4–12 hubs — same trust model as invitations). If this is too permissive in practice, restrict later to the original inviter; note the decision in docs/federation-api.md.

Implementation plan

This is a /federation/* contract change: read docs/federation-api.md FIRST, update it, and bump FEDERATION_API_VERSION in hub/src/version.ts in the same commit as the wire change (AGENTS.md rule). Handle version skew: older peers ignore unknown gossip fields — verify, don't assume.

Phase 1 — schema + state (no wire change)

• Migration in hub/src/db/client.ts migration path (check hub/src/db/schema.sql for current instances shape first): add instances.status TEXT NOT NULL DEFAULT 'active' (invited|active|disabled|tombstoned) + status_changed_at; new table peer_tombstones (instance_id PK, removed_by, reason, created_at, signature). SQLite pitfall: plain ADD COLUMN, index in a separate step.
• Gate every consumer on status = 'active':
  • hub/src/library/sync.ts / sync-peer.ts / hub/src/services/auto-sync.ts — skip non-active peers.
  • hub/src/library/merge.ts — exclude instance_* rows from non-active instances (decide: keep rows but exclude from merge, so re-enable is instant; document in docs/system-architecture.md).
  • Ed25519 request auth (wherever peer signatures are verified — locate via hub/test/federation-signing.test.ts) — reject requests from non-active peers with 403.
  • /proxy/* and /federation/stream/:id — same gate.

Phase 2 — admin surface

• hub/src/routes/admin.ts (hubAdminRoutes plugin): POST /api/admin/hub/peers/:id/disable, POST /api/admin/hub/peers/:id/enable, DELETE /api/admin/hub/peers/:id (creates tombstone). Owner-only, matching existing peer endpoints.
• Frontend frontend/src/features/hub-admin/ peers view: status badge + disable/enable/remove actions; API client functions in frontend/src/lib/api.ts. Show status beside Last Seen/Last Sync (addresses the UI: distinguish 'Last Seen' vs 'Last Sync' for peers #154 confusion).

Phase 3 — tombstone gossip (wire change; bump FEDERATION_API_VERSION)

• hub/src/federation/gossip.ts: include tombstones in the gossip payload; on receipt, verify signature against the remover's pinned pubkey, store, and demote a matching active instance to tombstoned. Handshake path (hub/test/handshake.test.ts locates it): refuse tombstoned ids unless the invitation post-dates the tombstone.
• Update docs/federation-api.md with the record format, verification rules, and re-admission semantics.

Phase 4 — per-peer rate limiting (#153)

• New hub/src/federation/rate-limit.ts: token bucket keyed by instance id, applied via onRequest hook to /federation/* and /proxy/* (after signature verification so the key is authenticated). Config via env (FEDERATION_RATE_LIMIT_RPS, default generous, e.g. 50 rps burst 100). 429 on exceed; never echo internals. .unref() any timers.

Phase 5 — backup/restore (#152)

• scripts/peer-backup.sh + owner-only GET /api/admin/hub/peers/export / POST …/import: JSON of instances + peer_tombstones + own keypair metadata (NOT private key material in the API response — script reads the key file directly with a loud warning). Import is additive + idempotent.

Testing plan

• hub/test/peer-lifecycle.test.ts (new): full state-machine walk; disabled peer's requests 403; disabled peer excluded from merge; re-enable restores without re-sync.
• hub/test/gossip.test.ts / handshake.test.ts (extend): tombstone propagates; tombstoned id refused at handshake; invitation newer than tombstone re-admits; forged tombstone signature rejected.
• hub/test/rate-limit.test.ts (new): bucket math, 429 behavior, per-peer isolation.
• pnpm test:federation is mandatory (docker hub-a/b/c): add a scenario — hub-a tombstones hub-c, assert hub-b learns via gossip and hub-c's proxied streams fail mesh-wide; version-skew smoke (new hub gossiping to a hub without tombstone support must not break admission).
• pnpm verify for everything else.

Acceptance criteria

• A removed peer cannot rejoin via gossip or an old invitation anywhere in the mesh.
• Disable/enable round-trip needs no re-sync.
• FEDERATION_API_VERSION bumped once; docs/federation-api.md fully describes tombstones and rate limits.

Agent execution notes

Read docs/federation-api.md, docs/system-architecture.md, docs/pitfalls.md ("Federation" table — TOCTOU on invitations, timer .unref(), transitive-trust pinning) before Phase 1. Feature branch + Draft PR; one commit per phase; phases 1–3 are the core and should not be split across PRs (stacked PRs are banned).

[benders]

FROM @claude:

Starting phases 1–3 (schema/lifecycle, admin surface, tombstone gossip) on feature/244-peer-lifecycle per owner decision — phases 4–5 (#153 rate limiting, #152 backup/restore) deferred to a follow-up PR, so this PR will close #150/#151 and ref this umbrella.

Plan correction from pre-implementation review: instances.status already exists (online|offline|degraded) and is churned by health checks/auto-sync every poll — the plan's ADD COLUMN status would collide with live liveness state. Lifecycle gets its own column (instances.lifecycle, default 'active'), keeping liveness and admission-state orthogonal.

Backwards compat per owner requirement: FEDERATION_API_VERSION → 7, floor stays 5; tombstones travel as an additive gossip field that v5/v6 parsers must ignore (verifying parser behavior, not assuming).

[benders]

FROM @claude:

Phases 1–3 are implemented and up as Draft PR #250 (closes #150/#151 on merge; #153 rate limiting and #152 backup/restore remain for a follow-up PR per owner decision).

Key deviations/discoveries during implementation:

• The plan's "add instances.status" collided with the existing liveness column — implemented as a separate lifecycle column instead (see earlier comment).
• Backwards compat verified: the tombstones wire change is a sibling field v5/v6 parsers never read; FEDERATION_API_VERSION is now 7 with the floor still 5. A pre-existing bug in proxy/auth.ts compared peers against the current version instead of the floor — the bump would have cut v6 peers off from /proxy/*; fixed to use MIN_FEDERATION_API_VERSION.
• The federation suite caught that /proxy/* has its own auth path the Phase 1 lifecycle gate didn't cover (an evicted peer could still stream/sync). Gate added there too; recorded in docs/pitfalls.md.
• Re-admission (invitation issued after the tombstone) works at both handshake and gossip paths, with a stale-tombstone rejection rule so the mesh converges instead of flip-flopping a re-admitted peer.
• Fixed fix(sync): instances.track_count inflated by duplicate upserts #157 (track_count inflation) in passing with a regression test.

Verification: pnpm verify 739 hub / 115 frontend green, lint zero, Docker federation suite green incl. py-sonic 84/84 and a new end-to-end eviction scenario.