[bug]: externalhosts — a resolved IP change is not broadcast to connected peers or persisted at change time

[ZZiigguurraatt]

Summary

When externalhosts (dynamic DNS) resolves a host to a new IP, lnd updates only its in-memory node announcement — it neither pushes the updated announcement to currently-connected peers nor writes it to the persisted graph. As a result, peers holding a stable connection, peers doing a fresh gossip sync, and lnd's own describegraph keep showing the old address until either that peer reconnects or the periodic ~24h keepalive rebroadcast fires (the only path that both persists and floods) — and that keepalive timer is reset by routine node-announcement regenerations, so it can be postponed well beyond 24h. In practice operators have to force an announcement (e.g. a color change) to make the new address take effect promptly.

lncli getinfo reflects the change within ~5 min (it reads the in-memory announcement), which masks the problem.

Scope (what is and isn't affected)

This is not a blanket network-wide propagation delay. A channel peer that (re)connects is actively pushed the fresh announcement (see maybeSendNodeAnn below) and relays it onward, so a node with peer churn does converge. The gap is specifically:

• peers that stay connected and don't reconnect,
• peers doing a timestamp-horizon gossip sync (served from the persisted graph), and
• lnd's own persisted graph / describegraph,

which all keep the old address until the change is either forced out or picked up by the (possibly long-delayed) keepalive rebroadcast.

Environment

• externalhosts=<host:port> configured.

Steps to reproduce

1. Run a node with externalhosts=mynode.example.com:9735 and steady peer connections; it advertises A.
2. Change the DNS record so the host now resolves to B.
3. Within ~5 min, lncli getinfo | jq '.uris' shows B (in-memory announcement updated).
4. lncli describegraph for our own node still shows A, and already-connected peers still show A; they do not update while the connection stays up.
5. Forcing an explicit announcement update makes B persist and flood to all connected peers immediately, e.g. a color change:

lncli --network regtest peers updatenodeannouncement --color "$(head -c3 /dev/urandom | xxd -p | tr 'a-f' 'A-F' | sed 's/^/#/')"

Root cause

The externalhosts HostAnnouncer applies a resolved change via s.genNodeAnnouncement(...), which only mutates the in-memory currentNodeAnn and re-signs — it calls neither SetSourceNode (persist) nor BroadcastMessage (push to connected peers). The IPAnnouncer wrapper discards the returned announcement. (server.go, the len(cfg.ExternalHosts) != 0 wiring block → genNodeAnnouncement; netann/host_ann.go IPAnnouncer.)

So at change time nothing is persisted and nothing is pushed to connected peers. The change reaches others only through one of:

• A peer reconnecting. On connect, if we share a confirmed public channel, maybeSendNodeAnn (peer/brontide.go) sends GenNodeAnnouncement() — the fresh in-memory announcement, with the new IP — to that peer, which then relays it. This works, but only fires on (re)connect, so it does nothing for peers that stay connected.
• The periodic keepalive rebroadcast (discovery/gossiper.go), which is the only path that calls addNode (persist) and Broadcast (flood all peers). It is gated on now - currentNodeAnn.Timestamp >= RebroadcastInterval (default 24h). But genNodeAnnouncement always applies NodeAnnSetTimestamp, so every regeneration (including each maybeSendNodeAnn on peer connect and channel funding/announce) bumps the timestamp and resets that 24h clock — postponing the persist+flood, potentially well beyond 24h on a busy node.

Separately, peers doing a fresh gossip sync are served node announcements from the persisted graph (discovery/chan_series.go, NodeUpdatesInHorizon), not from in-memory currentNodeAnn — so until the change is persisted they receive the old address too.

For comparison, the explicit RPC path updateAndBroadcastSelfNode (server.go) does both SetSourceNode and BroadcastMessage, which is why the --color workaround takes effect immediately.

This appears to be an oversight carried forward rather than a deliberate choice. The HostAnnouncer was wired to genNodeAnnouncement in its original commit ba3688c3b (2020), before updateAndBroadcastSelfNode existed; that persist+broadcast wrapper was added later for the UpdateNodeAnnouncement RPC, and the externalhosts path was never switched over. No code comment, TODO, or commit message marks the in-memory-only runtime behavior as intentional.

(Approximate locations; line numbers drift.)

Interaction with restart

Because the resolved change is never persisted at change time, a restart before the keepalive rebroadcast's addNode fires discards it: currentNodeAnn is rebuilt from the persisted graph, which still holds the old address. For externalhosts this is not permanent data loss — the HostAnnouncer re-resolves the host on its first refresh shortly after startup and re-derives the new address into the in-memory announcement again (so it returns within a few minutes, provided DNS still resolves to it). What the restart costs is that persistence/broadcast progress is thrown away and the ~24h keepalive clock restarts, and the stale old address is reloaded from the persisted graph (feeding the separate stale-accumulation problem in #10952 / #10968). If DNS resolution fails on that first post-restart refresh, the new address isn't reconstructed at all until DNS recovers, so the node advertises the stale address in the meantime.

The practical consequence: if a node restarts more often than the keepalive interval, the change may never get persisted or flood-broadcast at all — it only ever reaches peers that reconnect.

Is the keepalive/timer behavior intentional? (design context)

Partly yes, and the fix should respect it. The ~24h rebroadcast is a keepalive that avoids being pruned by the network (~2-week staleness) while limiting redundant self-floods, and bumping the timestamp on every regeneration is required for correctness — gossip peers only accept a node_announcement with a strictly newer timestamp. The flaw is not the keepalive; it's that a genuine content change (an address update) is routed through the passive refresh path (genNodeAnnouncement) instead of an explicit persist+broadcast, so it neither persists nor reaches already-connected peers at change time.

Proposed direction & tradeoffs

Route real content changes through an explicit persist+broadcast. Have the HostAnnouncer apply a resolved change via updateAndBroadcastSelfNode (persist via SetSourceNode + immediate BroadcastMessage to connected peers) rather than bare genNodeAnnouncement, decoupled from the keepalive timer.

• Pro: the change persists and reaches connected peers at once; describegraph and gossip-sync responses converge immediately; no reliance on reconnects or the keepalive.
• Con / to handle:
  • Flapping. Broadcasting on every 5-min re-detection of a churny IP could spam the network — add a modest debounce on the change path (not a 24h backoff).
  • Startup ordering. updateAndBroadcastSelfNode calls BroadcastMessage, which needs the gossiper/broadcast machinery running; confirm the HostAnnouncer's first refresh can't fire before that is up.
  • Leave the keepalive rebroadcast and its "skip if recently refreshed" guard unchanged; this change is orthogonal.

Possible follow-up: genNodeAnnouncement doubles as a "get current announcement" getter (maybeSendNodeAnn, funding manager) while having the side effect of minting a new timestamp and re-signing. Splitting the read-only getter from the mutating "produce a new update" operation would stop routine reads from resetting the keepalive timer.

Relationship to existing work

Distinct from #10952 / #10968 (stale-address accumulation), which fix which addresses are advertised (the set) by reconciling at startup. This issue is about when a change is persisted and pushed to connected peers (the timing); #10968's diff does not touch updateAndBroadcastSelfNode, BroadcastMessage, SetSourceNode, or the rebroadcast timer.

Possibly related to the older #7223 (closed, 2022) "IP & Alias changes not seen by network after changing IP", which reports a similar symptom but was left undiagnosed.

Workaround for operators today

Any explicit updatenodeannouncement call takes the persist+broadcast path (updateAndBroadcastSelfNode), forcing the current announcement out to connected peers and to disk immediately. Two options:

Add the new address explicitly (this also persists and broadcasts it). This works even if the HostAnnouncer has already put the new address in the in-memory announcement — the RPC re-signs, persists, and broadcasts regardless:

lncli peers updatenodeannouncement --address_add=<new_ip:port>

Or force a fresh announcement with a content-neutral change such as a random color (does not touch the address set):

lncli --network regtest peers updatenodeannouncement --color "$(head -c3 /dev/urandom | xxd -p | tr 'a-f' 'A-F' | sed 's/^/#/')"