[bug]: SQLSTATE 40001 still causes "invalid revocation" force-close on v0.21.0-beta despite retry logic (regression of #8049?)

[AirHache]

Pre-Submission Checklist

• I have searched the existing issues and believe this is a new bug.
• I am not asking a question about how to use lnd, but reporting a bug (otherwise open a discussion).

LND Version

0.21.0-beta

LND Configuration

/data/lnd/lnd.conf - géré par Ansible

[Application Options]
debuglevel=info
maxpendingchannels=5
listen=localhost
rpclisten=0.0.0.0:10009
tlsautorefresh=true
tlsdisableautofill=true
tlsextradomain=localhost
rpcmiddleware.enable=true
bitcoind.rpchost=127.0.0.1:8332
bitcoind.zmqpubrawblock=tcp://127.0.0.1:28332
bitcoind.zmqpubrawtx=tcp://127.0.0.1:28333
bitcoin.basefee=1000
bitcoin.feerate=1000
minchansize=1000000
accept-keysend=true
accept-amp=true
protocol.wumbo-channels=true
coop-close-target-confs=24
protocol.simple-taproot-chans=true
wtclient.active=true
gc-canceled-invoices-on-startup=true
gc-canceled-invoices-on-the-fly=true
ignore-historical-gossip-filters=1
stagger-initial-reconnect=true
payments-expiration-grace-period=600s
[bolt]
db.bolt.auto-compact=true
db.bolt.auto-compact-min-age=168h
[db]
db.backend=postgres
[postgres]
db.postgres.dsn=
db.postgres.timeout=10s
[Bitcoin]
bitcoin.active=true
bitcoin.mainnet=true
bitcoin.node=bitcoind
[tor]
tor.active=true
tor.v3=true
tor.streamisolation=false
tor.skip-proxy-for-clearnet-targets=true

Backend Version

v30.2.0

Backend Configuration

bitcoin.conf - géré par Ansible

server=1
txindex=1
deprecatedrpc=create_bdb

Network

listen=1
listenonion=1
proxy=127.0.0.1:9050
bind=127.0.0.1

Connections

rpcauth=
rpccookieperms=group
rpcbind=0.0.0.0

ZMQ

zmqpubrawblock=tcp://0.0.0.0:28332
zmqpubrawtx=tcp://0.0.0.0:28333

Electrs

whitelist=download@127.0.0.1

Performances

maxconnections=40
maxuploadtarget=5000
dbcache=2000
blocksonly=0
rpcthreads=16
rpcworkqueue=256

OS/Distribution

• database backend: PostgreSQL (via Patroni HA cluster, 3 nodes, synchronous_commit=on * PostgreSQL version: 18.4 (Ubuntu 18.4-1.pgdg24.04+1) * OS: Ubuntu (Proxmox VM)

Bug Details & Steps to Reproduce

We hit the same "invalid revocation" force-close described in #8049, despite running a version
(v0.21.0-beta) well past the retry-logic fixes referenced in that issue (#7927, #7992, milestone v0.18.0).
This may indicate the retry logic doesn't cover this specific query path, or that retries are
exhausted under prolonged lock contention.

Actual behaviour

A PostgreSQL checkpoint began at 21:46:23 CEST and took 115 seconds to complete (finished 21:48:19).
During this window, several concurrent transactions failed with SQLSTATE 40001. One of these, at
21:47:26.832 CEST, was processing an HTLC update and resulted in LND sending an "invalid revocation"
error to the peer, causing a force-close — even though the actual cause was local database
contention, not an invalid revocation from the peer.

LND log:
2026-07-25 21:47:26.832 [ERR] HSWC: ChannelLink(...): failing link: unable to accept revocation:
unable to restore remote unsigned local updates: ERROR: could not serialize access due to
read/write dependencies among transactions (SQLSTATE 40001) with error: invalid revocation
PostgreSQL log (same timestamp):
2026-07-25 21:47:26.832 CEST ERROR: could not serialize access due to read/write dependencies among transactions
DETAIL: Reason code: Canceled on identification as a pivot, during conflict in checking.
STATEMENT: INSERT INTO channeldb_kv (key, value, parent_id) VALUES($1, $2, $3)
ON CONFLICT (key, parent_id) WHERE parent_id IS NOT NULL DO UPDATE SET value=$2
WHERE channeldb_kv.value IS NOT NULL

Confirmed no Patroni leader change occurred during this window (continuous "I am the leader with
the lock" logs before/during/after, via journalctl -u patroni).

Expected Behavior

Per the retry logic introduced in #7927/#7992, SQLSTATE 40001 should be caught and retried
internally rather than surfaced to the peer as a protocol-level "invalid revocation" error
resulting in a force-close.

Debug Information

The remote peer (running CLN) reported receiving 6 similar "invalid revocation" errors from
various peers since the start of the year, suggesting this may not be isolated to our setup.

Environment

No response

[bitromortac]

I'm looking into this currently.

Edit: Think I have a repro for this and could be related to #10994.

[yyforyongyu]

Think it's related to #8531

[AirHache]

"Think it's related to #8531"

yes, I confirm

[AirHache]

Update: possible root cause is concurrency, not the checkpoint

Follow-up with additional context after digging further into this.

On the checkpoint theory (likely a red herring): the checkpoint at the time of our incident
was a scheduled/timed checkpoint (checkpoint starting: time in the postgres log), not an
emergency one triggered by max_wal_size, and it completed within its allotted budget
(checkpoint_completion_target=0.9). So the checkpoint itself probably isn't the real trigger —
more likely a coincidental overlap.

Stronger hypothesis: concurrent writes to the same channel state. Channel state updates for
a given channel should be inherently serialized by the single TCP connection with the peer, so a
genuine SQLSTATE 40001 conflict on that data is surprising. This matches a comment from
@Roasbeef on the original issue (#8049):

"Interesting that you're running into it as is though, since we have an in process mutex that
should limit to just a single writer."

If that in-process mutex has a gap — e.g. two goroutines updating the upstream and downstream
channel of a forwarded payment at the same time, or handling an incoming message before the
previous one on the same connection has fully committed — that would explain occurrences like
ours despite the mutex existing.

Related issues that may share a root cause family (DB/concurrency handling under load), though
via different trigger mechanisms:

• [bug]: Channel force close due to short database outage and resulting channel failure #8531 — force-close following a genuine ~13s Postgres outage; two of four affected channels
  never recovered, one via SQLSTATE 57P03 leading to the same "invalid revocation" pattern, the
  other via a subsequent HTLC sig-count mismatch after resync.
• [bug]: [wtclient] catch-up after a prolonged tower outage starves a running node — forwarding stops, gRPC hangs, one Postgres txn left open #10994 — a stuck watchtower catch-up left a single Postgres connection idle in transaction
  for 30+ minutes while the rest of the node was otherwise idle (low CPU, no iowait, no active
  queries) — i.e., lnd stopped issuing work while holding an open transaction, rather than
  being blocked waiting on the database. Different trigger, but consistent with the broader
  pattern of lnd's DB/goroutine coordination not being fully reliable under contention.

None of this pins down the exact code path, but the recurring theme across independent reports
is that lnd's assumption of single-writer-per-channel doesn't always hold in practice. Flagging
in case it's useful for narrowing down where to look.

Update: this pattern is common but rarely fatal — severity depends on peer implementation

Went back through logs on all three HA nodes to find every occurrence of this exact
SQLSTATE 40001 → "invalid revocation" pattern, not just the one that caused the force-close
reported above. Found ~12 additional occurrences over the past month, across 9 different
channels with various peers.

Only the original incident actually resulted in a force-close. In all ~12 other cases, the
same sequence happened (local SQLSTATE 40001 → failing link → invalid revocation sent to peer),
but the link simply reconnected and resynced (ChannelReestablish) within seconds to minutes,
and the channel kept operating normally afterward.

I don't have visibility into what implementations those other peers were running, so this isn't
conclusive, but one plausible explanation is a difference in how LND and CLN each handle a
received error message on their end — one giving the link a chance to reconnect/resync before
treating it as fatal, the other force-closing immediately per spec. If that's accurate, the actual
blast radius of this bug may depend heavily on the counterparty's tolerance for a transient error,
which seems worth knowing when prioritizing a fix: the underlying local DB hiccup is apparently
common, but only becomes destructive when the peer on the other end doesn't allow for a resync
attempt first.

[ziggie1984]

How often did this happen to you ?

Can you set the timeout to 0 as a mitigation.

db.postgres.timeout=0

[AirHache]

Thanks for the suggestion. Worth clarifying though: db.postgres.timeout is a
connection/query timeout, not something that governs SERIALIZABLE conflict
detection. Our log showed an explicit Postgres error — "could not serialize
access due to read/write dependencies among transactions (SQLSTATE 40001)" —
which is Postgres actively rejecting the transaction due to a detected conflict,
not a client-side timeout expiring. Setting timeout=0 wouldn't prevent that
specific failure mode.

There's also a documented downside to timeout=0 worth flagging: issue #10994
describes a case where timeout=0 let a Postgres transaction sit "idle in
transaction" for 30+ minutes with no way to self-clear, since nothing ever
times out. So it's not a free mitigation.

On frequency: I've had this happen at least twice with the same confirmed
signature — once in June, and the case documented in this report. I also just
had another force-close from a different peer moments ago, but that peer had
been unstable for several days beforehand, so I haven't confirmed yet whether
it's the same SQLSTATE 40001 signature or just ordinary instability — will
check the logs and follow up if it matches.

[whitslack]

Sorry if this is stating the obvious… LND shouldn't be transmitting any error message to the peer when a local database exception is caught. According to the LN protocol spec, transmitting an error message to the peer means, “I have encountered an unrecoverable error on this channel. Please broadcast your commitment transaction so as to unilaterally close this channel.” Clearly a database serialization conflict is not an unrecoverable condition, so LND should not be transmitting an error message to the peer. It could optionally transmit a warning message — which is not supposed to result in force-closure of the channel — or it could simply drop the connection so as to force a reconnection and resync with the peer. (Of course, it would be even better if LND would properly handle the database serialization conflict.)