[bug]: kv_invoice_migration fails on legacy AMP invoices with empty AMPState ("migrated record does not match original record")

[apotdevin]

Pre-Submission Checklist

• I have searched the existing issues and believe this is a new bug.
• I am not asking a question about how to use lnd, but reporting a bug (otherwise open a discussion).

LND Version

v0.21.1-beta (embedded in lightning-terminal v0.17.0-alpha, rev 08934d)

LND Configuration

Relevant parts of lnd.conf (running under litd; DSN and identity settings removed):

bitcoin.mainnet=true
bitcoin.node=neutrino
db.backend=postgres
db.use-native-sql=true
db.postgres.maxconnections=8

The database was migrated from bbolt to postgres with lndinit migrate-db
(which completed successfully); the failure happens afterwards, during lnd's
own kv→native-SQL invoice migration at first startup.

Backend Version

neutrino (built-in), against private Bitcoin Core v31.1 peers serving compact filters

Backend Configuration

Not relevant to this bug (it triggers during DB schema migration, before any
chain backend interaction), but for completeness: neutrino against private
Bitcoin Core peers with compact filters enabled.

[neutrino] connect=<private bitcoind peers>

OS/Distribution

Linux (Kubernetes StatefulSet, official lightninglabs docker images), postgres on AWS RDS

Bug Details & Steps to Reproduce

1. Take a long-lived node whose bbolt channel.db contains an AMP invoice settled on lnd 0.13.x (2021) — i.e. an invoice that has AMP HTLCs (htlc.AMP populated with root_share/set_id/child_index) but an empty AMPState map, because per-set-ID AMP state tracking didn't exist yet when it was settled.
2. Migrate the node's bbolt databases to postgres with lndinit migrate-db (succeeds).
3. Start lnd with db.backend=postgres and db.use-native-sql=true.
4. The native SQL migration kv_invoice_migration (schema version 7) fails its post-migration verification with migrated record does not match original record: the re-fetched SQL invoice has an empty Htlcs map, while the original KV invoice has the AMP HTLC. The migration transaction rolls back and lnd exits.

Root cause (from reading invoices/sql_migration.go): plain HTLCs are inserted into invoice_htlcs for every circuit key, but the AMP sub-invoice rows (InsertAMPSubInvoice / InsertAMPSubInvoiceHTLC) are only written by iterating invoice.AMPState:

for setID, ampState := range invoice.AMPState {
    ...
    err := tx.InsertAMPSubInvoice(ctx, params)
    ...
    // Now we can add the AMP HTLCs to the database.

For a legacy AMP invoice with empty AMPState, no AMP rows are written at all. Since fetching an AMP invoice reconstructs its HTLCs through the AMP tables, the verification fetch returns the invoice with zero HTLCs and the deep-compare fails. Any node that received AMP payments before AMPState tracking was introduced (~0.14) can never complete this migration.

The node is otherwise healthy — setting db.use-native-sql=false boots it fine on the KV schema (that's our current workaround).

Expected Behavior

The kv_invoice_migration should handle settled legacy AMP invoices whose AMPState map is empty — e.g. by reconstructing the sub-invoice state from the HTLCs' AMP records (set_id is available on each HTLC via htlc.AMP.Record.SetID()), or by explicitly skipping AMP-table population for these invoices and relaxing the verification accordingly — instead of failing the whole migration and leaving the node unable to start with db.use-native-sql=true.

Debug Information

[INF] SQLB: Closing database walletdb
[INF] SQLB: Closing database channeldb
[ERR] LTND: faild to run migrations for the native SQL store: error applying migration 'kv_invoice_migration' (version 7) to schema version 6: failed to migrate invoices to SQL: migrated record does not match original record: in
--- Expected
+++ Actual
@@ -10,38 +10,7 @@
SettleIndex: (uint64) 209,
State: (invoices.ContractState) Settled,

• AcceptTime: (time.Time) 2021-09-06 16:36:18.981662 +0000 UTC,
• ResolveTime: (time.Time) 2021-09-06 16:36:18.981662 +0000 UTC,
• Expiry: (uint32) 699386,
• State: (invoices.HtlcState) 2,
• CustomRecords: (record.CustomSet) (len=1) {
• (uint64) 65536: ([]uint8) (len=181 cap=1024) { <181-byte text payload> }
• },
• WireCustomRecords: (lnwire.CustomRecords) ,
• AMP: (*invoices.InvoiceHtlcAMPData)(...)({
• Record: (record.AMP) root_share= set_id= child_index=4138590185,
• Hash: (lntypes.Hash) ,
• Preimage: (*lntypes.Preimage)(...)()
• })
• })

• Htlcs: (map[models.CircuitKey]*invoices.InvoiceHTLC) {
  },
  AMPState: (invoices.AMPInvoiceState) {
  },

Note the original (Expected) record: the invoice HAS an AMP HTLC but its
`AMPState` is empty `{}` — the signature of an AMP invoice settled before
AMPState tracking existed. The invoice was settled 2021-09-06 (lnd 0.13.x era).

### Environment

lnd embedded in litd v0.17.0-alpha, running in Kubernetes (official docker images). Postgres 16 on AWS RDS. Databases migrated from bbolt with lndinit migrate-db immediately before first boot; lndinit completed successfully and the failure is reproducible on every subsequent start with db.use-native-sql=true. Setting db.use-native-sql=false works as expected.

[ziggie1984]

Thank you for reporting this issue. You can still set db.use-native-sql=false and run the node in kv-sql-schema until a fix is out which should be in the next couple of weeks.