[bug]: Fresh LND hangs on startup migrations when `db.postgres.maxconnections < # of migrations`

[bufo24]

Pre-Submission Checklist

• I have searched the existing issues and believe this is a new bug.
• I am not asking a question about how to use lnd, but reporting a bug (otherwise open a discussion).

LND Version

lnd: 0.21.1-beta, litd: 0.17.0-alpha

LND Configuration

litd.conf:

postgres.maxconnections=3
lnd.db.postgres.maxconnections=6
taproot-assets.postgres.maxconnections=2

Backend Version

Bitcoin core v31.1

Backend Configuration

unrelated

OS/Distribution

unrelated

Bug Details & Steps to Reproduce

1. Setting up a fresh litd node with integrated LND+TAPD
2. SQLD initiates migrations, but hangs after some without timeout or error. The service never gets online.

On the initial start, it managed to do 5 migrations, and after restarting it did 3. The connection limits are probably on the low side, but I can imagine there can be some improvements around either:

• SQL connection handling
• timeouts
• errors

SQLD Logs:

2026-07-30 09:24:32.817 [INF] SQLD: Applying migrations from version=8
2026-07-30 09:24:32.806 [INF] SQLD: Migrating SQL schema to version 8
2026-07-30 09:24:32.785 [INF] SQLD: 8/u graph (46.941011ms)
2026-07-30 09:24:32.737 [INF] SQLD: Applying migrations from version=7
2026-07-30 09:24:32.726 [INF] SQLD: Migrating SQL schema to version 8
2026-07-30 09:24:32.705 [INF] SQLD: 7/u invoice_add_settled_index (18.473711ms)
2026-07-30 09:24:32.685 [INF] SQLD: Applying migrations from version=6
2026-07-30 09:24:32.672 [INF] SQLD: Migrating SQL schema to version 7
2026-07-30 09:24:32.668 [INF] SQLD: Migration 'kv_invoice_migration' (version 7) applied
2026-07-30 09:24:32.653 [INF] SQLD: Applying custom migration 'kv_invoice_migration' (version 7) to schema version 6
2026-07-30 09:24:32.635 [INF] SQLD: Applying migrations from version=6
2026-07-30 09:24:32.623 [INF] SQLD: Migrating SQL schema to version 6
2026-07-30 09:24:32.601 [INF] SQLD: 6/u invoice_migration (22.325874ms)
2026-07-30 09:24:32.577 [INF] SQLD: Applying migrations from version=5
2026-07-30 09:24:32.566 [INF] SQLD: Migrating SQL schema to version 6
2026-07-30 09:24:32.542 [INF] SQLD: Applying migrations from version=5
2026-07-30 09:24:32.529 [INF] SQLD: Migrating SQL schema to version 5
2026-07-30 09:24:32.529 [INF] SQLD: Skipping migration '000004_invoice_expiry_fix' (version 4) as it has already been applied
2026-07-30 09:24:32.529 [INF] SQLD: Skipping migration '000003_invoice_events' (version 3) as it has already been applied
2026-07-30 09:24:32.529 [INF] SQLD: Skipping migration '000002_amp_invoices' (version 2) as it has already been applied
2026-07-30 09:24:32.529 [INF] SQLD: Skipping migration '000001_invoices' (version 1) as it has already been applied
2026-07-30 09:24:32.486 [INF] SQLD: Using SQL database 'postgres://xxx'
2026-07-30 08:39:08.509 [INF] SQLD: 5/u migration_tracker (14.241133ms)
2026-07-30 08:39:08.494 [INF] SQLD: Applying migrations from version=4
2026-07-30 08:39:08.483 [INF] SQLD: Migrating SQL schema to version 5
2026-07-30 08:39:08.462 [INF] SQLD: 4/u invoice_expiry_fix (15.47347ms)
2026-07-30 08:39:08.445 [INF] SQLD: Applying migrations from version=3
2026-07-30 08:39:08.433 [INF] SQLD: Migrating SQL schema to version 4
2026-07-30 08:39:08.413 [INF] SQLD: 3/u invoice_events (24.486699ms)
2026-07-30 08:39:08.387 [INF] SQLD: Applying migrations from version=2
2026-07-30 08:39:08.376 [INF] SQLD: Migrating SQL schema to version 3
2026-07-30 08:39:08.356 [INF] SQLD: 2/u amp_invoices (23.938654ms)
2026-07-30 08:39:08.330 [INF] SQLD: Applying migrations from version=1
2026-07-30 08:39:08.318 [INF] SQLD: Migrating SQL schema to version 2
2026-07-30 08:39:08.295 [INF] SQLD: 1/u invoices (32.835491ms)
2026-07-30 08:39:08.261 [INF] SQLD: Applying migrations from version=0
2026-07-30 08:39:08.233 [INF] SQLD: Migrating SQL schema to version 1
2026-07-30 08:39:08.233 [INF] SQLD: No database version found, using schema version -1 (dirty=false) as base version
2026-07-30 08:39:08.149 [INF] SQLD: Using SQL database 'postgres://xxx'

Expected Behavior

The service could:

• throw an error when it fails to get a connection after a timeout
• reuse SQL connections so migrations run successfully.

Debug Information

No response

Environment

No response

[ziggie1984]

Thanks for the report. We can confirm that this is caused by the current SQL migration implementation.

Each call to ExecuteMigrations creates a new golang-migrate Postgres driver. That driver reserves a dedicated connection from the database/sql pool for migration execution and advisory locking. The driver is neither closed nor reused, so every migration permanently consumes another pool connection for the duration of startup.

After applying each SQL migration, LND starts a separate transaction to update its migration_tracker table. Once all configured connections have been reserved by migration drivers, this transaction waits indefinitely for a connection. Because no context or connection-acquisition timeout applies to the reserved migration connection, no useful error is returned.

There are currently 18 entries in LND's migration configuration. For a completely fresh database, the current implementation can require up to 20 connections:

• one retained while determining the initial schema version;
• one retained for each of the 18 migration entries;
• one additional connection for the migration-tracker transaction.

This also explains why restarting allows the node to make further progress: restarting releases all previously retained connections, and already recorded migrations are skipped on the next attempt.

As an immediate workaround, lnd.db.postgres.maxconnections should be set to at least 20 when initializing a fresh v0.21.1 database. Using the default value of 50 also avoids the startup deadlock.

This is a known issue and was previously addressed by the connection-reuse work in #10426, specifically commit 214af567. That change was never merged, and the current master implementation still creates a separate migration driver for every entry.

We should address the underlying migration-driver lifecycle and connection reuse as part of the v0.22 release rather than relying on a sufficiently large connection pool. A proper fix should also add a regression test using a deliberately small Postgres connection pool.

[bufo24]

Hey @ziggie1984, thanks for the rundown. As a temp solution we've increased the connection limit again. I'll be looking forward to a more efficient way to handle these connection, since we're looking into a setup where 1 postgres instance can serve multiple LND nodes, but needing a big connection pool per node will make it difficult to run. Hopefully #10426 gets merged 🙏

[Vandit1604]

I've been digging into this and can reproduce it. Added a regression test that runs all migrations on a fresh Postgres DB with maxconnections=2: it deadlocks on master (hangs past a 60s deadline), and passes in ~2.5s once the migrate driver is reused. Full sqldb package stays green.

The fix is basically your 214af567 from #10426: build one migrate driver in ApplyAllMigrations and reuse it across the steps instead of creating a fresh one (and leaking its connection) per migration. I kept the per-call ExecuteMigrations/GetSchemaVersion/SetSchemaVersion methods since the test harness still uses them, and mirrored the same change for sqlite.

Since #10426 also bundles the default-maxconnections change and has been sitting a while, would you rather I open a focused PR for just the connection reuse plus the regression test here, or pick up #10426 and rebase it? Happy either way, just don't want to step on your work.