fix(backend): serialize concurrent migrations with a Postgres advisory lock

[feiiiiii5]

Fixes #2545

Problem

Every backend replica runs migrate.sh at container start, so rolling out a new version with multiple Cloud Run Job instances fires alembic upgrade head against the same database simultaneously. As reported in #2545, during the v0.13.0 rollout three backend pods ran the 582-line org migration concurrently and only avoided corruption because Postgres row-level locks serialized the writes and the migration guards happened to be idempotent.

Solution

Acquire a session-scoped pg_advisory_lock in Alembic's env.py (online mode) so exactly one process performs the upgrade while any other replica waits, then finds the database already at head:

• Dedicated autocommit connection holds the session-scoped lock for the whole upgrade; Alembic keeps using its own connection/transactions untouched.
• Crash-safe: the lock auto-releases if a holder dies mid-upgrade (session end).
• Bounded wait: pg_try_advisory_lock polling with ALEMBIC_LOCK_TIMEOUT (default 600s); on timeout it raises a clear RuntimeError instead of hanging replicas forever or racing through the upgrade.
• Non-PostgreSQL databases: locking is skipped entirely (dialect guard), so community/non-PG setups are unaffected.
• Covers every entry point: since the lock lives in env.py, manual alembic upgrade, CI jobs, and migrate.sh are all serialized — not just the shell script path.
• migrate.sh got a short comment pointing at the mechanism for discoverability.

Lock key is the fixed constant crc32(b"rhesis-backend-migrations") = 1260693429.

Testing

Verified against Postgres 16 (local Docker, full alembic upgrade head from an empty DB to head with the change active):

Scenario	Result
Fresh DB, single upgrade	✅ migrates to head, lock released after (0 rows in pg_locks)
External holder + ALEMBIC_LOCK_TIMEOUT=6	✅ logs "Another replica is running migrations; waiting up to 6.0s", times out with clear RuntimeError, exit 1
External holder releases mid-wait (ALEMBIC_LOCK_TIMEOUT=60)	✅ warning → "Migration advisory lock acquired" → completes at head, exit 0
Re-run at head	✅ no-op

WARNI [alembic.runtime.migration] Another replica is running migrations; waiting up to 6.0s for the migration advisory lock
INFO  [alembic.runtime.migration] Migration advisory lock acquired

Also: ruff check + ruff format --check clean, bash -n migrate.sh clean.

Note

LLM-assisted contribution, per repo guidelines.

[peqy]

[Improvement] The non-Postgres skip isn’t fully safe: _acquire_migration_advisory_lock() applies execution_options(isolation_level="AUTOCOMMIT") before checking dialect.name, which may fail on SQLite/other dialects. Please move the dialect guard ahead of the autocommit connection/options so non-Postgres setups truly bypass the locking path.

[peqy]

_acquire_migration_advisory_lock() calls connectable.connect().execution_options(isolation_level="AUTOCOMMIT") before the dialect guard. On non-Postgres (e.g. SQLite), that AUTOCOMMIT isolation level can raise and would contradict the “skipped entirely” promise in the PR description.

Fix: check connectable.dialect.name first, and only then open the lock connection + apply the Postgres-specific AUTOCOMMIT option.

[peqy]

Looks fixed in c367047: dialect is checked at the engine level before opening the lock connection / applying AUTOCOMMIT, so non-Postgres setups should now skip locking cleanly.

[feiiiiii5]

Fixed in c367047 — the dialect guard now runs at engine level (connectable.dialect.name) before the lock connection is opened, so non-PostgreSQL setups never touch execution_options(...) at all. Re-ran the full verification against Postgres 16 (fresh upgrade to head, contention wait/acquire, timeout path) after the change.

[peqy]

Yep, that addresses my concern — checking connectable.dialect.name before opening the connection / applying AUTOCOMMIT means non-Postgres setups truly skip the lock path. Thanks for the quick fix + re-test.

[peqy]

Improvement: The current implementation still sets isolation_level="AUTOCOMMIT" before the Postgres dialect check in _acquire_migration_advisory_lock(). SQLAlchemy may reject that option on some non-Postgres dialects, which would break the “skip entirely for non-PostgreSQL” guarantee.

Fix: check connectable.dialect.name (engine-level) before opening/configuring the lock connection, or guard the execution_options(...) call so non-Postgres never sees it.

[peqy]

Looks good. The advisory-lock approach in Alembic env.py should reliably serialize concurrent upgrades, and the non-Postgres guard is now done before opening/applying AUTOCOMMIT. Ship it.