port(phase 3): relay operations and database pressure from block/buzz - #785
Open
nocodeafrica wants to merge 17 commits into
Open
port(phase 3): relay operations and database pressure from block/buzz#785nocodeafrica wants to merge 17 commits into
nocodeafrica wants to merge 17 commits into
Conversation
## Why Database pressure currently collapses several distinct delays into one symptom. This adds the evidence layer needed to distinguish pool acquisition wait, logical database operation time, advisory-lock wait, and selected transaction duration before changing timeout or retry policy. This is the phase 2 Lane A observability bundle for [#26](TheSentinel454/buzz#26), [#28](TheSentinel454/buzz#28), and [#33](TheSentinel454/buzz#33). It is stacked on #6668. ## What - Record explicit reader/writer checkout wait and acquisition outcomes with `buzz_db_pool_acquire_wait_seconds` and `buzz_db_pool_acquisitions_total`. - Extend the compile-time `#[datastore_span(name = "...")]` seam with `buzz_db_operation_duration_seconds`, so operation labels remain static source literals instead of request data. - Route correctness-critical replacement, membership, push-gate, deletion, and migration/schema-safety advisory locks through one observer without changing their SQL, order, scope, or blocking behavior. - Measure six internally owned transaction lifetimes with `buzz_db_transaction_duration_seconds`, starting after `BEGIN` succeeds and ending after explicit commit/rollback or scope exit. - Emit root slow-operation warnings at 500 ms, logging the first slow completion and then 1/100 per call site with only `operation`, `outcome`, and `elapsed_ms`. - Document names, units, fixed label vocabularies, measurement boundaries, and blind spots in this PR description. Fixed labels are deliberately small: - `pool_role`: `writer`, `reader` - `lock_type`: `replacement`, `membership`, `push_gate`, `deletion`, `migration_schema_safety` - `outcome`: `success`, `error`, `timeout` where SQLx/PostgreSQL can distinguish it accurately - `operation`: compile-time datastore names plus the six closed transaction operation names documented in the runbook No metric or slow warning contains community IDs, event IDs, event kinds, coordinates, d-tags, SQL/query text, query IDs, returned errors, or event content. ## Coverage boundaries - Operation duration is the complete annotated logical function body, not pure SQL execution; it may include implicit checkout, lock wait, nested operations, and application work. Cancelled futures do not reach its completion hook. - Pool timing covers explicit helper checkouts, including proved-reader routing and selected writer-owned transactions. Implicit SQLx checkout through `&PgPool` remains folded into operation duration. - Lock timing covers application-side blocking locks in the five named families. Trigger/stored-procedure locks, channel-TTL locking, the usage try-lock, and the audit service session lock remain outside this slice. - Transaction timing covers only the six wholly owned boundaries documented in the runbook. It excludes pool wait, `BEGIN`, asynchronous rollback cleanup after an early return, and caller-owned `Db::begin_transaction` lifetime. ## Relationship to #6229 #6229 is the incident-driven timeout precursor. This PR does not add or change `statement_timeout`, `lock_timeout`, `idle_in_transaction_session_timeout`, retries, audit durability, or client-visible conflicts. It provides the missing distributions needed to evaluate those policies later and intentionally leaves #6229's open audit retry/durability finding untouched. The branches overlap in `crates/buzz-db/src/lib.rs` and `crates/buzz-db/src/migration.rs`, so a later rebase may need textual conflict resolution, but the behavior is complementary rather than duplicated. ## Risk assessment Moderate-low. The primary risk is instrumentation overhead and added static series. Cardinality is source-bounded, slow logs are sampled/redacted root events, and the lock/transaction changes wrap existing awaits without changing policy or ordering. ## Verification Author workstation: `buzz-tornquist-db-pressure-observability` (`2010927`), exact head `d7cf833e26c528adfcde3917ded80daf6f4ddac9`, parent `6f50e6b2b2a996349149af61d35bdd6a355f77fd`. - `cargo fmt --all --check` — passed - `cargo clippy -p buzz-datastore-tracing -p buzz-db -p buzz-audit -p buzz-search -p buzz-relay --all-targets -- -D warnings` — passed - `cargo test -p buzz-datastore-tracing --quiet` — 4 passed - `cargo test -p buzz-db --quiet` — 109 passed, 200 ignored - `cargo test -p buzz-audit -p buzz-search --quiet` — 16 passed, 25 ignored - `cargo test -p buzz-relay --lib --quiet -- --test-threads=1` — 906 passed, 48 ignored - Native PostgreSQL focused tests for pool success/timeout/error, lock success/contention/timeout/error, replacement, membership serialization, push ordering, deletion fencing, migration/schema exclusion, and reader fallback — 8 passed The default-parallel relay run passed once; subsequent runs exposed the existing load-sensitive `api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echo` 504 at the end of the suite. That test passes in isolation and the full relay suite passes serially. Independent exact-head review workstation: `buzz-tornquist-db-pressure-observability-review` (`2013067`). Formatting, the same all-target clippy command, datastore instrumentation tests, DB unit tests, source privacy guards, and diff/non-goal audits passed; no review findings. Generated with Codex --------- Signed-off-by: tornquist <tornquist@squareup.com> (cherry picked from commit 113a33b7e49b7173ee1767c49ef2f49c63803034) Signed-off-by: Basheer Phiri <phiribash@gmail.com> Colony port notes: - observability.rs lands at runtime/observability.rs (Colony's buzz-db is split into runtime/ and store/); re-exported as crate::observability from lib.rs. - Upstream's migration.rs and replaceable.rs hunks applied to runtime/migration.rs and store/replaceable.rs. - Upstream's lib.rs hunks applied at their Colony homes: runtime/mod.rs (reader boot ping, proved_reader acquire, begin_transaction), store/usage.rs (usage-metrics try-lock acquire), store/replaceable.rs (replace_addressable_event), store/relay_members.rs (publish_nip43_membership_locked). - tests/observability_source.rs include path updated to the new module path. Signed-off-by: Basheer Phiri <phiribash@gmail.com>
## Why
Buzz already reports coarse writer/reader database checkout waits, but
those
signals cannot explain which startup or serving operation is blocked by
pool
pressure. That makes rollout diagnosis and postmortems ambiguous: a
readiness
probe, NIP-42 authentication, authorization check, reconnect history
repair,
event write, and background maintenance can all wait on the same pool
while
appearing identical.
This PR implements Package 2A of the pod-handoff plan: operation-aware
pool
borrow causality. It is observability-only; it does not change
configured pool
sizes, SQL semantics, transaction ordering, or timeout policy. Physical
DNS/TCP/TLS/Postgres authentication and session initialization remain
the
separate Package 2B boundary.
## Metric contract
The final contract separates three questions:
| Question | Metric |
|---|---|
| How long did checkout wait? |
`buzz_db_pool_acquire_duration_seconds{pool_role,operation}` |
| How did the attempt end? |
`buzz_db_pool_acquire_attempts_total{pool_role,operation,outcome}` |
| Who is waiting now for a tracked operation? |
`buzz_db_pool_waiters{pool_role,operation}` |
Outcomes are `success`, `timeout`, `error`, and `cancelled`. Operations
are
`bootstrap`, `readiness`, `tenant_resolution`, `authentication`,
`authorization`, `subscription_history`, `event_write`, and
`maintenance`.
Only these eleven pool/operation pairs are constructible:
```text
writer/bootstrap reader/bootstrap
writer/readiness
writer/tenant_resolution
writer/authentication
writer/authorization reader/authorization
writer/subscription_history reader/subscription_history
writer/event_write
writer/maintenance
```
The duration histogram intentionally does not carry `outcome`. Result
data
remains available on the terminal counter for historical counts and
rates,
without multiplying the expensive histogram family. Nine finite buckets
plus
`+Inf`, sum, and count produce 12 duration series per valid pair.
Together with
four outcome counters and one waiter gauge, the new-family ceiling is
exactly
187 raw Prometheus series per pod, asserted from the production
exporter.
The existing coarse acquisition families remain temporarily for
dashboard
compatibility while the new series are validated in staging.
The operation-specific waiter family covers the explicitly routed,
deployment-critical operations above; it is not a census of every
possible
SQLx checkout in the process. The dashboard pairs it with SQLx pool
active/idle/max gauges for whole-pool capacity context, and treats a
missing
operation series as unknown rather than healthy zero.
## What changed
### Cancellation-safe acquisition ownership
- Add writer- and reader-specific typed operation APIs so invalid label
pairs
cannot be constructed and store modules cannot emit reader labels.
- Own every polled acquisition with one RAII terminal guard.
- Record exactly one duration and terminal outcome for success, timeout,
error,
or cancellation.
- Emit nothing for a future that is created but never polled.
- Balance the operation-specific waiter count exactly once on every
terminal or
dropped future.
- Periodically refresh every expected waiter pair, including healthy
zero, so
missing telemetry is not presented as zero. Reader pairs are emitted
only
when a distinct read pool is configured; a writer-only pod cannot
fabricate
healthy reader-zero state.
### Production attribution
Route the deployment-critical acquisition paths through caller-owned
semantic
entry points, including:
- writer and reader bootstrap;
- the real post-#7149 readiness acquisition and deletion-catalog
validation;
- tenant resolution and community lifecycle checks;
- NIP-42 allowlist authentication;
- membership, moderation, invite, operator, Git, agent-owner, and policy
authorization;
- operator community create/list/archive/unarchive, reverse host/channel
tenant
resolution, and REQ row-community conformance lookups;
- writer/reader subscription history, feed, thread, and routed fallback
paths;
- primary and command event writes, replaceable events, mention
indexing,
reaction/channel/member/archive side effects, and thread metadata;
- push matching, usage rollups/leadership, replica-fence startup and
recurring
probes, periodic reconciliation, channel/deletion reapers, partitions,
and
other bounded maintenance/bootstrap paths.
Shared helpers now accept caller-owned intent or expose named semantic
variants
instead of assigning one misleading operation to every caller. No known
P0 path
uses `other`.
### Readiness and size-one-pool correctness
- Rebase on the post-#7149 readiness implementation and instrument the
actual
`Db::readiness_check` acquisition rather than the superseded ping-only
seam.
- Acquire once for deletion-catalog validation and run its queries on
that
connection, preserving the shared readiness deadline.
- Scope the channel-roster catalog checkout before the behavior probe so
a
size-one writer pool cannot self-deadlock during startup verification.
### Exporter, documentation, and CI
- Register metric HELP/type/unit metadata through the production
Prometheus
builder.
- Configure dedicated checkout buckets at 1ms, 5ms, 10ms, 25ms, 50ms,
150ms,
500ms, 1s, and 3s.
- Add a production scrape-contract test for exact names, labels,
buckets,
valid pairs, sensitive-label exclusion, and the 187-series ceiling.
- Add source mutation guards for the P0 semantic entry points and
raw-checkout
bypasses.
- Add an exact backend-integration CI selector for the production
attribution,
cancellation, readiness, and size-one-pool PostgreSQL tests.
- Document the frozen label vocabulary, valid combinations, semantics,
and
cardinality budget in the Helm chart README.
## Dashboard intent
The new Stage 2 row in **Buzz Startup & Rollout Safety** is
deployment-first:
- baseline-versus-candidate attempts, failure rates, cancellation rates,
and
maximum wait by operation;
- outcome counts and percentages over time by SHA/ReplicaSet;
- acquisition wait heatmap, average, and maximum through the rollout;
- historical waiter pressure beside writer active/idle/max context;
- per-pod postmortem drilldown, including terminated pods;
- a smaller current-waiter table with explicit stale/missing semantics.
Percentile widgets remain disabled until Datadog metadata confirms
percentile
support for the new distribution. Current gauges use no fill,
interpolation, or
`default_zero`; missing means unknown.
## Risk assessment
Moderate. The patch touches many database acquisition call sites, but
preserves
the selected physical pool and executes the same SQL on the acquired
connection. The main risks are incorrect semantic attribution,
cancellation
double-counting, and a helper accidentally acquiring twice. Typed APIs,
production-method PostgreSQL tests, source guards, the raw scrape
contract, and
the size-one-pool regression cover those risks.
No tenant, community, user, pubkey, event, channel, SQL, URL, pod,
version,
ReplicaSet, or request-controlled value is emitted as an application
metric
label. Deployment identity is supplied by infrastructure enrichment.
## Verification
- `cargo fmt --all -- --check` — passed.
- `cargo clippy -p buzz-db -p buzz-relay --all-targets --all-features --
-D warnings`
— passed.
- `cargo test -p buzz-db` — 122 passed, 0 failed, 263 ignored;
source-contract integration test: 3 passed, 0 failed.
- Focused relay compatibility, metric-contract, and readiness tests —
passed.
- `scripts/test-postgres-test-discovery.sh` — passed.
- Full `buzz-relay` package run from the identical tree reached 1,015
passes;
the six media-test failures all stopped in their shared local PostgreSQL
setup with `Sqlx(PoolTimedOut)` because Docker/PostgreSQL was
unavailable.
The same six failed in isolation, while every changed exact test passed.
- Exact implementation head:
`f92910b353086e9edf85918ca5f72190edbbe22f`.
- Exact multi-architecture staging image:
`dev-sha-f92910b353086e9edf85918ca5f72190edbbe22f-run-33607968668-1`
(`sha256:161712c8ed2e265a15df9b63e02248d5973481f875ff129d7d2ae78a09d487a2`).
- Focused staging GitOps PR:
<https://github.com/squareup/builderbot-platform-core-infrastructure/pull/299>
— merged after renderer, inventory, infrastructure test, Kargo, Semgrep,
and
Intersect gates passed; the source/generated-artifact diff was exactly
two
image lines.
- Exact GitHub head reports 47 terminal checks: 35 successful and 12
intentionally skipped. PostgreSQL, unit, lint, security, both server
cross-compiles, backend integration, relay E2E, desktop, mobile, image,
Helm, Semgrep, zizmor, and DCO gates are green.
- Datadog readback identifies two exact-image pods,
`buzz-d79c8d8f7-ckv2l` and `buzz-d79c8d8f7-qzqdp`, in ReplicaSet
`buzz-d79c8d8f7`; both report the full source SHA above.
- Both pods report all eleven allowed pool/operation waiter pairs at
current
zero, with no invalid pair. The acceptance window observed nonzero
success
receipts for readiness, tenant resolution, authorization, subscription
history, event write, and maintenance, and no timeout, error, or
cancelled
outcome. Maximum observed wait was about 101 ms for maintenance and 50
ms
for reader subscription history.
The main **Buzz Startup & Rollout Safety** dashboard now has a live
Stage 2
database row with eight widgets and nineteen fully scoped queries. Final
readback preserved all seven top-level groups, found zero under-scoped
Row 6
queries, and confirmed the tracked-operation waiter boundary in the
panel
descriptions.
Generated with Codex.
---------
Signed-off-by: Ravneet Arora <rarora@squareup.com>
(cherry picked from commit 91ab9d31b8f7249ff1db141ae9d8bb3f2a20bcdd)
Signed-off-by: Basheer Phiri <phiribash@gmail.com>
Colony port notes:
- Skipped, because Colony has no such file or feature: crates/buzz-db/src/store/relay_operators.rs,
crates/buzz-relay/src/readiness.rs (arrives with #7149), the channel-roster fence and
large-roster reconciliation blocks in store/channel_members.rs (arrive with #5765/#6251),
channel canvas Db wrappers, and the Huddle-lifecycle ingest validation.
- Colony's Db::is_relay_member reads the writer directly (upstream routes it), so
ReaderOperation::Authorization is currently only reachable through the metric contract.
- Colony's Db::begin_transaction gains upstream's #[deprecated] alias; the six Colony call
sites now call begin_event_write_transaction, which is exactly what the alias does.
- Colony-only seams the source guard forces to carry an operation label:
store/event.rs insert_block_action_once (event_write), query_latest_owner_authored_heads,
get_last_authored_event_at, query_in_progress_task_heads, query_due_snoozed_task_heads,
claim_task_wake (maintenance), store/relay_members.rs list_relay_owners (maintenance),
store/usage.rs try_lock_usage_metrics (maintenance, per upstream).
- tests/observability_source.rs: dropped the relay_admin_actions/relay_operators span test,
and retargeted four anchors onto Colony's code (thread `impl Db`, channel listing, the
ensure-configured-community fetch_one, the boot-fence section). Two assertions that only
hold once #7149 lands are noted inline and moved to Colony's deletion-catalog readiness probe.
- runtime/observability.rs: the Postgres-only production-label test builds its writer pool
directly (Db::connect_writer_pool arrives with #6229) and asserts ping() instead of
readiness_check (arrives with #7149).
- deploy/charts/buzz/README.md: the new section is appended before "Relay Pod extensions"
since Colony has no readiness-metrics section to anchor it after.
Signed-off-by: Basheer Phiri <phiribash@gmail.com>
…ment) (#6229) ## Why A wedged relay boot pod holding a relation lock can park every other writer in the fleet behind it: DB load pins at pool capacity in `Lock:relation` waits while CPU stays flat, and nothing server-side releases the lock until the holder dies. We hit exactly this in production — ~1,400 sessions queued behind one crash-looping pod's boot transaction for ~20 minutes until kubelet killed the container. ## What Applies session-level Postgres timeouts to every **writer** connection inside the existing single `after_connect` hook in `buzz-db`, all env-tunable through the same `Config::from_env → DbConfig` path as the existing pool-size knobs: | Env var | GUC | Default | Effect | |---|---|---|---| | `BUZZ_DB_LOCK_TIMEOUT_MS` | `lock_timeout` | 5000 | statements waiting on any lock fail fast instead of parking behind a wedged holder | | `BUZZ_DB_IDLE_TXN_TIMEOUT_MS` | `idle_in_transaction_session_timeout` | 60000 | reaps wedged clients idling inside an open transaction while holding locks | | `BUZZ_DB_STATEMENT_TIMEOUT_MS` | `statement_timeout` | 0 (off) | opt-in runaway-statement cap; off by default because startup migrations/backfills legitimately run long statements | `0` disables a timeout (Postgres semantics) and deliberately passes through the env parsing — unlike the pool-size knobs where `0` falls back to the default. The reader pool is untouched: replica sessions never take contended locks and already fail acquire in 150 ms. Deployers tune these via plain env vars (`.env`, or `relay.extraEnv` in the Helm chart) — no code changes needed. ## Behavior change to note With the 5 s default `lock_timeout`, a boot-time migration or backfill that waits >5 s on a lock now errors (surfacing in logs / crash-looping the pod) instead of stalling silently. That is the intended visible-failure-over-fleet-stall tradeoff; deployers with slow contended migrations can set `BUZZ_DB_LOCK_TIMEOUT_MS=0`. ## Testing - `cargo test -p buzz-db -p buzz-relay` — buzz-db green; buzz-relay has 9 failures that also fail on clean `main` in this environment (api::admin/api::media/mesh_demo — unrelated, pre-existing). - New config test covers override / `0`-passthrough / invalid-fallback for all three env vars. - Extended the existing `writer_pool_safety_hook_is_single_and_composed` source-shape test so the timeouts can't drift out of the single `after_connect` hook (SQLx replaces hooks — a second hook would silently disarm the floor guard). - `cargo fmt --check` and `cargo clippy --all-targets` clean for the touched crates. Closest existing PR/issue: none found. Signed-off-by: Basheer Phiri <phiribash@gmail.com> --- **Update Aug 28, 17:06 EDT:** Rebased onto `main` at `a3730784fc` and addressed the latest correctness review. - Ported the timeout policy onto the refactored `buzz-db::runtime` pool constructor and kept the shared env overlay for relay, admin, deletion, and audit writers. - Migration/schema-destruction connections now disable `lock_timeout` and `statement_timeout` for their intentional long wait/DDL path. This supersedes the earlier “Behavior change to note”: contended boot migrations wait for the current migration owner rather than crash-looping after five seconds. - The audit worker now preserves and retries the same entry on PostgreSQL `55P03` lock timeouts, using exponential backoff capped at one second. Other database errors retain the existing terminal error behavior, and retries emit `buzz_audit_log_lock_retries_total`. - Added CI-backed PostgreSQL regressions for writer GUC installation/migration exemption, audit-pool lock timeouts, and worker recovery. The worker regression holds the real audit advisory lock past `lock_timeout`, observes a retry, releases the lock, and proves the original entry is appended exactly once. Current verification supersedes the earlier testing notes: workspace Rust clippy passed with warnings denied; all nine infrastructure-free backend unit-test lanes passed; all three focused PostgreSQL regressions passed against PostgreSQL 17; formatting, diff checks, file-size guards, and desktop frontend checks passed. The Linux Blox workstation could not run the unrelated Tauri native lane because `glib-2.0` is absent, so that platform check is left to PR CI. --- **Update Aug 31, 11:09 EDT:** Rebased onto current `main` at `c3132c3ee9` and reran the requested audit-lock contention scenario on Blox at head `896c3fe9ed`. - `git range-diff` reports both PR commits unchanged by the rebase; the branch remains two commits and the worktree is clean. - `cargo fmt --all -- --check` and clippy with warnings denied passed for `buzz-db`, `buzz-relay`, `buzz-admin`, and `buzz-deletion`. - All three focused PostgreSQL 17 regressions passed: writer session timeout/migration exemption, audit writer timeout bounds, and audit worker recovery of the original entry exactly once. - Live protocol verification used a head-built relay and CLI, native PostgreSQL 17/Redis, `BUZZ_DB_LOCK_TIMEOUT_MS=300`, and an eight-second hold on the community audit advisory lock. The real message was accepted and persisted once while the lock was held; its audit-row count remained zero during contention while retries accumulated. After release, exactly one `event_created` audit row appeared and remained exactly one after an additional two-second duplicate check. The run recorded nine lock-timeout retries, zero audit failures, and event ID `7f8c4ffae28e78555fcf2d56396d6e6c01b3712e5411288dc79e9a54af9d9444`. Generated with Codex --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> (cherry picked from commit 3ed623bb217bf9697b0ce4562529254977e0ea04) Signed-off-by: Basheer Phiri <phiribash@gmail.com> Colony port notes: - runtime/mod.rs: the cherry-pick's "empty HEAD vs upstream block" hunk carried a second `Db::new`. Colony's `Db::new` is kept and now calls `Self::connect_writer_pool(config)`; Colony's `Db::connect_pool(config, url, arm_floor_guard)` is removed, since upstream's `connect_writer_pool` replaces it (upstream has no `connect_pool` at this commit). - BEHAVIOUR CHANGE beyond the timeouts: `connect_writer_pool` also asserts `SHOW transaction_isolation == "read committed"` on every writer connection, which Colony's `connect_pool` did not. A database whose `default_transaction_isolation` is not READ COMMITTED will now fail relay startup. PostgreSQL's default is READ COMMITTED. - runtime/migration.rs: kept Colony's `acquire_writer_with_legacy_metrics` seam from #7195 and added upstream's `SET lock_timeout = 0; SET statement_timeout = 0` exemption on the detached migration connection. - runtime/tests.rs also gains two pre-existing upstream tests that #6229 only amends (`writer_pool_safety_hook_is_single_and_composed`, `writer_pool_rejects_non_read_committed_database_default`); Colony had neither, and the first is the source guard the commit depends on. - ci.yml: all three hunks fit Colony's Relay Suites job; both buzz-relay tests they select (`audit_writer_pool_installs_timeouts_and_bounds_advisory_lock_waits`, `audit_worker_retries_lock_timeout_until_original_entry_is_appended_once`) landed. Signed-off-by: Basheer Phiri <phiribash@gmail.com>
## Why `replica_heartbeat` is updated continuously, but autovacuum heap truncation can take an `ACCESS EXCLUSIVE` lock whose replay cancels concurrent hot-standby reads. Truncating a fixed, single-row table provides no meaningful space benefit. ## What - Add migration 0034 to set `vacuum_truncate = false` on `replica_heartbeat` - Keep the desired-state schema aligned for fresh installations - Extend migration coverage so both paths retain the reloption ## Risk Assessment Low — this disables only autovacuum's heap-truncation phase for a single-row table; normal vacuum cleanup remains enabled. The tradeoff is retaining at most the table's otherwise-truncatable tail pages. ## References - [Investigation thread](https://sq-block.slack.com/archives/C0B2WT43BDH/p1787760831375989) - [PostgreSQL table storage parameters](https://www.postgresql.org/docs/current/sql-createtable.html#SQL-CREATETABLE-STORAGE-PARAMETERS) Generated with Codex Signed-off-by: Basheer Phiri <phiribash@gmail.com> --- **Update Aug 27, 2026:** Hardened fresh `pgschema` bootstraps after review. - Renamed the shared post-apply script to `reconcile-schema-after-pgschema.sql` and made it restore the heartbeat reloption and singleton row idempotently. - Added live catalog and row assertions that stop bootstrap when the database does not match the required state. - Updated every repository `pgschema apply` caller, including the desktop release smoke path. - Added a regression test that requires future `pgschema apply` callers to run reconciliation. - Documented the `pgschema` DML and storage-parameter limitation in `AGENTS.md` and at the reconciliation entry point. - Merged current `main` and renumbered this migration to 0034 after `main` added migration 0033. --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> (cherry picked from commit 0ccf934b88f610f5f235862ecd51e2dcbae2cb74) Signed-off-by: Basheer Phiri <phiribash@gmail.com> Colony port notes: - Migration 0034 lands as migrations/0072_replica_heartbeat_vacuum_truncate.sql (Colony was at 0071); the embedded-migrator count assert goes 71 -> 72 and the new asserts read migrations[71].version == 72. Mirrored into schema/schema.sql. - scripts/attach-schema-partitions.sql is renamed to scripts/reconcile-schema-after-pgschema.sql, keeping Colony's partition-attach body and gaining upstream's heartbeat reloption/singleton reconciliation plus its catalog assertions. Upstream's diff only updated its own callers, so five further Colony references were repointed by hand: scripts/run-real-shell-e2e.sh (a live `psql <` that would have failed on a missing file), two ci.yml path filters (a silent `Detect Changed Paths` miss), and comments in scripts/create-required-extensions.sql, scripts/check-schema-drift.mjs and scripts/check-schema-drift.test.mjs. - The new `every_pgschema_apply_runs_post_apply_reconciliation` test failed on Colony's ci.yml: a three-line comment sat between `pgschema apply` and the reconcile `psql`, putting it one line outside the six-line window. The comment now precedes the apply line; no CI step ordering changed. - Skipped: the AGENTS.md hunk and scripts/run-desktop-release-smoke.sh (Colony has neither that doc item nor that script). Colony's scripts/start-isolated-test-relay.sh wins its conflict unchanged: it migrates with `buzz-admin migrate` rather than `pgschema apply`, so it has no reconcile step to add. Signed-off-by: Basheer Phiri <phiribash@gmail.com>
## Summary
This PR makes relay readiness failures diagnosable without weakening the
existing fail-closed readiness contract. It distinguishes Postgres pool
acquisition from query execution, Redis pool acquisition,
deletion-catalog validation, and the overall two-second deadline;
exports a bounded Prometheus contract for rollout dashboards; and fixes
the concurrency, shutdown, and listener-boundary semantics needed for
those signals to be trustworthy.
## Why
The previous `/_readiness` implementation exposed only an aggregate
ready/not-ready result. During a rollout, operators could not tell
whether a pod was blocked on:
- acquiring a Postgres connection;
- executing the Postgres readiness query;
- acquiring a Redis connection;
- validating the deletion catalog; or
- the shared readiness deadline.
Adding metrics to the existing handler also exposed three correctness
hazards that this PR resolves:
1. the same handler is mounted on both the public application listener
and the private Kubernetes health listener, so public requests could
otherwise distort rollout telemetry;
2. concurrent probes can finish out of order, allowing an older result
to overwrite newer current-state gauges; and
3. a probe started before SIGTERM can finish afterward, incorrectly
return `200 ready`, and resurrect ready gauges while the process is
draining.
## Behavior
### Readiness evaluation
- Postgres, Redis, and deletion-catalog checks still run under one
shared two-second deadline.
- Postgres distinguishes pool acquisition timeout/error from query
timeout/error.
- Redis distinguishes pool acquisition timeout/error. This does not
claim a Redis command round trip.
- The deletion catalog distinguishes operation timeout/error.
- Multiple failures are reported as `multiple_dependencies_failed`;
exhaustion of the shared deadline is reported as `overall_timeout` when
no more specific completed outcome wins.
- Readiness remains fail-closed: every dependency must succeed for `200
{"status":"ready"}`.
### Ordered publication and shutdown
`ReadinessCoordinator` is process-owned and uses one mutex as the
linearization point for probe generations, current-state publication,
and terminal shutdown.
- Every completed dependency attempt may contribute its truthful counter
and duration observation.
- Only the newest admissible probe generation may publish current-state
gauges.
- An older, slower probe cannot overwrite a newer probe's gauges.
- `begin_shutdown()` and probe commit serialize through the same
coordinator.
- Once shutdown commits, an in-flight probe cannot return ready or
publish ready/current dependency gauges, even if its dependency work
later succeeds.
Shutdown without dependency evaluation records only:
- `buzz_readiness_checks_total{reason="shutting_down"}`; and
- `buzz_readiness_state{check="overall"} = 0`.
It does **not** fabricate dependency failures, dependency state changes,
or zero-duration latency samples. If shutdown wins after an in-flight
evaluation actually ran, those completed dependency attempts may remain
as attempt telemetry, but they cannot overwrite shutdown-dominant
current state.
### Listener and response contract
- The private health listener's `/_readiness` route is the sole
authority for rollout readiness telemetry.
- The public application listener retains `/_readiness` for
compatibility, evaluates the same dependencies, and preserves the
existing response shape, but it emits no `buzz_readiness_*` metrics.
- Ready responses remain `200 {"status":"ready"}`.
- Shutdown responses remain `503 {"status":"shutting_down"}`.
- Failed private-health responses include the bounded `reason` plus
`postgres`, `redis`, and `deletion_catalog` booleans.
- Failed public compatibility responses retain the dependency booleans
but omit the new detailed reason.
- No header, query parameter, path value, or other request-controlled
value becomes a metric label.
## Prometheus contract
The final schema is intentionally capped at **99 raw Prometheus series
per pod**.
| Metric | Type | Labels | Raw series/pod |
|---|---|---|---:|
| `buzz_readiness_checks_total` | counter | `reason` | 12 |
| `buzz_readiness_dependency_checks_total` | counter | `dependency`,
typed `outcome` | 11 |
| `buzz_readiness_check_duration_seconds` | histogram | `check` | 72 |
| `buzz_readiness_state` | gauge | `check` | 4 |
| **Total** | | | **99** |
### Closed label sets
`reason`:
```text
ready
shutting_down
postgres_pool_timeout
postgres_pool_error
postgres_query_timeout
postgres_query_error
redis_pool_timeout
redis_pool_error
deletion_catalog_timeout
deletion_catalog_error
overall_timeout
multiple_dependencies_failed
```
Valid `dependency` / `outcome` pairs are enforced by typed enums:
- `postgres`: `success | pool_timeout | pool_error | operation_timeout |
operation_error`
- `redis`: `success | pool_timeout | pool_error`
- `deletion_catalog`: `success | operation_timeout | operation_error`
`check` is `overall | postgres | redis | deletion_catalog`.
The readiness histogram has 15 configured buckets concentrated around
the two-second deadline:
```text
0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5,
0.75, 1.0, 1.25, 1.5, 1.75, 2.0, 2.5, +Inf
```
The contract deliberately removes the redundant overall `result` label
and the histogram `outcome` label. Pod, ReplicaSet, version, rollout,
raw error, SQL, URL, tenant, user, community, pubkey, and
request-controlled values are prohibited application labels;
infrastructure enrichment can supply deployment identity outside the
application metric.
## Postgres production seam and CI
The real `Db::readiness_check` path now supports deterministic
production-seam testing while preserving the same acquisition/query
implementation used by the relay. The isolated PostgreSQL lane
automatically discovers and executes three ignored integration tests
covering:
- a held sole connection causing pool timeout, followed by recovery
after release;
- closed-pool acquisition error;
- acquisition success followed by query timeout;
- classified query error;
- cancellation while waiting for a connection;
- cancellation during an in-flight query; and
- eventual pool recovery with waiter/in-flight state balanced.
This prevents the central SQLx/Postgres behavior from being merely
compiled but never executed in CI.
## Tests and verification
- `cargo fmt --all -- --check`
- `./scripts/test-postgres-test-discovery.sh`
- complete `cargo test -p buzz-db`
- focused `cargo test -p buzz-relay readiness`
- real-router `cargo test -p buzz-relay real_health_route_`
- controlled PostgreSQL execution of all three ignored readiness tests
- production health router -> real `GET /_readiness` -> Prometheus
render assertions
- public-router requests proving zero rollout telemetry
- deterministic out-of-order A/B probe tests
- SIGTERM-during-probe tests proving shutdown dominance
- exported metric name/type/exact-label/bucket assertions
- exported raw-series allowlist and 99-series ceiling assertion
The route-to-scrape regression test uses the production Prometheus
builder and verifies the `2`, `2.5`, and `+Inf` readiness buckets. It
fails if the health recording call, health route, public/health
boundary, generation fence, shutdown fence, or bucket override is
removed.
## Risk assessment
**Medium.** This changes the live readiness publication path and adds
synchronization around probe commit/shutdown. The risk is bounded by:
- preserving the existing dependencies and shared two-second deadline;
- keeping readiness fail-closed;
- using a short, process-local mutex only at begin/commit linearization
points, not across dependency awaits;
- retaining the public compatibility endpoint while isolating its
telemetry;
- enforcing typed, low-cardinality labels and an exported series
ceiling; and
- covering the production router, Prometheus exposition, real PostgreSQL
seam, concurrency ordering, and shutdown races.
## Operational notes
- Dashboards should treat `buzz_readiness_state` as the latest sampled
current state, not as an event stream.
- `shutting_down` should be excluded from dependency-failure alerts
because no dependency failure is implied.
- Do not use `default_zero()` for missing current-state data;
missing/stale is unknown, not healthy.
- Use histogram buckets, heatmaps, max, or average until the Datadog
distribution metadata confirms percentiles are enabled; do not title a
widget p95 before that live readback.
## Non-goals
This PR does not add the broader process-startup lifecycle,
worker/listener supervision, shutdown coordinator, WebSocket/huddle
handoff, client recovery telemetry, dashboard mutations, Datadog
configuration changes, or deployment changes. Those remain separate
follow-up work.
## References
- [Staging dev relay image
runbook](https://github.com/block/buzz/blob/main/docs/staging-dev-relay-images.md)
- Readiness telemetry contract: `deploy/charts/buzz/README.md`
---------
Signed-off-by: Ravneet Arora <rarora@squareup.com>
(cherry picked from commit beb76406c12ab8a7af9b2fcf7547c353e3369c34)
Signed-off-by: Basheer Phiri <phiribash@gmail.com>
Colony port notes:
- crates/buzz-relay/src/readiness.rs is ported at its POST-#7195 shape, not #7149's
original. #7149 supplies 836 lines; #7195 (91ab9d31b8) supplies three further hunks that
this branch already owes: the `DbError` import, `deletion_catalog_check` rewritten to call
`Db::validate_deletion_serving_catalog_for_readiness(deadline)` through the new
`classify_deletion_catalog_result`, and its unit test
`deletion_catalog_deadline_is_a_timeout_not_an_operation_error`.
- runtime/mod.rs: the cherry-pick again produced an "empty HEAD vs upstream block" hunk
carrying duplicate `migrate`/`ping`/`pool_stats`/`read_pool_stats`/`begin_transaction`/
`insert_event_with_serving_write_guard`. Dropped it; only `readiness_check` and
`readiness_check_sql` are new, and they are inserted in #7195's shape
(`observability::acquire_writer_until` with `WriterOperation::Readiness`), not #7149's
bare `tokio::time::timeout_at(deadline, self.pool.acquire())`.
- router.rs: the same misalignment dragged in pre-existing upstream test scaffolding Colony
cannot compile: `spa_state` (needs `config::AdminAuth`), `write_bundle`/`spa_response` and
the four admin-CSP tests (need `ADMIN_CSP`, `header::`), and
`status_payload_exposes_source_and_build_identity` (needs `status_payload`). None belong to
#7149; all dropped. Every test #7149 actually adds is kept.
- This step clears the three #7195 debts recorded in 058f02d: readiness.rs labels (above),
the `acquire_writer_until` assertion in tests/observability_source.rs restored to point at
runtime/mod.rs, the `Db::readiness_check` assertion restored in the observability.rs
production-label test, and metrics.rs `contract_tests` (the 187-series budget) added now
that `readiness_test_recorder` and `configured_prometheus_builder` exist.
- deploy/charts/buzz/README.md keeps both contracts, readiness first then the pool contract,
matching upstream's ordering.
Signed-off-by: Basheer Phiri <phiribash@gmail.com>
## Why Early relay failures can currently appear as a container restart without a trustworthy in-process account of whether crypto, structured logging, configuration, relay identity, or the metrics listener failed. Most of those steps happen before the Prometheus exporter exists, so their chronology belongs in logs rather than metrics. Implements the logs-only early-startup slice of #7238. Post-bind Prometheus exporter supervision is tracked separately in #7284. ## What changed - create a process lifecycle recorder before the Tokio runtime and emit a fixed, versioned JSON schema directly to stderr; - record started and exactly one terminal event for `crypto_init`, `tracing_init`, `config_load`, `key_load`, `metrics_bind`, and the aggregate `process_telemetry` phase; - keep every status and reason bounded and suppress raw errors that could contain credentials, keys, URLs, or other secrets; - return typed metrics-install errors so `metrics_bind` can be classified without logging raw values, while preserving the existing public `metrics::install` API; - document the logs-only evidence contract and add real child-process regressions for success and failure paths. This PR adds **no startup metric families** and no dashboard contract. Existing application metrics remain unchanged. ## Verification Exact head: `8faf7526822a119efa035e58b2b3c59aa67fc81d` - `cargo fmt --all -- --check` - `cargo clippy -p buzz-relay --all-targets -- -D warnings` - relay binary target: 13 passed, 1 PostgreSQL-only test ignored - real relay child-process lifecycle target: 9 passed - full relay package library target: 1,023 passed, 89 ignored; the same six media tests failed at `crates/buzz-relay/src/api/media.rs:1145` with `Sqlx(PoolTimedOut)` because local PostgreSQL is unavailable - three independent exact-head reviews found no correctness, security, compatibility, lifecycle-accounting, logs-only-scope, or test-adequacy finding All exact-head GitHub CI gates are green, including lint, unit tests, PostgreSQL, relay/backend/desktop integration, both Linux server cross-compiles, Windows/macOS builds, and security checks. ## Staging verification - exact multi-architecture image: `dev-sha-8faf7526822a119efa035e58b2b3c59aa67fc81d-run-33708188952-1` - immutable manifest: `sha256:26cad28266a6bb0b0e7081eb6091d374e5489f8bb78c475a4a65737dee86cc67` - image workflow: https://github.com/block/buzz/actions/runs/33708188952 - focused staging deployment: https://github.com/squareup/builderbot-platform-core-infrastructure/pull/314 - replacement ReplicaSet `buzz-d68764bc7` has two Ready pods with zero restarts - Datadog received one complete, contiguous sequence 1-12 from each pod; both end with `process_telemetry/terminal/succeeded` at 3 ms - queries scoped to the replacement ReplicaSet return no data for the removed `buzz_startup_phase_terminal` or `buzz_startup_phase_duration_seconds` families The experimental Row 7 was removed from the Buzz Startup & Rollout Safety dashboard. This logs-only PR deliberately adds no replacement dashboard row. Signed-off-by: Basheer Phiri <phiribash@gmail.com> --- **Update Sep 3, 12:26 ET:** Clarified the review boundary: this PR does not close the broader #7238. Later exporter-task termination is pre-existing runtime behavior and is now explicitly tracked in #7284; no production code or staged image changed in this update. Generated with Codex Signed-off-by: Ravneet Arora <rarora@squareup.com> (cherry picked from commit 88687876f7808a2fd742b7eb2e4b9f87d999ad8d) Signed-off-by: Basheer Phiri <phiribash@gmail.com> Colony port notes: - crates/buzz-relay/src/lifecycle.rs and crates/buzz-relay/tests/boot_lifecycle.rs are byte-identical to upstream. main.rs, metrics.rs and telemetry.rs auto-merged onto Colony's boot order (crypto_init, tracing_init, config_load, key_load, metrics_bind) with no adaptation needed; Colony's Postgres/Redis/migration steps come after metrics_bind and this commit deliberately does not instrument them. - crates/buzz-relay/src/test_support.rs is created from upstream's version minus `database_url()` and its DEFAULT_DATABASE_URL constant: metrics.rs's isolated-child test is the only Colony caller, and the unused helper would fail `-D warnings`. The module is declared `#[cfg(test)] mod test_support;` and the now-redundant per-item `#[cfg(test)]` gates are dropped. - lib.rs conflict: keep Colony's `job_runtime` module declaration AND add `lifecycle`. - deploy/charts/buzz/README.md was expected to conflict and did not; its ten-line early-startup telemetry section is kept, sitting alongside the readiness and pool contracts added in steps 5 and 2. - BEYOND UPSTREAM'S DIFF: ci.yml. Colony's nextest archive names its integration tests explicitly, so `boot_lifecycle` would have been compiled and never run. Added `--test boot_lifecycle` to the archive and a `Startup lifecycle evidence` step in Relay Suites. Upstream runs these nine tests through its own default test selection and touches no workflow file here. Signed-off-by: Basheer Phiri <phiribash@gmail.com>
Pinky, an AI agent, updated this description on Wes's behalf after
taking over the startup investigation.
**Category:** fix
**User Impact:** An EVENT refused by WebSocket admission or handler
saturation receives a correlated `OK(event_id, false, reason)` instead
of an uncorrelated NOTICE, so the client can settle that refusal without
waiting for its publish timeout. Rate-limited refusals also arm client
backoff. This fixes a protocol failure mechanism; it does not establish
that every startup send will succeed or that the reported Desktop
startup incident is fully resolved.
**Problem:** Startup opens several live subscriptions and publishes at
once, and the relay's WebSocket admission gate is a fixed 5-second
window (`ws_admission_budget` = `human_ws_events_per_sec * 5`). If that
shared per-principal quota is exhausted, `enforce_ws_admission`
previously rejected an EVENT with a bare `["NOTICE", reason]`. Quota
pressure is a possible trigger, not proof of the original incident's
complete cause.
A NOTICE carries no event id. Both clients settle a pending publish
*only* from an `OK` keyed by event id (desktop `pendingEvents`, mobile
`_pendingEvents`), so nothing settled — and `handle_text_message`
returns early, so no `OK` ever followed either. The send **could not
fail**; it could only time out at `PUBLISH_TIMEOUT_MS` = 25s. That
explains how this rejection mechanism can produce a roughly 25-second
timeout; attributing the original report to it still requires the actual
startup/send workflow.
The handler-semaphore saturation path had the identical defect, and that
one needs no quota burst to fire.
**Solution:** NIP-01 gives each request type its own acknowledgement
channel, and a rejection is only actionable on the same one. Reject a
REQ with `CLOSED`, an EVENT with `OK(id, false, reason)`, and fall back
to `NOTICE` only where no per-request correlation exists. COUNT refusals
now also use `CLOSED(query_id, reason)` per NIP-45, covering both quota
admission and handler saturation (added in
`cd12c93804b87a24b61075dfd171dc471a0a527f`).
Reason strings are unchanged, so the `rate-limited:` prefix and `retry
in {N}s` hint that existing client gates parse keep working (desktop
`parseRateLimitHint`, mobile `RelayRateLimitGate`, buzz-acp
`set_rate_limit_gate`). Only the frame *type* changes, so
`docs/multi-tenant-relay.md` L7 stays satisfied.
Two notes on how this landed, both worth a reviewer's attention:
1. **A survived mutation became a design change.**
`send_admission_result` originally took a `RejectionTarget` parameter,
and reverting the *second* call site (the per-minute message quota)
survived the whole suite — with Redis unreachable the first quota check
short-circuits, so that line is unreachable in test. Rather than test
around it, the parameter is gone: the target is derived from the frame,
so no call site can name the wrong channel.
2. **The relay fix would have caused a client regression on its own.**
Gate arming lived only in the NOTICE branch. Once rejections arrive as
`OK:false`, `handleOk` failed the send without ever backing off — the
client would retry straight into the same quota. Desktop and Mobile now
arm on a `rate-limited:` OK rejection. ACP was subsequently fixed in
`3b06dd32493596ec650f20abf8805791c50fdc24`: it arms the gate and
re-parks only the refused observer frame, preserving other in-flight
frames. Desktop gets `activateRateLimitIfSignalled` as the single owner
of that prefix test, called from both `handleOk` and the NOTICE branch.
<details>
<summary>File changes</summary>
**crates/buzz-relay/src/rejection.rs** (new)
Owns the admission-rejection concern: `RejectionTarget`,
`rejection_target_for`, `request_rejection_message`,
`send_admission_result`, and `enforce_ws_admission`, moved out of
`connection.rs`. Six tests, two of which drive the real
`enforce_ws_admission` against a real `AppState`.
**crates/buzz-relay/src/connection.rs**
Fix the EVENT handler-semaphore rejection to correlate to the event id;
delegate admission to the new module. Add two tests that drive the real
`handle_text_message` with every handler permit held. Down from 1319 to
1116 lines.
**crates/buzz-relay/src/state.rs**
Widen the existing `test_state` helper to `pub(crate)` so the rejection
tests reuse it rather than adding a ninth copy of `AppState`
construction.
**desktop/src/shared/api/relayRateLimitGate.ts**
Add `activateRateLimitIfSignalled` — one owner for the `rate-limited:`
prefix test, since three inbound frame types now carry it.
**desktop/src/shared/api/relayClientSession.ts**
Arm the gate on a rate-limited OK rejection; route the NOTICE branch
through the same helper. Net zero lines, which keeps this
already-oversized file within the differential ratchet.
**desktop/src/shared/api/relayClientPublishRejection.test.mjs** (new)
Four tests against the real `RelayClient`: a rate-limited OK settles the
pending publish and arms the gate; an ordinary rejection does not arm
it; an accepted OK still resolves.
**mobile/lib/shared/relay/relay_session.dart**
Arm the gate in `_handleOk` for a rate-limited rejection.
**mobile/test/shared/relay/relay_session_test.dart**
Two tests driving the real `publish` + `debugHandleMessage` path.
</details>
<details>
<summary>Validation</summary>
**Mutation-tested — 5 mutations, all now killed.** Each production call
site was reverted to the defective behaviour to confirm a test fails.
This caught two false-negative tests:
| # | Mutation | Result |
|---|----------|--------|
| 1 | `rejection_target_for`: EVENT → `Connection` | 4 tests fail |
| 2 | EVENT handler-semaphore call site → bare NOTICE | **survived at
first** |
| 3 | per-minute quota call site → `Connection` | **survived**; fixed by
removing the parameter |
| 4 | desktop `handleOk` gate arming removed | 1 test fails |
| 5 | mobile `_handleOk` gate arming removed | 1 test fails |
Mutation 2 is the lesson: my first saturation test called
`request_rejection_message` directly, so reverting the real call site
inside the `match` arm left it green. It now drives
`handle_text_message` itself and dies on that mutation.
- `cargo test -p buzz-relay` — 928 passed, 1 failed:
`api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echo`,
**pre-existing**, reproduced with all changes stashed at `4dd4d73de`.
- `cd desktop && npm test` — 5721 passed, 0 failed (full suite).
- `cd mobile && flutter test` — 1876 passed, 0 failed (full suite).
- `just fmt-check`, `just clippy`, `just desktop-check`, `just
mobile-check`, `just file-size-check` — clean. Desktop's 5 biome
warnings are pre-existing (reproduced with changes stashed).
- All 9 pre-push lanes green, including `rust-tests` and
`desktop-tauri-checks`.
**Not verified:** not reproduced end-to-end against a live relay under a
forced quota burst. The causal chain is source-proven and
mutation-proven at the frame level; the ~25s attribution follows from
`PUBLISH_TIMEOUT_MS` but is not directly measured. A packaged-build
click-through would close that gap.
</details>
Related work: #6957 bounds Desktop HTTP event submission, but safe
retained-operation recovery after exhausted/ambiguous outcomes remains
unfinished. #6998 is the separately reviewable Desktop
readiness/duplicate-subscription slice. Neither is claimed to complete
native before/after startup-send validation.
Diagnosis note: `RESEARCH/DESKTOP_STARTUP_SEND_STALL_2026_08_27.md`
(Brain's workspace).
## Current review disposition (2026-08-28)
The [review on
`cd12c938`](block/buzz#6961 (review))
identified ACP's missing rate-limited-OK handling. Commit
`3b06dd32493596ec650f20abf8805791c50fdc24` fixes gate arming, re-parking
the specifically refused observer frame, and the stale NOTICE comment.
Two regressions drive the real frame dispatcher. See [the implementation
and validation
response](block/buzz#6961 (comment)).
The Mobile generation-check inline thread is resolved: its `async
publish` returns a failed Future when superseded; it does not throw
synchronously at invocation. No further production change was indicated
by that comment.
The validation counts above describe the original slice, not a new
rerun. At `3b06dd324`, the current GitHub check rollup has successful
completed test/build checks (non-applicable jobs skipped). The
security-review comment still requires review for the current base/head
range; do not read a green authorization job as a completed security
review. Approval and merge remain human decisions.
---------
Signed-off-by: Brain <1a02c72794dcd0f07058a353bc3a81f4028b8c77c92c87fce6d5c8b85970a20b@buzz.block.builderlab.xyz>
Signed-off-by: Wes <wesbillman@users.noreply.github.com>
Co-authored-by: Brain <1a02c72794dcd0f07058a353bc3a81f4028b8c77c92c87fce6d5c8b85970a20b@buzz.block.builderlab.xyz>
Co-authored-by: Carl <9d00794d3df50972eb8b615511783cab12a77a8fd5dd5edd58073ec73b54bd8b@buzz.block.builderlab.xyz>
(cherry picked from commit 2f3dd850db3afe27e56f18cbcd3548eabdd9b9c2)
Signed-off-by: Basheer Phiri <phiribash@gmail.com>
Colony port notes:
- Colony had already fixed the relay half independently in d030dfe (2026-09-10,
"fix(blocks): acknowledge and retry rate-limited action publications"): a refused EVENT
got NOTICE + OK(id,false,reason) from `early_rejection_messages` in connection.rs. This
adopts upstream's `rejection.rs` shape instead, so future upstream work lands cleanly.
WIRE BEHAVIOUR CHANGE: an over-quota or saturated EVENT now receives ONLY
OK(id,false,reason); the additional NOTICE is gone. Verified safe for every Colony client:
desktop arms its gate from the OK reason (relayEventPublisher.ts) and still from NOTICE for
connection-scoped limits (relayClientInbound.ts), buzz-acp arms from the OK
(`rate_limited_ok_arms_gate_and_reparks_refused_observer_frame`), and mobile reads neither
today. Colony's `early_rejection_messages` and its three tests are removed as superseded.
- Desktop: Colony's `relayEventPublisher.ts` is KEPT in full. Upstream ships a
same-named module of a different design (session-epoch ownership, settle-on-rejection);
Colony's retries a rate-limited publish up to three times behind the shared gate, keys
retries to an exact-id `RelayEventRejection`, and classifies terminal rejections. Its 21
tests plus `relayClientSession.publish.test.mjs` cover this commit's ground against
Colony's actual design, so upstream's `relayClientPublishRejection.test.mjs` is dropped
rather than adapted: it asserts upstream's settle-immediately contract.
`relayClientSession.ts` keeps Colony's side on all six hunks (its inbound dispatch and
publish were already extracted into relayClientInbound.ts and relayEventPublisher.ts).
Taken from upstream: `activateRateLimitIfSignalled` in relayRateLimitGate.ts, the
ReadOnlyRelayClient gate wait plus OK-signal arming, and
`readOnlyRelayClientPublishRejection.test.mjs` - retargeted onto Colony's NativeBridge
(`setNativeBridge` + `createMockNativeBridge`) because Colony no longer reaches into
`window.__TAURI_INTERNALS__`.
- buzz-acp: upstream's two new tests call `handle_ws_message` with its 9-argument signature.
Colony's takes 10 (an extra `ask_tx` and a `RelayPin` instead of a URL string), so both
calls are adapted the same way an existing Colony test in that file already does.
- MOBILE SKIPPED ENTIRELY, and `mobile/lib/shared/relay/relay_session.dart` reverted with it.
The auto-merged hunks reference `_rateLimitGate` and `RelayRateLimitGate`, neither of which
exists in Colony's mobile tree - Colony's mobile relay has no rate-limit gate at all - so
the merged file would not have analyzed. The four tests #6961 adds depend on the same
missing scaffolding (`_ManualTimer`, `_RecordingRelaySocket`, `debugAttachSocketForTest`,
`RelayRateLimitGate(...)`). Porting mobile back-pressure handling is its own piece of work.
Signed-off-by: Basheer Phiri <phiribash@gmail.com>
…ents #6961 moved an EVENT refusal onto the OK channel and dropped the NOTICE that used to accompany it. Current clients settle the pending publish and arm their rate-limit gate from the OK reason, but the relay outlives the app versions connected to it: shipped Colony.app and Canary installs from before Colony started arming from OK (d030dfe) read back-pressure from NOTICE alone, and would retry straight back into the same quota. `send_request_rejection` now emits the correlated OK first, then repeats the reason as a NOTICE, for EVENT targets only. A REQ or COUNT refusal stays a single CLOSED: it already names the subscription, and a second frame would arm the gate twice for one refusal. Remove the compatibility frame once every channel has been on an OK-arming client for a release. Signed-off-by: Basheer Phiri <phiribash@gmail.com>
## Summary - return complete channel rosters instead of truncating at 1,000 members - chunk `event_mentions` inserts inside one transaction so large kind `39002` snapshots remain discoverable by every `p` tag - add a targeted `buzz-admin reconcile-channels --channel <uuid>` force-republish path for stale discovery snapshots - cover a 1,501-member roster, 11,000-tag mention index, and kind `39002` tag construction past member 1,000 ## Why The relay builds NIP-29 discovery and several authorization decisions from `get_members()`, but that helper silently returned only the first 1,000 active members. Desktop then counted the truncated kind `39002` event, while late members could be rejected by roster-scanning member actions. Removing the roster cap exposes PostgreSQL's 65,535 bind-parameter ceiling in mention indexing, so the insert is chunked transactionally to preserve all-or-nothing indexing. The existing reconcilers only fill missing discovery events. The targeted admin option bypasses the separately known 1,000-channel reconciliation-list ceiling and replaces an existing channel snapshot using the configured production relay key. ## Attribution This supersedes and builds on #3166 by @LordMelkor. Thank you for identifying the roster boundary and contributing the original complete-roster and mention-index patch. The production roster/query changes and the two PostgreSQL regressions retain that work's shape; this PR rebases it onto current `main`, adds relay coverage, and adds the targeted repair operation requested for rollout. ## Validation Exact pushed head: `24d02e4f3824150ed84913c9d230e675502e5b12` - `cargo check -p buzz-db -p buzz-admin` - `cargo test -p buzz-db channel::tests::get_members_returns_full_roster_beyond_1000 -- --ignored --exact --nocapture` - `cargo test -p buzz-db feed::tests::insert_mentions_indexes_rosters_past_bind_parameter_cap -- --ignored --exact --nocapture` - `cargo test -p buzz-relay --lib handlers::side_effects::tests::group_members_snapshot_keeps_members_past_one_thousand -- --exact` - `cargo run -q -p buzz-admin -- reconcile-channels --help` - mandatory pre-push hook: branch-skew, desktop checks/typecheck/tests, mobile tests, Rust tests, and desktop Tauri checks all passed on the pushed head ## Rollout 1. Deploy the relay/backend build. 2. Run `buzz-admin reconcile-channels --channel <general-channel-uuid>` with `BUZZ_RELAY_PRIVATE_KEY` configured. 3. Verify the replacement kind `39002` roster count matches the active database membership count. No schema migration or desktop release is required. Fixes #3156 Supersedes #3166 --------- Signed-off-by: Wes <wesbillman@users.noreply.github.com> Co-authored-by: Carl <c7ebe626f000404285d3686e1dc74cc07cc60a9754a150041ba132e14bd3e2ec@buzz.block.builderlab.xyz> (cherry picked from commit e0940927ff381f6a353c637732c7a81886f9639d) Signed-off-by: Basheer Phiri <phiribash@gmail.com> Colony port notes: - The whole buzz-db `lib.rs` half of this commit was ALREADY in Colony, so that conflict resolves to Colony's side untouched: `insert_mentions_in_transaction` (runtime/mod.rs:46), the `MENTION_INSERT_CHUNK_ROWS = 5_000` chunk loop binding through `&mut **tx` (runtime/mod.rs:102), and mention indexing inside the replacement transaction (store/replaceable.rs). The three conflict hunks were upstream's pre-Phase-2 monolithic lib.rs, which Colony has split across runtime/ and store/. - The roster cap itself: upstream's `channel.rs` hunk landed in `store/channel_members.rs`, dropping `LIMIT 1000` from `get_members_with_operation` while keeping the `WriterOperation` parameter #7195 added. `LIMIT 1000` remains on `get_accessible_channels` (3), `get_bot_members` (1) and `list_channels_with_operation` (2) exactly as upstream leaves them: those cap channel lists, not member rosters. - buzz-admin `main.rs`: only the dispatch arm conflicted, because Colony has an `OperatorAnalytics` command where upstream has `Deletions`. Kept Colony's arm and took upstream's `{ channel, relay_key }` destructuring; the whole `reconcile_channels` body, including the `--channel` force-republish path and its refusal to run on an ephemeral key, auto-merged. - buzz-admin `Cargo.toml`: upstream's hunk adds `uuid`, which Colony already depends on further down the list. Taking both produced a duplicate-key manifest error, so the file is left exactly as Colony had it. Signed-off-by: Basheer Phiri <phiribash@gmail.com>
## Summary - detect relay-authored NIP-29 kind 39002 roster snapshots truncated by the former 1,000-member query cap and repair stale large rosters during relay startup - serialize canonical roster capture and replacement with membership writes, preserving tenant, channel, signer, pubkey, and role boundaries through mixed-version deployments - install migration 0032's fail-closed roster fence on the partitioned events table and verify its catalog shape plus behavior before opening relay listeners ## Rollout Migration 0032 is a hard schema-before-code compatibility boundary. Apply migrations before rolling this relay version. Startup refuses to open listeners when the parent/partition triggers are missing, disabled, mis-shaped, or behaviorally inert. For large installations, prefer `buzz-admin migrate` and monitor lock acquisition as documented in the chart README. ## Validation Exact head: `bcbba271f54bc0046a6683007e5a2b70403a11d5` - rebased onto `569308c23c9c2bf620dd3a9a5e4baecbcfa22e16`; the nine-file feature patch is byte-identical to pre-rebase head `be8ea0084f4d4c78c7c2550baad4399e4df8ce73` - pre-push hook passed at exact head: branch-skew, file-size, full Rust unit suite, Desktop Tauri clippy, and Desktop Tauri tests - `cargo fmt --all -- --check` - `cargo test -p buzz-relay group_members_snapshot_keeps_members_past_one_thousand -- --nocapture` - focused CI-mode Playwright regression: `selected relay agents revoked after the invite prompt cause no side effects` passed at exact head - prior exact-patch validation: `large_roster_reconciliation_candidates_respect_snapshot_count_and_signer`, mixed-writer locking/rollback, migration admission, partition trigger coverage, and desired-schema parity regressions ## Review Independent DB/relay review found no blocking issues in the exact feature patch. The concurrency fence holds the established replacement and membership locks on one transaction/connection through replacement; failures roll back both soft-delete and insert. Reconciliation remains tenant/channel/signer scoped and validates exact normalized pubkey-plus-role membership. The prior red Desktop shard was unrelated to this backend-only diff: its mocked mention test exercises no relay, database, or migration path. It reproduced as a timing flake on the old head, passed on retry/base, and now passes locally after rebasing onto current main. --------- Signed-off-by: Wes <wesbillman@users.noreply.github.com> Co-authored-by: Carl <32a2e2c9d428ee08902cab75d956da2c1d235a22d4766b0dd4138bf6e2e5db1d@buzz.block.builderlab.xyz> (cherry picked from commit 24ec6a468ec9d0d425ee58fbfc4d416412c446ad) Signed-off-by: Basheer Phiri <phiribash@gmail.com> Colony port notes: - Migration 0032 lands as migrations/0073_channel_roster_snapshot_fence.sql, body unchanged. Embedded-migrator count 72 -> 73, asserts read migrations[72].version == 73, and the `extract_roster_fence` equality test proves schema/schema.sql installs the byte-identical trigger function so a pgschema bootstrap and a migrated database cannot drift. scripts/reconcile-schema-after-pgschema.sql (step 4's rename of attach-schema-partitions.sql) gains the roster-fence DROP TRIGGER on all eight partitions, keeping Colony's existing drops alongside it. - buzz-db lib.rs was the pre-Phase-2 monolith again in all four conflicts; resolved to Colony's tree. Upstream's lib.rs additions land where Colony keeps them: the three `impl Db` wrappers (`verify_channel_roster_fence`, `lock_member_snapshot`, `list_large_channel_rosters_needing_reconciliation`) go in store/channel_members.rs, and `event_replacement_lock_key` is already `pub(crate)` in store/replaceable.rs. - store/channel_members.rs: the five conflicts carried upstream's whole channel.rs prefix. Kept Colony's `lock_member_snapshot` with its #7195 labels (conflicts 2-4 were upstream's pre-#7195 `pool.begin()` / bare `event_replacement_lock_key` path), took only `verify_channel_roster_fence_catalog`, `verify_channel_roster_fence_behavior`, `LargeChannelRoster` and `list_large_channel_rosters_needing_reconciliation`, and dropped a duplicate `list_channels` and a duplicate CHANNEL_MEMBERSHIP_LOCK_NAMESPACE the merge dragged in (Colony holds `list_channels` in store/channel.rs). - #7195 LABEL DEBT FROM STEP 2 IS NOW PAID: both new functions arrived in their pre-#7195 form and are labelled here - `verify_channel_roster_fence_behavior` acquires through `WriterOperation::Bootstrap`, `list_large_channel_rosters_needing_reconciliation` through `WriterOperation::Maintenance`. buzz-db's source guard enforces it. - `run_migrations_through` is adapted to Colony's pool-based `run_migrations_locked` signature (upstream's takes `&mut PgConnection`). - side_effects.rs references `buzz_db::channel::LockedMemberSnapshot`; retargeted to `buzz_db::channel_members::LockedMemberSnapshot`, which is where Phase 2 put the type. - The startup repair is NOT routed through the lifecycle module. `lifecycle`'s phase vocabulary is frozen and asserted by `schema_and_vocabulary_are_frozen`, it covers only pre-exporter startup, and `boot.finish()` has already run by main.rs:240 while the fence check and repair sit at 770-790. Adding a phase would break the frozen schema and diverge from upstream. The repair is bounded by design: it only visits channels whose canonical roster exceeds the legacy 1,000 cap AND whose live snapshot tag count differs, and it logs through the ordinary tracing path. Signed-off-by: Basheer Phiri <phiribash@gmail.com>
## Problem Kind `30179` (NIP-PMA private managed agent — NIP-44 ciphertext carrying the agent nsec, env vars, prompt) joined `AUTHOR_ONLY_KINDS` in #4593, but only **fresh** installs stopped indexing it: - `migrations/0008` installs the positive FTS allowlist only when `events` is empty; - `migrations/0014` wraps the retained brownfield expression for `30350` alone; - `schema/schema.sql` still carries the legacy negative skip-set without `30179`. So a relay upgraded in place keeps tokenizing `30179` ciphertext into `events.search_tsv`. Found by Wren while reviewing #4999 (which activates publication of 30179 from Desktop). No readable leak: `/query` applies `search_hit_accepted` + `event_visible_to_reader` before serialization and live foreign searches against builderlab returned `[]` — this is the storage-layer privacy invariant (`docs/nips/NIP-PMA.md` deployment step 2) plus wasted FTS work. ## Fix - `migrations/0033_private_managed_agent_fts.sql` — same shape as 0014: capture the current generated expression via `pg_get_expr`, drop/re-add `search_tsv` wrapped with `kind = 30179 → NULL`. Every other kind keeps whatever policy the database already had (fresh allowlist or brownfield skip-set). Rebuilds the GIN index; no heap-wide policy rewrite. - `schema/schema.sql` — add `30179` to the desired-state skip-set. ## Tests - `populated_upgrade_preserves_search_policy_except_for_private_kinds` (renamed from `…_push_leases`): provisions the **legacy negative expression** (migrations 1–7), inserts kind 1 / 30179 / 30350 rows, checkpoints after 0014 (30179 still searchable, 30350 not), then runs to head and asserts 30179 is NULL. **Proven red** with the 0033 file removed: `left: [(1, Some(true)), (30179, Some(true)), (30350, None)]`. - `embedded_migrator_contains_consolidated_initial_schema`: count 32→33, asserts 0033 shape and `schema.sql` parity. - `buzz-search/tests/fts_integration.rs` setup applies 0033 so the FTS tripwires run against the real chain. ## Verification (HEAD 6154cc33d, same shell) - `cargo test -p buzz-db -- --include-ignored --test-threads=1`: 305 passed / 3 failed — the 3 (`create_community_with_owner_enforces_per_owner_limit`, `insert_mentions_indexes_rosters_past_bind_parameter_cap`, `transfer_ownership_returns_limit_reached_for_maxed_transferee`) fail identically on pristine `main` f24971033 with a fresh DB; unrelated to this diff. - `cargo test -p buzz-search -- --include-ignored`: 19 passed (incl. `author_only_kinds_are_storage_level_unsearchable`, `p_gated_persistent_kinds_have_storage_null_tsvector`). - `desired_state_schema_bootstrap_progresses_beyond_fencing` + `run_migrations_applies_consolidated_initial_schema_on_fresh_database`: pass. - `cargo fmt --all --check` clean; `cargo clippy -p buzz-db -p buzz-search --all-targets -D warnings` clean. Independent of #4999 — both main and #4999 are brownfield-exposed today; this lands either order. --------- Signed-off-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz> Signed-off-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Co-authored-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz> Co-authored-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> (cherry picked from commit b3baa56ba83d01a0b7292d81aae9370f5616c62b) Signed-off-by: Basheer Phiri <phiribash@gmail.com> Colony port notes: - KIND REMAP. Upstream's private managed-agent definition is kind 30179. Here 30179 is KIND_COMPANY_PROFILE and the private managed-agent definition is KIND_PRIVATE_MANAGED_AGENT = 30194 (crates/buzz-core/src/kind.rs:121). Every 30179 in the migration becomes 30194; the only remaining 30179 mentions in this commit are comments and an assertion string that pin why. A new assert ties the SQL literal to the Rust constant, so a future kind renumber fails the migrator test instead of silently un-excluding the ciphertext. - Migration 0033 lands as migrations/0074_private_managed_agent_fts.sql (Colony already has its own 0033_discovery_business_search.sql). Count 73 -> 74, asserts read migrations[73].version == 74, migrations dir contiguous 1..74. - schema/schema.sql NEEDS NO CHANGE and the upstream hunk is deliberately not applied. Upstream's desired-state schema carries a negative skip-set and had to gain 30179; Colony's carries a POSITIVE allowlist, `CASE WHEN kind IN (0, 9, 40002, 45001, 45003)`, so a fresh install never indexed 30194 in the first place. Two asserts pin that: the allowlist shape must stay, and it must never gain 30194. - The brownfield gap this migration closes is REAL here. migrations/0008 only installs the positive allowlist when `events` is empty; a populated database keeps the legacy negative skip-set from 0001/0005 (`kind IN (1059, 30300, 30622, 44100, 44101, 44200)`), which tokenizes everything else - including kind 30194 ciphertext. Colony's production relay is populated, so it is on the legacy expression today. - crates/buzz-search/tests/fts_integration.rs: kept Colony's four discovery/usage migration constants and added 0074 after them, so the suite applies the wrap over Colony's real expression chain rather than upstream's. - BEYOND UPSTREAM'S DIFF: ci.yml. `buzz-search` is in neither `just test-unit` nor the nextest archive, so fts_integration.rs compiled and never ran - the same silent-skip trap as boot_lifecycle in #7258. Added `-p buzz-search --test fts_integration` to the archive and a `Full-text search policy` step in Relay Suites. Its `BUZZ_TEST_DATABASE_URL` matches the suite's Postgres, and the file's hardcoded fallback already resolved to the same credentials. - The gate lists `populated_upgrade_preserves_search_policy_except_for_push_leases` as lost; that is upstream's own rename in this commit to `..._except_for_private_kinds`, verified absent upstream at b3baa56ba8 and present under the new name. Signed-off-by: Basheer Phiri <phiribash@gmail.com>
## Summary - apply the existing 2160×3840 video resolution envelope independent of orientation - accept portrait recordings whose short/long edges fit that envelope - retain rejection coverage for either edge exceeding its limit - make the relay error describe the orientation-independent limits ## Why The iOS simulator produces portrait H.264 recordings such as 1206×2622. The relay previously checked `width <= 3840 && height <= 2160`, so the equivalent portrait dimensions were rejected while landscape dimensions were accepted. ## Validation - `bin/just ci` - `cargo test -p buzz-media` (120 passed; one MinIO integration test ignored by its existing live-service guard) - `cargo clippy -p buzz-media --all-targets -- -D warnings` - exact blocked artifact `/Users/judeedwards/.buzz/.scratch/pr5874-mongo-evidence/native/head-20260816T050317Z/video-bitexact.mp4` validated through `validate_video_file`: 1206×2622 accepted Signed-off-by: Carl <5f365698229751c0461f57bb03a4e93134e6e936bd7039ebe7b737282a43c754@buzz.block.builderlab.xyz> Co-authored-by: Carl <5f365698229751c0461f57bb03a4e93134e6e936bd7039ebe7b737282a43c754@buzz.block.builderlab.xyz> (cherry picked from commit 196d62f97c21d053ddf8715d75ef57e92bd0051f) Signed-off-by: Basheer Phiri <phiribash@gmail.com> Colony port note: clean cherry-pick, no conflicts and no adaptation. error.rs is byte-identical to upstream; validation.rs differs only by Colony's own WAV and SVG validators, which this commit does not touch. The resolution envelope, the error text and the three new tests are exactly upstream's. Signed-off-by: Basheer Phiri <phiribash@gmail.com>
…phase3-relay-ops Signed-off-by: Basheer Phiri <phiribash@gmail.com>
Relay E2E failed at e2e_relay.rs:848 with:
expected quota NOTICE, got Ok(OkResponse { accepted: false,
message: "rate-limited: quota exceeded; retry in 5s" })
The test predates #6961 and 1117094: it read the first frame as a NOTICE
because that is the order d030dfe emitted. A refused EVENT is now answered
on its own acknowledgement channel first, so a client keyed by event id can
settle the exact pending publish without waiting out its publish timeout, and
the NOTICE follows as the compatibility frame for clients shipped before Colony
armed back-pressure from the OK reason.
Both frames stay asserted, in order, and the NOTICE must repeat the OK's reason
verbatim, so the compatibility frame keeps end-to-end coverage until it is
removed. The retry-after-expiry path is untouched.
Signed-off-by: Basheer Phiri <phiribash@gmail.com>
Backend Integration step "Database pressure observability PostgreSQL tests" reported "Starting 0 tests across 1 binary (412 tests and 10 binaries skipped)" and exited 4. The filter named two tests under observability::tests::, but #7195 moved them into observability::tests::postgres_tests:: and added three more, so the regex matched nothing. Match the module instead of the two names. That also picks up the three tests CI has never run: deletion_catalog_readiness_records_timeout_and_recovers, production_db_methods_emit_exact_pool_operation_labels and serving_write_gate_records_cancel_timeout_success_and_recovery. Proven locally with cargo-nextest 0.9.136, the version CI installs: old filter -> 0 tests new filter -> 5 tests and every other Relay Suites filter re-checked the same way, all non-zero: session timeouts 1, audit writer pool 1, audit worker retry 1, boot_lifecycle 9, fts_integration 20. Signed-off-by: Basheer Phiri <phiribash@gmail.com>
production_db_methods_emit_exact_pool_operation_labels failed at
observability.rs:1380 with:
real startup fence verification succeeds:
InvalidData("buzz.created_at_floor GUC not set on this pool")
Not a production regression. Db::new -> connect_writer_pool still sets
buzz.created_at_floor as the first statement of its single after_connect hook,
next to the three session timeouts and the isolation assertion. The test was
building a bare PgPoolOptions: a deferral I left in the #7195 port because
connect_writer_pool only arrived with #6229 one step later, and never came back
to. It went unnoticed because the CI filter for these tests matched nothing
until df3cdcd.
The probe now builds its writer pool through Db::connect_writer_pool, so it
exercises the pool the relay actually creates. A live assertion is added to
session_timeouts_install_through_db_new_and_bound_lock_waits, which already
goes through Db::new: SQLx keeps exactly one after_connect hook, so asserting
the timeouts alone would still pass if a future edit dropped the floor guard and
left the serving write fence unarmed on every writer connection.
Two further first-run findings from the same test, which had never executed
here:
1. reader/authorization is unreachable in Colony. Db::is_relay_member reads the
writer directly; upstream routes that one permission read on the bounded
replica arm "by explicit product decision", and its own comment calls it
"not precedent for routing other permission reads". No route_read has ever
existed in relay_members.rs here, at this branch or before the Phase 2 store
split, so nothing was dropped in the port. Routing it would make a revoked
membership readable for up to the freshness budget: a product call, not a
port detail. The attempt and duration assertions are narrowed to the ten
pairs Colony reaches; the waiter-gauge assertion keeps all eleven, because
refresh_pool_waiters must publish the full vocabulary for a healthy zero to
be distinguishable from a missing series. POOL_ACQUIRE_VALID_PAIRS and the
187-series budget are unchanged, and an assert keeps the excluded pair a
valid label combination so the exclusion cannot outlive the routing decision.
2. serving_write_gate_records_cancel_timeout_success_and_recovery calls
db.migrate() unless BUZZ_TEST_SCHEMA_MODE=desired. Relay Suites provisions
Postgres from schema/schema.sql via pgschema, so migration 0001 collides with
42710, type "channel_type" already exists. The env var is now set on that CI
step.
All five postgres_tests verified locally against a scratch database on the local
Postgres, provisioned exactly as CI does (create-required-extensions.sql,
schema/schema.sql, reconcile-schema-after-pgschema.sql): 5 passed.
Signed-off-by: Basheer Phiri <phiribash@gmail.com>
All nine buzz-relay::boot_lifecycle tests failed in Backend Integration with
`spawn buzz-relay child process: Os { code: 13, kind: PermissionDenied }` at
boot_lifecycle.rs:47.
Not an archive problem. actions/upload-artifact does not preserve file modes,
so `target/ci/buzz-relay` and `target/ci/git-credential-nostr` arrive 0644 from
`Download relay artifacts`. This workflow already chmods in all four of its
"Start relay" steps for exactly that reason, but those run near the end of the
job; the nextest steps come first, and boot_lifecycle spawns the relay through
env!("CARGO_BIN_EXE_buzz-relay"), which resolves to that same downloaded file.
Restoring the bit immediately after the download covers every later consumer
rather than only the relay start, and it also covers git-credential-nostr,
which this job never chmods at all.
Verified locally:
- the archive route itself is sound: `cargo nextest archive -p buzz-relay
--bin buzz-relay --test boot_lifecycle` then `cargo nextest run
--archive-file ...` ran 9 tests, 9 passed, so nextest does carry the
non-test binary and extracts it executable
- re-running that with an existing 0644 copy at the extraction path also
passed, with the mode corrected to 0755 on the way through
- and the mechanism directly: exec'ing the relay binary at 0644 raises
PermissionError 13, the exact error CI reported; at 0755 it execs and exits
1 on missing config
Signed-off-by: Basheer Phiri <phiribash@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 3 of the block/buzz upstream port: the instruments and guard rails for relay operations and database pressure. The August incident (relay 500s from a CPU-starved Fly Postgres) had no way to tell pool wait from query time from lock wait; this phase ports the evidence layer, then the policies that use it.
Ten upstream commits, cherry-picked one at a time with
-x -s, each audited before the next began.What landed
113a33b7e42064b65c03runtime/observability.rs.91ab9d31b8058f02d94bbuzz_db_pool_acquire_*by(pool_role, operation), 187 raw series ceiling.3ed623bb21d0a437cb7clock_timeout,idle_in_transaction_session_timeout,statement_timeouton every writer connection.0ccf934b881d7d85a045beb76406c1beedd2db15/_readinesssplit by Postgres pool, Postgres query, Redis, deletion catalog, deadline. Newreadiness.rs.88687876f74fafbedd09lifecycle.rs.2f3dd850dbada188d57crejection.rs.11170947f1e0940927ffac87fe87a6buzz-admin reconcile-channels --channel.24ec6a468e6f51c6998cb3baa56ba861eb0b3c90196d62f97c3dc976a746Migrations
Three, all mirrored into
schema/schema.sql. The migrator count assert moves 71 → 74 and the directory stays contiguous 1..74.0072_replica_heartbeat_vacuum_truncate.sql(upstream 0034). One-lineALTER TABLE, instant.0073_channel_roster_snapshot_fence.sql(upstream 0032). A hard schema-before-code boundary.run_migrations_lockedand relay startup both refuse to proceed when the trigger is missing, disabled, mis-shaped or behaviourally inert, so a relay from this branch will not open listeners against a database that has not applied it. Apply migrations before rolling the relay.0074_private_managed_agent_fts.sql(upstream 0033). Expensive on a populated database:DROP COLUMN+ADD ... GENERATED ... STOREDrewrites the wholeeventsheap and rebuilds the GIN index under ACCESS EXCLUSIVE inside the migration transaction, with nolock_timeoutand noCREATE INDEX CONCURRENTLY. Downtime is proportional to the size ofevents. Size production before the window; 0073 and 0074 land in the same one.A ported test asserts
schema.sqlinstalls the byte-identical roster-fence trigger function as migration 0073, so apgschemabootstrap and a migrated database cannot drift.Kind remap: upstream 30179 → Colony 30194
Upstream's private managed-agent definition is kind 30179. Here 30179 is
KIND_COMPANY_PROFILEand the private managed-agent definition isKIND_PRIVATE_MANAGED_AGENT= 30194. Migration 0074 and its tests use 30194 throughout. Two asserts keep it that way:assert_eq!(buzz_core::kind::KIND_PRIVATE_MANAGED_AGENT, 30194)— a renumber fails the migrator test instead of silently un-excluding the ciphertext.assert!(!private_agent_fts.contains("30179"))— 30179 must stay indexable.The brownfield gap is real here:
0008_fresh_install_search_allowlist.sqlinstalls the positive allowlist only wheneventsis empty, so a populated database keeps the legacy negative skip-set from 0001/0005 and tokenizes kind-30194 ciphertext today. The rewrite purges it in the same statement that installs the exclusion.The NOTICE compatibility frame, and when to remove it
#6961 moves an EVENT refusal onto the
OKchannel and drops theNOTICEthat used to accompany it. Current clients settle the pending publish and arm their rate-limit gate from theOKreason, but the relay outlives the app versions connected to it: shipped Colony.app and Canary installs from befored030dfe39fread back-pressure fromNOTICEalone and would retry straight back into the same quota.So
send_request_rejectionemits the correlatedOKfirst, then repeats the reason as aNOTICE, for EVENT targets only — a REQ or COUNT refusal stays a singleCLOSED, which already correlates. Two tests pin both halves.Remove the second frame once every channel has been on an OK-arming client for a release. The condition is written in the code comment and the commit body as well as here.
Deliberate divergences from upstream
schema/schema.sqlkeeps Colony's positive allowlist. #6822 adds 30179 to upstream's negative skip-set; Colony's schema isCASE WHEN kind IN (0, 9, 40002, 45001, 45003) THEN to_tsvector(...) ELSE NULL, so a fresh install never indexed 30194 and there is no list to add to. Applying upstream's line would have replaced the allowlist with a skip-set and started indexing every kind Colony deliberately excludes. Two asserts pin the allowlist shape and forbid 30194 joining it.relayEventPublisher.tsis kept in full. Upstream #6961 ships a same-named desktop module of a different design; Colony's retries a rate-limited publish up to three times behind the shared gate, keys retries to an exact-idRelayEventRejection, and classifiesinvalid:/auth-required:/restricted:as terminal. Upstream'srelayClientPublishRejection.test.mjsasserts the settle-immediately contract Colony does not have and is not taken; Colony's 21-test suite covers the same ground.mobile/lib/shared/relay/relay_session.dartand left it referencing_rateLimitGateandRelayRateLimitGate, neither of which exists in Colony's mobile tree — the file would not have analyzed. Colony's mobile relay has no rate-limit gate at all, before or after. Porting mobile back-pressure handling is its own piece of work.lifecycle. That module's phase vocabulary is frozen by test and covers only pre-exporter startup;boot.finish()runs atmain.rs:240while the repair sits at ~770. It is bounded by query instead: only channels whose canonical roster exceeds 1,000 members and whose live snapshot differs are visited, so a healthy relay does zero work.Not taken
57216c942f— skipped in Phase 2; both hunks land in tests Colony never had.d12dea4e67, community deletion in versioned media buckets — deliberately not taken. Colony's media bucket on Tigris has no versioning, and communities here are archived rather than deleted, so the code would be unreachable and untestable. Revisit if either changes.CI wiring beyond the upstream diffs
Three suites would have compiled and never run, because Colony's nextest archive names its integration tests explicitly:
--test boot_lifecycle+ a Startup lifecycle evidence step (9 child-process tests, no database needed).-p buzz-search --test fts_integration+ a Full-text search policy step — the only place in CI where the brownfield search-policy migrations are executed against a real database.--test fts_integration's credentials were checked againstBUZZ_TEST_POSTGRES_PASSWORD, not assumed.Also:
Detect Changed Pathsfilters follow step 4's rename ofscripts/attach-schema-partitions.sqltoscripts/reconcile-schema-after-pgschema.sql, and a ported test caught that a three-line comment had pushed the reconcile step outside its six-line window in both CI blocks.Verification
Every step ran an identifier gate (each file's
fn/struct/enum/trait/type/const/static/modnames at the previous commit vs now) plus a whole-crate sweep. Across all eleven commits the gate listed exactly five identifiers, each an intended upstream replacement, each verified absent upstream at its own commit:enum PoolRoleWriterOperation/ReaderOperation(#7195)fn connect_poolDb::connect_writer_pool(#6229)fn readiness_handlerpublic_readiness_handler+kubernetes_readiness_handler(#7149)fn early_rejection_messages+ 3 testsrejection.rs'sRejectionTargetand its 7 tests (#6961)populated_upgrade_preserves_search_policy_except_for_push_leases..._except_for_private_kinds(#6822)Per step:
cargo checkandcargo clippy --all-targets -- -D warningsclean,cargo fmtwith no drift, and a symbol census against the upstream tree so a dropped hunk shows up as a count mismatch rather than a silent gap.Final state:
cargo check/clippy -D warningsclean forbuzz-db,buzz-relay,buzz-admin,buzz-searchandbuzz-media;cargo test -p buzz-db --lib132 passed; the observability source guard, the 187-series budget test, all 9boot_lifecycleintegration tests and 51 desktop node tests green.origin/canary/upstream-portmerged in ata00235c9b3with no conflicts; the native inventory regenerates to no diff.Not covered here: everything
#[ignore = "requires Postgres"], which is most of the new roster, fence and FTS coverage. Relay Suites is the gate for those.🤖 Generated with Claude Code