Skip to content

Fix workflow scheduler pagination - #5092

Draft
jacobandjacob wants to merge 1 commit into
block:mainfrom
jacobandjacob:big-sol/fix-workflow-scheduler-pagination
Draft

Fix workflow scheduler pagination#5092
jacobandjacob wants to merge 1 commit into
block:mainfrom
jacobandjacob:big-sol/fix-workflow-scheduler-pagination

Conversation

@jacobandjacob

Copy link
Copy Markdown

What changed

  • Replace the globally capped schedule scan with keyset pagination ordered by the globally unique (created_at, community_id, id) tuple.
  • Process each page incrementally while retaining only active workflow identities for interval-state pruning.
  • Add a matching partial PostgreSQL index in migration 0029 and the canonical schema.
  • Anchor cron recovery to the exact previous successful scan and apply a documented latest-only catch-up policy.

Why

The scheduler previously loaded only the 1,000 oldest active schedules. Once the global fleet exceeds that limit, every newer schedule is permanently invisible. Long or failed passes could also leave uncovered cron time.

This change removes the starvation boundary without unbounded definition memory. Tenant-scoped workflow IDs remain safe at page boundaries, and delayed passes attempt the latest missed occurrence without replaying every occurrence in a storm.

The hosted incident that led to this investigation still requires production schedule counts or relay logs for definitive attribution. After deployment, an automatic canary should pass before hosted schedules are considered reliable.

Validation

  • cargo +stable test -p buzz-db -p buzz-workflow — 94 + 156 passed at commit 5198decfee48177af95b410ed1a40ea3467ee20c
  • Fresh PostgreSQL migration regression — passed with migration 0029
  • Live PostgreSQL 1,001-row pagination regression — passed with the same workflow UUID and timestamp in two communities straddling the page boundary
  • Independent review by Big Sol — approved with no further code changes requested

Co-authored-by: Jacob Greif <greifj@gmail.com>
Signed-off-by: Jacob Greif <greifj@gmail.com>
@Chkhikvadze

Copy link
Copy Markdown

Production follow-up (2026-08-19): hosted native schedule fires are still not reaching newer definitions; a 120-second external recovery watchdog remains the effective launcher. This fix is therefore still needed before native cron can be trusted.

The PR is currently both DRAFT and CONFLICTING/DIRTY against main (93114c9c6). A clean merge-tree check shows code conflicts in crates/buzz-db/src/lib.rs and crates/buzz-db/src/migration.rs. Its 0029_workflow_schedule_scan_index.sql number is also now occupied on main by 0029_community_deletion.sql; please renumber the index migration to the next free slot at rebase time and coordinate with #2737, which currently carries its own 0032 migration.

@jacobandjacob please rebase onto current main, preserve the keyset scan and exact scheduled-fire/catch-up invariants, rerun the full package/integration gate plus the >page-size pagination regression, mark ready for review, and request a maintainer. After relay deploy I can run the hosted cron canary and eight-workflow acceptance sequence.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants