feat: seed orchestrator with auto-seeding, persistence, and management CLI#1940
Open
jrtorrez31337 wants to merge 17 commits intokoala73:mainfrom
Open
feat: seed orchestrator with auto-seeding, persistence, and management CLI#1940jrtorrez31337 wants to merge 17 commits intokoala73:mainfrom
jrtorrez31337 wants to merge 17 commits intokoala73:mainfrom
Conversation
|
@jrtorrez31337 is attempting to deploy a commit to the Elie Team on Vercel. A member of the Team first needs to authorize it. |
…supervisord, copy scripts into image
Adds scripts/seed-orchestrator.mjs with:
- classifySeeders: classify seeders into active/skipped by env vars
- buildStartupSummary: human-readable startup report
- Tiered cold start (hot/warm/cold/frozen with per-tier concurrency)
- Freshness check via seed-meta keys before running stale seeders
- Steady-state scheduling with setTimeout-based recurring timers
- Overlap protection, retry-after-60s, consecutive failure demotion
- Global concurrency cap of 5 with queue-based overflow
- Graceful shutdown on SIGTERM/SIGINT (15s drain timeout)
- Meta writing for null-metaKey seeders to seed-meta:orchestrator:{name}
jrtorrez31337
force-pushed
the
feat/seed-orchestrator
branch
from
3c80992 to
b7eefe8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The self-hosted Docker stack currently requires manually running 42 seed scripts after every
container start, has no data persistence across restarts, and no mechanism to keep data fresh
while running. This makes the stack unusable out of the box for homelab deployments and blocks
planned work to significantly expand the data ingestion pipeline.
This PR adds automated seed lifecycle management — turning the stack from "panels show nothing
until you manually run scripts" into a turnkey deployment that populates data on boot, keeps it
fresh, and survives restarts.
What changed
Seed Orchestrator (
scripts/seed-orchestrator.mjs)Redis Persistence
REDIS_MAXMEMORY)Seed Manager CLI (
wmsm.sh)status— freshness dashboard for all 42 seedersschedule— refresh intervals with next-run estimatesrefresh <name|--all>— force re-seed with typo correctionflush— wipe and re-seed from scratchlogs— filtered orchestrator outputWarm-ping fix
in Docker mode instead of api.worldmonitor.app
Docker changes
SEED_TURBO=real|dryfor compressed-interval testingTest plan
node --test tests/seed-orchestrator*.test.mjs)./wmsm.sh statusshows correct freshness for all seedersSEED_TURBO=drycompletes full scheduling lifecycle in ~5 minutes