feat: make load shedding queue- and priority-aware - #816
Open
theo372001 wants to merge 1 commit into
Open
Conversation
Load shedding previously used a single flat queue-depth threshold and only checked LLM provider health for the synchronous verify route. This tunes it with two more signals: - Queue-depth threshold now scales by job priority (low/normal/high/ urgent) so low-value backlog is shed well before high-priority work. The precise, priority-aware check runs in ensure_queue_capacity right before a job is queued; the middleware pre-check stays a cheap, priority-agnostic outer gate since it runs before the body is parsed. - Provider health is now also checked for async job creation (/v1/ai/inference with type=humanitarian_verification), not just the synchronous verify route, so LLM-dependent jobs are shed up front instead of being queued, retried, and eventually dead-lettered. Both paths reuse the existing LoadShedError -> ErrorEnvelope plumbing, and queue-pressure sheds are now also broken out by priority in a new queue_shed_by_priority_total metric. Closes Pulsefy#777
|
@theo372001 Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits. You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀 |
Contributor
|
@theo372001 please resolve conflicts |
Cedarich
approved these changes
Jul 24, 2026
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #777.
Load shedding previously used a single flat Celery queue-depth threshold
and only checked LLM provider health for the synchronous verify route,
relying on coarse request-level thresholds alone. This tunes it with
queue depth, provider health, and job priority signals:
services/load_shedder.pynowscales the queue-depth threshold by the job's declared
priority(
low=0.5x,normal=1x,high=1.5x,urgent=3x), so low-valuebacklog is shed well before high-priority work. The middleware
pre-check (
evaluate_load_shed) stays a cheap, priority-agnosticouter gate since it runs before the request body is parsed; the
precise, priority-aware check runs in
ensure_queue_capacity,wired up from
tasks.create_taskright before a job is queued.health was only checked on the synchronous
/ai/humanitarian/verifyroute. It's now also checked when queuing async jobs via
/v1/ai/inferencewithtype=humanitarian_verification, soLLM-dependent work is shed up front instead of being queued, retried,
and eventually dead-lettered.
LoadShedError->ErrorEnvelopeplumbing (so shed responses stay consistent with therest of the service's error envelope), and queue-pressure sheds are
now additionally broken out by priority tier in a new
queue_shed_by_priority_total{priority}counter alongside theexisting
requests_shed_total{reason,method,endpoint}.app/ai-service/README.md.Test plan
tests/test_load_shedder.py- 22 new tests (30 total): prioritymultiplier resolution, priority-scaled queue-depth thresholds
(low sheds sooner, urgent tolerates more), provider-aware job
creation (sheds LLM-dependent types, ignores non-LLM types,
queue check short-circuits before the provider check), the new
priority-labeled metric, and two end-to-end HTTP tests proving
envelope consistency for both new shed paths.
pytestsuite run locally - no regressions in any testtouching load shedding, tasks, or metrics.
flake8 --select=E9,F63,F7,F82clean on all changed files.🌊 Stellar Wave application: 668468d4-8d29-4f21-a1a1-9d2b5736ddbb