Skip to content

feat: make load shedding queue- and priority-aware - #816

Open
theo372001 wants to merge 1 commit into
Pulsefy:mainfrom
theo372001:feat/issue-777-queue-aware-load-shedding
Open

feat: make load shedding queue- and priority-aware#816
theo372001 wants to merge 1 commit into
Pulsefy:mainfrom
theo372001:feat/issue-777-queue-aware-load-shedding

Conversation

@theo372001

Copy link
Copy Markdown
Contributor

Summary

Closes #777.

Load shedding previously used a single flat Celery queue-depth threshold
and only checked LLM provider health for the synchronous verify route,
relying on coarse request-level thresholds alone. This tunes it with
queue depth, provider health, and job priority signals:

  • Priority-scaled queue threshold - services/load_shedder.py now
    scales the queue-depth threshold by the job's declared priority
    (low=0.5x, normal=1x, high=1.5x, urgent=3x), so low-value
    backlog is shed well before high-priority work. The middleware
    pre-check (evaluate_load_shed) stays a cheap, priority-agnostic
    outer gate since it runs before the request body is parsed; the
    precise, priority-aware check runs in ensure_queue_capacity,
    wired up from tasks.create_task right before a job is queued.
  • Provider health for async job creation - previously provider
    health was only checked on the synchronous /ai/humanitarian/verify
    route. It's now also checked when queuing async jobs via
    /v1/ai/inference with type=humanitarian_verification, so
    LLM-dependent work is shed up front instead of being queued, retried,
    and eventually dead-lettered.
  • Metrics - both paths reuse the existing LoadShedError ->
    ErrorEnvelope plumbing (so shed responses stay consistent with the
    rest of the service's error envelope), and queue-pressure sheds are
    now additionally broken out by priority tier in a new
    queue_shed_by_priority_total{priority} counter alongside the
    existing requests_shed_total{reason,method,endpoint}.
  • Documented the tuned behavior in app/ai-service/README.md.

Test plan

  • tests/test_load_shedder.py - 22 new tests (30 total): priority
    multiplier resolution, priority-scaled queue-depth thresholds
    (low sheds sooner, urgent tolerates more), provider-aware job
    creation (sheds LLM-dependent types, ignores non-LLM types,
    queue check short-circuits before the provider check), the new
    priority-labeled metric, and two end-to-end HTTP tests proving
    envelope consistency for both new shed paths.
  • Full pytest suite run locally - no regressions in any test
    touching load shedding, tasks, or metrics.
  • flake8 --select=E9,F63,F7,F82 clean on all changed files.

🌊 Stellar Wave application: 668468d4-8d29-4f21-a1a1-9d2b5736ddbb

Load shedding previously used a single flat queue-depth threshold and
only checked LLM provider health for the synchronous verify route.
This tunes it with two more signals:

- Queue-depth threshold now scales by job priority (low/normal/high/
  urgent) so low-value backlog is shed well before high-priority work.
  The precise, priority-aware check runs in ensure_queue_capacity right
  before a job is queued; the middleware pre-check stays a cheap,
  priority-agnostic outer gate since it runs before the body is parsed.
- Provider health is now also checked for async job creation
  (/v1/ai/inference with type=humanitarian_verification), not just the
  synchronous verify route, so LLM-dependent jobs are shed up front
  instead of being queued, retried, and eventually dead-lettered.

Both paths reuse the existing LoadShedError -> ErrorEnvelope plumbing,
and queue-pressure sheds are now also broken out by priority in a new
queue_shed_by_priority_total metric.

Closes Pulsefy#777
@drips-wave

drips-wave Bot commented Jul 24, 2026

Copy link
Copy Markdown

@theo372001 Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

@Cedarich

Copy link
Copy Markdown
Contributor

@theo372001 please resolve conflicts

@Cedarich

Copy link
Copy Markdown
Contributor

@theo372001

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make Load Shedding Queue-Aware

2 participants