Skip to content

buzz-acp: session-limit failures are dead-lettered ~25 min into a multi-hour limit — the message is lost, never reprocessed when quota returns #5918

Description

@eireal

Summary

When a turn fails with the provider's session limit error, buzz-acp retries it on the ordinary
budget and dead-letters it ~25 minutes later. The limit resets hours later, so the retry window
can never outlive it: the event is discarded before the agent could possibly have answered it, and
it is never reprocessed once quota returns. The message is lost for good.

The error text carries the reset instant, so the harness knows exactly how long it would have to
wait — it just doesn't use it.

What I observed

Desktop 0.5.11 on macOS, managed agents against a Claude subscription. One agent, one DM channel,
three days (channel ids redacted, timestamps UTC):

dead-lettered events discarded
2026-08-12 15:15:40Z 2
2026-08-12 17:19:15Z 1
2026-08-13 16:36:45Z 1
2026-08-14 13:46:34Z 1

Five of the owner's DMs to that agent, gone. Two more batches (a project channel and another
channel) were dead-lettered the same morning for the same reason.

Every one of the 10 retries preceding each dead-letter logged the same failure:

WARN buzz_acp: agent_returned (application error — pipe intact) agent=0 outcome="error"
     configured_model=claude-fable-5[1m] pid=… error=Agent reported error (code -32603):
     Internal error: You've hit your session limit · resets 1:50pm (America/Buenos_Aires)
WARN buzz_acp::queue: requeueing failed batch with backoff channel_id=… attempt=1  max=10 delay_secs=4.09
…
WARN buzz_acp::queue: requeueing failed batch with backoff channel_id=… attempt=10 max=10 delay_secs=246.46
ERROR buzz_acp::queue: dead-lettering batch after 10 retries — discarding 1 events channel_id=… attempt=11 events=1

First requeue 13:20:59Z → dead-letter 13:46:34Z: 25m35s of retrying, against a limit that reset
at 16:50Z — 3h04m later. Not one of the ten attempts had any chance of succeeding, and the batch
was discarded before the only moment it could have worked.

Why this is structural, not bad luck

crates/buzz-acp/src/queue.rs:29-36:

pub(crate) const MAX_RETRIES: u32 = 10;
const BASE_RETRY_DELAY_SECS: u64 = 5;    // doubled each attempt
const MAX_RETRY_DELAY_SECS: u64 = 300;

5 + 10 + 20 + 40 + 80 + 160 + 300 + 300 + 300 + 300 = 1515s ≈ 25 min, and that is the ceiling,
not a typical case. A subscription session limit is measured in hours. So for this error class the
outcome is deterministic: retry budget exhausted, batch discarded, message never seen by the agent.

The user does get the ⚠️ I couldn't process the last request after multiple retries (…) notice, so
this is not a silent drop — but "please re-send" arrives 25 minutes into a multi-hour outage, and if
the sender isn't watching (a DM overnight, an owner on another machine), the request is simply gone.

Where it would fit

handle_prompt_result in crates/buzz-acp/src/lib.rs already classifies failures per error type
rather than treating them uniformly — is_auth_error at line 3956 dead-letters immediately, because
"the token won't self-repair between retries, so requeueing only wastes attempt slots". Session-limit
errors are the mirror image of that case: they do self-repair, at a known instant, just not inside
the retry budget. Today they fall through to the generic queue.requeue(batch) at line 3971.

Proposal

  1. Preferred — classify the limit error and park the batch until the reset instead of spending
    retries on it. EventQueue already has a per-channel retry_after; the reset time is right there
    in the error string. Parse it (fall back to a fixed backoff like 1h when it can't be parsed), set
    retry_after to it, and don't count the attempt against MAX_RETRIES. The message is then
    answered when quota returns, which is what the sender expects from an agent that is merely
    throttled.
  2. Minimum — if parking is unwanted, treat it like the auth class: dead-letter immediately with a
    notice that names the reset time ("quota resets at 1:50pm — please re-send after that"), instead
    of 25 minutes of silence followed by an undated "re-send".

Option 1 needs a per-adapter way to recognise the error, which is the same detection problem raised
in #5605 (there, quota exhaustion arrives as a successful turn whose text is the limit notice; here
it arrives as a -32603 error). If the adapter surfaced quota exhaustion distinguishably, both
issues could key on it. Distinct from #5708 (missing notice on the panic path) and #4127 (dead
session never recycled) — here the session is fine and the notice is posted; the defect is that the
retry policy cannot span the outage it is retrying against.

Happy to send a PR for either option.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions