fix(llm-client): detect native sglang context-overflow messages

[fparisio]

SGLang reports context-window overflows with wording that matches nothing in OPENAI_OVERFLOW_PHRASES, so the failure is classified as a generic upstream 400 instead of ContextWindowExceeded, and context-window fallback routing never fires for SGLang backends.

SGLang rejects an over-length request from two places. The KV-pool admission check in srt/managers/utils.py returns:

Input length (700001 tokens) exceeds the maximum allowed length (536826 tokens). Use a shorter input or enable --allow-auto-truncate.

The declared-context check in srt/managers/tokenizer_manager.py, hit when the server runs with an explicit --context-length, returns:

The input (12345 tokens) is longer than the model's context length (8192 tokens).

The closest phrase in the list today is exceeds the maximum allowed input length, the LiteLLM-wrapped variant, which differs from the first message by one word.

This PR adds one phrase for each message. The wording is identical in v0.5.15.post1 and v0.5.17 (current latest), and SGLang's own test fixtures assert on "maximum allowed length", so a silent rewording upstream is unlikely.

The first test fixture is a verbatim capture from a live deployment. Both fixtures use SGLang's top-level {"object":"error",...} envelope with no error key, so they also cover the raw-body fallback in is_overflow_body.

Validation:

• cargo test -p switchyard-llm-client: 56 passed
• cargo test --workspace passes; cargo fmt --check and cargo clippy are clean
• Tested end to end against the live deployment through a passthrough route: a 700k-token request now returns the typed 400 context_length_exceeded; before the patch it returned the SGLang body wrapped as a generic upstream error.

Summary by CodeRabbit

• Bug Fixes
  • Improved handling of SGLang errors when an input exceeds the permitted length or model context limit.
  • Added coverage for multiple SGLang error response formats.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: c45f5914-3a93-452c-b524-75da065bccc8

📥 Commits

Reviewing files that changed from the base of the PR and between a17efa9 and d8f4f08.

📒 Files selected for processing (1)

• crates/libsy-llm-client/src/backend.rs

---

Walkthrough

The backend adds two input-length phrases to OpenAI overflow detection. Regression tests cover native SGLang top-level errors for maximum-length and model-context-length rejections.

Changes

Overflow detection

Layer / File(s)	Summary
Overflow phrases and regression tests crates/libsy-llm-client/src/backend.rs	The backend recognizes messages stating that input exceeds the maximum allowed length or the model context length. Tests cover both native SGLang error responses.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to d8f4f

This change classifies two native SGLang context-overflow responses as typed context-window errors so fallback routing can work correctly. The change is localized and validated; no actionable merge-blocking risk remains.

Poem

I’m a rabbit in the code,
Counting tokens as they flowed.
Long inputs now raise a sign,
SGLang errors match just fine.
Hop, hop—tests keep the path aligned!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely identifies the main change: detecting native SGLang context-overflow messages.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

---

_{Comment @coderabbitai help to get the list of available commands.}