feat(runtime-fallback): auto-retry, generic retry detection, and timeout toggle

[youngbinkim0]

Summary

This PR implements runtime fallback auto-retry with full agent preservation. The fix chain enables end-to-end runtime fallback to work correctly in production.

What Changed

Bug Fixes (9 total)

1. Nested status code detection - Anthropic errors have data.statusCode structure
2. Regex pattern relaxed - Match "credit balance too low" with flexible spacing
3. Schema limit bumped - max_fallback_attempts now supports 20 (was 10)
4. Fallback models discovery - Handle sparse session.error events without agent info
5. Model fallback init - Derive model from agent config when events lack data
6. Auto-retry with promptAsync - Resend last user message after fallback selection
7. Persistent fallback - Fallback model now persists across messages in session
8. Manual model override - Detect user-initiated model changes and reset fallback state
9. Agent preservation - Resolve correct agent for auto-retry (was defaulting to sisyphus)

New Features

• Provider-agnostic retry detection - Generalized pattern for Copilot, OpenAI, and future providers
• Timeout toggle - timeout_seconds: 0 disables quota-based fallback escalation
• Agent/category fallback_models - Support custom fallback chains per agent/category
• Provider-aware model resolution - Check connectivity before matching fallback models

Testing

✅ 44 runtime-fallback tests passing
✅ 46 schema tests passing
✅ TypeScript compilation clean
✅ Manual end-to-end verification (Anthropic error → auto-fallback → agent preserved → manual override works)

Related

Builds on #1408 (feat/runtime-fallback-only)

[github-actions]

All contributors have signed the CLA. Thank you! ✅
_{Posted by the CLA Assistant Lite bot.}

[youngbinkim0]

I have read the CLA Document and I hereby sign the CLA

[youngbinkim0]

recheck

[youngbinkim0]

recheck

[cubic-dev-ai]

2 issues found across 27 files

Confidence score: 3/5

• Potential user-impacting inconsistency in src/shared/model-resolution-pipeline.ts: new userFallbackModels reads connected providers only from cache, ignoring constraints.connectedProviders, which could change model resolution behavior.
• Code duplication in src/hooks/runtime-fallback/index.ts auto-retry handlers raises maintenance risk and could lead to subtle divergence, though it’s not an immediate blocker.
• Pay close attention to src/shared/model-resolution-pipeline.ts and src/hooks/runtime-fallback/index.ts - connected provider handling and duplicated retry logic.

Prompt for AI agents (all issues)

Check if these issues are valid — if so, understand the root cause of each and fix them.


<file name="src/hooks/runtime-fallback/index.ts">

<violation number="1" location="src/hooks/runtime-fallback/index.ts:473">
P2: Significant code duplication in auto-retry logic between `session.error` and `message.updated` event handlers. The logic for fetching session messages, extracting the last user message, parsing parts, and calling `promptAsync` is duplicated almost verbatim (approximately 40+ lines). This increases maintenance burden and risk of inconsistencies.</violation>
</file>

<file name="src/shared/model-resolution-pipeline.ts">

<violation number="1" location="src/shared/model-resolution-pipeline.ts:105">
P1: Inconsistent handling of `constraints.connectedProviders` - the new `userFallbackModels` logic reads connected providers only from cache, ignoring the `constraints.connectedProviders` parameter that is respected in `categoryDefaultModel` and `fallbackChain` logic. This prevents callers from enforcing specific provider constraints in the user fallback code path.</violation>
</file>

---

Since this is your first cubic review, here's how it works:

• cubic automatically reviews your code and comments on bugs and improvements
• Teach cubic by replying to its comments. cubic learns from your replies and gets better over time
• Add one-off context when rerunning by tagging @cubic-dev-ai with guidance or docs links (including llms.txt)
• Ask questions if you need clarification on any suggestion

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}

[youngbinkim0]

One situation that can get a little messy and i'm not sure it is working perfectly is when using prometheus to create a plan. At the end it'll give you the start work slash command to run. When you run the slash command, if the model for atlas is rate limited, you'll get the message about falling back to the next model. However, it won't switch to atlas and instead it will change the model and insert a message of continue but Prometheus will recieve the message instead and prometheus will say, you need to run the slash command to start work.

When you run a slash command that targets an agent and the model used is rate limited, we need to switch to the agent, then rotate to the next model, then enter the message of continue.

Issue 2 i noticed is there is odd behavior around when the compact command runs. It won't actually generate the compacted contents if the model that is going to be used for compacting is rate limited. It just kind of hanges there with no contents and you have to undo and recover the session state by running compact manually then continuing.

@rothnic

1. Great catch — I've seen this too. When /start-work fires and atlas's model is rate-limited, the fallback rotates the model but doesn't actually switch the agent, so prometheus ends up receiving the "continue" message and loops back to "run the slash command."
   This is a real issue but I think it deserves its own PR. The current runtime-fallback work (this PR) fixes agent resolution after a session is established — the scenario you're describing is about the agent switch + model selection failing atomically during session creation, which touches different parts of the flow (/start-work orchestration, session creation timing). Mixing the two would make both harder to review and test.
   Would you be open to filing a separate issue for this? Happy to collaborate on it once this PR lands.

2. Separately — the compaction hanging when the compaction model is rate-limited is also worth tracking, but that lives in a completely different subsystem (anthropic-context-window-limit-recovery / preemptive-compaction), not runtime-fallback. Probably best as its own issue too so it gets the right eyes on it.

[cubic-dev-ai]

1 issue found across 51 files

Confidence score: 4/5

• Schema duplication in assets/oh-my-opencode.schema.json repeats fallback_models 15 times instead of using $defs/$ref, which raises maintenance risk and increases chance of future inconsistency if edits are missed
• Overall risk appears low because this is a schema maintainability concern rather than a direct runtime regression, so this PR still seems safe to merge
• Pay close attention to assets/oh-my-opencode.schema.json - consider consolidating fallback_models into $defs to reduce duplication

Prompt for AI agents (all issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="assets/oh-my-opencode.schema.json">

<violation number="1" location="assets/oh-my-opencode.schema.json:304">
P2: Significant schema duplication: `fallback_models` definition is repeated 15 times across agent configurations instead of using JSON Schema `$defs` with `$ref` references</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}

[rothnic]

1. Great catch — I've seen this too. When /start-work fires and atlas's model is rate-limited, the...

@youngbinkim0 one other case i encountered i wanted to mention to make sure it would also go into a separate issue is for the todo continuation and generally how the fallback works.

Let's say you have an agent working on something and it doesn't hit rate limits. However, let's assume you just used up your last request, so the next request will hit the rate limits and the agent stops working with an incomplete todo list. The model that is chosen is the one that has reached the usage limits, so it has to fallback. Sometimes it seems like at this point while waiting for the fallback OmO will try to do the ToDo continuation again and it seems to reconsider the fallback all over again, so it kind of gets in a loop of not allowing the fallback to occur and then the todo continuation kicks in each time.

In other words, there is a couple issues there potentially:

• The selected model isn't updated with the model we are trying to fallback to
• The todo continuation functionality seems almost unaware of a fallback being in progress

[davidakerr]

I seen some interrupted messaging on the fallbacks.

[rothnic]

@davidakerr is that using the opencode desktop application? I just tried it out for the first time earlier and also ended up with an issue where a session was interrupted in a similar situation.

[davidakerr]

For these tests I'm running opencode serve's WebUI

[youngbinkim0]

@davidakerr @rothnic Thanks for the reports and screenshots! We've filed three follow-up issues to track the problems you've been hitting:

• fix: agent pin races with rate-limit error during /start-work #2060 — Agent pin races with rate-limit during /start-work (wrong agent gets the retry)
• fix: compaction hangs when compaction model is rate-limited #2062 — Compaction hangs when compaction model is rate-limited
• fix: todo-continuation fires during model fallback causing retry loop #2063 — Todo-continuation fires during model fallback causing retry loop

Your screenshots show MiniMax M2.5 Free · Interrupted — this looks like a mid-stream interruption (the model starts responding then gets cut off), which is a slightly different path from the pre-request rate-limit errors (429s) that this PR's patterns handle. The patterns added here (/quota.?protection/i, /key.?limit.?exceeded/i) catch errors returned before any content streams. Mid-stream disconnects from free-tier models are an additional edge case that #2063 in particular would help with — smoother auto-recovery instead of dropping to "Interrupted" and requiring a manual "continue" click.

We'll keep this PR focused on the error pattern additions and address the interruption/recovery flow in those follow-up issues.