fix(tools): recover forced-tool arguments when the backend ignores tool_choice - #72
Open
walcz-de wants to merge 3 commits into
Open
fix(tools): recover forced-tool arguments when the backend ignores tool_choice#72walcz-de wants to merge 3 commits into
walcz-de wants to merge 3 commits into
Conversation
…ol_choice Some llama.cpp chat templates do not honor a named/required tool_choice at generation time (observed with Qwen3-family templates when thinking is disabled via chat_template_kwargs): the forced decision free-runs plain text and returns without any tool call, which callers surface as 'no parameters generated for tool X' and retry until the generation cap. Add a backend-agnostic fallback in decision()/decisionWithStreaming(): when a forced decision yields no tool call, re-ask with a response_format JSON schema built from the forced tool's parameter schema — the plain structured-output grammar path, which those templates do honor — and parse the arguments from the content. Well-behaved backends never take the fallback (gate-tested both ways). Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: stefanwalcz <stefan.walcz@walcz.de>
…ree-running to the cap A backend that ignores a named/required tool_choice streams prose instead of tool-call deltas — previously the doomed generation ran all the way to the per-completion cap (minutes at local speeds) before the schema fallback could recover. Detect the violation after 512 bytes of plain content with no tool-call delta, cancel the stream, and go straight to the response_format schema fallback. Reasoning deltas do not trigger the abort, so thinking models that reason before a honored forced call are unaffected. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: stefanwalcz <stefan.walcz@walcz.de>
…tent Some pipelines deliver a thinking model's <think> block as plain content deltas, not reasoning events — the early-abort then misfired on models that DO honor forced tool_choice (they reason at length before the call) and sent them into the fallback needlessly. Count only prose outside a leading think block toward the abort threshold. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: stefanwalcz <stefan.walcz@walcz.de>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Some OpenAI-compatible backends do not honor a named/required
tool_choiceat generation time. We hit this with llama.cpp: Qwen3-family chat templates with thinking disabled viachat_template_kwargsignore the constraint entirely and free-run plain text (reported upstream to llama.cpp with a minimal repro). For cogito this was fatal: every forced decision indecision()/decisionWithStreaming()came back without tool calls, surfaced asno parameters generated for tool X, and retried until the generation cap — the whole tool loop died in minutes-long narration loops.Three backend-agnostic hardening steps, none of which change behavior on well-behaved backends:
forcedToolParamsViaSchema): when a forced decision yields no tool call, re-ask with aresponse_formatJSON schema built from the forced tool's parameter schema — the plain structured-output grammar path, which the affected templates do honor — and parse the arguments from the content.<think>…</think>block does not count toward the abort threshold — thinking models that legitimately reason at length before an honored forced call are unaffected (reasoning deltas never counted anyway; this covers pipelines that deliver reasoning as content).Testing
Unit tests cover: fallback fires exactly once for a forced no-call decision and recovers valid arguments; unforced text answers never take the fallback; the think-block prose counter. Verified end-to-end on a LocalAI + llama.cpp deployment where the agent tool loop went from all-red (retry-to-cap on every task) to 8/10 tasks passing with 10/10 clean finalizations.
Assisted-by: Claude (Anthropic) — implementation done with AI assistance; the failure analysis, repro and end-to-end verification were performed against real hardware.