improvements to tool calling logic (merged changes from old PR branch) #1855
+133
−33
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ng irrelevant to toolcall decision
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
as we've discussed in discord,
this change alters the way koboldcpp determines what tool to use in some pretty drastic ways that vastly improve it's accuracy, especially with small LLM's. instead of doing one request to the LLM to prompt it and ask it if a tool should be used, and forcing it down to 5 tokens with grammar forcing a simple "yes/no" answer.. it now gives the LLM full freedom to write out it's decision and why it took that decision, with it's final decision text always added at the end of the response. then we take that response and use the yes/no grammar on that instead!
(this is a redo of the pull request because due to inexperience with git and github i messed up my branch with too many merges with upstream)