Issues with Using Ollama as LLM Backend

Hello,

I am using Ollama as the LLM backend, but I am encountering issues when trying to run certain models. Here are the specific problems:

Llama3 and Phi3.5 Models: These models are not supported by Ollama and throw an unsupported error.
Llama3.2 Model: This model throws an error, as shown in the attached image.

I am aware that the author recommends using llama-cpp-python instead of Ollama. However, my computer does not meet the hardware requirements for llama-cpp-python, so I am limited to using Ollama.

Is there a workaround for these errors, or could you recommend a suitable model that is compatible with Ollama?

Thank you for any assistance you can provide!

[andreashappe]

Hi @superwow1, what usecase have you been trying? Hm, @Neverbolt the llama3.2 error looks like as if we're getting multiple tool-calls as a result and we're not currently able to deal with that.. what's your long-term plan with using instructor BTW? I remember that you wanted to remove it?

[Neverbolt]

Hi, yeah I wanted to remove instructor as it is no longer necessary (we have our own tool calling now). However, I will only be able to do so in ~2-3 weeks, as I am on holiday until then.
As a quick fix I could recommend switching from MinimalWebTesting to WebTestingWithExplanation if you're using that, as the WebTestingWithExplanation uses our internal tool calling method.

I tried running web testing on a simple website using the command: hackingBuddyGPT MinimalWebTesting --host "link-to-web"

The .env file structure is configured as shown in the attached image.

[Neverbolt]

If you run it with WebTestingWithExplanation does the error persist? Also please try to publish as little of your openai token as possible!

[andreashappe]

@superwow1 erm, revoke your openAPI key (sk-proj-..) as other people might now use it.

Ok I crop the picture so the key is not full. When I run WebTestingWithExplanation using the llama3.2 model, it successfully starts hackingBuddyGPT. However, there seem to be issues with the explanations it generates—they don't always seem accurate, which I suspect might be due to the model choice. Additionally, I frequently encounter the warning:

"WARNING: Did not get usage information in the stream response".

Is there a recommended model or configuration to resolve these issues and improve the accuracy of the explanations?