Add support for Palm, Claude-2, Llama2, CodeLlama (100+LLMs)

[ishaan-jaff]

This PR adds support for the above mentioned LLMs using LiteLLM https://github.com/BerriAI/litellm/
Example

# openai call
response = completion(model="gpt-3.5-turbo", messages=messages)
# cohere call
response = completion(model="command-nightly", messages=messages)
# anthropic
response = completion(model="claude-2", messages=messages)

In addition LiteLLM client allows you to:

• A/B test LLMs in production
• Dynamically control each LLMs prompt, temperature, top_k etc in our UI (no need to re-deploy code)
• Logging to view input/outputs for each LLM

Here's a link to a live demo of litellm client: https://admin.litellm.ai/

[ishaan-jaff]

@joonspk-research can you please take a look at this PR when possible ? Happy to add more docs/tests if this initial commit looks good😊

[mikeyang01]

did you test which model can run this agent without crash or error?

[Miru302]

From my experience of changing to llama-2 you'll need to rewrite all the prompt templates for it to work more or less stably. The code is still VERY brittle.

[mikeyang01]

From my experience of changing to llama-2 you'll need to rewrite all the prompt templates for it to work more or less stably. The code is still VERY brittle.

+1

[sardanian]

Has anyone gotten this working well with Llama2? and if so, is this commit all that is needed or was there a lot of other rewritings?

[Miru302]

I have a fork that sometimes runs. It doesn't work well, often errors out because output parsing often breaks. Innitially, used another PR(#64), not this one. You can clone that, fix paths to your LLM and embedding models. You may also need to fix prompt templates that I didn't get to yet (I used Llama2-chat, it uses specific tags in prompts). I stopped working on it because I got frustrated with the quality and architecture of the code in this repo and decided to write my own system.