Description
Title: Support for Multiple LLM AI API Endpoints for Self-Hosting and Model Selection
Feature Request
Description:
We would like to propose the addition of a new feature to ChatDev that enables users to configure and utilize multiple Language Model (LLM) AI API endpoints for self-hosting and experimentation with different models. This feature would enhance the flexibility and versatility of ChatDev for developers and researchers working with LLMs.
Feature Details:
-
Endpoint Configuration:
- Allow users to store API keys and configure multiple endpoints.
- Implement support for an environment (env) file to securely store sensitive information locally, facilitating scripted pass-through of values. Add to .gitignore
-
Custom Endpoint Names:
- Provide the ability to assign user-friendly names to each configured endpoint. This helps users easily identify and differentiate between endpoints. It would also allow multiple models to be leveraged on the same endpoint by having different configurations for each. A check could occur to validate that the endpoint has the model expected, and if not, do a quick unload/load of the desired model.
-
Chat Parameters:
- Integrate settings for chat parameters, such as temperature and other relevant options, that can be adjusted per endpoint. This allows users to fine-tune model behavior.
-
Model Selection (if applicable):
- If applicable to the specific LLM, enable users to preset a model for each endpoint. This feature can be especially useful when working with multiple LLMs simultaneously.
-
API Key Management (if applicable):
- For LLM services like OpenAI that require API keys, provide a dedicated parameter in each endpoint for users to input and manage their API keys for each endpoint.
-
Endpoint Address:
- Allow users to specify the endpoint address (URL) to which API requests should be sent. This flexibility is crucial for self-hosted instances or when working with different LLM providers.
-
Optional - Endpoint Tagging:
- Allowing us to add tags like #code #logic, or #budget could let us give key indicators of where a model's strengths are in, and select from a pool of models with a particular benefit, allowing more diverse outcomes. It could also allow for side-by-side comparisons. This could allow future result tracking/scoring to better identify which models are best at particular features, by having multiple #code models, and testing each's results you can identify and retrain or replace under-performing models to build an optimum workflow.
Expected Benefits:
This feature will benefit developers, researchers, and users who work with LLMs by offering a centralized and user-friendly interface for managing multiple AI API endpoints. It enhances the ability to experiment with various models, configurations, and providers while maintaining security and simplicity. This could allow different characters to leverage specific fine-tuned models rather than the same model for each. This could also allow self-hosted users to experiment with expand the number of repeated looped calls without drastically increasing the bill.
Additional Notes:
Consider implementing an intuitive user interface for configuring and managing these endpoints within the GitHub platform, making it accessible to both novice and experienced users.
References:
Include any relevant resources or references that support the need for this feature, such as the growing popularity of LLMs in various fields and the demand for flexible API management solutions.
Related Issues/Pull Requests:
#27
#33
Azure OpenAI #55
Assignees:
If you permit this ticket to remain open, I will assemble some links and resources, as well as opening another ticket to handle TextGenWebUI with relevant links there to implementing it. I can try implementing and doing a PR if someone else doesn't get to it first.
Thank you for considering this feature request. I believe that this enhancement will greatly benefit the ChatDev community and its users working with Language Model AI API endpoints.