Implement Evaluator-Optimizer Workflow for Enhanced CodeBuddy Responses

[olasunkanmi-SE]

Feature Request: Implement Evaluator-Optimizer Workflow for Enhanced CodeBuddy Responses

Issue Description

This issue proposes the implementation of an "Evaluator-Optimizer" workflow to enhance the quality and reliability of responses generated by Codebuddy.

Background

Currently, Codebuddy uses a more straightforward approach to answering user queries. For instance, when a user asks a question like "What is Kahn's algorithm?", Codebuddy performs a web search, process the results with an LLM, and provide a single-pass generated answer.

While this approach can be effective for many queries, it may fall short when:

• Nuance is required: For complex topics or questions requiring subtle understanding and accurate details, a single LLM pass might miss crucial aspects.
• Iterative refinement is beneficial: Similar to how human writers refine their work through review and feedback, an iterative process can significantly improve the quality of AI-generated content.
• Clear evaluation criteria exist: For many coding-related questions, we can define criteria to evaluate the correctness and completeness of an answer.

Proposed Solution: Evaluator-Optimizer Workflow

We should implement an "Evaluator-Optimizer" workflow as depicted in the diagram below (and as discussed previously):

Solution

[User Query] --> [Orchestrator, decides where to route request]--> [LLM Call Generator] -->[Orchestrator] --> [LLM Call Evaluator] --> [Out] (Accepted)

[User Query] --> [Orchestrator, decides where to route request]--> [LLM Call Generator] --> [Orchestrator] --> [LLM Call Evaluator] --> [Out] ( Rejected + Feedback)--> [Orchestrator] --> [Orchestrator, decides where to route request]

Workflow Breakdown:

1. User Input: The user provides a question (e.g., "What is Kahn's algorithm?").
2. LLM Call Generator:
   • The AI agent uses its web search tool to retrieve relevant information based on the user's question.
   • An LLM (Generator LLM) is prompted to generate an initial answer based on the search results.
3. LLM Call Evaluator:
   • Another LLM (Evaluator LLM) is prompted to evaluate the generated answer against the original question and predefined criteria.
   • The evaluator determines if the answer is "Accepted" or "Rejected + Feedback".
4. Feedback Loop:
   • Accepted: If the evaluator accepts the answer, it is returned to the user.
   • Rejected + Feedback: If the evaluator rejects the answer, it provides feedback indicating areas for improvement. This feedback can be used (in more advanced implementations) to refine the next iteration of answer generation. For a simpler initial implementation, we can iterate a fixed number of times or until acceptance.
5. Output: The final (accepted or best-attempt) answer is provided to the user.

Example Scenario: "What is Kahn's Algorithm?"

1. Question: User asks "What is Kahn's algorithm?".
2. Search: The AI agent searches the web for "Kahn's algorithm".
3. Generate (Initial Answer): The Generator LLM creates an answer based on search snippets.
4. Evaluate: The Evaluator LLM checks if the answer:
   • Accurately defines Kahn's algorithm.
   • The checks will be based on general software engineering design patterns etc
5. Outcome:
   • If the initial answer is deemed insufficient, the Evaluator rejects it with feedback.
   • The system could then (in a future enhancement) use this feedback to refine the prompt for the Generator LLM for a second attempt, or in this simpler implementation, iterate to a maximum number of attempts.
   • Eventually, a satisfactory answer is accepted and presented to the user.

Benefits of Implementing Evaluator-Optimizer:

• Improved Accuracy and Correctness: Iterative evaluation and refinement can catch and correct inaccuracies or incomplete information in the initial LLM response.
• Enhanced Relevance and Depth: The feedback loop allows for focusing the answer more precisely on the user's question and providing more comprehensive information.
• Increased Reliability: By having an evaluation step, we can have greater confidence in the quality of the generated answers, especially for complex or critical queries.
• Better User Experience: Users receive more accurate, complete, and reliable answers, leading to a more positive and productive interaction with the AI assistant.
• Foundation for Future Enhancements: This workflow provides a robust framework for further improvements, such as incorporating more sophisticated feedback mechanisms, dynamic prompt refinement, and personalized evaluation criteria.

Implementation Considerations (Production Readiness):

• Robust Error Handling: Implement comprehensive error handling for all steps, including search API calls, LLM API calls, and workflow logic.
• Asynchronous Operations: Ensure all operations are asynchronous to maintain responsiveness and handle API delays effectively.
• Configuration Management: Externalize configuration parameters (API keys, LLM model names, iteration limits, evaluation criteria) for easy adjustments and environment management.
• Rate Limiting and Retries: Implement strategies to handle API rate limits and transient errors, including retry mechanisms with exponential backoff.
• Detailed Logging and Monitoring: Integrate thorough logging throughout the workflow to track progress, debug issues, and monitor performance metrics (iteration counts, acceptance rates, error rates).
• Input Sanitization: Sanitize user input to prevent prompt injection vulnerabilities.
• Prompt Engineering: Carefully design prompts for both the Generator LLM and Evaluator LLM to ensure effective answer generation and accurate evaluation.
• Performance Optimization: Consider potential performance bottlenecks and optimize for speed and efficiency, especially if iterations are involved.

Acceptance Criteria:

• A functional Evaluator-Optimizer workflow is implemented for question-answering tasks.
• The workflow successfully utilizes the agent orchestrator, a web search tool, and any other required tools by CodeBuddyToolProvider, a Generator LLM, and an Evaluator LLM (or mock implementations initially for testing).
• The system demonstrates iterative answer refinement based on evaluator feedback (even if initially limited to a fixed number of iterations).
• Basic logging is implemented to track workflow execution.
• The implementation is documented with clear instructions for setup and usage.
• Initial testing shows improved answer quality for complex queries compared to a single-pass approach (qualitative assessment).