[SC-16225] Add Gemini support#513
Conversation
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
PR SummaryThis pull request introduces substantial enhancements to how the ValidMind Library handles LLM configurations. The changes add support for the Gemini provider alongside the existing OpenAI and Azure integrations. Key updates include: • Modification of environment-based provider resolution, now checking for Gemini API keys (via GOOGLE_API_KEY or GEMINI_API_KEY) and setting default Gemini models and embeddings. • Updates to the client/model initialization logic in the ai/utils module. In particular, the new helper functions (_get_configured_provider, _build_gemini_judge_config, _build_openai_judge_config) abstract the creation of judge models, while get_deepeval_model now handles Gemini-specific DeepEval model instantiation. • Adjustments for LangChain compatibility within the judge configuration, including unwrapping of Functional models to ensure they contain the correct types. • Extensive modifications to DeepEval scorer modules: calls to get_client_and_model() have been replaced by get_deepeval_model() to ensure consistent configuration for Gemini and other providers. • Updates in unit tests for both RAGAS and AI utils to cover the new Gemini configuration scenarios, as well as tests for the custom • Dependency updates in configuration files (pyproject.toml and uv.lock) to include the new packages: langchain-google-genai and google-genai, ensuring proper support of the Gemini provider. Overall, the PR robustly expands LLM functionality by integrating Gemini support, refactors judge configuration logic, and broadens the test coverage to verify correct behavior under different provider environments. Test Suggestions
|
Pull Request Description
What and why?
Added Gemini support across the library’s three LLM evaluation paths: prompt evaluation tests, RAGAS tests, and DeepEval scorers.
Before this change, the shared judge config only supported OpenAI/Azure and Gemini was not wired consistently across all evaluators. After this change, all three evaluation paths can use Gemini, RAGAS handles Gemini finish reasons more reliably, and the judge-configuration notebook now documents the current environment variables and defaults used by the library.
How to test
OPENAI_API_KEYkey in your.envGEMINI_API_KEY, and optionallyGEMINI_MODELandGEMINI_EMBEDDINGS_MODELnotebooks/how_to/run_tests/configure_tests/configure_judge_llms.ipynbWhat needs special review?
Dependencies, breaking changes, and deployment notes
Release notes
Added Gemini support for ValidMind prompt evaluation tests, RAGAS tests, and DeepEval scorers, and updated the judge-configuration notebook to reflect the current environment variables and defaults.
Checklist