Skip to content

[SC-16225] Add Gemini support#513

Open
juanmleng wants to merge 2 commits into
mainfrom
juan/sc-16225/add-llm-config-support
Open

[SC-16225] Add Gemini support#513
juanmleng wants to merge 2 commits into
mainfrom
juan/sc-16225/add-llm-config-support

Conversation

@juanmleng
Copy link
Copy Markdown
Contributor

Pull Request Description

What and why?

Added Gemini support across the library’s three LLM evaluation paths: prompt evaluation tests, RAGAS tests, and DeepEval scorers.

Before this change, the shared judge config only supported OpenAI/Azure and Gemini was not wired consistently across all evaluators. After this change, all three evaluation paths can use Gemini, RAGAS handles Gemini finish reasons more reliably, and the judge-configuration notebook now documents the current environment variables and defaults used by the library.

How to test

  1. Comment OPENAI_API_KEY key in your .env
  2. Set GEMINI_API_KEY, and optionally GEMINI_MODEL and GEMINI_EMBEDDINGS_MODEL
  3. Run notebooks/how_to/run_tests/configure_tests/configure_judge_llms.ipynb

What needs special review?

Dependencies, breaking changes, and deployment notes

Release notes

Added Gemini support for ValidMind prompt evaluation tests, RAGAS tests, and DeepEval scorers, and updated the judge-configuration notebook to reflect the current environment variables and defaults.

Checklist

  • What and why
  • Screenshots or videos (Frontend)
  • How to test
  • What needs special review
  • Dependencies, breaking changes, and deployment notes
  • Labels applied
  • PR linked to Shortcut
  • Unit tests added (Backend)
  • Tested locally
  • Documentation updated (if required)
  • Environment variable additions/changes documented (if required)

juanmleng and others added 2 commits May 19, 2026 14:15
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@juanmleng juanmleng self-assigned this May 19, 2026
@juanmleng juanmleng added the enhancement New feature or request label May 19, 2026
@github-actions
Copy link
Copy Markdown
Contributor

PR Summary

This pull request introduces substantial enhancements to how the ValidMind Library handles LLM configurations. The changes add support for the Gemini provider alongside the existing OpenAI and Azure integrations. Key updates include:

• Modification of environment-based provider resolution, now checking for Gemini API keys (via GOOGLE_API_KEY or GEMINI_API_KEY) and setting default Gemini models and embeddings.

• Updates to the client/model initialization logic in the ai/utils module. In particular, the new helper functions (_get_configured_provider, _build_gemini_judge_config, _build_openai_judge_config) abstract the creation of judge models, while get_deepeval_model now handles Gemini-specific DeepEval model instantiation.

• Adjustments for LangChain compatibility within the judge configuration, including unwrapping of Functional models to ensure they contain the correct types.

• Extensive modifications to DeepEval scorer modules: calls to get_client_and_model() have been replaced by get_deepeval_model() to ensure consistent configuration for Gemini and other providers.

• Updates in unit tests for both RAGAS and AI utils to cover the new Gemini configuration scenarios, as well as tests for the custom _ragas_is_finished_parser that accepts finish reasons from Gemini responses.

• Dependency updates in configuration files (pyproject.toml and uv.lock) to include the new packages: langchain-google-genai and google-genai, ensuring proper support of the Gemini provider.

Overall, the PR robustly expands LLM functionality by integrating Gemini support, refactors judge configuration logic, and broadens the test coverage to verify correct behavior under different provider environments.

Test Suggestions

  • Write tests to verify correct behavior when only GEMINI_API_KEY is set (and neither OPENAI_API_KEY nor AZURE_OPENAI_KEY is provided).
  • Add integration tests for _ragas_is_finished_parser with finish reasons from both Gemini and OpenAI responses.
  • Simulate failure scenarios by unsetting required environment variables to ensure proper exception messages are raised.
  • Create tests to verify that get_deepeval_model returns the appropriate model instance based on the provider (Gemini vs OpenAI).
  • Verify logging outputs for judge configuration warnings/errors when invalid types are passed.

Copy link
Copy Markdown
Contributor

@johnwalz97 johnwalz97 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very nice! lgtm

Copy link
Copy Markdown
Contributor

@AnilSorathiya AnilSorathiya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@nrichers nrichers added the support Support-related PR label May 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request support Support-related PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants