diff --git a/notebooks/how_to/tests/run_tests/configure_tests/configure_judge_llms.ipynb b/notebooks/how_to/tests/run_tests/configure_tests/configure_judge_llms.ipynb new file mode 100644 index 000000000..3e8d27bcd --- /dev/null +++ b/notebooks/how_to/tests/run_tests/configure_tests/configure_judge_llms.ipynb @@ -0,0 +1,827 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "0935afb5", + "metadata": {}, + "source": [ + "# Configure judge LLM and judge embeddings\n", + "\n", + "This notebook shows how to configure and validate the default judge LLM and judge embeddings used by the ValidMind Library for LLM-focused tests.\n", + "\n", + "It exercises three important paths:\n", + "1. Prompt-validation tests, which depend on the default judge LLM.\n", + "2. RAGAS-based tests, which depend on both the default judge LLM and the default judge embeddings model.\n", + "3. DeepEval scorers, which depend on the default local scorer model path.\n", + "\n", + "The notebook automatically selects the available provider from your environment, with OpenAI taking precedence when both OpenAI and Gemini keys are set, to match the library's default-provider logic." + ] + }, + { + "cell_type": "markdown", + "id": "1f2befa6", + "metadata": {}, + "source": [ + "::: {.content-hidden when-format=\"html\"}\n", + "## Contents\n", + "- [Introduction](#toc1__)\n", + "- [About ValidMind](#toc2__)\n", + " - [Before you begin](#toc2_1__)\n", + " - [New to ValidMind?](#toc2_2__)\n", + " - [Key concepts](#toc2_3__)\n", + "- [Setting up](#toc3__)\n", + " - [Install the ValidMind Library](#toc3_1__)\n", + " - [Connect to the ValidMind Platform](#toc3_2__)\n", + " - [Register or select a model](#toc3_2_1__)\n", + " - [Choose a documentation template](#toc3_2_2__)\n", + " - [Get your code snippet](#toc3_2_3__)\n", + " - [Initialize the notebook environment](#toc3_3__)\n", + "- [Getting to know ValidMind](#toc4__)\n", + " - [Preview the documentation template](#toc4_1__)\n", + " - [View model documentation in the ValidMind Platform](#toc4_2__)\n", + "- [Configure the judge provider](#toc5__)\n", + "- [Prompt-validation tests](#toc6__)\n", + "- [RAGAS tests](#toc7__)\n", + "- [DeepEval scorers](#toc8__)\n", + "- [In summary](#toc9__)\n", + "- [Next steps](#toc10__)\n", + " - [Discover more learning resources](#toc10_1__)\n", + "- [Upgrade ValidMind](#toc11__)\n", + "\n", + ":::\n", + "\n", + "" + ] + }, + { + "cell_type": "markdown", + "id": "b77005b8", + "metadata": {}, + "source": [ + "\n", + "\n", + "## Introduction\n", + "\n", + "This notebook shows how to configure and validate the default judge LLM and judge embeddings used by the ValidMind Library for LLM-focused tests.\n", + "\n", + "It walks through the provider configuration used by three important evaluation paths:\n", + "- prompt-validation tests\n", + "- RAGAS-based tests\n", + "- DeepEval scorers\n", + "\n", + "Along the way, you will initialize ValidMind model and dataset objects, inspect the resolved judge configuration, run representative tests, and optionally log the results to the ValidMind Platform. By the end of the notebook, you will have a practical reference for configuring judge models and understanding how those settings affect different LLM evaluation workflows." + ] + }, + { + "cell_type": "markdown", + "id": "56ecff8d", + "metadata": {}, + "source": [ + "\n", + "\n", + "## About ValidMind\n", + "\n", + "ValidMind is a suite of tools for managing model risk, including risk associated with AI and statistical models. \n", + "\n", + "You use the ValidMind Library to automate documentation and validation tests, and then use the ValidMind Platform to collaborate on model documentation. Together, these products simplify model risk management, facilitate compliance with regulations and institutional standards, and enhance collaboration between yourself and model validators." + ] + }, + { + "cell_type": "markdown", + "id": "e8743d30", + "metadata": {}, + "source": [ + "\n", + "\n", + "### Before you begin\n", + "\n", + "Before running this notebook, make sure you have:\n", + "- a Python environment with the ValidMind Library and its LLM dependencies installed\n", + "- access to a ValidMind account if you want to log results to the ValidMind Platform\n", + "- credentials for one supported judge provider in your environment\n", + "\n", + "This notebook supports:\n", + "- OpenAI via `OPENAI_API_KEY`, with optional `OPENAI_MODEL` and `OPENAI_EMBEDDINGS_MODEL` overrides. The current default judge model is `gpt-4.1` and the default embeddings model is `text-embedding-3-small`.\n", + "- Gemini via `GOOGLE_API_KEY` or `GEMINI_API_KEY`, with optional `GEMINI_MODEL` and `GEMINI_EMBEDDINGS_MODEL` overrides. The current defaults are `gemini-2.5-pro` and `models/text-embedding-004`.\n", + "- Azure OpenAI via `AZURE_OPENAI_KEY`, `AZURE_OPENAI_ENDPOINT`, and `AZURE_OPENAI_MODEL`. The current default embeddings model is `text-embedding-3-small`.\n", + "\n", + "You can still run the notebook locally without connecting to the ValidMind Platform, but connecting a model document makes it easier to review and share results after the tests complete." + ] + }, + { + "cell_type": "markdown", + "id": "ee479eb5", + "metadata": {}, + "source": [ + "\n", + "\n", + "### New to ValidMind?\n", + "\n", + "If you are new to the ValidMind Library, start with the [ValidMind Library overview](https://docs.validmind.ai/developer/validmind-library.html). It introduces the core workflow for initializing models and datasets, running tests, and logging outputs back to the ValidMind Platform.\n", + "\n", + "
You only need a ValidMind account if you want to log results to the ValidMind Platform.\n", + "

\n", + "Register with ValidMind
" + ] + }, + { + "cell_type": "markdown", + "id": "689e55db", + "metadata": {}, + "source": [ + "\n", + "\n", + "### Key concepts\n", + "\n", + "**Judge LLM**: The language model used by ValidMind to evaluate prompts, answers, contexts, and other LLM outputs.\n", + "\n", + "**Judge embeddings**: The embeddings model used when a test requires semantic similarity or retrieval-based comparison.\n", + "\n", + "**Provider credentials**: Environment variables that tell ValidMind which provider to use for judge evaluation. In this notebook, the provider is resolved automatically from the credentials available in your environment.\n", + "\n", + "**ValidMind dataset**: A dataset initialized with [`vm.init_dataset()`](https://docs.validmind.ai/validmind/validmind.html#init_dataset). Wrapping a pandas DataFrame this way lets you pass the dataset into ValidMind tests with the metadata those tests expect.\n", + "\n", + "**ValidMind model**: A model initialized with [`vm.init_model()`](https://docs.validmind.ai/validmind/validmind.html#init_model). In this notebook, we use a lightweight model object to run prompt-validation tests against a prompt template.\n", + "\n", + "**Prompt-validation tests**: Tests that evaluate prompt quality and instructions, such as clarity or bias, using a judge LLM.\n", + "\n", + "**RAGAS tests**: Retrieval-augmented generation tests that can rely on both a judge LLM and judge embeddings.\n", + "\n", + "**DeepEval scorers**: LLM-based scorers used for tasks such as answer relevancy and hallucination detection. These use the evaluation model path but do not require judge embeddings." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "\n", + "## Setting up" + ] + }, + { + "cell_type": "markdown", + "id": "8d6a8300", + "metadata": {}, + "source": [ + "\n", + "\n", + "### Install the ValidMind Library\n", + "\n", + "
Recommended Python versions\n", + "

\n", + "Python 3.8 <= x <= 3.14
\n", + "\n", + "Install the ValidMind Library with the optional LLM dependencies so the notebook can run prompt-validation tests, RAGAS tests, and DeepEval scorers:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "da644666", + "metadata": {}, + "outputs": [], + "source": [ + "%pip install -q \"validmind[llm]\"" + ] + }, + { + "cell_type": "markdown", + "id": "17c1c7d2", + "metadata": {}, + "source": [ + "\n", + "\n", + "### Connect to the ValidMind Platform\n", + "\n", + "If you want to log notebook outputs to the ValidMind Platform, start by selecting an existing model in your inventory or registering a new one. This notebook can run without platform connectivity, but linking it to a model document gives you a place to review the results after the examples finish.\n", + "\n", + "\n", + "\n", + "#### Register or select a model\n", + "\n", + "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n", + "2. Open **Inventory** and either select an existing model or click **+ Register Model**.\n", + "3. Complete the model details and stakeholder assignments if you are registering a new model.\n", + "4. Open the document where you want notebook results to be logged.\n", + "\n", + "Using a real model document is especially helpful in this notebook because it lets you compare the locally executed tests with the sections available in your template." + ] + }, + { + "cell_type": "markdown", + "id": "dc628c6c", + "metadata": {}, + "source": [ + "\n", + "\n", + "#### Choose a documentation template\n", + "\n", + "If you plan to log results from this notebook, make sure your model document uses a template that includes sections for the LLM evaluation results you want to capture.\n", + "\n", + "This is important because tests that are not included in the selected template will not appear automatically in the Platform document, even if you run and log them successfully from the notebook. If you want to document those results as well, you can add the relevant sections or tests manually in the Platform.\n", + "\n", + "Before running the notebook, preview the template structure and confirm that the document has the sections you expect for your workflow." + ] + }, + { + "cell_type": "markdown", + "id": "702f5196", + "metadata": {}, + "source": [ + "\n", + "\n", + "#### Get your code snippet\n", + "\n", + "Initialize the ValidMind Library with the code snippet associated with your model document so that test results are uploaded to the correct destination in the ValidMind Platform.\n", + "\n", + "1. In the model sidebar, open **Getting Started**.\n", + "2. Select the document you want to update.\n", + "3. Copy the generated code snippet.\n", + "4. Load the values from an `.env` file or replace the placeholders in the example below with your own values.\n", + "\n", + "Using environment variables is usually the easiest way to keep the notebook portable across environments and avoid hard-coding connection details in the notebook itself." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c52a42d0", + "metadata": {}, + "outputs": [], + "source": [ + "# Load your model identifier credentials from an `.env` file\n", + "\n", + "%load_ext dotenv\n", + "%dotenv .env\n", + "\n", + "# Or replace with your code snippet\n", + "\n", + "import validmind as vm\n", + "\n", + "vm.init(\n", + " api_host=\"http://localhost:5000/api/v1/tracking\",\n", + " api_key=\"..\",\n", + " api_secret=\"..\",\n", + " document=\"documentation\", # requires library >=2.12.0\n", + " model=\"..\",\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "3657a7a4", + "metadata": {}, + "source": [ + "\n", + "\n", + "### Initialize the notebook environment\n", + "\n", + "Load environment variables and prepare the notebook session. In the execution cells that follow, you will import the libraries needed for this walkthrough, inspect the configured judge provider, and create the ValidMind objects used by the example tests.\n", + "\n", + "This section is also where the notebook becomes reproducible: once your credentials and dependencies are in place, the remaining sections can be run top to bottom." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "979988a9", + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "\n", + "import pandas as pd\n", + "\n", + "from validmind.ai import utils as ai_utils\n", + "from validmind.models import Prompt\n", + "from validmind.tests import run_test" + ] + }, + { + "cell_type": "markdown", + "id": "3db58d74", + "metadata": {}, + "source": [ + "\n", + "\n", + "## Getting to know ValidMind" + ] + }, + { + "cell_type": "markdown", + "id": "45450d55", + "metadata": {}, + "source": [ + "\n", + "\n", + "### Preview the documentation template\n", + "\n", + "If you have already connected this notebook to a model document, you can preview the active template structure directly from the library.\n", + "\n", + "This is useful for confirming where logged results will appear before you run the prompt-validation, RAGAS, and DeepEval examples below. It also helps you spot gaps early if a test you plan to run is not represented in the current template:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "98f0b602", + "metadata": {}, + "outputs": [], + "source": [ + "vm.preview_template()" + ] + }, + { + "cell_type": "markdown", + "id": "58e1d75f", + "metadata": {}, + "source": [ + "\n", + "\n", + "### View model documentation in the ValidMind Platform\n", + "\n", + "After you run the notebook and log results, open your model document in the ValidMind Platform to review how the test outputs were added.\n", + "\n", + "Comparing the template preview with the rendered document is a good way to confirm that your notebook is writing results to the expected sections. If a result does not appear automatically, check whether the corresponding test is part of the selected template before troubleshooting the notebook run itself." + ] + }, + { + "cell_type": "markdown", + "id": "86038351", + "metadata": {}, + "source": [ + "\n", + "\n", + "## Configure the judge provider\n", + "\n", + "The next cells load your environment variables, resolve the judge provider from the credentials available in your session, and initialize the ValidMind Library for result logging.\n", + "\n", + "This notebook uses the same provider resolution logic as the library itself:\n", + "- OpenAI is selected when `OPENAI_API_KEY` is available, with `OPENAI_MODEL` as an optional override. The current default judge model is `gpt-4.1`.\n", + "- Azure OpenAI is selected when Azure credentials are available, using `AZURE_OPENAI_MODEL` for the judge model.\n", + "- Gemini is selected when `GOOGLE_API_KEY` or `GEMINI_API_KEY` is available, with optional `GEMINI_MODEL` and `GEMINI_EMBEDDINGS_MODEL` overrides. The current defaults are `gemini-2.5-pro` and `models/text-embedding-004`.\n", + "\n", + "If more than one provider is configured, OpenAI takes precedence to match the library default.\n", + "\n", + "This matters because the same default judge configuration is reused across multiple evaluation paths, so checking it once here makes the later test results easier to interpret." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a3efda1f", + "metadata": {}, + "outputs": [], + "source": [ + "# Optional: override the default judge models for this notebook session.\n", + "# os.environ[\"OPENAI_MODEL\"] = \"gpt-4.1\"\n", + "# os.environ[\"GEMINI_MODEL\"] = \"gemini-2.5-pro\"\n", + "# os.environ[\"GEMINI_EMBEDDINGS_MODEL\"] = \"models/text-embedding-004\"" + ] + }, + { + "cell_type": "markdown", + "id": "f0438cf0", + "metadata": {}, + "source": [ + "The next cells import the required libraries, inspect the resolved provider configuration, and connect the notebook to the ValidMind Platform. Reading the printed provider and class names is a quick sanity check that your environment is using the judge setup you expect before any tests are executed." + ] + }, + { + "cell_type": "markdown", + "id": "089f10fa", + "metadata": {}, + "source": [ + "\n", + "\n", + "### Load credentials and resolve the provider\n", + "\n", + "Run the next cells to:\n", + "- import the libraries used in this notebook\n", + "- inspect the provider selected from your environment\n", + "- inspect the resolved judge LLM and judge embeddings classes\n", + "- initialize the ValidMind Library with your platform credentials\n", + "\n", + "If both OpenAI and Gemini credentials are available, OpenAI will be selected to match the default provider precedence used by the library.\n", + "\n", + "This section gives you a concrete view of the effective configuration that the later prompt-validation, RAGAS, and DeepEval examples will use." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f1479922", + "metadata": {}, + "outputs": [], + "source": [ + "from validmind.ai.utils import get_client_and_model, get_judge_config\n", + "\n", + "client, model = get_client_and_model()\n", + "judge_llm, judge_embeddings = get_judge_config()\n", + "\n", + "print(\"resolved_model:\", model)\n", + "print(\"judge_llm_type:\", type(judge_llm).__name__)\n", + "print(\"judge_embeddings_type:\", type(judge_embeddings).__name__)\n", + "\n", + "# Useful for Gemini/OpenAI/Azure debugging\n", + "print(\"judge_llm:\", judge_llm)\n", + "print(\"judge_embeddings:\", judge_embeddings)" + ] + }, + { + "cell_type": "markdown", + "id": "e7868c71", + "metadata": {}, + "source": [ + "\n", + "\n", + "## Prompt-validation tests\n", + "\n", + "This section validates the default judge LLM path with two representative prompt-validation tests. For this smoke test, we use a simple prompt-only model because these tests evaluate the prompt template itself and do not require model predictions.\n", + "\n", + "The example below creates a ValidMind model with `vm.init_model()` and attaches a prompt template to it. That gives the tests a standard object to inspect, even though there is no real predictive model behind the example.\n", + "\n", + "- `Clarity` checks whether the prompt instructions are clear and well-scoped.\n", + "- `Bias` checks whether the prompt structure or examples could induce biased behavior." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "7cc07ba8", + "metadata": {}, + "outputs": [], + "source": [ + "system_prompt = \"\"\"\n", + "You are an AI assistant specialized in sentiment analysis for financial news.\n", + "You will classify each sentence as positive, negative, or neutral.\n", + "Respond only with the sentiment label.\n", + "\"\"\".strip()\n", + "\n", + "\n", + "def noop_predict(_):\n", + " return \"dummy\"\n", + "\n", + "\n", + "vm_prompt_model = vm.init_model(\n", + " input_id=\"judge_prompt_model\",\n", + " predict_fn=noop_predict,\n", + " prompt=Prompt(template=system_prompt, variables=[]),\n", + ")\n", + "\n", + "vm_prompt_model.prompt.template" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "40298f6b", + "metadata": {}, + "outputs": [], + "source": [ + "run_test(\n", + " test_id=\"validmind.prompt_validation.Clarity\",\n", + " inputs={\"model\": vm_prompt_model},\n", + ").log()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "4180b0f1", + "metadata": {}, + "outputs": [], + "source": [ + "run_test(\n", + " test_id=\"validmind.prompt_validation.Bias\",\n", + " inputs={\"model\": vm_prompt_model},\n", + ").log()" + ] + }, + { + "cell_type": "markdown", + "id": "9935d075", + "metadata": {}, + "source": [ + "\n", + "\n", + "## RAGAS tests\n", + "\n", + "This section validates the default judge LLM plus default judge embeddings path. The selected tests are useful because they exercise the RAGAS integration that historically depended on the default OpenAI setup.\n", + "\n", + "The example data is wrapped with `vm.init_dataset()`, which turns the pandas DataFrame into a ValidMind dataset object that can be passed directly into these tests.\n", + "\n", + "- `ResponseRelevancy` exercises the judge LLM and embeddings path.\n", + "- `AnswerCorrectness` exercises semantic and factual comparison with judge embeddings.\n", + "- `Faithfulness` is a companion smoke test for the judge LLM path on RAG data.\n", + "\n", + "These tests produce Plotly figures, so this notebook focuses on running and logging the results rather than comparing visual output in detail." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "17cbf0e3", + "metadata": {}, + "outputs": [], + "source": [ + "rag_df = pd.DataFrame(\n", + " {\n", + " \"user_input\": [\n", + " \"What happened to the company's revenue guidance?\",\n", + " \"Why did the bank's stock decline?\",\n", + " \"What was the announced dividend decision?\",\n", + " ],\n", + " \"retrieved_contexts\": [\n", + " [\n", + " \"The company raised its full-year revenue guidance after reporting strong demand in the enterprise segment.\",\n", + " \"Management said the improved forecast was driven by larger-than-expected renewals.\",\n", + " ],\n", + " [\n", + " \"The bank's stock declined after it reported higher-than-expected credit losses in its consumer portfolio.\",\n", + " \"Executives also warned that provisions may remain elevated next quarter.\",\n", + " ],\n", + " [\n", + " \"The board announced that it would keep the quarterly dividend unchanged.\",\n", + " \"Management said capital return policy remains the same for now.\",\n", + " ],\n", + " ],\n", + " \"response\": [\n", + " \"The company increased its full-year revenue guidance after stronger enterprise demand.\",\n", + " \"The bank's stock fell because it disclosed higher-than-expected credit losses.\",\n", + " \"The company kept its dividend unchanged.\",\n", + " ],\n", + " \"reference\": [\n", + " \"The company raised its full-year revenue guidance because demand in the enterprise segment was strong.\",\n", + " \"The bank's shares dropped after it reported higher-than-expected credit losses.\",\n", + " \"The board decided to leave the quarterly dividend unchanged.\",\n", + " ],\n", + " }\n", + ")\n", + "\n", + "vm_rag_ds = vm.init_dataset(\n", + " dataset=rag_df,\n", + " input_id=\"judge_rag_dataset\",\n", + " text_column=\"user_input\",\n", + " target_column=\"reference\",\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "fcdb6232", + "metadata": {}, + "outputs": [], + "source": [ + "run_test(\n", + " test_id=\"validmind.model_validation.ragas.ResponseRelevancy\",\n", + " inputs={\"dataset\": vm_rag_ds},\n", + ").log()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "25124a2f", + "metadata": {}, + "outputs": [], + "source": [ + "run_test(\n", + " test_id=\"validmind.model_validation.ragas.AnswerCorrectness\",\n", + " inputs={\"dataset\": vm_rag_ds},\n", + ").log()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "3a58bd42", + "metadata": {}, + "outputs": [], + "source": [ + "run_test(\n", + " test_id=\"validmind.model_validation.ragas.Faithfulness\",\n", + " inputs={\"dataset\": vm_rag_ds},\n", + ").log()" + ] + }, + { + "cell_type": "markdown", + "id": "8b65420f", + "metadata": {}, + "source": [ + "\n", + "\n", + "## DeepEval scorers\n", + "\n", + "This section validates the default local scorer model path used by DeepEval-based scorers in `validmind.scorers.llm.deepeval`.\n", + "\n", + "As in the RAGAS example, we create a ValidMind dataset with `vm.init_dataset()` so the scorer workflow runs against the same kind of object customers would use in their own notebooks.\n", + "\n", + "These scorers do not use the judge embeddings object. For this notebook, we use two representative examples:\n", + "- `AnswerRelevancy`\n", + "- `Hallucination`\n", + "\n", + "They are included here so the notebook covers all three LLM evaluation surfaces:\n", + "- prompt-validation\n", + "- RAGAS\n", + "- DeepEval scorers" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c34f2484", + "metadata": {}, + "outputs": [], + "source": [ + "deepeval_df = pd.DataFrame(\n", + " {\n", + " \"input\": [\n", + " \"What is the capital of France?\",\n", + " \"Why did the company raise its full-year guidance?\",\n", + " \"What did the board decide about the quarterly dividend?\",\n", + " ],\n", + " \"actual_output\": [\n", + " \"The capital of France is Paris.\",\n", + " \"The company raised guidance because enterprise demand was stronger than expected.\",\n", + " \"The board kept the quarterly dividend unchanged.\",\n", + " ],\n", + " \"context\": [\n", + " [\"France's capital city is Paris.\"],\n", + " [\n", + " \"Management raised its full-year guidance after reporting stronger-than-expected demand in the enterprise segment.\"\n", + " ],\n", + " [\n", + " \"The board announced that the quarterly dividend would remain unchanged.\"\n", + " ],\n", + " ],\n", + " }\n", + ")\n", + "\n", + "vm_deepeval_ds = vm.init_dataset(\n", + " dataset=deepeval_df,\n", + " input_id=\"judge_deepeval_dataset\",\n", + " text_column=\"input\",\n", + " target_column=\"actual_output\",\n", + ")\n", + "\n", + "deepeval_df" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "9a3cdae0", + "metadata": {}, + "outputs": [], + "source": [ + "vm_deepeval_ds.assign_scores(metrics=[\n", + " \"validmind.scorers.llm.deepeval.Hallucination\",\n", + " \"validmind.scorers.llm.deepeval.AnswerRelevancy\"\n", + "])" + ] + }, + { + "cell_type": "markdown", + "id": "d86a90ab", + "metadata": {}, + "source": [ + "\n", + "\n", + "## In summary\n", + "\n", + "In this notebook, you learned how to:\n", + "- [x] configure the judge provider from environment credentials\n", + "- [x] override the default judge LLM and judge embeddings models\n", + "- [x] initialize ValidMind model and dataset objects for LLM evaluation workflows\n", + "- [x] run prompt-validation tests that use the judge LLM\n", + "- [x] run RAGAS tests that use the judge LLM and judge embeddings\n", + "- [x] run DeepEval scorers that use the local scorer model path" + ] + }, + { + "cell_type": "markdown", + "id": "c7b72b3e", + "metadata": {}, + "source": [ + "\n", + "\n", + "## Next steps\n", + "\n", + "You can use this notebook as a starting point for your own LLM evaluation workflows. A few practical follow-ups are:\n", + "- replace the sample prompt and datasets with your own evaluation inputs\n", + "- set `OPENAI_MODEL` / `OPENAI_EMBEDDINGS_MODEL` when you want to override the OpenAI judge pair, or `GEMINI_MODEL` / `GEMINI_EMBEDDINGS_MODEL` when you want to standardize the Gemini judge pair used across notebooks or environments\n", + "- expand the set of tests and scorers based on your use case" + ] + }, + { + "cell_type": "markdown", + "id": "e5eb12d8", + "metadata": {}, + "source": [ + "\n", + "\n", + "### Discover more learning resources\n", + "\n", + "To continue learning about testing and evaluation with the ValidMind Library, explore:\n", + "\n", + "- [Run tests and test suites](https://docs.validmind.ai/developer/how-to/testing-overview.html)\n", + "- [ValidMind Library overview](https://docs.validmind.ai/developer/validmind-library.html)\n", + "- [Use ValidMind Library features](https://docs.validmind.ai/developer/how-to/feature-overview.html)\n", + "- [Code samples by use case](https://docs.validmind.ai/guide/samples-jupyter-notebooks.html)\n", + "\n", + "You can also visit the [ValidMind documentation](https://docs.validmind.ai/) for broader guidance on configuration, testing workflows, and model documentation." + ] + }, + { + "cell_type": "markdown", + "id": "99a11a0e", + "metadata": {}, + "source": [ + "\n", + "\n", + "## Upgrade ValidMind\n", + "\n", + "
After installing ValidMind, periodically check that you are using a recent version so you can access the latest provider integrations, tests, and product improvements.
\n", + "\n", + "Retrieve the information for the currently installed version of ValidMind:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cfed92f5", + "metadata": {}, + "outputs": [], + "source": [ + "%pip show validmind" + ] + }, + { + "cell_type": "markdown", + "id": "58cc2437", + "metadata": {}, + "source": [ + "If the version returned is lower than the version indicated in our [production open-source code](https://github.com/validmind/validmind-library/blob/prod/validmind/__version__.py), restart your notebook and run:\n", + "\n", + "```bash\n", + "%pip install --upgrade validmind\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "aa5c4672", + "metadata": {}, + "source": [ + "You may need to restart your kernel after running the upgrade package for changes to be applied." + ] + }, + { + "cell_type": "markdown", + "id": "copyright-fe0b013da3464949b043e9dbdd34b608", + "metadata": {}, + "source": [ + "\n", + "\n", + "\n", + "\n", + "***\n", + "\n", + "Copyright © 2023-2026 ValidMind Inc. All rights reserved.
\n", + "Refer to [LICENSE](https://github.com/validmind/validmind-library/blob/main/LICENSE) for details.
\n", + "SPDX-License-Identifier: AGPL-3.0 AND ValidMind Commercial" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": ".venv-py31111", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.11" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/pyproject.toml b/pyproject.toml index 03325b4b8..bb4891fee 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "validmind" -version = "2.13.2" +version = "2.13.3" description = "ValidMind Library" readme = "README.pypi.md" requires-python = ">=3.9,<3.15" @@ -46,6 +46,8 @@ all = [ "langchain-community (>=0.2.0,<0.4.0)", "sentencepiece (>=0.2.0,<0.3.0)", "langchain-openai (>=0.1.8)", + "langchain-google-genai", + "google-genai", "scipy", "statsmodels (>=0.14.2,<0.15.0)", "langdetect", @@ -79,6 +81,8 @@ llm = [ "datasets (>=4.4,<5.0.0) ; python_version >= '3.14'", "sentencepiece (>=0.2.0,<0.3.0)", "langchain-openai (>=0.1.8)", + "langchain-google-genai", + "google-genai", "pyarrow (<16) ; python_version < '3.13'", "pyarrow (>=20,<21) ; python_version == '3.13'", "pyarrow (>=22,<23) ; python_version >= '3.14'", diff --git a/r/validmind/DESCRIPTION b/r/validmind/DESCRIPTION index 7b44879de..470383be9 100644 --- a/r/validmind/DESCRIPTION +++ b/r/validmind/DESCRIPTION @@ -1,7 +1,7 @@ Package: validmind Type: Package Title: Interface to the 'ValidMind' Platform -Version: 2.13.2 +Version: 2.13.3 Authors@R: c(person("Andres", "Rodriguez", role = c("aut", "cre","cph"), email = "andres@validmind.ai")) Maintainer: Andres Rodriguez diff --git a/tests/unit_tests/model_validation/ragas/test_utils.py b/tests/unit_tests/model_validation/ragas/test_utils.py new file mode 100644 index 000000000..bc27a95c9 --- /dev/null +++ b/tests/unit_tests/model_validation/ragas/test_utils.py @@ -0,0 +1,48 @@ +from types import SimpleNamespace + +from langchain_core.messages import AIMessage +from langchain_core.outputs import ChatGeneration + +from validmind.tests.model_validation.ragas.utils import _ragas_is_finished_parser + + +def test_ragas_is_finished_parser_accepts_gemini_stop_reason(): + response = SimpleNamespace( + flatten=lambda: [ + SimpleNamespace( + generations=[ + [ + ChatGeneration( + message=AIMessage( + content="done", + response_metadata={"finish_reason": "STOP"}, + ) + ) + ] + ] + ) + ] + ) + + assert _ragas_is_finished_parser(response) is True + + +def test_ragas_is_finished_parser_accepts_max_tokens_reason(): + response = SimpleNamespace( + flatten=lambda: [ + SimpleNamespace( + generations=[ + [ + ChatGeneration( + message=AIMessage( + content="partial", + response_metadata={"finish_reason": "MAX_TOKENS"}, + ) + ) + ] + ] + ) + ] + ) + + assert _ragas_is_finished_parser(response) is True diff --git a/tests/unit_tests/test_ai_utils.py b/tests/unit_tests/test_ai_utils.py new file mode 100644 index 000000000..bdc857bd5 --- /dev/null +++ b/tests/unit_tests/test_ai_utils.py @@ -0,0 +1,99 @@ +import os +import sys +import types +from unittest import mock + +from validmind.ai import utils as ai_utils + + +def _reset_ai_utils_state(): + ai_utils.__dict__["__client"] = None + ai_utils.__dict__["__model"] = None + ai_utils.__dict__["__judge_llm"] = None + ai_utils.__dict__["__judge_embeddings"] = None + ai_utils.__dict__["__ack"] = None + + +def test_get_client_and_model_supports_gemini_env(): + _reset_ai_utils_state() + + with mock.patch.dict(os.environ, {"GEMINI_API_KEY": "test-key"}, clear=True): + client, model = ai_utils.get_client_and_model() + + assert client is None + assert model == ai_utils.GEMINI_MODEL + + +def test_get_judge_config_builds_gemini_models(): + _reset_ai_utils_state() + + class FakeChatGoogleGenerativeAI: + def __init__(self, **kwargs): + self.kwargs = kwargs + + class FakeGoogleGenerativeAIEmbeddings: + def __init__(self, **kwargs): + self.kwargs = kwargs + + fake_module = types.SimpleNamespace( + ChatGoogleGenerativeAI=FakeChatGoogleGenerativeAI, + GoogleGenerativeAIEmbeddings=FakeGoogleGenerativeAIEmbeddings, + ) + + with mock.patch.dict( + os.environ, + {"GEMINI_API_KEY": "test-key", "GEMINI_MODEL": "gemini-test-model"}, + clear=True, + ), mock.patch.dict(sys.modules, {"langchain_google_genai": fake_module}): + judge_llm, judge_embeddings = ai_utils.get_judge_config() + + assert isinstance(judge_llm, FakeChatGoogleGenerativeAI) + assert judge_llm.kwargs == { + "model": "gemini-test-model", + "api_key": "test-key", + } + assert isinstance(judge_embeddings, FakeGoogleGenerativeAIEmbeddings) + assert judge_embeddings.kwargs == { + "model": ai_utils.GEMINI_EMBEDDINGS_MODEL, + "google_api_key": "test-key", + } + + +def test_is_configured_uses_resolved_judge_model(): + _reset_ai_utils_state() + + class FakeJudgeLLM: + def invoke(self, messages): + assert messages == [("user", "ping")] + return types.SimpleNamespace(content="pong") + + with mock.patch.object( + ai_utils, + "get_judge_config", + return_value=(FakeJudgeLLM(), None), + ): + assert ai_utils.is_configured() is True + + +def test_get_deepeval_model_supports_gemini_env(): + _reset_ai_utils_state() + + class FakeGeminiModel: + def __init__(self, **kwargs): + self.kwargs = kwargs + + fake_module = types.SimpleNamespace(GeminiModel=FakeGeminiModel) + + with mock.patch.dict( + os.environ, + {"GEMINI_API_KEY": "test-key", "GEMINI_MODEL": "gemini-test-model"}, + clear=True, + ), mock.patch.dict(sys.modules, {"deepeval.models": fake_module}): + model = ai_utils.get_deepeval_model() + + assert isinstance(model, FakeGeminiModel) + assert model.kwargs == { + "model": "gemini-test-model", + "api_key": "test-key", + "temperature": 0, + } diff --git a/uv.lock b/uv.lock index 68bd4000c..91551f66f 100644 --- a/uv.lock +++ b/uv.lock @@ -2167,6 +2167,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/81/47/dd9a212ef6e343a6857485ffe25bba537304f1913bdbed446a23f7f592e1/filelock-3.29.0-py3-none-any.whl", hash = "sha256:96f5f6344709aa1572bbf631c640e4ebeeb519e08da902c39a001882f30ac258", size = 39812, upload-time = "2026-04-19T15:39:08.752Z" }, ] +[[package]] +name = "filetype" +version = "1.2.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/bb/29/745f7d30d47fe0f251d3ad3dc2978a23141917661998763bebb6da007eb1/filetype-1.2.0.tar.gz", hash = "sha256:66b56cd6474bf41d8c54660347d37afcc3f7d1970648de365c102ef77548aadb", size = 998020, upload-time = "2022-11-02T17:34:04.141Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/18/79/1b8fa1bb3568781e84c9200f951c735f3f157429f44be0495da55894d620/filetype-1.2.0-py2.py3-none-any.whl", hash = "sha256:7ce71b6880181241cf7ac8697a2f1eb6a8bd9b429f7ad6d27b8db9ba5f1c2d25", size = 19970, upload-time = "2022-11-02T17:34:01.425Z" }, +] + [[package]] name = "flake8" version = "7.3.0" @@ -2505,6 +2514,147 @@ http = [ { name = "aiohttp", marker = "python_full_version >= '3.14'" }, ] +[[package]] +name = "google-ai-generativelanguage" +version = "0.11.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "google-api-core", extra = ["grpc"] }, + { name = "google-auth", version = "2.50.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.10'" }, + { name = "google-auth", version = "2.52.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.10'" }, + { name = "grpcio" }, + { name = "proto-plus", version = "1.27.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.10'" }, + { name = "proto-plus", version = "1.28.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.10'" }, + { name = "protobuf" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/b3/36/9365ca196c4753d0d22951ee0147ecb2124382a010d5dbfb9f3ecd57f2cf/google_ai_generativelanguage-0.11.0.tar.gz", hash = "sha256:d9e24e9836e894a85b52ca03d03530988aeb492d48df71cd1573dc1c3b6d81fc", size = 1539696, upload-time = "2026-03-30T22:51:43.143Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/06/8c/dc1faf90d7a6ec77e09a6706abefabb41ae5b8d838e5e5b16914da26ee1e/google_ai_generativelanguage-0.11.0-py3-none-any.whl", hash = "sha256:f797f307f0969ae49622e09c1d1a23aa86c7538ffae881279506548166e91d45", size = 1432183, upload-time = "2026-03-30T22:47:56.02Z" }, +] + +[[package]] +name = "google-api-core" +version = "2.30.3" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "google-auth", version = "2.50.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.10'" }, + { name = "google-auth", version = "2.52.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.10'" }, + { name = "googleapis-common-protos" }, + { name = "proto-plus", version = "1.27.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.10'" }, + { name = "proto-plus", version = "1.28.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.10'" }, + { name = "protobuf" }, + { name = "requests" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/16/ce/502a57fb0ec752026d24df1280b162294b22a0afb98a326084f9a979138b/google_api_core-2.30.3.tar.gz", hash = "sha256:e601a37f148585319b26db36e219df68c5d07b6382cff2d580e83404e44d641b", size = 177001, upload-time = "2026-04-10T00:41:28.035Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/03/15/e56f351cf6ef1cfea58e6ac226a7318ed1deb2218c4b3cc9bd9e4b786c5a/google_api_core-2.30.3-py3-none-any.whl", hash = "sha256:a85761ba72c444dad5d611c2220633480b2b6be2521eca69cca2dbb3ffd6bfe8", size = 173274, upload-time = "2026-04-09T22:57:16.198Z" }, +] + +[package.optional-dependencies] +grpc = [ + { name = "grpcio" }, + { name = "grpcio-status" }, +] + +[[package]] +name = "google-auth" +version = "2.50.0" +source = { registry = "https://pypi.org/simple" } +resolution-markers = [ + "python_full_version < '3.10'", +] +dependencies = [ + { name = "cryptography", version = "44.0.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.10'" }, + { name = "pyasn1-modules", marker = "python_full_version < '3.10'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/5f/18/238d7021d151bdab868f23433817b027dd759135202f4dfce0670d1230ca/google_auth-2.50.0.tar.gz", hash = "sha256:f35eafb191195328e8ce10a7883970877e7aeb49c2bfaa54aa0e394316d353d0", size = 336523, upload-time = "2026-04-30T21:19:29.659Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/37/cf/4880c2137c14280b2f59975cdf12cc442bc0ae1f9ea473a26eaa0c146786/google_auth-2.50.0-py3-none-any.whl", hash = "sha256:04382175e28b94f49694977f0a792688b59a668def1499e9d8de996dc9ce5b15", size = 246495, upload-time = "2026-04-30T21:19:27.664Z" }, +] + +[[package]] +name = "google-auth" +version = "2.52.0" +source = { registry = "https://pypi.org/simple" } +resolution-markers = [ + "python_full_version >= '3.14' and platform_machine == 'x86_64' and sys_platform == 'darwin'", + "(python_full_version >= '3.14' and platform_machine != 'x86_64') or (python_full_version >= '3.14' and sys_platform != 'darwin')", + "python_full_version == '3.13.*' and platform_machine == 'x86_64' and sys_platform == 'darwin'", + "(python_full_version == '3.13.*' and platform_machine != 'x86_64') or (python_full_version == '3.13.*' and sys_platform != 'darwin')", + "python_full_version == '3.12.*' and platform_machine == 'x86_64' and sys_platform == 'darwin'", + "(python_full_version == '3.12.*' and platform_machine != 'x86_64') or (python_full_version == '3.12.*' and sys_platform != 'darwin')", + "python_full_version == '3.11.*' and platform_machine == 'x86_64' and sys_platform == 'darwin'", + "(python_full_version == '3.11.*' and platform_machine != 'x86_64') or (python_full_version == '3.11.*' and sys_platform != 'darwin')", + "python_full_version == '3.10.*'", +] +dependencies = [ + { name = "cryptography", version = "48.0.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.10'" }, + { name = "pyasn1-modules", marker = "python_full_version >= '3.10'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/d4/f8/80d2493cbedece1c623dc3e3cb1883300871af0dcdae254409522985ac23/google_auth-2.52.0.tar.gz", hash = "sha256:01f30e1a9e3638698d89464f5e603ce29d18e1c0e63ec31ac570aba4e164aaf5", size = 335027, upload-time = "2026-05-07T19:45:24.033Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ee/fc/2cdc74252746f547f81ff3f02d4d4234a3f411b5de5b61af97e633a060b9/google_auth-2.52.0-py3-none-any.whl", hash = "sha256:aee92803ba0ff93a70a3b8a35c7b4797837751cd6380b63ff38372b98f3ed627", size = 245614, upload-time = "2026-05-07T19:45:21.914Z" }, +] + +[package.optional-dependencies] +requests = [ + { name = "requests", marker = "python_full_version >= '3.10'" }, +] + +[[package]] +name = "google-genai" +version = "1.47.0" +source = { registry = "https://pypi.org/simple" } +resolution-markers = [ + "python_full_version < '3.10'", +] +dependencies = [ + { name = "anyio", version = "4.12.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.10'" }, + { name = "google-auth", version = "2.50.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.10'" }, + { name = "httpx", marker = "python_full_version < '3.10'" }, + { name = "pydantic", marker = "python_full_version < '3.10'" }, + { name = "requests", marker = "python_full_version < '3.10'" }, + { name = "tenacity", version = "9.1.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.10'" }, + { name = "typing-extensions", marker = "python_full_version < '3.10'" }, + { name = "websockets", version = "15.0.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.10'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/9f/97/784fba9bc6c41263ff90cb9063eadfdd755dde79cfa5a8d0e397b067dcf9/google_genai-1.47.0.tar.gz", hash = "sha256:ecece00d0a04e6739ea76cc8dad82ec9593d9380aaabef078990e60574e5bf59", size = 241471, upload-time = "2025-10-29T22:01:02.88Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/89/ef/e080e8d67c270ea320956bb911a9359664fc46d3b87d1f029decd33e5c4c/google_genai-1.47.0-py3-none-any.whl", hash = "sha256:e3851237556cbdec96007d8028b4b1f2425cdc5c099a8dc36b72a57e42821b60", size = 241506, upload-time = "2025-10-29T22:01:00.982Z" }, +] + +[[package]] +name = "google-genai" +version = "2.0.1" +source = { registry = "https://pypi.org/simple" } +resolution-markers = [ + "python_full_version >= '3.14' and platform_machine == 'x86_64' and sys_platform == 'darwin'", + "(python_full_version >= '3.14' and platform_machine != 'x86_64') or (python_full_version >= '3.14' and sys_platform != 'darwin')", + "python_full_version == '3.13.*' and platform_machine == 'x86_64' and sys_platform == 'darwin'", + "(python_full_version == '3.13.*' and platform_machine != 'x86_64') or (python_full_version == '3.13.*' and sys_platform != 'darwin')", + "python_full_version == '3.12.*' and platform_machine == 'x86_64' and sys_platform == 'darwin'", + "(python_full_version == '3.12.*' and platform_machine != 'x86_64') or (python_full_version == '3.12.*' and sys_platform != 'darwin')", + "python_full_version == '3.11.*' and platform_machine == 'x86_64' and sys_platform == 'darwin'", + "(python_full_version == '3.11.*' and platform_machine != 'x86_64') or (python_full_version == '3.11.*' and sys_platform != 'darwin')", + "python_full_version == '3.10.*'", +] +dependencies = [ + { name = "anyio", version = "4.13.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.10'" }, + { name = "distro", marker = "python_full_version >= '3.10'" }, + { name = "google-auth", version = "2.52.0", source = { registry = "https://pypi.org/simple" }, extra = ["requests"], marker = "python_full_version >= '3.10'" }, + { name = "httpx", marker = "python_full_version >= '3.10'" }, + { name = "pydantic", marker = "python_full_version >= '3.10'" }, + { name = "requests", marker = "python_full_version >= '3.10'" }, + { name = "sniffio", marker = "python_full_version >= '3.10'" }, + { name = "tenacity", version = "9.1.4", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.10'" }, + { name = "typing-extensions", marker = "python_full_version >= '3.10'" }, + { name = "websockets", version = "16.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.10'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/44/ae/8504f6fa44aae887909c3fda1d49c6ffe129225b68f6f63b8904c49e7e90/google_genai-2.0.1.tar.gz", hash = "sha256:32cec7c07157c0e65e4dfc740e3288ff8e8bfc2d506cde49f884d79ed8377867", size = 537456, upload-time = "2026-05-09T01:37:12.693Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a1/20/7f427041c3660fabd4c396e80b27dd40b7d89b121ba384d69005a764910d/google_genai-2.0.1-py3-none-any.whl", hash = "sha256:5cb61ff5b8d33129bb7f5df0b5384ed2e71e5dd06ccc012cdbad28b070f6ce99", size = 791449, upload-time = "2026-05-09T01:37:10.841Z" }, +] + [[package]] name = "googleapis-common-protos" version = "1.74.0" @@ -2765,6 +2915,20 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/e6/ff/33f6a8823f06c6a1d1f530c1531e563b76c02091525e36255c08575ae775/grpcio-1.80.0-cp39-cp39-win_amd64.whl", hash = "sha256:05d55e1798756282cddd52d56c896b3e7d673e3a8798c2f1cd05ba249a3bb4de", size = 4892359, upload-time = "2026-03-30T08:49:06.902Z" }, ] +[[package]] +name = "grpcio-status" +version = "1.80.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "googleapis-common-protos" }, + { name = "grpcio" }, + { name = "protobuf" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/b1/ed/105f619bdd00cb47a49aa2feea6232ea2bbb04199d52a22cc6a7d603b5cb/grpcio_status-1.80.0.tar.gz", hash = "sha256:df73802a4c89a3ea88aa2aff971e886fccce162bc2e6511408b3d67a144381cd", size = 13901, upload-time = "2026-03-30T08:54:34.784Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/76/80/58cd2dfc19a07d022abe44bde7c365627f6c7cb6f692ada6c65ca437d09a/grpcio_status-1.80.0-py3-none-any.whl", hash = "sha256:4b56990363af50dbf2c2ebb80f1967185c07d87aa25aa2bea45ddb75fc181dbe", size = 14638, upload-time = "2026-03-30T08:54:01.569Z" }, +] + [[package]] name = "h11" version = "0.16.0" @@ -4201,6 +4365,21 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/b1/71/cadea475bda84fd4cc6fe46b347fd643be8b323fc8c413f4ccbbe2717db6/langchain_core-0.3.85-py3-none-any.whl", hash = "sha256:3ff7d8da9645cc8ff221d3db7fcccb9794ea4b84834b3714071459c254ed9061", size = 460237, upload-time = "2026-05-05T20:43:19.294Z" }, ] +[[package]] +name = "langchain-google-genai" +version = "2.1.12" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "filetype" }, + { name = "google-ai-generativelanguage" }, + { name = "langchain-core" }, + { name = "pydantic" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/09/38/8b3a71c729bd03e9eb0fd8bdb19e06a074c35bc2eaa61b1b9edfa863f38d/langchain_google_genai-2.1.12.tar.gz", hash = "sha256:4a98371e545eb97fcdf483086a4aebbb8eceeb9597ca5a9c4c35e92f4fbbd271", size = 77566, upload-time = "2025-09-17T01:27:11.747Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e1/8d/9dd9653e5414e73cae3480e5947bbbbd94ba7fa824efdf46e7ff2c0faef2/langchain_google_genai-2.1.12-py3-none-any.whl", hash = "sha256:4c07630419a8fbe7a2ec512c6dea68289663bfe7d5fae0ba431d2cd59a0d0880", size = 50746, upload-time = "2025-09-17T01:27:10.653Z" }, +] + [[package]] name = "langchain-openai" version = "0.3.35" @@ -7005,6 +7184,44 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/5b/5a/bc7b4a4ef808fa59a816c17b20c4bef6884daebbdf627ff2a161da67da19/propcache-0.4.1-py3-none-any.whl", hash = "sha256:af2a6052aeb6cf17d3e46ee169099044fd8224cbaf75c76a2ef596e8163e2237", size = 13305, upload-time = "2025-10-08T19:49:00.792Z" }, ] +[[package]] +name = "proto-plus" +version = "1.27.2" +source = { registry = "https://pypi.org/simple" } +resolution-markers = [ + "python_full_version < '3.10'", +] +dependencies = [ + { name = "protobuf", marker = "python_full_version < '3.10'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/81/0d/94dfe80193e79d55258345901acd2917523d56e8381bc4dee7fd38e3868a/proto_plus-1.27.2.tar.gz", hash = "sha256:b2adde53adadf75737c44d3dcb0104fde65250dfc83ad59168b4aa3e574b6a24", size = 57204, upload-time = "2026-03-26T22:18:57.174Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/84/f3/1fba73eeffafc998a25d59703b63f8be4fe8a5cb12eaff7386a0ba0f7125/proto_plus-1.27.2-py3-none-any.whl", hash = "sha256:6432f75893d3b9e70b9c412f1d2f03f65b11fb164b793d14ae2ca01821d22718", size = 50450, upload-time = "2026-03-26T22:13:42.927Z" }, +] + +[[package]] +name = "proto-plus" +version = "1.28.0" +source = { registry = "https://pypi.org/simple" } +resolution-markers = [ + "python_full_version >= '3.14' and platform_machine == 'x86_64' and sys_platform == 'darwin'", + "(python_full_version >= '3.14' and platform_machine != 'x86_64') or (python_full_version >= '3.14' and sys_platform != 'darwin')", + "python_full_version == '3.13.*' and platform_machine == 'x86_64' and sys_platform == 'darwin'", + "(python_full_version == '3.13.*' and platform_machine != 'x86_64') or (python_full_version == '3.13.*' and sys_platform != 'darwin')", + "python_full_version == '3.12.*' and platform_machine == 'x86_64' and sys_platform == 'darwin'", + "(python_full_version == '3.12.*' and platform_machine != 'x86_64') or (python_full_version == '3.12.*' and sys_platform != 'darwin')", + "python_full_version == '3.11.*' and platform_machine == 'x86_64' and sys_platform == 'darwin'", + "(python_full_version == '3.11.*' and platform_machine != 'x86_64') or (python_full_version == '3.11.*' and sys_platform != 'darwin')", + "python_full_version == '3.10.*'", +] +dependencies = [ + { name = "protobuf", marker = "python_full_version >= '3.10'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/c9/56/e647b0c675392d2da368da7b6f158f7368b18542fd6f7d7400a2f39de000/proto_plus-1.28.0.tar.gz", hash = "sha256:38e5696342835b08fc116f30a25665b29531cda9d5d5643e9b81fc312385abd9", size = 57221, upload-time = "2026-05-07T08:04:50.811Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7c/20/b122d4626976acb81132036d2ad1bb35a1a8775fceb837ec30964622516a/proto_plus-1.28.0-py3-none-any.whl", hash = "sha256:a630604310899e73c59ec302e5765c058d412b2f090b9c79c8822589f14955b8", size = 50410, upload-time = "2026-05-07T08:03:31.962Z" }, +] + [[package]] name = "protobuf" version = "6.33.6" @@ -7324,6 +7541,27 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/7b/03/f335d6c52b4a4761bcc83499789a1e2e16d9d201a58c327a9b5cc9a41bd9/pyarrow-22.0.0-cp314-cp314t-win_amd64.whl", hash = "sha256:0c34fe18094686194f204a3b1787a27456897d8a2d62caf84b61e8dfbc0252ae", size = 29185594, upload-time = "2025-10-24T10:09:53.111Z" }, ] +[[package]] +name = "pyasn1" +version = "0.6.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/5c/5f/6583902b6f79b399c9c40674ac384fd9cd77805f9e6205075f828ef11fb2/pyasn1-0.6.3.tar.gz", hash = "sha256:697a8ecd6d98891189184ca1fa05d1bb00e2f84b5977c481452050549c8a72cf", size = 148685, upload-time = "2026-03-17T01:06:53.382Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/5d/a0/7d793dce3fa811fe047d6ae2431c672364b462850c6235ae306c0efd025f/pyasn1-0.6.3-py3-none-any.whl", hash = "sha256:a80184d120f0864a52a073acc6fc642847d0be408e7c7252f31390c0f4eadcde", size = 83997, upload-time = "2026-03-17T01:06:52.036Z" }, +] + +[[package]] +name = "pyasn1-modules" +version = "0.4.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "pyasn1" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/e9/e6/78ebbb10a8c8e4b61a59249394a4a594c1a7af95593dc933a349c8d00964/pyasn1_modules-0.4.2.tar.gz", hash = "sha256:677091de870a80aae844b1ca6134f54652fa2c8c5a52aa396440ac3106e941e6", size = 307892, upload-time = "2025-03-28T02:41:22.17Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/47/8d/d529b5d697919ba8c11ad626e835d4039be708a35b0d22de83a269a6682c/pyasn1_modules-0.4.2-py3-none-any.whl", hash = "sha256:29253a9207ce32b64c3ac6600edc75368f98473906e8fd1043bd6b5b1de2c14a", size = 181259, upload-time = "2025-03-28T02:41:19.028Z" }, +] + [[package]] name = "pycares" version = "4.11.0" @@ -11059,7 +11297,7 @@ wheels = [ [[package]] name = "validmind" -version = "2.13.2" +version = "2.13.3" source = { editable = "." } dependencies = [ { name = "aiohttp", extra = ["speedups"] }, @@ -11098,9 +11336,12 @@ all = [ { name = "datasets", version = "2.21.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.14'" }, { name = "datasets", version = "4.8.5", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.14'" }, { name = "evaluate" }, + { name = "google-genai", version = "1.47.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.10'" }, + { name = "google-genai", version = "2.0.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.10'" }, { name = "langchain" }, { name = "langchain-community", version = "0.3.21", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version == '3.13.*'" }, { name = "langchain-community", version = "0.3.31", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version != '3.13.*'" }, + { name = "langchain-google-genai" }, { name = "langchain-openai" }, { name = "langdetect" }, { name = "nltk", version = "3.9.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.10'" }, @@ -11149,9 +11390,12 @@ llm = [ { name = "datasets", version = "2.21.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.14'" }, { name = "datasets", version = "4.8.5", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.14'" }, { name = "deepeval" }, + { name = "google-genai", version = "1.47.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.10'" }, + { name = "google-genai", version = "2.0.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.10'" }, { name = "langchain" }, { name = "langchain-community", version = "0.3.21", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version == '3.13.*'" }, { name = "langchain-community", version = "0.3.31", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version != '3.13.*'" }, + { name = "langchain-google-genai" }, { name = "langchain-openai" }, { name = "pyarrow", version = "15.0.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.13'" }, { name = "pyarrow", version = "20.0.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version == '3.13.*'" }, @@ -11243,12 +11487,16 @@ requires-dist = [ { name = "deepeval", marker = "extra == 'llm'", specifier = ">=3.7.0,<=3.7.5" }, { name = "evaluate", marker = "extra == 'all'" }, { name = "evaluate", marker = "extra == 'nlp'" }, + { name = "google-genai", marker = "extra == 'all'" }, + { name = "google-genai", marker = "extra == 'llm'" }, { name = "ipywidgets" }, { name = "kaleido", specifier = ">=1.2.0,<2.0.0" }, { name = "langchain", marker = "extra == 'all'", specifier = ">=0.2.0,<0.4.0" }, { name = "langchain", marker = "extra == 'llm'", specifier = ">=0.2.0,<0.4.0" }, { name = "langchain-community", marker = "extra == 'all'", specifier = ">=0.2.0,<0.4.0" }, { name = "langchain-community", marker = "extra == 'llm'", specifier = ">=0.2.0,<0.4.0" }, + { name = "langchain-google-genai", marker = "extra == 'all'" }, + { name = "langchain-google-genai", marker = "extra == 'llm'" }, { name = "langchain-openai", marker = "extra == 'all'", specifier = ">=0.1.8" }, { name = "langchain-openai", marker = "extra == 'llm'", specifier = ">=0.1.8" }, { name = "langdetect", marker = "extra == 'all'" }, @@ -11482,6 +11730,164 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/34/db/b10e48aa8fff7407e67470363eac595018441cf32d5e1001567a7aeba5d2/websocket_client-1.9.0-py3-none-any.whl", hash = "sha256:af248a825037ef591efbf6ed20cc5faa03d3b47b9e5a2230a529eeee1c1fc3ef", size = 82616, upload-time = "2025-10-07T21:16:34.951Z" }, ] +[[package]] +name = "websockets" +version = "15.0.1" +source = { registry = "https://pypi.org/simple" } +resolution-markers = [ + "python_full_version < '3.10'", +] +sdist = { url = "https://files.pythonhosted.org/packages/21/e6/26d09fab466b7ca9c7737474c52be4f76a40301b08362eb2dbc19dcc16c1/websockets-15.0.1.tar.gz", hash = "sha256:82544de02076bafba038ce055ee6412d68da13ab47f0c60cab827346de828dee", size = 177016, upload-time = "2025-03-05T20:03:41.606Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/1e/da/6462a9f510c0c49837bbc9345aca92d767a56c1fb2939e1579df1e1cdcf7/websockets-15.0.1-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:d63efaa0cd96cf0c5fe4d581521d9fa87744540d4bc999ae6e08595a1014b45b", size = 175423, upload-time = "2025-03-05T20:01:35.363Z" }, + { url = "https://files.pythonhosted.org/packages/1c/9f/9d11c1a4eb046a9e106483b9ff69bce7ac880443f00e5ce64261b47b07e7/websockets-15.0.1-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:ac60e3b188ec7574cb761b08d50fcedf9d77f1530352db4eef1707fe9dee7205", size = 173080, upload-time = "2025-03-05T20:01:37.304Z" }, + { url = "https://files.pythonhosted.org/packages/d5/4f/b462242432d93ea45f297b6179c7333dd0402b855a912a04e7fc61c0d71f/websockets-15.0.1-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:5756779642579d902eed757b21b0164cd6fe338506a8083eb58af5c372e39d9a", size = 173329, upload-time = "2025-03-05T20:01:39.668Z" }, + { url = "https://files.pythonhosted.org/packages/6e/0c/6afa1f4644d7ed50284ac59cc70ef8abd44ccf7d45850d989ea7310538d0/websockets-15.0.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0fdfe3e2a29e4db3659dbd5bbf04560cea53dd9610273917799f1cde46aa725e", size = 182312, upload-time = "2025-03-05T20:01:41.815Z" }, + { url = "https://files.pythonhosted.org/packages/dd/d4/ffc8bd1350b229ca7a4db2a3e1c482cf87cea1baccd0ef3e72bc720caeec/websockets-15.0.1-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:4c2529b320eb9e35af0fa3016c187dffb84a3ecc572bcee7c3ce302bfeba52bf", size = 181319, upload-time = "2025-03-05T20:01:43.967Z" }, + { url = "https://files.pythonhosted.org/packages/97/3a/5323a6bb94917af13bbb34009fac01e55c51dfde354f63692bf2533ffbc2/websockets-15.0.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ac1e5c9054fe23226fb11e05a6e630837f074174c4c2f0fe442996112a6de4fb", size = 181631, upload-time = "2025-03-05T20:01:46.104Z" }, + { url = "https://files.pythonhosted.org/packages/a6/cc/1aeb0f7cee59ef065724041bb7ed667b6ab1eeffe5141696cccec2687b66/websockets-15.0.1-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:5df592cd503496351d6dc14f7cdad49f268d8e618f80dce0cd5a36b93c3fc08d", size = 182016, upload-time = "2025-03-05T20:01:47.603Z" }, + { url = "https://files.pythonhosted.org/packages/79/f9/c86f8f7af208e4161a7f7e02774e9d0a81c632ae76db2ff22549e1718a51/websockets-15.0.1-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:0a34631031a8f05657e8e90903e656959234f3a04552259458aac0b0f9ae6fd9", size = 181426, upload-time = "2025-03-05T20:01:48.949Z" }, + { url = "https://files.pythonhosted.org/packages/c7/b9/828b0bc6753db905b91df6ae477c0b14a141090df64fb17f8a9d7e3516cf/websockets-15.0.1-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:3d00075aa65772e7ce9e990cab3ff1de702aa09be3940d1dc88d5abf1ab8a09c", size = 181360, upload-time = "2025-03-05T20:01:50.938Z" }, + { url = "https://files.pythonhosted.org/packages/89/fb/250f5533ec468ba6327055b7d98b9df056fb1ce623b8b6aaafb30b55d02e/websockets-15.0.1-cp310-cp310-win32.whl", hash = "sha256:1234d4ef35db82f5446dca8e35a7da7964d02c127b095e172e54397fb6a6c256", size = 176388, upload-time = "2025-03-05T20:01:52.213Z" }, + { url = "https://files.pythonhosted.org/packages/1c/46/aca7082012768bb98e5608f01658ff3ac8437e563eca41cf068bd5849a5e/websockets-15.0.1-cp310-cp310-win_amd64.whl", hash = "sha256:39c1fec2c11dc8d89bba6b2bf1556af381611a173ac2b511cf7231622058af41", size = 176830, upload-time = "2025-03-05T20:01:53.922Z" }, + { url = "https://files.pythonhosted.org/packages/9f/32/18fcd5919c293a398db67443acd33fde142f283853076049824fc58e6f75/websockets-15.0.1-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:823c248b690b2fd9303ba00c4f66cd5e2d8c3ba4aa968b2779be9532a4dad431", size = 175423, upload-time = "2025-03-05T20:01:56.276Z" }, + { url = "https://files.pythonhosted.org/packages/76/70/ba1ad96b07869275ef42e2ce21f07a5b0148936688c2baf7e4a1f60d5058/websockets-15.0.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:678999709e68425ae2593acf2e3ebcbcf2e69885a5ee78f9eb80e6e371f1bf57", size = 173082, upload-time = "2025-03-05T20:01:57.563Z" }, + { url = "https://files.pythonhosted.org/packages/86/f2/10b55821dd40eb696ce4704a87d57774696f9451108cff0d2824c97e0f97/websockets-15.0.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:d50fd1ee42388dcfb2b3676132c78116490976f1300da28eb629272d5d93e905", size = 173330, upload-time = "2025-03-05T20:01:59.063Z" }, + { url = "https://files.pythonhosted.org/packages/a5/90/1c37ae8b8a113d3daf1065222b6af61cc44102da95388ac0018fcb7d93d9/websockets-15.0.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d99e5546bf73dbad5bf3547174cd6cb8ba7273062a23808ffea025ecb1cf8562", size = 182878, upload-time = "2025-03-05T20:02:00.305Z" }, + { url = "https://files.pythonhosted.org/packages/8e/8d/96e8e288b2a41dffafb78e8904ea7367ee4f891dafc2ab8d87e2124cb3d3/websockets-15.0.1-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:66dd88c918e3287efc22409d426c8f729688d89a0c587c88971a0faa2c2f3792", size = 181883, upload-time = "2025-03-05T20:02:03.148Z" }, + { url = "https://files.pythonhosted.org/packages/93/1f/5d6dbf551766308f6f50f8baf8e9860be6182911e8106da7a7f73785f4c4/websockets-15.0.1-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:8dd8327c795b3e3f219760fa603dcae1dcc148172290a8ab15158cf85a953413", size = 182252, upload-time = "2025-03-05T20:02:05.29Z" }, + { url = "https://files.pythonhosted.org/packages/d4/78/2d4fed9123e6620cbf1706c0de8a1632e1a28e7774d94346d7de1bba2ca3/websockets-15.0.1-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:8fdc51055e6ff4adeb88d58a11042ec9a5eae317a0a53d12c062c8a8865909e8", size = 182521, upload-time = "2025-03-05T20:02:07.458Z" }, + { url = "https://files.pythonhosted.org/packages/e7/3b/66d4c1b444dd1a9823c4a81f50231b921bab54eee2f69e70319b4e21f1ca/websockets-15.0.1-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:693f0192126df6c2327cce3baa7c06f2a117575e32ab2308f7f8216c29d9e2e3", size = 181958, upload-time = "2025-03-05T20:02:09.842Z" }, + { url = "https://files.pythonhosted.org/packages/08/ff/e9eed2ee5fed6f76fdd6032ca5cd38c57ca9661430bb3d5fb2872dc8703c/websockets-15.0.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:54479983bd5fb469c38f2f5c7e3a24f9a4e70594cd68cd1fa6b9340dadaff7cf", size = 181918, upload-time = "2025-03-05T20:02:11.968Z" }, + { url = "https://files.pythonhosted.org/packages/d8/75/994634a49b7e12532be6a42103597b71098fd25900f7437d6055ed39930a/websockets-15.0.1-cp311-cp311-win32.whl", hash = "sha256:16b6c1b3e57799b9d38427dda63edcbe4926352c47cf88588c0be4ace18dac85", size = 176388, upload-time = "2025-03-05T20:02:13.32Z" }, + { url = "https://files.pythonhosted.org/packages/98/93/e36c73f78400a65f5e236cd376713c34182e6663f6889cd45a4a04d8f203/websockets-15.0.1-cp311-cp311-win_amd64.whl", hash = "sha256:27ccee0071a0e75d22cb35849b1db43f2ecd3e161041ac1ee9d2352ddf72f065", size = 176828, upload-time = "2025-03-05T20:02:14.585Z" }, + { url = "https://files.pythonhosted.org/packages/51/6b/4545a0d843594f5d0771e86463606a3988b5a09ca5123136f8a76580dd63/websockets-15.0.1-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:3e90baa811a5d73f3ca0bcbf32064d663ed81318ab225ee4f427ad4e26e5aff3", size = 175437, upload-time = "2025-03-05T20:02:16.706Z" }, + { url = "https://files.pythonhosted.org/packages/f4/71/809a0f5f6a06522af902e0f2ea2757f71ead94610010cf570ab5c98e99ed/websockets-15.0.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:592f1a9fe869c778694f0aa806ba0374e97648ab57936f092fd9d87f8bc03665", size = 173096, upload-time = "2025-03-05T20:02:18.832Z" }, + { url = "https://files.pythonhosted.org/packages/3d/69/1a681dd6f02180916f116894181eab8b2e25b31e484c5d0eae637ec01f7c/websockets-15.0.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:0701bc3cfcb9164d04a14b149fd74be7347a530ad3bbf15ab2c678a2cd3dd9a2", size = 173332, upload-time = "2025-03-05T20:02:20.187Z" }, + { url = "https://files.pythonhosted.org/packages/a6/02/0073b3952f5bce97eafbb35757f8d0d54812b6174ed8dd952aa08429bcc3/websockets-15.0.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:e8b56bdcdb4505c8078cb6c7157d9811a85790f2f2b3632c7d1462ab5783d215", size = 183152, upload-time = "2025-03-05T20:02:22.286Z" }, + { url = "https://files.pythonhosted.org/packages/74/45/c205c8480eafd114b428284840da0b1be9ffd0e4f87338dc95dc6ff961a1/websockets-15.0.1-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:0af68c55afbd5f07986df82831c7bff04846928ea8d1fd7f30052638788bc9b5", size = 182096, upload-time = "2025-03-05T20:02:24.368Z" }, + { url = "https://files.pythonhosted.org/packages/14/8f/aa61f528fba38578ec553c145857a181384c72b98156f858ca5c8e82d9d3/websockets-15.0.1-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:64dee438fed052b52e4f98f76c5790513235efaa1ef7f3f2192c392cd7c91b65", size = 182523, upload-time = "2025-03-05T20:02:25.669Z" }, + { url = "https://files.pythonhosted.org/packages/ec/6d/0267396610add5bc0d0d3e77f546d4cd287200804fe02323797de77dbce9/websockets-15.0.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:d5f6b181bb38171a8ad1d6aa58a67a6aa9d4b38d0f8c5f496b9e42561dfc62fe", size = 182790, upload-time = "2025-03-05T20:02:26.99Z" }, + { url = "https://files.pythonhosted.org/packages/02/05/c68c5adbf679cf610ae2f74a9b871ae84564462955d991178f95a1ddb7dd/websockets-15.0.1-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:5d54b09eba2bada6011aea5375542a157637b91029687eb4fdb2dab11059c1b4", size = 182165, upload-time = "2025-03-05T20:02:30.291Z" }, + { url = "https://files.pythonhosted.org/packages/29/93/bb672df7b2f5faac89761cb5fa34f5cec45a4026c383a4b5761c6cea5c16/websockets-15.0.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:3be571a8b5afed347da347bfcf27ba12b069d9d7f42cb8c7028b5e98bbb12597", size = 182160, upload-time = "2025-03-05T20:02:31.634Z" }, + { url = "https://files.pythonhosted.org/packages/ff/83/de1f7709376dc3ca9b7eeb4b9a07b4526b14876b6d372a4dc62312bebee0/websockets-15.0.1-cp312-cp312-win32.whl", hash = "sha256:c338ffa0520bdb12fbc527265235639fb76e7bc7faafbb93f6ba80d9c06578a9", size = 176395, upload-time = "2025-03-05T20:02:33.017Z" }, + { url = "https://files.pythonhosted.org/packages/7d/71/abf2ebc3bbfa40f391ce1428c7168fb20582d0ff57019b69ea20fa698043/websockets-15.0.1-cp312-cp312-win_amd64.whl", hash = "sha256:fcd5cf9e305d7b8338754470cf69cf81f420459dbae8a3b40cee57417f4614a7", size = 176841, upload-time = "2025-03-05T20:02:34.498Z" }, + { url = "https://files.pythonhosted.org/packages/cb/9f/51f0cf64471a9d2b4d0fc6c534f323b664e7095640c34562f5182e5a7195/websockets-15.0.1-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:ee443ef070bb3b6ed74514f5efaa37a252af57c90eb33b956d35c8e9c10a1931", size = 175440, upload-time = "2025-03-05T20:02:36.695Z" }, + { url = "https://files.pythonhosted.org/packages/8a/05/aa116ec9943c718905997412c5989f7ed671bc0188ee2ba89520e8765d7b/websockets-15.0.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:5a939de6b7b4e18ca683218320fc67ea886038265fd1ed30173f5ce3f8e85675", size = 173098, upload-time = "2025-03-05T20:02:37.985Z" }, + { url = "https://files.pythonhosted.org/packages/ff/0b/33cef55ff24f2d92924923c99926dcce78e7bd922d649467f0eda8368923/websockets-15.0.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:746ee8dba912cd6fc889a8147168991d50ed70447bf18bcda7039f7d2e3d9151", size = 173329, upload-time = "2025-03-05T20:02:39.298Z" }, + { url = "https://files.pythonhosted.org/packages/31/1d/063b25dcc01faa8fada1469bdf769de3768b7044eac9d41f734fd7b6ad6d/websockets-15.0.1-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:595b6c3969023ecf9041b2936ac3827e4623bfa3ccf007575f04c5a6aa318c22", size = 183111, upload-time = "2025-03-05T20:02:40.595Z" }, + { url = "https://files.pythonhosted.org/packages/93/53/9a87ee494a51bf63e4ec9241c1ccc4f7c2f45fff85d5bde2ff74fcb68b9e/websockets-15.0.1-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:3c714d2fc58b5ca3e285461a4cc0c9a66bd0e24c5da9911e30158286c9b5be7f", size = 182054, upload-time = "2025-03-05T20:02:41.926Z" }, + { url = "https://files.pythonhosted.org/packages/ff/b2/83a6ddf56cdcbad4e3d841fcc55d6ba7d19aeb89c50f24dd7e859ec0805f/websockets-15.0.1-cp313-cp313-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0f3c1e2ab208db911594ae5b4f79addeb3501604a165019dd221c0bdcabe4db8", size = 182496, upload-time = "2025-03-05T20:02:43.304Z" }, + { url = "https://files.pythonhosted.org/packages/98/41/e7038944ed0abf34c45aa4635ba28136f06052e08fc2168520bb8b25149f/websockets-15.0.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:229cf1d3ca6c1804400b0a9790dc66528e08a6a1feec0d5040e8b9eb14422375", size = 182829, upload-time = "2025-03-05T20:02:48.812Z" }, + { url = "https://files.pythonhosted.org/packages/e0/17/de15b6158680c7623c6ef0db361da965ab25d813ae54fcfeae2e5b9ef910/websockets-15.0.1-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:756c56e867a90fb00177d530dca4b097dd753cde348448a1012ed6c5131f8b7d", size = 182217, upload-time = "2025-03-05T20:02:50.14Z" }, + { url = "https://files.pythonhosted.org/packages/33/2b/1f168cb6041853eef0362fb9554c3824367c5560cbdaad89ac40f8c2edfc/websockets-15.0.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:558d023b3df0bffe50a04e710bc87742de35060580a293c2a984299ed83bc4e4", size = 182195, upload-time = "2025-03-05T20:02:51.561Z" }, + { url = "https://files.pythonhosted.org/packages/86/eb/20b6cdf273913d0ad05a6a14aed4b9a85591c18a987a3d47f20fa13dcc47/websockets-15.0.1-cp313-cp313-win32.whl", hash = "sha256:ba9e56e8ceeeedb2e080147ba85ffcd5cd0711b89576b83784d8605a7df455fa", size = 176393, upload-time = "2025-03-05T20:02:53.814Z" }, + { url = "https://files.pythonhosted.org/packages/1b/6c/c65773d6cab416a64d191d6ee8a8b1c68a09970ea6909d16965d26bfed1e/websockets-15.0.1-cp313-cp313-win_amd64.whl", hash = "sha256:e09473f095a819042ecb2ab9465aee615bd9c2028e4ef7d933600a8401c79561", size = 176837, upload-time = "2025-03-05T20:02:55.237Z" }, + { url = "https://files.pythonhosted.org/packages/36/db/3fff0bcbe339a6fa6a3b9e3fbc2bfb321ec2f4cd233692272c5a8d6cf801/websockets-15.0.1-cp39-cp39-macosx_10_9_universal2.whl", hash = "sha256:5f4c04ead5aed67c8a1a20491d54cdfba5884507a48dd798ecaf13c74c4489f5", size = 175424, upload-time = "2025-03-05T20:02:56.505Z" }, + { url = "https://files.pythonhosted.org/packages/46/e6/519054c2f477def4165b0ec060ad664ed174e140b0d1cbb9fafa4a54f6db/websockets-15.0.1-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:abdc0c6c8c648b4805c5eacd131910d2a7f6455dfd3becab248ef108e89ab16a", size = 173077, upload-time = "2025-03-05T20:02:58.37Z" }, + { url = "https://files.pythonhosted.org/packages/1a/21/c0712e382df64c93a0d16449ecbf87b647163485ca1cc3f6cbadb36d2b03/websockets-15.0.1-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:a625e06551975f4b7ea7102bc43895b90742746797e2e14b70ed61c43a90f09b", size = 173324, upload-time = "2025-03-05T20:02:59.773Z" }, + { url = "https://files.pythonhosted.org/packages/1c/cb/51ba82e59b3a664df54beed8ad95517c1b4dc1a913730e7a7db778f21291/websockets-15.0.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d591f8de75824cbb7acad4e05d2d710484f15f29d4a915092675ad3456f11770", size = 182094, upload-time = "2025-03-05T20:03:01.827Z" }, + { url = "https://files.pythonhosted.org/packages/fb/0f/bf3788c03fec679bcdaef787518dbe60d12fe5615a544a6d4cf82f045193/websockets-15.0.1-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:47819cea040f31d670cc8d324bb6435c6f133b8c7a19ec3d61634e62f8d8f9eb", size = 181094, upload-time = "2025-03-05T20:03:03.123Z" }, + { url = "https://files.pythonhosted.org/packages/5e/da/9fb8c21edbc719b66763a571afbaf206cb6d3736d28255a46fc2fe20f902/websockets-15.0.1-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ac017dd64572e5c3bd01939121e4d16cf30e5d7e110a119399cf3133b63ad054", size = 181397, upload-time = "2025-03-05T20:03:04.443Z" }, + { url = "https://files.pythonhosted.org/packages/2e/65/65f379525a2719e91d9d90c38fe8b8bc62bd3c702ac651b7278609b696c4/websockets-15.0.1-cp39-cp39-musllinux_1_2_aarch64.whl", hash = "sha256:4a9fac8e469d04ce6c25bb2610dc535235bd4aa14996b4e6dbebf5e007eba5ee", size = 181794, upload-time = "2025-03-05T20:03:06.708Z" }, + { url = "https://files.pythonhosted.org/packages/d9/26/31ac2d08f8e9304d81a1a7ed2851c0300f636019a57cbaa91342015c72cc/websockets-15.0.1-cp39-cp39-musllinux_1_2_i686.whl", hash = "sha256:363c6f671b761efcb30608d24925a382497c12c506b51661883c3e22337265ed", size = 181194, upload-time = "2025-03-05T20:03:08.844Z" }, + { url = "https://files.pythonhosted.org/packages/98/72/1090de20d6c91994cd4b357c3f75a4f25ee231b63e03adea89671cc12a3f/websockets-15.0.1-cp39-cp39-musllinux_1_2_x86_64.whl", hash = "sha256:2034693ad3097d5355bfdacfffcbd3ef5694f9718ab7f29c29689a9eae841880", size = 181164, upload-time = "2025-03-05T20:03:10.242Z" }, + { url = "https://files.pythonhosted.org/packages/2d/37/098f2e1c103ae8ed79b0e77f08d83b0ec0b241cf4b7f2f10edd0126472e1/websockets-15.0.1-cp39-cp39-win32.whl", hash = "sha256:3b1ac0d3e594bf121308112697cf4b32be538fb1444468fb0a6ae4feebc83411", size = 176381, upload-time = "2025-03-05T20:03:12.77Z" }, + { url = "https://files.pythonhosted.org/packages/75/8b/a32978a3ab42cebb2ebdd5b05df0696a09f4d436ce69def11893afa301f0/websockets-15.0.1-cp39-cp39-win_amd64.whl", hash = "sha256:b7643a03db5c95c799b89b31c036d5f27eeb4d259c798e878d6937d71832b1e4", size = 176841, upload-time = "2025-03-05T20:03:14.367Z" }, + { url = "https://files.pythonhosted.org/packages/02/9e/d40f779fa16f74d3468357197af8d6ad07e7c5a27ea1ca74ceb38986f77a/websockets-15.0.1-pp310-pypy310_pp73-macosx_10_15_x86_64.whl", hash = "sha256:0c9e74d766f2818bb95f84c25be4dea09841ac0f734d1966f415e4edfc4ef1c3", size = 173109, upload-time = "2025-03-05T20:03:17.769Z" }, + { url = "https://files.pythonhosted.org/packages/bc/cd/5b887b8585a593073fd92f7c23ecd3985cd2c3175025a91b0d69b0551372/websockets-15.0.1-pp310-pypy310_pp73-macosx_11_0_arm64.whl", hash = "sha256:1009ee0c7739c08a0cd59de430d6de452a55e42d6b522de7aa15e6f67db0b8e1", size = 173343, upload-time = "2025-03-05T20:03:19.094Z" }, + { url = "https://files.pythonhosted.org/packages/fe/ae/d34f7556890341e900a95acf4886833646306269f899d58ad62f588bf410/websockets-15.0.1-pp310-pypy310_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:76d1f20b1c7a2fa82367e04982e708723ba0e7b8d43aa643d3dcd404d74f1475", size = 174599, upload-time = "2025-03-05T20:03:21.1Z" }, + { url = "https://files.pythonhosted.org/packages/71/e6/5fd43993a87db364ec60fc1d608273a1a465c0caba69176dd160e197ce42/websockets-15.0.1-pp310-pypy310_pp73-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:f29d80eb9a9263b8d109135351caf568cc3f80b9928bccde535c235de55c22d9", size = 174207, upload-time = "2025-03-05T20:03:23.221Z" }, + { url = "https://files.pythonhosted.org/packages/2b/fb/c492d6daa5ec067c2988ac80c61359ace5c4c674c532985ac5a123436cec/websockets-15.0.1-pp310-pypy310_pp73-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:b359ed09954d7c18bbc1680f380c7301f92c60bf924171629c5db97febb12f04", size = 174155, upload-time = "2025-03-05T20:03:25.321Z" }, + { url = "https://files.pythonhosted.org/packages/68/a1/dcb68430b1d00b698ae7a7e0194433bce4f07ded185f0ee5fb21e2a2e91e/websockets-15.0.1-pp310-pypy310_pp73-win_amd64.whl", hash = "sha256:cad21560da69f4ce7658ca2cb83138fb4cf695a2ba3e475e0559e05991aa8122", size = 176884, upload-time = "2025-03-05T20:03:27.934Z" }, + { url = "https://files.pythonhosted.org/packages/b7/48/4b67623bac4d79beb3a6bb27b803ba75c1bdedc06bd827e465803690a4b2/websockets-15.0.1-pp39-pypy39_pp73-macosx_10_15_x86_64.whl", hash = "sha256:7f493881579c90fc262d9cdbaa05a6b54b3811c2f300766748db79f098db9940", size = 173106, upload-time = "2025-03-05T20:03:29.404Z" }, + { url = "https://files.pythonhosted.org/packages/ed/f0/adb07514a49fe5728192764e04295be78859e4a537ab8fcc518a3dbb3281/websockets-15.0.1-pp39-pypy39_pp73-macosx_11_0_arm64.whl", hash = "sha256:47b099e1f4fbc95b701b6e85768e1fcdaf1630f3cbe4765fa216596f12310e2e", size = 173339, upload-time = "2025-03-05T20:03:30.755Z" }, + { url = "https://files.pythonhosted.org/packages/87/28/bd23c6344b18fb43df40d0700f6d3fffcd7cef14a6995b4f976978b52e62/websockets-15.0.1-pp39-pypy39_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:67f2b6de947f8c757db2db9c71527933ad0019737ec374a8a6be9a956786aaf9", size = 174597, upload-time = "2025-03-05T20:03:32.247Z" }, + { url = "https://files.pythonhosted.org/packages/6d/79/ca288495863d0f23a60f546f0905ae8f3ed467ad87f8b6aceb65f4c013e4/websockets-15.0.1-pp39-pypy39_pp73-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:d08eb4c2b7d6c41da6ca0600c077e93f5adcfd979cd777d747e9ee624556da4b", size = 174205, upload-time = "2025-03-05T20:03:33.731Z" }, + { url = "https://files.pythonhosted.org/packages/04/e4/120ff3180b0872b1fe6637f6f995bcb009fb5c87d597c1fc21456f50c848/websockets-15.0.1-pp39-pypy39_pp73-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4b826973a4a2ae47ba357e4e82fa44a463b8f168e1ca775ac64521442b19e87f", size = 174150, upload-time = "2025-03-05T20:03:35.757Z" }, + { url = "https://files.pythonhosted.org/packages/cb/c3/30e2f9c539b8da8b1d76f64012f3b19253271a63413b2d3adb94b143407f/websockets-15.0.1-pp39-pypy39_pp73-win_amd64.whl", hash = "sha256:21c1fa28a6a7e3cbdc171c694398b6df4744613ce9b36b1a498e816787e28123", size = 176877, upload-time = "2025-03-05T20:03:37.199Z" }, + { url = "https://files.pythonhosted.org/packages/fa/a8/5b41e0da817d64113292ab1f8247140aac61cbf6cfd085d6a0fa77f4984f/websockets-15.0.1-py3-none-any.whl", hash = "sha256:f7a866fbc1e97b5c617ee4116daaa09b722101d4a3c170c787450ba409f9736f", size = 169743, upload-time = "2025-03-05T20:03:39.41Z" }, +] + +[[package]] +name = "websockets" +version = "16.0" +source = { registry = "https://pypi.org/simple" } +resolution-markers = [ + "python_full_version >= '3.14' and platform_machine == 'x86_64' and sys_platform == 'darwin'", + "(python_full_version >= '3.14' and platform_machine != 'x86_64') or (python_full_version >= '3.14' and sys_platform != 'darwin')", + "python_full_version == '3.13.*' and platform_machine == 'x86_64' and sys_platform == 'darwin'", + "(python_full_version == '3.13.*' and platform_machine != 'x86_64') or (python_full_version == '3.13.*' and sys_platform != 'darwin')", + "python_full_version == '3.12.*' and platform_machine == 'x86_64' and sys_platform == 'darwin'", + "(python_full_version == '3.12.*' and platform_machine != 'x86_64') or (python_full_version == '3.12.*' and sys_platform != 'darwin')", + "python_full_version == '3.11.*' and platform_machine == 'x86_64' and sys_platform == 'darwin'", + "(python_full_version == '3.11.*' and platform_machine != 'x86_64') or (python_full_version == '3.11.*' and sys_platform != 'darwin')", + "python_full_version == '3.10.*'", +] +sdist = { url = "https://files.pythonhosted.org/packages/04/24/4b2031d72e840ce4c1ccb255f693b15c334757fc50023e4db9537080b8c4/websockets-16.0.tar.gz", hash = "sha256:5f6261a5e56e8d5c42a4497b364ea24d94d9563e8fbd44e78ac40879c60179b5", size = 179346, upload-time = "2026-01-10T09:23:47.181Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/20/74/221f58decd852f4b59cc3354cccaf87e8ef695fede361d03dc9a7396573b/websockets-16.0-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:04cdd5d2d1dacbad0a7bf36ccbcd3ccd5a30ee188f2560b7a62a30d14107b31a", size = 177343, upload-time = "2026-01-10T09:22:21.28Z" }, + { url = "https://files.pythonhosted.org/packages/19/0f/22ef6107ee52ab7f0b710d55d36f5a5d3ef19e8a205541a6d7ffa7994e5a/websockets-16.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:8ff32bb86522a9e5e31439a58addbb0166f0204d64066fb955265c4e214160f0", size = 175021, upload-time = "2026-01-10T09:22:22.696Z" }, + { url = "https://files.pythonhosted.org/packages/10/40/904a4cb30d9b61c0e278899bf36342e9b0208eb3c470324a9ecbaac2a30f/websockets-16.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:583b7c42688636f930688d712885cf1531326ee05effd982028212ccc13e5957", size = 175320, upload-time = "2026-01-10T09:22:23.94Z" }, + { url = "https://files.pythonhosted.org/packages/9d/2f/4b3ca7e106bc608744b1cdae041e005e446124bebb037b18799c2d356864/websockets-16.0-cp310-cp310-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:7d837379b647c0c4c2355c2499723f82f1635fd2c26510e1f587d89bc2199e72", size = 183815, upload-time = "2026-01-10T09:22:25.469Z" }, + { url = "https://files.pythonhosted.org/packages/86/26/d40eaa2a46d4302becec8d15b0fc5e45bdde05191e7628405a19cf491ccd/websockets-16.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:df57afc692e517a85e65b72e165356ed1df12386ecb879ad5693be08fac65dde", size = 185054, upload-time = "2026-01-10T09:22:27.101Z" }, + { url = "https://files.pythonhosted.org/packages/b0/ba/6500a0efc94f7373ee8fefa8c271acdfd4dca8bd49a90d4be7ccabfc397e/websockets-16.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:2b9f1e0d69bc60a4a87349d50c09a037a2607918746f07de04df9e43252c77a3", size = 184565, upload-time = "2026-01-10T09:22:28.293Z" }, + { url = "https://files.pythonhosted.org/packages/04/b4/96bf2cee7c8d8102389374a2616200574f5f01128d1082f44102140344cc/websockets-16.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:335c23addf3d5e6a8633f9f8eda77efad001671e80b95c491dd0924587ece0b3", size = 183848, upload-time = "2026-01-10T09:22:30.394Z" }, + { url = "https://files.pythonhosted.org/packages/02/8e/81f40fb00fd125357814e8c3025738fc4ffc3da4b6b4a4472a82ba304b41/websockets-16.0-cp310-cp310-win32.whl", hash = "sha256:37b31c1623c6605e4c00d466c9d633f9b812ea430c11c8a278774a1fde1acfa9", size = 178249, upload-time = "2026-01-10T09:22:32.083Z" }, + { url = "https://files.pythonhosted.org/packages/b4/5f/7e40efe8df57db9b91c88a43690ac66f7b7aa73a11aa6a66b927e44f26fa/websockets-16.0-cp310-cp310-win_amd64.whl", hash = "sha256:8e1dab317b6e77424356e11e99a432b7cb2f3ec8c5ab4dabbcee6add48f72b35", size = 178685, upload-time = "2026-01-10T09:22:33.345Z" }, + { url = "https://files.pythonhosted.org/packages/f2/db/de907251b4ff46ae804ad0409809504153b3f30984daf82a1d84a9875830/websockets-16.0-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:31a52addea25187bde0797a97d6fc3d2f92b6f72a9370792d65a6e84615ac8a8", size = 177340, upload-time = "2026-01-10T09:22:34.539Z" }, + { url = "https://files.pythonhosted.org/packages/f3/fa/abe89019d8d8815c8781e90d697dec52523fb8ebe308bf11664e8de1877e/websockets-16.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:417b28978cdccab24f46400586d128366313e8a96312e4b9362a4af504f3bbad", size = 175022, upload-time = "2026-01-10T09:22:36.332Z" }, + { url = "https://files.pythonhosted.org/packages/58/5d/88ea17ed1ded2079358b40d31d48abe90a73c9e5819dbcde1606e991e2ad/websockets-16.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:af80d74d4edfa3cb9ed973a0a5ba2b2a549371f8a741e0800cb07becdd20f23d", size = 175319, upload-time = "2026-01-10T09:22:37.602Z" }, + { url = "https://files.pythonhosted.org/packages/d2/ae/0ee92b33087a33632f37a635e11e1d99d429d3d323329675a6022312aac2/websockets-16.0-cp311-cp311-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:08d7af67b64d29823fed316505a89b86705f2b7981c07848fb5e3ea3020c1abe", size = 184631, upload-time = "2026-01-10T09:22:38.789Z" }, + { url = "https://files.pythonhosted.org/packages/c8/c5/27178df583b6c5b31b29f526ba2da5e2f864ecc79c99dae630a85d68c304/websockets-16.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7be95cfb0a4dae143eaed2bcba8ac23f4892d8971311f1b06f3c6b78952ee70b", size = 185870, upload-time = "2026-01-10T09:22:39.893Z" }, + { url = "https://files.pythonhosted.org/packages/87/05/536652aa84ddc1c018dbb7e2c4cbcd0db884580bf8e95aece7593fde526f/websockets-16.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:d6297ce39ce5c2e6feb13c1a996a2ded3b6832155fcfc920265c76f24c7cceb5", size = 185361, upload-time = "2026-01-10T09:22:41.016Z" }, + { url = "https://files.pythonhosted.org/packages/6d/e2/d5332c90da12b1e01f06fb1b85c50cfc489783076547415bf9f0a659ec19/websockets-16.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:1c1b30e4f497b0b354057f3467f56244c603a79c0d1dafce1d16c283c25f6e64", size = 184615, upload-time = "2026-01-10T09:22:42.442Z" }, + { url = "https://files.pythonhosted.org/packages/77/fb/d3f9576691cae9253b51555f841bc6600bf0a983a461c79500ace5a5b364/websockets-16.0-cp311-cp311-win32.whl", hash = "sha256:5f451484aeb5cafee1ccf789b1b66f535409d038c56966d6101740c1614b86c6", size = 178246, upload-time = "2026-01-10T09:22:43.654Z" }, + { url = "https://files.pythonhosted.org/packages/54/67/eaff76b3dbaf18dcddabc3b8c1dba50b483761cccff67793897945b37408/websockets-16.0-cp311-cp311-win_amd64.whl", hash = "sha256:8d7f0659570eefb578dacde98e24fb60af35350193e4f56e11190787bee77dac", size = 178684, upload-time = "2026-01-10T09:22:44.941Z" }, + { url = "https://files.pythonhosted.org/packages/84/7b/bac442e6b96c9d25092695578dda82403c77936104b5682307bd4deb1ad4/websockets-16.0-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:71c989cbf3254fbd5e84d3bff31e4da39c43f884e64f2551d14bb3c186230f00", size = 177365, upload-time = "2026-01-10T09:22:46.787Z" }, + { url = "https://files.pythonhosted.org/packages/b0/fe/136ccece61bd690d9c1f715baaeefd953bb2360134de73519d5df19d29ca/websockets-16.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:8b6e209ffee39ff1b6d0fa7bfef6de950c60dfb91b8fcead17da4ee539121a79", size = 175038, upload-time = "2026-01-10T09:22:47.999Z" }, + { url = "https://files.pythonhosted.org/packages/40/1e/9771421ac2286eaab95b8575b0cb701ae3663abf8b5e1f64f1fd90d0a673/websockets-16.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:86890e837d61574c92a97496d590968b23c2ef0aeb8a9bc9421d174cd378ae39", size = 175328, upload-time = "2026-01-10T09:22:49.809Z" }, + { url = "https://files.pythonhosted.org/packages/18/29/71729b4671f21e1eaa5d6573031ab810ad2936c8175f03f97f3ff164c802/websockets-16.0-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:9b5aca38b67492ef518a8ab76851862488a478602229112c4b0d58d63a7a4d5c", size = 184915, upload-time = "2026-01-10T09:22:51.071Z" }, + { url = "https://files.pythonhosted.org/packages/97/bb/21c36b7dbbafc85d2d480cd65df02a1dc93bf76d97147605a8e27ff9409d/websockets-16.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e0334872c0a37b606418ac52f6ab9cfd17317ac26365f7f65e203e2d0d0d359f", size = 186152, upload-time = "2026-01-10T09:22:52.224Z" }, + { url = "https://files.pythonhosted.org/packages/4a/34/9bf8df0c0cf88fa7bfe36678dc7b02970c9a7d5e065a3099292db87b1be2/websockets-16.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:a0b31e0b424cc6b5a04b8838bbaec1688834b2383256688cf47eb97412531da1", size = 185583, upload-time = "2026-01-10T09:22:53.443Z" }, + { url = "https://files.pythonhosted.org/packages/47/88/4dd516068e1a3d6ab3c7c183288404cd424a9a02d585efbac226cb61ff2d/websockets-16.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:485c49116d0af10ac698623c513c1cc01c9446c058a4e61e3bf6c19dff7335a2", size = 184880, upload-time = "2026-01-10T09:22:55.033Z" }, + { url = "https://files.pythonhosted.org/packages/91/d6/7d4553ad4bf1c0421e1ebd4b18de5d9098383b5caa1d937b63df8d04b565/websockets-16.0-cp312-cp312-win32.whl", hash = "sha256:eaded469f5e5b7294e2bdca0ab06becb6756ea86894a47806456089298813c89", size = 178261, upload-time = "2026-01-10T09:22:56.251Z" }, + { url = "https://files.pythonhosted.org/packages/c3/f0/f3a17365441ed1c27f850a80b2bc680a0fa9505d733fe152fdf5e98c1c0b/websockets-16.0-cp312-cp312-win_amd64.whl", hash = "sha256:5569417dc80977fc8c2d43a86f78e0a5a22fee17565d78621b6bb264a115d4ea", size = 178693, upload-time = "2026-01-10T09:22:57.478Z" }, + { url = "https://files.pythonhosted.org/packages/cc/9c/baa8456050d1c1b08dd0ec7346026668cbc6f145ab4e314d707bb845bf0d/websockets-16.0-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:878b336ac47938b474c8f982ac2f7266a540adc3fa4ad74ae96fea9823a02cc9", size = 177364, upload-time = "2026-01-10T09:22:59.333Z" }, + { url = "https://files.pythonhosted.org/packages/7e/0c/8811fc53e9bcff68fe7de2bcbe75116a8d959ac699a3200f4847a8925210/websockets-16.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:52a0fec0e6c8d9a784c2c78276a48a2bdf099e4ccc2a4cad53b27718dbfd0230", size = 175039, upload-time = "2026-01-10T09:23:01.171Z" }, + { url = "https://files.pythonhosted.org/packages/aa/82/39a5f910cb99ec0b59e482971238c845af9220d3ab9fa76dd9162cda9d62/websockets-16.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:e6578ed5b6981005df1860a56e3617f14a6c307e6a71b4fff8c48fdc50f3ed2c", size = 175323, upload-time = "2026-01-10T09:23:02.341Z" }, + { url = "https://files.pythonhosted.org/packages/bd/28/0a25ee5342eb5d5f297d992a77e56892ecb65e7854c7898fb7d35e9b33bd/websockets-16.0-cp313-cp313-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:95724e638f0f9c350bb1c2b0a7ad0e83d9cc0c9259f3ea94e40d7b02a2179ae5", size = 184975, upload-time = "2026-01-10T09:23:03.756Z" }, + { url = "https://files.pythonhosted.org/packages/f9/66/27ea52741752f5107c2e41fda05e8395a682a1e11c4e592a809a90c6a506/websockets-16.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c0204dc62a89dc9d50d682412c10b3542d748260d743500a85c13cd1ee4bde82", size = 186203, upload-time = "2026-01-10T09:23:05.01Z" }, + { url = "https://files.pythonhosted.org/packages/37/e5/8e32857371406a757816a2b471939d51c463509be73fa538216ea52b792a/websockets-16.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:52ac480f44d32970d66763115edea932f1c5b1312de36df06d6b219f6741eed8", size = 185653, upload-time = "2026-01-10T09:23:06.301Z" }, + { url = "https://files.pythonhosted.org/packages/9b/67/f926bac29882894669368dc73f4da900fcdf47955d0a0185d60103df5737/websockets-16.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:6e5a82b677f8f6f59e8dfc34ec06ca6b5b48bc4fcda346acd093694cc2c24d8f", size = 184920, upload-time = "2026-01-10T09:23:07.492Z" }, + { url = "https://files.pythonhosted.org/packages/3c/a1/3d6ccdcd125b0a42a311bcd15a7f705d688f73b2a22d8cf1c0875d35d34a/websockets-16.0-cp313-cp313-win32.whl", hash = "sha256:abf050a199613f64c886ea10f38b47770a65154dc37181bfaff70c160f45315a", size = 178255, upload-time = "2026-01-10T09:23:09.245Z" }, + { url = "https://files.pythonhosted.org/packages/6b/ae/90366304d7c2ce80f9b826096a9e9048b4bb760e44d3b873bb272cba696b/websockets-16.0-cp313-cp313-win_amd64.whl", hash = "sha256:3425ac5cf448801335d6fdc7ae1eb22072055417a96cc6b31b3861f455fbc156", size = 178689, upload-time = "2026-01-10T09:23:10.483Z" }, + { url = "https://files.pythonhosted.org/packages/f3/1d/e88022630271f5bd349ed82417136281931e558d628dd52c4d8621b4a0b2/websockets-16.0-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:8cc451a50f2aee53042ac52d2d053d08bf89bcb31ae799cb4487587661c038a0", size = 177406, upload-time = "2026-01-10T09:23:12.178Z" }, + { url = "https://files.pythonhosted.org/packages/f2/78/e63be1bf0724eeb4616efb1ae1c9044f7c3953b7957799abb5915bffd38e/websockets-16.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:daa3b6ff70a9241cf6c7fc9e949d41232d9d7d26fd3522b1ad2b4d62487e9904", size = 175085, upload-time = "2026-01-10T09:23:13.511Z" }, + { url = "https://files.pythonhosted.org/packages/bb/f4/d3c9220d818ee955ae390cf319a7c7a467beceb24f05ee7aaaa2414345ba/websockets-16.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:fd3cb4adb94a2a6e2b7c0d8d05cb94e6f1c81a0cf9dc2694fb65c7e8d94c42e4", size = 175328, upload-time = "2026-01-10T09:23:14.727Z" }, + { url = "https://files.pythonhosted.org/packages/63/bc/d3e208028de777087e6fb2b122051a6ff7bbcca0d6df9d9c2bf1dd869ae9/websockets-16.0-cp314-cp314-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:781caf5e8eee67f663126490c2f96f40906594cb86b408a703630f95550a8c3e", size = 185044, upload-time = "2026-01-10T09:23:15.939Z" }, + { url = "https://files.pythonhosted.org/packages/ad/6e/9a0927ac24bd33a0a9af834d89e0abc7cfd8e13bed17a86407a66773cc0e/websockets-16.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:caab51a72c51973ca21fa8a18bd8165e1a0183f1ac7066a182ff27107b71e1a4", size = 186279, upload-time = "2026-01-10T09:23:17.148Z" }, + { url = "https://files.pythonhosted.org/packages/b9/ca/bf1c68440d7a868180e11be653c85959502efd3a709323230314fda6e0b3/websockets-16.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:19c4dc84098e523fd63711e563077d39e90ec6702aff4b5d9e344a60cb3c0cb1", size = 185711, upload-time = "2026-01-10T09:23:18.372Z" }, + { url = "https://files.pythonhosted.org/packages/c4/f8/fdc34643a989561f217bb477cbc47a3a07212cbda91c0e4389c43c296ebf/websockets-16.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:a5e18a238a2b2249c9a9235466b90e96ae4795672598a58772dd806edc7ac6d3", size = 184982, upload-time = "2026-01-10T09:23:19.652Z" }, + { url = "https://files.pythonhosted.org/packages/dd/d1/574fa27e233764dbac9c52730d63fcf2823b16f0856b3329fc6268d6ae4f/websockets-16.0-cp314-cp314-win32.whl", hash = "sha256:a069d734c4a043182729edd3e9f247c3b2a4035415a9172fd0f1b71658a320a8", size = 177915, upload-time = "2026-01-10T09:23:21.458Z" }, + { url = "https://files.pythonhosted.org/packages/8a/f1/ae6b937bf3126b5134ce1f482365fde31a357c784ac51852978768b5eff4/websockets-16.0-cp314-cp314-win_amd64.whl", hash = "sha256:c0ee0e63f23914732c6d7e0cce24915c48f3f1512ec1d079ed01fc629dab269d", size = 178381, upload-time = "2026-01-10T09:23:22.715Z" }, + { url = "https://files.pythonhosted.org/packages/06/9b/f791d1db48403e1f0a27577a6beb37afae94254a8c6f08be4a23e4930bc0/websockets-16.0-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:a35539cacc3febb22b8f4d4a99cc79b104226a756aa7400adc722e83b0d03244", size = 177737, upload-time = "2026-01-10T09:23:24.523Z" }, + { url = "https://files.pythonhosted.org/packages/bd/40/53ad02341fa33b3ce489023f635367a4ac98b73570102ad2cdd770dacc9a/websockets-16.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:b784ca5de850f4ce93ec85d3269d24d4c82f22b7212023c974c401d4980ebc5e", size = 175268, upload-time = "2026-01-10T09:23:25.781Z" }, + { url = "https://files.pythonhosted.org/packages/74/9b/6158d4e459b984f949dcbbb0c5d270154c7618e11c01029b9bbd1bb4c4f9/websockets-16.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:569d01a4e7fba956c5ae4fc988f0d4e187900f5497ce46339c996dbf24f17641", size = 175486, upload-time = "2026-01-10T09:23:27.033Z" }, + { url = "https://files.pythonhosted.org/packages/e5/2d/7583b30208b639c8090206f95073646c2c9ffd66f44df967981a64f849ad/websockets-16.0-cp314-cp314t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:50f23cdd8343b984957e4077839841146f67a3d31ab0d00e6b824e74c5b2f6e8", size = 185331, upload-time = "2026-01-10T09:23:28.259Z" }, + { url = "https://files.pythonhosted.org/packages/45/b0/cce3784eb519b7b5ad680d14b9673a31ab8dcb7aad8b64d81709d2430aa8/websockets-16.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:152284a83a00c59b759697b7f9e9cddf4e3c7861dd0d964b472b70f78f89e80e", size = 186501, upload-time = "2026-01-10T09:23:29.449Z" }, + { url = "https://files.pythonhosted.org/packages/19/60/b8ebe4c7e89fb5f6cdf080623c9d92789a53636950f7abacfc33fe2b3135/websockets-16.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:bc59589ab64b0022385f429b94697348a6a234e8ce22544e3681b2e9331b5944", size = 186062, upload-time = "2026-01-10T09:23:31.368Z" }, + { url = "https://files.pythonhosted.org/packages/88/a8/a080593f89b0138b6cba1b28f8df5673b5506f72879322288b031337c0b8/websockets-16.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:32da954ffa2814258030e5a57bc73a3635463238e797c7375dc8091327434206", size = 185356, upload-time = "2026-01-10T09:23:32.627Z" }, + { url = "https://files.pythonhosted.org/packages/c2/b6/b9afed2afadddaf5ebb2afa801abf4b0868f42f8539bfe4b071b5266c9fe/websockets-16.0-cp314-cp314t-win32.whl", hash = "sha256:5a4b4cc550cb665dd8a47f868c8d04c8230f857363ad3c9caf7a0c3bf8c61ca6", size = 178085, upload-time = "2026-01-10T09:23:33.816Z" }, + { url = "https://files.pythonhosted.org/packages/9f/3e/28135a24e384493fa804216b79a6a6759a38cc4ff59118787b9fb693df93/websockets-16.0-cp314-cp314t-win_amd64.whl", hash = "sha256:b14dc141ed6d2dde437cddb216004bcac6a1df0935d79656387bd41632ba0bbd", size = 178531, upload-time = "2026-01-10T09:23:35.016Z" }, + { url = "https://files.pythonhosted.org/packages/72/07/c98a68571dcf256e74f1f816b8cc5eae6eb2d3d5cfa44d37f801619d9166/websockets-16.0-pp311-pypy311_pp73-macosx_10_15_x86_64.whl", hash = "sha256:349f83cd6c9a415428ee1005cadb5c2c56f4389bc06a9af16103c3bc3dcc8b7d", size = 174947, upload-time = "2026-01-10T09:23:36.166Z" }, + { url = "https://files.pythonhosted.org/packages/7e/52/93e166a81e0305b33fe416338be92ae863563fe7bce446b0f687b9df5aea/websockets-16.0-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:4a1aba3340a8dca8db6eb5a7986157f52eb9e436b74813764241981ca4888f03", size = 175260, upload-time = "2026-01-10T09:23:37.409Z" }, + { url = "https://files.pythonhosted.org/packages/56/0c/2dbf513bafd24889d33de2ff0368190a0e69f37bcfa19009ef819fe4d507/websockets-16.0-pp311-pypy311_pp73-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:f4a32d1bd841d4bcbffdcb3d2ce50c09c3909fbead375ab28d0181af89fd04da", size = 176071, upload-time = "2026-01-10T09:23:39.158Z" }, + { url = "https://files.pythonhosted.org/packages/a5/8f/aea9c71cc92bf9b6cc0f7f70df8f0b420636b6c96ef4feee1e16f80f75dd/websockets-16.0-pp311-pypy311_pp73-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0298d07ee155e2e9fda5be8a9042200dd2e3bb0b8a38482156576f863a9d457c", size = 176968, upload-time = "2026-01-10T09:23:41.031Z" }, + { url = "https://files.pythonhosted.org/packages/9a/3f/f70e03f40ffc9a30d817eef7da1be72ee4956ba8d7255c399a01b135902a/websockets-16.0-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:a653aea902e0324b52f1613332ddf50b00c06fdaf7e92624fbf8c77c78fa5767", size = 178735, upload-time = "2026-01-10T09:23:42.259Z" }, + { url = "https://files.pythonhosted.org/packages/6f/28/258ebab549c2bf3e64d2b0217b973467394a9cea8c42f70418ca2c5d0d2e/websockets-16.0-py3-none-any.whl", hash = "sha256:1637db62fad1dc833276dded54215f2c7fa46912301a24bd94d45d46a011ceec", size = 171598, upload-time = "2026-01-10T09:23:45.395Z" }, +] + [[package]] name = "wheel" version = "0.47.0" diff --git a/validmind/__version__.py b/validmind/__version__.py index 200cbc3be..09c419b16 100644 --- a/validmind/__version__.py +++ b/validmind/__version__.py @@ -1 +1 @@ -__version__ = "2.13.2" +__version__ = "2.13.3" diff --git a/validmind/ai/utils.py b/validmind/ai/utils.py index a3241d5b2..e717ebe59 100644 --- a/validmind/ai/utils.py +++ b/validmind/ai/utils.py @@ -2,6 +2,7 @@ # Refer to the LICENSE file in the root of this repository for details. # SPDX-License-Identifier: AGPL-3.0 AND ValidMind Commercial +import importlib import os from openai import AzureOpenAI, OpenAI @@ -16,7 +17,10 @@ __model = None __judge_llm = None __judge_embeddings = None -EMBEDDINGS_MODEL = "text-embedding-3-small" +OPENAI_MODEL = "gpt-4.1" +OPENAI_EMBEDDINGS_MODEL = "text-embedding-3-small" +GEMINI_MODEL = "gemini-2.5-pro" +GEMINI_EMBEDDINGS_MODEL = "models/text-embedding-004" # can be None, True or False (ternary to represent initial state, ack and failed ack) __ack = None @@ -44,6 +48,23 @@ def get_description(self): return md_to_html(description[0], mathml=True), description[1] +def _get_google_api_key(): + return os.getenv("GOOGLE_API_KEY") or os.getenv("GEMINI_API_KEY") + + +def _get_configured_provider(): + if os.getenv("OPENAI_API_KEY"): + return "openai" + + if os.getenv("AZURE_OPENAI_KEY"): + return "azure" + + if _get_google_api_key(): + return "gemini" + + return None + + def get_client_and_model(): """Get model and client to use for generating interpretations. @@ -52,16 +73,18 @@ def get_client_and_model(): """ global __client, __model - if __client and __model: + if __model is not None: return __client, __model - if "OPENAI_API_KEY" in os.environ: + provider = _get_configured_provider() + + if provider == "openai": __client = OpenAI(api_key=os.getenv("OPENAI_API_KEY")) - __model = os.getenv("VM_OPENAI_MODEL", "gpt-4o") + __model = os.getenv("OPENAI_MODEL", OPENAI_MODEL) logger.debug(f"Using OpenAI {__model} for generating descriptions") - elif "AZURE_OPENAI_KEY" in os.environ: + elif provider == "azure": if "AZURE_OPENAI_ENDPOINT" not in os.environ: raise ValueError( "AZURE_OPENAI_ENDPOINT must be set to run LLM tests with Azure" @@ -81,71 +104,189 @@ def get_client_and_model(): logger.debug(f"Using Azure OpenAI {__model} for generating descriptions") + elif provider == "gemini": + __client = None + __model = os.getenv("GEMINI_MODEL", GEMINI_MODEL) + + logger.debug(f"Using Gemini {__model} for generating descriptions") + else: raise ValueError( - "OPENAI_API_KEY, AZURE_OPENAI_KEY must be setup to use LLM features" + "OPENAI_API_KEY, AZURE_OPENAI_KEY, GOOGLE_API_KEY, or GEMINI_API_KEY " + "must be setup to use LLM features" ) return __client, __model -def get_judge_config(judge_llm=None, judge_embeddings=None): +def _import_judge_dependencies(): try: from langchain_core.embeddings import Embeddings from langchain_core.language_models.chat_models import BaseChatModel - from langchain_openai import ChatOpenAI, OpenAIEmbeddings from validmind.models.function import FunctionModel except ImportError: raise ImportError("Please run `pip install validmind[llm]` to use LLM tests") - if judge_llm is not None or judge_embeddings is not None: - if isinstance(judge_llm, FunctionModel) and judge_llm is not None: - if isinstance(judge_llm.model, BaseChatModel): - judge_llm = judge_llm.model - else: - raise ValueError( - "The ValidMind Functional model provided does not have have a langchain compatible LLM as a model attribute." - "To use default ValidMind LLM, do not set judge_llm/judge_embedding parameter, " - "ensure that you are connected to the ValidMind API and confirm ValidMind AI is enabled for your account." - ) - if isinstance(judge_embeddings, FunctionModel) and judge_embeddings is not None: - if isinstance(judge_embeddings.model, Embeddings): - judge_embeddings = judge_embeddings.model - else: - raise ValueError( - "The ValidMind Functional model provided does not have have a langchain compatible embeddings model as a model attribute." - "To use default ValidMind LLM, do not set judge_embedding parameter, " - "ensure that you are connected to the ValidMind API and confirm ValidMind AI is enabled for your account." - ) - - if (isinstance(judge_llm, BaseChatModel) or judge_llm is None) and ( - isinstance(judge_embeddings, Embeddings) or judge_embeddings is None - ): - return judge_llm, judge_embeddings - else: - raise ValueError( - "Provided Judge LLM/Embeddings are not Langchain compatible. Ensure the judge LLM/embedding provided are an instance of " - "Langchain BaseChatModel and LangchainEmbeddings. To use default ValidMind LLM, do not set judge_llm/judge_embedding parameter, " - "ensure that you are connected to the ValidMind API and confirm ValidMind AI is enabled for your account." - ) + return Embeddings, BaseChatModel, FunctionModel + + +def _unwrap_functional_judge_model( + judge_model, + expected_type, + model_kind, + FunctionModel, +): + if not isinstance(judge_model, FunctionModel): + return judge_model + + if isinstance(judge_model.model, expected_type): + return judge_model.model + + raise ValueError( + "The ValidMind Functional model provided does not have have a langchain " + f"compatible {model_kind} model as a model attribute." + "To use default ValidMind LLM, do not set judge_llm/judge_embedding parameter, " + "ensure that you are connected to the ValidMind API and confirm ValidMind AI " + "is enabled for your account." + ) + + +def _normalize_judge_overrides( + judge_llm, + judge_embeddings, + Embeddings, + BaseChatModel, + FunctionModel, +): + if judge_llm is None and judge_embeddings is None: + return None + + judge_llm = _unwrap_functional_judge_model( + judge_llm, + BaseChatModel, + "LLM", + FunctionModel, + ) + judge_embeddings = _unwrap_functional_judge_model( + judge_embeddings, + Embeddings, + "embeddings", + FunctionModel, + ) + + if (isinstance(judge_llm, BaseChatModel) or judge_llm is None) and ( + isinstance(judge_embeddings, Embeddings) or judge_embeddings is None + ): + return judge_llm, judge_embeddings + + raise ValueError( + "Provided Judge LLM/Embeddings are not Langchain compatible. Ensure the judge " + "LLM/embedding provided are an instance of Langchain BaseChatModel and " + "LangchainEmbeddings. To use default ValidMind LLM, do not set " + "judge_llm/judge_embedding parameter, ensure that you are connected to the " + "ValidMind API and confirm ValidMind AI is enabled for your account." + ) + + +def _build_gemini_judge_config(model): + try: + langchain_google_genai = importlib.import_module("langchain_google_genai") + except ImportError: + raise ImportError( + "Please run `pip install validmind[llm]` to use Gemini LLM tests" + ) + + ChatGoogleGenerativeAI = getattr(langchain_google_genai, "ChatGoogleGenerativeAI") + GoogleGenerativeAIEmbeddings = getattr( + langchain_google_genai, "GoogleGenerativeAIEmbeddings" + ) + google_api_key = _get_google_api_key() + + return ( + ChatGoogleGenerativeAI( + model=model, + api_key=google_api_key, + ), + GoogleGenerativeAIEmbeddings( + model=os.getenv("GEMINI_EMBEDDINGS_MODEL", GEMINI_EMBEDDINGS_MODEL), + google_api_key=google_api_key, + ), + ) + + +def _build_openai_judge_config(client, model): + try: + from langchain_openai import ChatOpenAI, OpenAIEmbeddings + except ImportError: + raise ImportError("Please run `pip install validmind[llm]` to use LLM tests") + + if client is not None and getattr(client, "base_url", None) is not None: + os.environ["OPENAI_API_BASE"] = str(client.base_url) + + return ( + ChatOpenAI(api_key=client.api_key, model=model), + OpenAIEmbeddings(api_key=client.api_key, model=OPENAI_EMBEDDINGS_MODEL), + ) + + +def get_judge_config(judge_llm=None, judge_embeddings=None): + Embeddings, BaseChatModel, FunctionModel = _import_judge_dependencies() + + normalized_overrides = _normalize_judge_overrides( + judge_llm, + judge_embeddings, + Embeddings, + BaseChatModel, + FunctionModel, + ) + if normalized_overrides is not None: + return normalized_overrides # grab default values if not passed at run time global __judge_llm, __judge_embeddings if __judge_llm and __judge_embeddings: return __judge_llm, __judge_embeddings + provider = _get_configured_provider() client, model = get_client_and_model() - os.environ["OPENAI_API_BASE"] = str(client.base_url) - __judge_llm = ChatOpenAI(api_key=client.api_key, model=model) - __judge_embeddings = OpenAIEmbeddings( - api_key=client.api_key, model=EMBEDDINGS_MODEL - ) + if provider == "gemini": + __judge_llm, __judge_embeddings = _build_gemini_judge_config(model) + else: + __judge_llm, __judge_embeddings = _build_openai_judge_config(client, model) return __judge_llm, __judge_embeddings +def get_deepeval_model(): + """Get the model object expected by DeepEval scorers. + + OpenAI/Azure scorers currently pass a model string. Gemini support requires a + native DeepEval model object so the provider can be configured correctly. + """ + + provider = _get_configured_provider() + _, model = get_client_and_model() + + if provider == "gemini": + try: + deepeval_models = importlib.import_module("deepeval.models") + except ImportError: + raise ImportError( + "Please run `pip install validmind[llm]` to use Gemini DeepEval scorers" + ) + + GeminiModel = getattr(deepeval_models, "GeminiModel") + return GeminiModel( + model=model, + api_key=_get_google_api_key(), + temperature=0, + ) + + return model + + def set_judge_config(judge_llm, judge_embeddings): global __judge_llm, __judge_embeddings try: @@ -181,19 +322,16 @@ def is_configured(): return True try: - client, model = get_client_and_model() - # send an empty message with max_tokens=1 to "ping" the API - response = client.chat.completions.create( - model=model, - messages=[{"role": "user", "content": ""}], - max_tokens=1, + judge_llm, _ = get_judge_config() + response = judge_llm.invoke( + [("user", "ping")], ) logger.debug( - f"Received response from OpenAI: {response.choices[0].message.content}" + f"Received response from judge LLM: {getattr(response, 'content', response)}" ) __ack = True except Exception as e: - logger.debug(f"Failed to connect to OpenAI: {e}") + logger.debug(f"Failed to connect to judge LLM: {e}") __ack = False return __ack diff --git a/validmind/scorers/llm/deepeval/AnswerRelevancy.py b/validmind/scorers/llm/deepeval/AnswerRelevancy.py index 22627a3ce..4a6de7e64 100644 --- a/validmind/scorers/llm/deepeval/AnswerRelevancy.py +++ b/validmind/scorers/llm/deepeval/AnswerRelevancy.py @@ -5,7 +5,7 @@ from typing import Any, Dict, List from validmind import tags, tasks -from validmind.ai.utils import get_client_and_model +from validmind.ai.utils import get_deepeval_model from validmind.errors import MissingDependencyError from validmind.tests.decorator import scorer from validmind.vm_models.dataset import VMDataset @@ -69,7 +69,7 @@ def AnswerRelevancy( f"Actual output column '{actual_output_column}' not found in dataset. Available columns: {dataset.df.columns.tolist()}" ) - _, model = get_client_and_model() + model = get_deepeval_model() metric = AnswerRelevancyMetric( threshold=threshold, model=model, include_reason=True, verbose_mode=False diff --git a/validmind/scorers/llm/deepeval/ArgumentCorrectness.py b/validmind/scorers/llm/deepeval/ArgumentCorrectness.py index 424c15ef8..c453bee91 100644 --- a/validmind/scorers/llm/deepeval/ArgumentCorrectness.py +++ b/validmind/scorers/llm/deepeval/ArgumentCorrectness.py @@ -5,7 +5,7 @@ from typing import Any, Dict, List from validmind import tags, tasks -from validmind.ai.utils import get_client_and_model +from validmind.ai.utils import get_deepeval_model from validmind.errors import MissingDependencyError from validmind.tests.decorator import scorer from validmind.vm_models.dataset import VMDataset @@ -86,7 +86,7 @@ def ArgumentCorrectness( f"Available columns: {dataset._df.columns.tolist()}" ) - _, llm_model = get_client_and_model() + llm_model = get_deepeval_model() results: List[Dict[str, Any]] = [] for _, row in dataset._df.iterrows(): diff --git a/validmind/scorers/llm/deepeval/Bias.py b/validmind/scorers/llm/deepeval/Bias.py index 5f43d1a7e..533d4d862 100644 --- a/validmind/scorers/llm/deepeval/Bias.py +++ b/validmind/scorers/llm/deepeval/Bias.py @@ -5,7 +5,7 @@ from typing import Any, Dict, List from validmind import tags, tasks -from validmind.ai.utils import get_client_and_model +from validmind.ai.utils import get_deepeval_model from validmind.errors import MissingDependencyError from validmind.tests.decorator import scorer from validmind.vm_models.dataset import VMDataset @@ -71,7 +71,7 @@ def Bias( f"Actual output column '{actual_output_column}' not found in dataset. Available columns: {dataset.df.columns.tolist()}" ) - _, model = get_client_and_model() + model = get_deepeval_model() metric = BiasMetric( threshold=threshold, diff --git a/validmind/scorers/llm/deepeval/ContextualPrecision.py b/validmind/scorers/llm/deepeval/ContextualPrecision.py index 2684fb55c..304816a4c 100644 --- a/validmind/scorers/llm/deepeval/ContextualPrecision.py +++ b/validmind/scorers/llm/deepeval/ContextualPrecision.py @@ -5,7 +5,7 @@ from typing import Any, Dict, List from validmind import tags, tasks -from validmind.ai.utils import get_client_and_model +from validmind.ai.utils import get_deepeval_model from validmind.errors import MissingDependencyError from validmind.tests.decorator import scorer from validmind.vm_models.dataset import VMDataset @@ -69,7 +69,7 @@ def ContextualPrecision( f"Available columns: {dataset.df.columns.tolist()}" ) - _, model = get_client_and_model() + model = get_deepeval_model() metric = ContextualPrecisionMetric( threshold=threshold, diff --git a/validmind/scorers/llm/deepeval/ContextualRecall.py b/validmind/scorers/llm/deepeval/ContextualRecall.py index 663cf75bb..ef88a3da5 100644 --- a/validmind/scorers/llm/deepeval/ContextualRecall.py +++ b/validmind/scorers/llm/deepeval/ContextualRecall.py @@ -5,7 +5,7 @@ from typing import Any, Dict, List from validmind import tags, tasks -from validmind.ai.utils import get_client_and_model +from validmind.ai.utils import get_deepeval_model from validmind.errors import MissingDependencyError from validmind.tests.decorator import scorer from validmind.vm_models.dataset import VMDataset @@ -69,7 +69,7 @@ def ContextualRecall( f"Available columns: {dataset.df.columns.tolist()}" ) - _, model = get_client_and_model() + model = get_deepeval_model() metric = ContextualRecallMetric( threshold=threshold, diff --git a/validmind/scorers/llm/deepeval/ContextualRelevancy.py b/validmind/scorers/llm/deepeval/ContextualRelevancy.py index ed8ae0ed1..d1c9e91f8 100644 --- a/validmind/scorers/llm/deepeval/ContextualRelevancy.py +++ b/validmind/scorers/llm/deepeval/ContextualRelevancy.py @@ -5,7 +5,7 @@ from typing import Any, Dict, List from validmind import tags, tasks -from validmind.ai.utils import get_client_and_model +from validmind.ai.utils import get_deepeval_model from validmind.errors import MissingDependencyError from validmind.tests.decorator import scorer from validmind.vm_models.dataset import VMDataset @@ -69,7 +69,7 @@ def ContextualRelevancy( f"Available columns: {dataset.df.columns.tolist()}" ) - _, model = get_client_and_model() + model = get_deepeval_model() metric = ContextualRelevancyMetric( threshold=threshold, diff --git a/validmind/scorers/llm/deepeval/Faithfulness.py b/validmind/scorers/llm/deepeval/Faithfulness.py index b1705e066..555455d4c 100644 --- a/validmind/scorers/llm/deepeval/Faithfulness.py +++ b/validmind/scorers/llm/deepeval/Faithfulness.py @@ -5,7 +5,7 @@ from typing import Any, Dict, List from validmind import tags, tasks -from validmind.ai.utils import get_client_and_model +from validmind.ai.utils import get_deepeval_model from validmind.errors import MissingDependencyError from validmind.tests.decorator import scorer from validmind.vm_models.dataset import VMDataset @@ -69,7 +69,7 @@ def Faithfulness( f"Available columns: {dataset.df.columns.tolist()}" ) - _, model = get_client_and_model() + model = get_deepeval_model() metric = FaithfulnessMetric( threshold=threshold, diff --git a/validmind/scorers/llm/deepeval/GEval.py b/validmind/scorers/llm/deepeval/GEval.py index 5649d372e..44a02dabe 100644 --- a/validmind/scorers/llm/deepeval/GEval.py +++ b/validmind/scorers/llm/deepeval/GEval.py @@ -5,7 +5,7 @@ from typing import Any, Dict, List from validmind import tags, tasks -from validmind.ai.utils import get_client_and_model +from validmind.ai.utils import get_deepeval_model from validmind.errors import MissingDependencyError from validmind.tests.decorator import scorer from validmind.vm_models.dataset import VMDataset @@ -91,7 +91,7 @@ def GEval( ... threshold=0.7 ... ) """ - _, model = get_client_and_model() + model = get_deepeval_model() results: List[Dict[str, Any]] = [] evaluation_params_dict = { diff --git a/validmind/scorers/llm/deepeval/Hallucination.py b/validmind/scorers/llm/deepeval/Hallucination.py index 2e0936557..a7c9a824e 100644 --- a/validmind/scorers/llm/deepeval/Hallucination.py +++ b/validmind/scorers/llm/deepeval/Hallucination.py @@ -5,7 +5,7 @@ from typing import Any, Dict, List from validmind import tags, tasks -from validmind.ai.utils import get_client_and_model +from validmind.ai.utils import get_deepeval_model from validmind.errors import MissingDependencyError from validmind.tests.decorator import scorer from validmind.vm_models.dataset import VMDataset @@ -69,7 +69,7 @@ def Hallucination( f"Available columns: {dataset.df.columns.tolist()}" ) - _, model = get_client_and_model() + model = get_deepeval_model() metric = HallucinationMetric( threshold=threshold, diff --git a/validmind/scorers/llm/deepeval/PlanAdherence.py b/validmind/scorers/llm/deepeval/PlanAdherence.py index 32896e9c1..e729da171 100644 --- a/validmind/scorers/llm/deepeval/PlanAdherence.py +++ b/validmind/scorers/llm/deepeval/PlanAdherence.py @@ -6,7 +6,7 @@ from typing import Any, Dict, List, Optional from validmind import tags, tasks -from validmind.ai.utils import get_client_and_model +from validmind.ai.utils import get_deepeval_model from validmind.errors import MissingDependencyError from validmind.tests.decorator import scorer from validmind.vm_models import VMModel @@ -79,7 +79,7 @@ def PlanAdherence( extra="llm", ) from None - _, llm_model = get_client_and_model() + llm_model = get_deepeval_model() results: List[Dict[str, Any]] = [] # Run one golden at a time so the metric runs after each predict_fn and diff --git a/validmind/scorers/llm/deepeval/PlanQuality.py b/validmind/scorers/llm/deepeval/PlanQuality.py index 08a0d8eb3..8890b9f7b 100644 --- a/validmind/scorers/llm/deepeval/PlanQuality.py +++ b/validmind/scorers/llm/deepeval/PlanQuality.py @@ -6,7 +6,7 @@ from typing import Any, Dict, List, Optional from validmind import tags, tasks -from validmind.ai.utils import get_client_and_model +from validmind.ai.utils import get_deepeval_model from validmind.errors import MissingDependencyError from validmind.tests.decorator import scorer from validmind.vm_models import VMModel @@ -88,9 +88,8 @@ def PlanQuality( extra="llm", ) from None - get_client_and_model() results: List[Dict[str, Any]] = [] - _, llm_model = get_client_and_model() + llm_model = get_deepeval_model() # Run one golden at a time so the metric runs after each predict_fn and # metric.score is set when the iterator exits (evals_iterator runs the diff --git a/validmind/scorers/llm/deepeval/Summarization.py b/validmind/scorers/llm/deepeval/Summarization.py index 3d384d10c..56b9ba918 100644 --- a/validmind/scorers/llm/deepeval/Summarization.py +++ b/validmind/scorers/llm/deepeval/Summarization.py @@ -5,7 +5,7 @@ from typing import Any, Dict, List, Optional from validmind import tags, tasks -from validmind.ai.utils import get_client_and_model +from validmind.ai.utils import get_deepeval_model from validmind.errors import MissingDependencyError from validmind.tests.decorator import scorer from validmind.vm_models.dataset import VMDataset @@ -73,7 +73,7 @@ def Summarization( f"Available columns: {dataset.df.columns.tolist()}" ) - _, model = get_client_and_model() + model = get_deepeval_model() # Build metric with optional parameters metric_kwargs: Dict[str, Any] = dict( diff --git a/validmind/scorers/llm/deepeval/TaskCompletion.py b/validmind/scorers/llm/deepeval/TaskCompletion.py index 1fb70d7ee..7f34c65ae 100644 --- a/validmind/scorers/llm/deepeval/TaskCompletion.py +++ b/validmind/scorers/llm/deepeval/TaskCompletion.py @@ -6,7 +6,7 @@ from typing import Any, Dict, List, Optional from validmind import tags, tasks -from validmind.ai.utils import get_client_and_model +from validmind.ai.utils import get_deepeval_model from validmind.errors import MissingDependencyError from validmind.tests.decorator import scorer from validmind.vm_models import VMModel @@ -76,7 +76,7 @@ def TaskCompletion( extra="llm", ) from None - _, llm_model = get_client_and_model() + llm_model = get_deepeval_model() results: List[Dict[str, Any]] = [] for _, row in dataset._df.iterrows(): diff --git a/validmind/scorers/llm/deepeval/ToolCorrectness.py b/validmind/scorers/llm/deepeval/ToolCorrectness.py index c77c9e212..5f7a085e8 100644 --- a/validmind/scorers/llm/deepeval/ToolCorrectness.py +++ b/validmind/scorers/llm/deepeval/ToolCorrectness.py @@ -5,7 +5,7 @@ from typing import Any, Dict, List from validmind import tags, tasks -from validmind.ai.utils import get_client_and_model +from validmind.ai.utils import get_deepeval_model from validmind.errors import MissingDependencyError from validmind.tests.decorator import scorer from validmind.vm_models.dataset import VMDataset @@ -88,7 +88,7 @@ def ToolCorrectness( f"Available columns: {dataset._df.columns.tolist()}" ) - _, llm_model = get_client_and_model() + llm_model = get_deepeval_model() results: List[Dict[str, Any]] = [] for _, row in dataset._df.iterrows(): diff --git a/validmind/tests/model_validation/ragas/utils.py b/validmind/tests/model_validation/ragas/utils.py index b716173b8..182b5006a 100644 --- a/validmind/tests/model_validation/ragas/utils.py +++ b/validmind/tests/model_validation/ragas/utils.py @@ -2,14 +2,69 @@ # Refer to the LICENSE file in the root of this repository for details. # SPDX-License-Identifier: AGPL-3.0 AND ValidMind Commercial +from langchain_core.outputs import ChatGeneration + from validmind.ai.utils import get_judge_config EMBEDDINGS_MODEL = "text-embedding-3-small" +def _ragas_is_finished_parser(response) -> bool: + """Accept finish signals used by Gemini as well as OpenAI-style responses.""" + + is_finished_list = [] + accepted_finish_reasons = { + "stop", + "STOP", + "end_turn", + "END_TURN", + "max_tokens", + "MAX_TOKENS", + } + + for generation_group in response.flatten(): + response_generation = generation_group.generations[0][0] + generation_info = getattr(response_generation, "generation_info", None) or {} + + finish_reason = generation_info.get("finish_reason") + if finish_reason is not None: + is_finished_list.append(finish_reason in accepted_finish_reasons) + continue + + if isinstance(response_generation, ChatGeneration): + response_metadata = ( + getattr(response_generation.message, "response_metadata", None) or {} + ) + finish_reason = response_metadata.get("finish_reason") + stop_reason = response_metadata.get("stop_reason") + + if finish_reason is not None: + is_finished_list.append(finish_reason in accepted_finish_reasons) + continue + + if stop_reason is not None: + is_finished_list.append(stop_reason in accepted_finish_reasons) + continue + + is_finished_list.append(True) + + return all(is_finished_list) + + def get_ragas_config(judge_llm=None, judge_embeddings=None): judge_llm, judge_embeddings = get_judge_config(judge_llm, judge_embeddings) - return {"llm": judge_llm, "embeddings": judge_embeddings} + + try: + from ragas.llms.base import LangchainLLMWrapper + except ImportError: + raise ImportError("Please run `pip install validmind[llm]` to use RAGAS tests") + + return { + "llm": LangchainLLMWrapper( + judge_llm, is_finished_parser=_ragas_is_finished_parser + ), + "embeddings": judge_embeddings, + } def make_sub_col_udf(root_col, sub_col): diff --git a/validmind/tests/prompt_validation/ai_powered_test.py b/validmind/tests/prompt_validation/ai_powered_test.py index d0d843ba1..b3ad420be 100644 --- a/validmind/tests/prompt_validation/ai_powered_test.py +++ b/validmind/tests/prompt_validation/ai_powered_test.py @@ -32,9 +32,10 @@ def call_model( # Only check OpenAI config if no custom judge_llm is provided if judge_llm is None and not is_configured(): raise ValueError( - "LLM is not configured. Please set an `OPENAI_API_KEY` environment variable " - "or ensure that you are connected to the ValidMind API and ValidMind AI is " - "enabled for your account." + "LLM is not configured. Please set an `OPENAI_API_KEY`, " + "`AZURE_OPENAI_KEY`, `GOOGLE_API_KEY`, or `GEMINI_API_KEY` " + "environment variable or ensure that you are connected to the " + "ValidMind API and ValidMind AI is enabled for your account." ) judge_llm, judge_embeddings = get_judge_config(judge_llm, judge_embeddings)