From dabc6da5ce2447d67b839fcc6351601a262cc1eb Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Tue, 2 Jun 2026 11:02:14 -0700 Subject: [PATCH 1/2] Merge pull request #1359 from validmind/beck/sc-15995/documentation-primary-record-types-phase MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit docs: Models > Records — Reporting, Monitoring, Attestation --- site/guide/attestation/_attestations.qmd | 2 +- .../attestation/_configure-attestations.qmd | 34 +++++++------- .../attestation/approve-attestations.qmd | 10 ++-- .../guide/attestation/manage-attestations.qmd | 16 +++---- .../guide/attestation/review-attestations.qmd | 10 ++-- .../guide/attestation/submit-attestations.qmd | 16 +++---- .../attestation/working-with-attestations.qmd | 30 ++++++------ site/guide/chatbot-capabilities.qmd | 8 ++-- site/guide/monitoring/enable-monitoring.qmd | 26 +++++------ site/guide/monitoring/ongoing-monitoring.qmd | 46 +++++++++---------- .../monitoring/review-monitoring-results.qmd | 14 +++--- .../monitoring/set-thresholds-and-alerts.qmd | 16 +++---- .../work-with-metrics-over-time.qmd | 28 +++++------ .../reporting/_create-a-visualization.qmd | 8 ++-- site/guide/reporting/_reports-grouping.qmd | 2 +- site/guide/reporting/_reports-metrics.qmd | 2 +- site/guide/reporting/_view-report-data.qmd | 6 +-- site/guide/reporting/export-analytics.qmd | 4 +- site/guide/reporting/export-artifacts.qmd | 10 ++-- site/guide/reporting/export-documents.qmd | 10 ++-- site/guide/reporting/export-inventory.qmd | 10 ++-- site/guide/reporting/generating-exports.qmd | 2 +- site/guide/reporting/view-report-data.qmd | 4 +- .../reporting/working-with-analytics.qmd | 30 ++++++------ site/guide/workflows/_workflow-step-types.qmd | 2 +- .../customizing-your-inventory.qmd | 31 +++++++------ .../onboarding-your-organization.qmd | 8 ++-- .../organizational-oversight-reporting.qmd | 14 +++--- .../using-validmind-for-risk-management.qmd | 30 ++++++------ .../ai-governance/managing-ai-use-cases.qmd | 4 +- .../common-slides/_review-model-activity.qmd | 2 +- .../using-validmind-for-validation.qmd | 6 +-- 32 files changed, 223 insertions(+), 218 deletions(-) diff --git a/site/guide/attestation/_attestations.qmd b/site/guide/attestation/_attestations.qmd index e662fd8197..f656e22dae 100644 --- a/site/guide/attestation/_attestations.qmd +++ b/site/guide/attestation/_attestations.qmd @@ -2,4 +2,4 @@ Refer to the LICENSE file in the root of this repository for details. SPDX-License-Identifier: AGPL-3.0 AND ValidMind Commercial --> -Attestation enables submitters, reviewers, and approvers to formally certify, review, and sign off on model attributes at specific points in time, supporting compliance and governance processes. \ No newline at end of file +Attestation enables submitters, reviewers, and approvers to formally certify, review, and sign off on record (model) attributes at specific points in time, supporting compliance and governance processes. \ No newline at end of file diff --git a/site/guide/attestation/_configure-attestations.qmd b/site/guide/attestation/_configure-attestations.qmd index 2f86dc345f..9fe22985f3 100644 --- a/site/guide/attestation/_configure-attestations.qmd +++ b/site/guide/attestation/_configure-attestations.qmd @@ -57,20 +57,20 @@ Assign users and roles responsible for this attestation. Each participant type h ### Stakeholder versus role -- **Submitter** — A model stakeholder assigned directly to a model +- **Submitter** — A stakeholder assigned directly to a record - **Reviewer** — A role responsible for reviewing attestations within their assigned groups - **Approver** — A role responsible for approving attestations within their assigned groups ___ -Attestation submissions are organized by owner and group. This approach ensures reviewers and approvers have appropriate permissions for models in their groups. +Attestation submissions are organized by owner and group. This approach ensures reviewers and approvers have appropriate permissions for records in their groups. ::: -a. Under Attestation Participants, select the **[submitter]{.smallcaps}** from the model stakeholders dropdown: +a. Under Attestation Participants, select the **[submitter]{.smallcaps}** from the inventory record stakeholders: - - Model Owner - - Model Developer - - Model Validator + - Record Owner + - Record Developer + - Record Validator b. Select the **[reviewer]{.smallcaps}** from the roles dropdown. @@ -80,7 +80,7 @@ d. Click **Save**. ### 3. Set inventory scope -Define which models are included in the attestation process. By default, all models are included. Use filters to narrow the scope based on model fields, stages, or custom attributes. +Define which records are included in the attestation process. By default, all records are included. Use filters to narrow the scope based on inventory record fields, stages, or custom attributes. a. Under Inventory Scope, click **{{< fa plus >}} Add Rule** to add a filter condition. @@ -94,7 +94,7 @@ a. Under Inventory Scope, click **{{< fa plus >}} Add Rule** to add a filter con b. Click **Save**. -c. Optional: Click **See Models** to preview which models match your filter criteria. +c. Optional: Click **View Records** to preview which records match your filter criteria. ::: {.embed-tip} ### How do I filter for groups? @@ -106,12 +106,12 @@ To filter for groups, you first need to define these user groups in your organiz ### 4. Add attestation fields -By default, attestations include the model name. Add additional model inventory fields as needed: +By default, attestations include the record name. Add additional inventory record fields as needed: -a. Under Relevant Attestation Fields, drag model inventory fields between columns: +a. Under Relevant Attestation Fields, drag inventory record fields between columns: - - **Model Inventory Fields** — Available fields from the model inventory - - **Relevant Attestation Fields** — Selected fields to display in model snapshots + - **Inventory Record Fields** — Available fields from the inventory + - **Relevant Attestation Fields** — Selected fields to display in record snapshots b. Optional: Reorder the fields to change how they display in the attestation. @@ -123,7 +123,7 @@ Define the structured fields that submitters fill out as part of the attestation ### Tip -Work with your model risk management team to create fields that accurately reflect your organization's attestation requirements. +Work with your risk management team to create fields that accurately reflect your organization's attestation requirements. ___ @@ -175,7 +175,7 @@ a. Click **Add Period**. #### Assign attestation participants -a. Under Attestation Participants, select the **[submitter]{.smallcaps}** from the model stakeholders dropdown. +a. Under Attestation Participants, select the **[submitter]{.smallcaps}** from the inventory record stakeholders. a. Select the **[reviewer]{.smallcaps}** from the roles dropdown. @@ -193,10 +193,10 @@ a. Click **Save**. #### Add attestation fields -a. Under Relevant Attestation Fields, drag model inventory fields between columns: +a. Under Relevant Attestation Fields, drag inventory record fields between columns: - - **Model Inventory Fields** — Available fields from the model inventory - - **Relevant Attestation Fields** — Selected fields to display in model snapshots + - **Inventory Record Fields** — Available fields from the inventory + - **Relevant Attestation Fields** — Selected fields to display in record snapshots a. Optional: Reorder the fields to change how they display in the attestation. diff --git a/site/guide/attestation/approve-attestations.qmd b/site/guide/attestation/approve-attestations.qmd index 1fad77f2ab..afb67632f9 100644 --- a/site/guide/attestation/approve-attestations.qmd +++ b/site/guide/attestation/approve-attestations.qmd @@ -6,16 +6,16 @@ title: "Approve attestations" date: last-modified --- -Sign off on reviewed attestation questionnaires to formally approve the accuracy of key model information at a point in time. +Sign off on reviewed attestation questionnaires to formally approve the accuracy of key record (model) information at a point in time. -As an approver, you provide the final sign-off on attestation questionnaires that have been reviewed, completing the attestation process and creating a permanent record in the model's history. +As an approver, you provide the final sign-off on attestation questionnaires that have been reviewed, completing the attestation process and creating a permanent audit entry in the record's history. ::: {.attn} ## Prerequisites - [x] {{< var link.login >}} -- [x] Models registered in the inventory require attestation.[^1] +- [x] Records registered in the inventory require attestation.[^1] - [x] Attestation has been set up.[^2] - [x] An attestation instance is in the [{{< fa clipboard-check >}} Reviewed]{.bubble} state.[^3] - [x] You are an approver.[^4] @@ -29,11 +29,11 @@ As an approver, you provide the final sign-off on attestation questionnaires tha - In the left sidebar, click **{{< fa clipboard-check >}} Attestations**, then click on the submission in the **Reviewed** column. - In the left sidebar, click **{{< fa inbox >}} My Inbox**, then under **Tasks**, locate the attestation with a **Reviewed** status. -2. Review the questionnaire responses and model context to verify the attestation is ready for approval. +2. Review the questionnaire responses and record context to verify the attestation is ready for approval. 3. Click **Sign Off**. -After you sign off, the attestation is recorded in the model's history. Attestation sign-offs are permanent and cannot be changed. +After you sign off, the attestation is cataloged in the record's history. Attestation sign-offs are permanent and cannot be changed. diff --git a/site/guide/attestation/manage-attestations.qmd b/site/guide/attestation/manage-attestations.qmd index 786bd96651..1074b92d41 100644 --- a/site/guide/attestation/manage-attestations.qmd +++ b/site/guide/attestation/manage-attestations.qmd @@ -25,7 +25,7 @@ Setting up attestations involves adding an attestation template and completing t ## Prerequisites - [x] {{< var link.login >}} -- [x] Models registered in the inventory require attestation. [^1] +- [x] Records (models) registered in the inventory require attestation. [^1] - [x] You are a [{{< fa hand >}} Customer Admin]{.bubble} or assigned another role with sufficient permissions to set up attestations.[^2] ::: @@ -116,7 +116,7 @@ The Progress tab displays a swimlane view of all attestation submissions organiz - **Reviewed** — Submissions awaiting approver sign-off. - **Signed Off** — Completed submissions. -Click any submission card to view its details, including the questionnaire responses and associated models. +Click any submission card to view its details, including the questionnaire responses and associated records. ### Responses @@ -134,11 +134,11 @@ Use this view to: ![Before and after snapshot comparison](attestations-before-after.png){fig-alt="Before and after toggle for comparing snapshots" .screenshot width=50%} ::: -The Inventory tab shows a snapshot of all models included in the attestation scope at the time the period started. This provides an organization-wide view of the model inventory state. +The Inventory tab shows a snapshot of all records included in the attestation scope at the time the period started. This provides an organization-wide view of the inventory state. -Click any model to view its snapshot details in a modal, which displays the model field values captured at that point in time. Click **View Live Data** to navigate to the current model inventory record. +Click any record to view its snapshot details in a modal, which displays the record field values captured at that point in time. Click **View Live Data** to navigate to the current record. -For completed attestation periods, use the **Before** and **After** toggle to compare the model state at the start versus the completion of the period. +For completed attestation periods, use the **Before** and **After** toggle to compare the record state at the start versus the completion of the period. ### Validation Issues @@ -147,12 +147,12 @@ The Validation Issues tab shows all validation issues that were open at the time - Title and artifact ID - Severity and status -- Associated inventory model +- Associated inventory record - Risk area, assignee, and due date ### Artifacts -The Artifacts tab displays model artifacts captured in the snapshot, organized by artifact type. +The Artifacts tab displays artifacts (findings) captured in the snapshot, organized by artifact type. ::: @@ -173,7 +173,7 @@ Export attestation data in CSV or Excel format for external reporting and compli When all submissions in an attestation period have been signed off, the period is automatically marked as complete. Administrators can also manually complete a period before all submissions are finished. ::: {.callout-important title="Completing a period triggers a final snapshot"} -When a period is completed, the platform captures a final snapshot of all models and artifacts. This enables before/after comparison in the Inventory tab. +When a period is completed, the platform captures a final snapshot of all records and artifacts. This enables before/after comparison in the Inventory tab. ::: To manually complete an attestation period: diff --git a/site/guide/attestation/review-attestations.qmd b/site/guide/attestation/review-attestations.qmd index bdc32ff2ef..be23ac1952 100644 --- a/site/guide/attestation/review-attestations.qmd +++ b/site/guide/attestation/review-attestations.qmd @@ -6,7 +6,7 @@ title: "Review attestations" date: last-modified --- -Review attestation questionnaires to verify the accuracy of key model information at a point in time. +Review attestation questionnaires to verify the accuracy of key record (model) information at a point in time. As a reviewer, you assess the submitter's attestation, including ownership, usage, performance, controls, and documentation. You can provide feedback and request changes to support a traceable and accountable audit trail. @@ -15,7 +15,7 @@ As a reviewer, you assess the submitter's attestation, including ownership, usag ## Prerequisites - [x] {{< var link.login >}} -- [x] Models registered in the inventory require attestation.[^1] +- [x] Records registered in the inventory require attestation.[^1] - [x] Attestation has been set up.[^2] - [x] An attestation instance is in the [{{< fa clipboard-check >}} Ready For Review]{.bubble} state.[^3] - [x] You are a reviewer.[^4] @@ -31,11 +31,11 @@ As a reviewer, you assess the submitter's attestation, including ownership, usag 2. Review the submitter's questionnaire responses displayed on the right side of the screen. -3. Review the model context using the tabs on the left: +3. Review the record context using the tabs on the left: - - **Inventory** — Shows the models included in the attestation with snapshot details. + - **Inventory** — Shows the records included in the attestation with snapshot details. - **Validation Issues** — Shows open validation issues at the time of the snapshot. - - **Artifacts** — Shows model artifacts captured in the snapshot. + - **Artifacts** — Shows artifacts (findings) captured in the snapshot. 4. Optional: Click **Activity** to review the attestation history: diff --git a/site/guide/attestation/submit-attestations.qmd b/site/guide/attestation/submit-attestations.qmd index 9ce6419629..995f1aa69a 100644 --- a/site/guide/attestation/submit-attestations.qmd +++ b/site/guide/attestation/submit-attestations.qmd @@ -6,16 +6,16 @@ title: "Submit attestations" date: last-modified --- -Certify the accuracy of key model information at a point in time by submitting the attestation questionnaire. +Certify the accuracy of key record (model) information at a point in time by submitting the attestation questionnaire. -As a submitter, this process is part of the audit trail for your model's information, supporting accountability across its lifecycle. Attestation typically includes details about ownership, usage, performance, controls, and documentation. +As a submitter, this process is part of the audit trail for your record's information, supporting accountability across its lifecycle. Attestation typically includes details about ownership, usage, performance, controls, and documentation. ::: {.attn} ## Prerequisites - [x] {{< var link.login >}} -- [x] Models registered in the inventory require attestation.[^1] +- [x] Records registered in the inventory require attestation.[^1] - [x] Attestation has been set up.[^2] - [x] An attestation instance is in the [{{< fa clipboard-check >}} Not Started]{.bubble} state. - [x] You are a submitter.[^3] @@ -33,16 +33,16 @@ As a submitter, this process is part of the audit trail for your model's informa After you start the attestation, the questionnaire fields are available to complete. -3. Review the model context using the tabs on the left side of the screen: +3. Review the record context using the tabs on the left side of the screen: - - **Inventory** — Shows the models included in your attestation with snapshot details. - - **Validation Issues** — Shows open validation issues for your models at the time of the snapshot. - - **Artifacts** — Shows model artifacts captured in the snapshot. + - **Inventory** — Shows the records included in your attestation with snapshot details. + - **Validation Issues** — Shows open validation issues for your records at the time of the snapshot. + - **Artifacts** — Shows artifacts (findings) captured in the snapshot. 4. Complete the attestation questionnaire by filling in each field. Required fields are marked with an asterisk (*). ::: {.callout} - Click on any model in the Inventory tab to view its snapshot details. Use the **View Live Data** link to navigate to the current model record if you need additional context. + Click on any record in the Inventory tab to view its snapshot details. Use the **View Live Data** link to navigate to the current record if you need additional context. ::: 5. Verify that your answers are complete and click **Submit for Review**. diff --git a/site/guide/attestation/working-with-attestations.qmd b/site/guide/attestation/working-with-attestations.qmd index b089c5f0e7..ccf3095924 100644 --- a/site/guide/attestation/working-with-attestations.qmd +++ b/site/guide/attestation/working-with-attestations.qmd @@ -25,8 +25,8 @@ listing: ## Prerequisites - [x] {{< var link.login >}} -- [x] Models registered in the inventory require attestation.[^1] -- [x] If applicable, model artifacts have been documented.[^2] +- [x] Records registered in the inventory require attestation.[^1] +- [x] If applicable, artifacts (findings) have been documented.[^2] - [x] You are assigned a role with sufficient permissions to perform attestation tasks:[^3] - [{{< fa hand >}} Customer Admin]{.bubble} — Sets up attestations - [Submitter]{.bubble} — Stakeholder who submits attestation questionnaires @@ -50,7 +50,7 @@ When an attestation period is active or has been completed in the past, **{{< fa - A swimlane view of all attestation submissions organized by status - Consolidated questionnaire responses across all submissions -- Organization-wide snapshots of models, validation issues, and artifacts +- Organization-wide snapshots of records, validation issues, and artifacts - Export capabilities for compliance reporting Attestation participants can also access their individual tasks via **{{< fa inbox >}} My Inbox** in the left sidebar, where tasks appear under the Tasks tab with the current attestation status. @@ -61,7 +61,7 @@ The Attestations sidebar item appears only when there is _at least one_ active o ### How does the attestation process work? -Attestation on the {{< var validmind.platform >}} begins at the start of the attestation period after the process has been set up initially. The platform captures a snapshot of each model and its artifacts, preserving the state at that point in time. A second snapshot is captured when the attestation period is completed, enabling comparative analysis between the start and end states. +Attestation on the {{< var validmind.platform >}} begins at the start of the attestation period after the process has been set up initially. The platform captures a snapshot of each record and its artifacts, preserving the state at that point in time. A second snapshot is captured when the attestation period is completed, enabling comparative analysis between the start and end states. Responsibilities for the attestation process are shared: @@ -94,42 +94,42 @@ Plan your attestation periods ahead of time to ensure they can be completed on t Attestation submissions are organized by owner AND group. This structure supports organizations where: -- Model owners may own models across multiple groups, such as Fraud and Know-Your-Customer (KYC). +- Record owners may own records across multiple groups, such as Fraud and Know-Your-Customer (KYC). - Each group has its own validation and governance team. - Reviewers and approvers are roles scoped to specific groups. ::: {.callout title="Example scenario"} -Steven owns 10 models: 4 in the Fraud group and 6 in the KYC group. Emma is the Governance Lead for Fraud, and Kam is the Governance Lead for KYC. +Steven owns 10 records: 4 in the Fraud group and 6 in the KYC group. Emma is the Governance Lead for Fraud, and Kam is the Governance Lead for KYC. -If attestations were organized by owner alone, neither Emma nor Kam would have permissions to see all of Steven's models. With owner+group organization: +If attestations were organized by owner alone, neither Emma nor Kam would have permissions to see all of Steven's records. With owner+group organization: -- Steven submits Attestation #1 for his 4 Fraud models → Emma approves -- Steven submits Attestation #2 for his 6 KYC models → Kam approves +- Steven submits Attestation #1 for his 4 Fraud records → Emma approves +- Steven submits Attestation #2 for his 6 KYC records → Kam approves -This approach ensures 100% accountability per group, where Emma approves all Fraud models and Kam approves all KYC models. If Fraud has two co-leads (Emma and Andres), either can approve. +This approach ensures 100% accountability per group, where Emma approves all Fraud records and Kam approves all KYC records. If Fraud has two co-leads (Emma and Andres), either can approve. ::: ## How do I create meaningful attestation questionnaires? -Attestation questionnaires are typically used in model risk management to confirm that key controls, governance processes and documentation requirements have been followed throughout a model's lifecycle. These questionnaires are part of a broader effort to ensure accountability and regulatory compliance. +Attestation questionnaires are typically used in risk management to confirm that key controls, governance processes and documentation requirements have been followed throughout a record's lifecycle. These questionnaires are part of a broader effort to ensure accountability and regulatory compliance. Questionnaires are composed of structured fields that you define when configuring the attestation template. Each field can be a text input, dropdown, checkbox, or other field type, with optional validation rules. Responses are collected consistently across all submissions and can be viewed in the consolidated Responses tab or exported for external reporting. Areas typically covered in the questionnaire: -- **Verification of responsibilities** — Attests that model owners, developers and validators have performed their roles in accordance with internal policies and procedures. +- **Verification of responsibilities** — Attests that record owners, developers and validators have performed their roles in accordance with internal policies and procedures. -- **Documentation of compliance** — Confirms that required documentation — such as model documentation, validation reports and change logs — has been completed and properly archived. +- **Documentation of compliance** — Confirms that required documentation — such as documentation, validation reports and change logs — has been completed and properly archived. - **Evidence of control operation** — Serves as formal evidence that key controls (for example: backtesting, performance monitoring, version control) are in place and functioning as intended. -- **Governance checkpoints** — Used as checkpoints at critical stages, such as model approval, annual reviews or retirement, to ensure all required steps have been completed before proceeding. +- **Governance checkpoints** — Used as checkpoints at critical stages, such as approval, annual reviews or retirement, to ensure all required steps have been completed before proceeding. - **Regulatory reporting** — In regulated environments, provides a documented trail that can be reviewed during internal audits or by external regulators. - **Risk ownership and accountability** — By signing off on questionnaires, individuals acknowledge their responsibilities and support a strong control culture. -Work with your model risk management team to create a template that accurately reflects your organization's attestation requirements. +Work with your risk management team to create a template that accurately reflects your organization's attestation requirements. ## Manage attestations diff --git a/site/guide/chatbot-capabilities.qmd b/site/guide/chatbot-capabilities.qmd index 10db25ba70..70f663da79 100644 --- a/site/guide/chatbot-capabilities.qmd +++ b/site/guide/chatbot-capabilities.qmd @@ -37,7 +37,7 @@ Keep these limitations in mind when using the assistant: - **Hallucination risk** — AI-generated responses may occasionally contain inaccuracies. Verify critical information before acting on it. - **Documentation lag** — The assistant's knowledge is based on published documentation, which may not reflect the latest release features. Check [release notes](/releases/all-releases.qmd) for recent changes. - **Not a compliance substitute** — The assistant provides guidance but cannot replace professional compliance judgment or regulatory expertise. -- **Limited live query scope** — When querying your organization's data, the assistant can currently access model inventory and artifacts only. Questions about workflows, analytics, or other platform areas are not yet supported. +- **Limited live query scope** — When querying your organization's data, the assistant can currently access inventory records and artifacts only. Questions about workflows, analytics, or other platform areas are not yet supported. ## In-app assistant vs. MCP @@ -46,11 +46,11 @@ Keep these limitations in mind when using the assistant: | Feature | In-app assistant | MCP integration | |---------|------------------|-----------------| | Access method | Floating chat window in platform | AI assistants (Cursor, Claude Code) | -| Primary use | Product help and guidance | Model inventory management | -| Data access | Documentation + your current context | Your models, artifacts, templates | +| Primary use | Product help and guidance | Inventory management | +| Data access | Documentation + your current context | Your records (models), artifacts (findings), templates | | Actions | Read-only guidance | Create, read, update operations | -The in-app assistant (described on this page) is focused on helping you use the platform. For programmatic access to your model inventory, refer to [Connect AI assistants via MCP](/guide/mcp/connect-ai-assistants-via-mcp.qmd). +The in-app assistant (described on this page) is focused on helping you use the platform. For programmatic access to your inventory, refer to [Connect AI assistants via MCP](/guide/mcp/connect-ai-assistants-via-mcp.qmd). ## Get help diff --git a/site/guide/monitoring/enable-monitoring.qmd b/site/guide/monitoring/enable-monitoring.qmd index 0b835bec0c..07e64065a2 100644 --- a/site/guide/monitoring/enable-monitoring.qmd +++ b/site/guide/monitoring/enable-monitoring.qmd @@ -6,9 +6,9 @@ title: "Enable monitoring" date: last-modified --- -To start uploading ongoing monitoring results for a model to {{< var vm.product >}}, enable monitoring in your code snippet within the {{< var validmind.platform >}} and then select a monitoring template. You can enable monitoring for both new and existing models. +To start uploading ongoing monitoring results for a record (model) to {{< var vm.product >}}, enable monitoring in your code snippet within the {{< var validmind.platform >}} and then select a monitoring template. You can enable monitoring for both new and existing records. -::: {.callout title="To try out monitoring, check out the code sample for ongoing monitoring of models."} +::: {.callout title="To try out monitoring, check out the code sample for ongoing monitoring."} [Quickstart for ongoing monitoring of models with {{< var vm.product >}} {{< fa hand-point-right >}}](/notebooks/use_cases/ongoing_monitoring/quickstart_customer_churn_ongoing_monitoring.ipynb) ::: @@ -18,7 +18,7 @@ To start uploading ongoing monitoring results for a model to {{< var vm.product - [x] {{< var link.login >}} - [x] You are a [{{< fa code >}} Developer]{.bubble} or assigned another role with sufficient permissions to perform the tasks in this guide.[^1] -- [x] The model is in the pre-approval stage for performance testing or the model has been approved and is in production. +- [x] The record is in the pre-approval stage for performance testing or the record has been approved and is in production. ::: @@ -28,16 +28,16 @@ To start uploading ongoing monitoring results for a model to {{< var vm.product {{< include /developer/quickstart/_code-snippet.qmd >}} -To enable ongoing monitoring for a model, add `monitoring=True` to your code snippet: +To enable ongoing monitoring for a record, add `monitoring=True` to your code snippet: -a. In the {{< var validmind.platform >}}, either register a new model[^2] or locate an existing model: +a. In the {{< var validmind.platform >}}, either register a new record[^2] or locate an existing record: - In the left sidebar, click **{{< fa cubes >}} Inventory**. - - Select a model or find your model by applying a filter or searching for it.[^3] + - Select a record or find your record by applying a filter or searching for it.[^3] -b. Copy the code snippet for the model: +b. Copy the code snippet for your record: - - In the left sidebar that appears for your model, click **{{< fa rocket >}} Getting Started**. + - In the left sidebar that appears for your record, click **{{< fa rocket >}} Getting Started**. - Select a Monitoring document type from the **[document]{.smallcaps}** drop-down menu.[^4] - Set **[enable monitoring]{.smallcaps}[?]{.smallercaps}** to true. - Click **{{< fa regular copy >}} Copy snippet to clipboard**. @@ -48,9 +48,9 @@ Before you can start sending ongoing monitoring data using the {{< var validmind a. In the {{< var validmind.platform >}}, click **{{< fa cubes >}} Inventory** in the left sidebar. -b. Select a model or find your model by applying a filter or searching for it.[^6] +b. Select a record or find your record by applying a filter or searching for it.[^6] -c. In the left sidebar that appears for your model, click **{{< fa file >}} Documents** and select the **Latest** tab.[^7] +c. In the left sidebar that appears for your record, click **{{< fa file >}} Documents** and select the **Latest** tab.[^7] d. Click on a Monitoring type file.[^8] @@ -89,11 +89,11 @@ b. Confirm that `monitoring=True` is present in the `vm.init` method, similar to ::: {.w-60-ns .pr3} - Load your model credentials from your `.env` file, then remove the inline credentials from `vm.init()` after importing the {{< var vm.developer >}}. + Load your record credentials from your `.env` file, then remove the inline credentials from `vm.init()` after importing the {{< var vm.developer >}}. You'll still need to include your document key and the monitoring flag. - [Store model credentials in `.env` files](/developer/quickstart/store-credentials-in-env-file.qmd){.button .button-green} + [Store record credentials in `.env` files](/developer/quickstart/store-credentials-in-env-file.qmd){.button .button-green} ::: @@ -119,7 +119,7 @@ b. Confirm that `monitoring=True` is present in the `vm.init` method, similar to ::: ::: -c. Run the code snippet to initialize the {{< var validmind.developer >}} and connect to the selected monitoring document for your model in the {{< var validmind.platform >}}.[^5] +c. Run the code snippet to initialize the {{< var validmind.developer >}} and connect to the selected monitoring document for your record in the {{< var validmind.platform >}}.[^5] ::: {.callout} ## A template must already be applied to your selected document to populate monitoring test results in the {{< var validmind.platform >}}. diff --git a/site/guide/monitoring/ongoing-monitoring.qmd b/site/guide/monitoring/ongoing-monitoring.qmd index e4cb438f01..6f3ad83e14 100644 --- a/site/guide/monitoring/ongoing-monitoring.qmd +++ b/site/guide/monitoring/ongoing-monitoring.qmd @@ -38,29 +38,29 @@ listing: fields: [title, description] --- -Monitoring of model performance in model risk management involves regularly assessing a model's accuracy, stability, and robustness to ensure it remains reliable after deployment. +Monitoring of record (model) performance in risk management involves regularly assessing a record's accuracy, stability, and robustness to ensure it remains reliable after deployment. -Monitoring is a critical component of model risk management, as emphasized in regulations such as SR 26-2, SS1/23, and E-24,[^1] and includes: +Monitoring is a critical component of risk management, as emphasized in regulations such as SR 26-2, SS1/23, and E-24,[^1] and includes: -- Tracking key performance indicators, detecting model drift, recalibrating as needed, and validating assumptions. -- Backtesting to verify predictive power, maintaining transparent reporting and governance, and ensuring ongoing compliance with regulatory standards. -- Continuous monitoring to sustain the model's integrity and mitigating risks associated with its use as they arise. +- Tracking key performance indicators, detecting model drift, recalibrating as needed, and validating assumptions. +- Backtesting to verify predictive power, maintaining transparent reporting and governance, and ensuring ongoing compliance with regulatory standards. +- Continuous monitoring to sustain the record's integrity and mitigating risks associated with its use as they arise. ## Monitoring scenarios Scenarios where ongoing monitoring is warranted: -- **Pre-approval monitoring of new models** — New models should undergo a trial phase of monitoring before full approval and subsequent deployment, particularly for high-risk or regulatory models, to ensure reliability before deployment. +- **Pre-approval monitoring of new records** — New records such as models should undergo a trial phase of monitoring before full approval and subsequent deployment, particularly for high-risk or regulatory records, to ensure reliability before deployment. - Trial phases where a model is subject to ongoing monitoring are typically fairly short, ranging from a few days to several weeks. + Trial phases where a record is subject to ongoing monitoring are typically fairly short, ranging from a few days to several weeks. -- **Monitoring during significant updates** — When a model undergoes a significant update, ongoing monitoring should compare the updated model's performance to the original. This process, called _parallel runs_, involves running both models simultaneously for a set period. +- **Monitoring during significant updates** — When a record undergoes a significant update, ongoing monitoring should compare the updated record's performance to the original. This process, called _parallel runs_, involves running both versions of records such as AI models simultaneously for a set period. - Model outputs should be closely monitored to assess whether the update improves performance or introduces new risks. The results help determine if the updated model should replace the original or if further adjustments are needed. Parallel runs are especially important for regulatory or critical models, ensuring changes don't harm performance. + Record outputs should be closely monitored to assess whether the update improves performance or introduces new risks. The results help determine if the updated record should replace the original or if further adjustments are needed. Parallel runs are especially important for regulatory or critical records, ensuring changes don't harm performance. -- **Post-production monitoring** — After deployment into production, models should be regularly tested against performance benchmarks to identify deviations, enabling timely recalibrations or adjustments. +- **Post-production monitoring** — After deployment into production, records should be regularly tested against performance benchmarks to identify deviations, enabling timely recalibrations or adjustments. - The model's output should be assessed regularly against predefined performance benchmarks to ensure it meets the required standards. Any deviations from the expected performance should be quickly identified, allowing for timely intervention. + The record's output should be assessed regularly against predefined performance benchmarks to ensure it meets the required standards. Any deviations from the expected performance should be quickly identified, allowing for timely intervention. ::: {.column-margin} ![Histogram of prediction probabilities to indicate model drift](prediction-probabilities-histogram-drift.png){fig-alt="A visualisation of prediction probabilities histogram drift for a binary classification model. It compares the distribution of predicted probabilities between two datasets: a reference dataset (baseline) and a monitoring dataset (current data)." .screenshot} @@ -71,14 +71,14 @@ Scenarios where ongoing monitoring is warranted: ## Ongoing monitoring plan -A robust ongoing monitoring plan is crucial for maintaining model accuracy and reliability. Model developers should start regular monitoring from the outset, ideally during the pre-approval phase of model development, refining the plan as new insights are gained. This plan should be included in the initial model documentation, with clear instructions for execution and use of results.[^2] +A robust ongoing monitoring plan is crucial for maintaining record accuracy and reliability. Developers should start regular monitoring from the outset, ideally during the pre-approval phase of development, refining the plan as new insights are gained. This plan should be included in the initial documentation, with clear instructions for execution and use of results.[^2] -As monitoring continues, it's important to report model status to key stakeholders, such as your model risk management committee. Regular updates with summary metrics will keep stakeholder informed of artifacts and emerging risks, highlighting significant trends or issues that may need action. +As monitoring continues, it's important to report record status to key stakeholders, such as your risk management committee. Regular updates with summary metrics will keep stakeholder informed of artifacts and emerging risks, highlighting significant trends or issues that may need action. Your ongoing monitoring plan should define: -- **Metrics and models** — Specify which models and performance metrics will be monitored. -- **Monitoring frequency** — Set how often each metric will be monitored, based on the model's risk and importance. +- **Metrics and records (models)** — Specify which records and performance metrics will be monitored. +- **Monitoring frequency** — Set how often each metric will be monitored, based on the record's risk and importance. - **Risk thresholds** — Establish thresholds that trigger alerts or actions when metrics deviate from expected ranges. - **Notification system** — Implement a system to notify stakeholders when significant issues or deviations occur. - **Regular reports**— Present monitoring results using visual tools for clear and accessible decision-making. @@ -116,22 +116,22 @@ Your ongoing monitoring plan should define: ### Design and implementation -The design of your ongoing monitoring plan overall should be a collaborative effort between the first line of defense, typically business units or model owners, and the second line of defense, namely your risk management team. Together, they should select the performance metrics, determine monitoring frequency, and tailor the ongoing monitoring plan to the model's specific use case and risk profile. +The design of your ongoing monitoring plan overall should be a collaborative effort between the first line of defense, typically business units or record (model) owners, and the second line of defense, namely your risk management team. Together, they should select the performance metrics, determine monitoring frequency, and tailor the ongoing monitoring plan to the record's specific use case and risk profile. -This entails that the ongoing monitoring plan is primarily designed and implemented by the developers involved in the model's development and deployment. Their work is then reviewed during model validation to ensure the robustness of the ongoing monitoring plan and alignment with risk management goals. +This entails that the ongoing monitoring plan is primarily designed and implemented by the developers involved in the record's development and deployment. Their work is then reviewed during validation to ensure the robustness of the ongoing monitoring plan and alignment with risk management goals. -The implementation of the ongoing monitoring plan typically also falls on model developers. This effort includes executing the monitoring activities, collecting performance data, and generating reports. The model developers also ensure that the monitoring is carried out according to the established schedule and that any anomalies or deviations are promptly identified and addressed. +The implementation of the ongoing monitoring plan typically also falls on developers. This effort includes executing the monitoring activities, collecting performance data, and generating reports. The developers also ensure that the monitoring is carried out according to the established schedule and that any anomalies or deviations are promptly identified and addressed. ### Testing -Model monitoring should incorporate a variety of tests to ensure ongoing accuracy and reliability. +Monitoring should incorporate a variety of tests to ensure ongoing accuracy and reliability. Key test areas to pay attention to: -- **Benchmarking** — Compare model estimates with alternative estimates to validate their accuracy. -- **Sensitivity testing** — Reaffirm the model's robustness and stability under different conditions. -- **Analysis of overrides** — Evaluate any adjustments made to the model to understand their impact. -- **Parallel outcomes analysis** — Assess whether new data should be included in the model’s calibration to enhance its performance. +- **Benchmarking** — Compare record estimates with alternative estimates to validate their accuracy. +- **Sensitivity testing** — Reaffirm the record's robustness and stability under different conditions. +- **Analysis of overrides** — Evaluate any adjustments made to the record to understand their impact. +- **Parallel outcomes analysis** — Assess whether new data should be included in the record’s calibration to enhance its performance. ::: {.callout} To try out monitoring, check out the code sample for ongoing monitoring of models.[^3] diff --git a/site/guide/monitoring/review-monitoring-results.qmd b/site/guide/monitoring/review-monitoring-results.qmd index f9dbf3118e..5897073034 100644 --- a/site/guide/monitoring/review-monitoring-results.qmd +++ b/site/guide/monitoring/review-monitoring-results.qmd @@ -6,13 +6,13 @@ title: "Review monitoring results" date: last-modified --- -To ensure your model continues to perform as expected, it's important to regularly review the monitoring results generated during the ongoing monitoring process. +To ensure your record (model) continues to perform as expected, it's important to regularly review the monitoring results generated during the ongoing monitoring process. -- As your code runs, the monitoring template for your model automatically populates with data, providing a comprehensive view of your model's performance over time. +- As your code runs, the monitoring template for your record automatically populates with data, providing a comprehensive view of your record's performance over time. - You access and examine these results within the {{< var validmind.platform >}}, allowing you to identify any deviations from expected performance and take corrective actions as needed. -::: {.callout title="To try out monitoring, check out the code sample for ongoing monitoring of models."} +::: {.callout title="To try out monitoring, check out the code sample for ongoing monitoring."} [Quickstart for ongoing monitoring of models with {{< var vm.product >}} {{< fa hand-point-right >}}](/notebooks/use_cases/ongoing_monitoring/quickstart_customer_churn_ongoing_monitoring.ipynb) ::: @@ -22,9 +22,9 @@ To ensure your model continues to perform as expected, it's important to regular - [x] {{< var link.login >}} - [x] The record you want to review monitoring results for is registered in the inventory.[^1] -- [x] Monitoring has been enabled for your model.[^2] +- [x] Monitoring has been enabled for your record.[^2] - [x] You are a [{{< fa code >}} Developer]{.bubble} or assigned another role with sufficient permissions to perform the tasks in this guide.[^3] -- [x] The model is in the pre-approval stage for performance testing or the model has been approved and is in production. +- [x] The record is in the pre-approval stage for performance testing or the record has been approved and is in production. ::: @@ -36,9 +36,9 @@ For results to be present and available for review, ongoing monitoring must be e 1. In the {{< var validmind.platform >}}, click **{{< fa cubes >}} Inventory** in the left sidebar. -2. Select a model or find your model by applying a filter or searching for it.[^5] +2. Select a record or find your record by applying a filter or searching for it.[^5] -3. In the left sidebar that appears for your model, click **{{< fa file >}} Documents** and select the **Latest** tab.[^6] +3. In the left sidebar that appears for your record, click **{{< fa file >}} Documents** and select the **Latest** tab.[^6] 3. Select the Monitoring type file you previously sent monitoring test results to.[^7] diff --git a/site/guide/monitoring/set-thresholds-and-alerts.qmd b/site/guide/monitoring/set-thresholds-and-alerts.qmd index afc310165e..cc00201e1d 100644 --- a/site/guide/monitoring/set-thresholds-and-alerts.qmd +++ b/site/guide/monitoring/set-thresholds-and-alerts.qmd @@ -8,16 +8,16 @@ date: last-modified When logging a metric, you can define thresholds or use the `passed` parameter to flag whether the metric meets performance criteria. If a metric breaches a threshold, stakeholders receive email alert notifications. -These thresholds and alerts apply to metrics over time blocks[^1], helping you track model performance and identify issues more easily. Thresholds help you flag values that suggest drift, underperformance, or other types of risk—for example, by signaling low, medium, or high risk based on how a metric evolves over time. +These thresholds and alerts apply to metrics over time blocks[^1], helping you track record (model) performance and identify issues more easily. Thresholds help you flag values that suggest drift, underperformance, or other types of risk—for example, by signaling low, medium, or high risk based on how a metric evolves over time. -Together, thresholds and notifications improve your visibility into model performance and compliance risk, enabling timely intervention when needed. +Together, thresholds and notifications improve your visibility into record performance and compliance risk, enabling timely intervention when needed. ::: {.attn} ## Prerequisites - [x] {{< var link.login >}} -- [x] Metrics over time have already been logged via the {{< var validmind.developer >}} for your model.[^1] +- [x] Metrics over time have already been logged via the {{< var validmind.developer >}} for your record.[^1] - [x] You are a [{{< fa code >}} Developer]{.bubble} or assigned another role with sufficient permissions to perform the tasks in this guide.[^2] ::: @@ -101,7 +101,7 @@ Here, the GINI score fluctuates over time but stays above the medium risk thresh :::: {.flex .flex-wrap .justify-around .w-100} ::: {.w-50-ns .pa2} -Here, the GINI score drops to `0.5`, breaching the threshold. This indicates potential performance degradation or model risk, as indicated in the upper-right corner. +Here, the GINI score drops to `0.5`, breaching the threshold. This indicates potential performance degradation or risk, as indicated in the upper-right corner. ::: ::: {.w-50-ns .pa2} @@ -119,13 +119,13 @@ The `passed` boolean is determined by you, either through a custom function, suc A metric is considered to have breached a threshold when its value falls within the high or medium risk range, that is, when it is below the low risk threshold (in this example, below `0.8`). Breaching a threshold affects visual risk indicators and triggers alert notifications, independently of whether the passed flag is `True` or `False`. -An email is sent to model stakeholders notifying them that the model has a metric that did not pass an ongoing monitoring threshold and requires attention. +An email is sent to inventory record stakeholders notifying them that the record has a metric that did not pass an ongoing monitoring threshold and requires attention. Stakeholders who receive email alert notification include: -- The model owners -- The model developers -- The validators +- The record owners +- The record developers +- The record validators ::: diff --git a/site/guide/monitoring/work-with-metrics-over-time.qmd b/site/guide/monitoring/work-with-metrics-over-time.qmd index 1fce02b5aa..09ba39ddc2 100644 --- a/site/guide/monitoring/work-with-metrics-over-time.qmd +++ b/site/guide/monitoring/work-with-metrics-over-time.qmd @@ -8,17 +8,17 @@ date: last-modified Once generated via the {{< var validmind.developer >}}, view and add metrics over time to your ongoing monitoring reports in the {{< var validmind.platform >}}. -Metrics over time refers to the continued monitoring of a model's performance once it is deployed. Tracking how a model performs as new data is introduced or conditions change ensures that it remains accurate and reliable in real-world environments where data distributions or market conditions shift. +Metrics over time refers to the continued monitoring of a record's performance once it is deployed. Tracking how a record, such as a model, performs as new data is introduced or conditions change ensures that it remains accurate and reliable in real-world environments where data distributions or market conditions shift. -- Model performance is determined by continuously measuring metrics and comparing them over time to detect degradation, bias, or shifts in the model's output. +- Record performance is determined by continuously measuring metrics and comparing them over time to detect degradation, bias, or shifts in the record's output. - Performance data is collected and tracked over time, often using a rolling window approach or real-time monitoring tools with the same metrics used in testing, but observed across different periods. -- Continuous tracking helps to identify if and when a model needs to be recalibrated, retrained, or even replaced due to performance deterioration or changing conditions. +- Continuous tracking helps to identify if and when a record needs to be recalibrated, retrained, or even replaced due to performance deterioration or changing conditions. ::: {.column-margin} ::: {.callout} ## **[Log metrics over time {{< fa hand-point-right >}}](/notebooks/how_to/metrics/log_metrics_over_time.ipynb)** -Learn how to log metrics over time, set thresholds, and analyze model performance trends with our Jupyter Notebook sample. +Learn how to log metrics over time, set thresholds, and analyze record performance trends with our Jupyter Notebook sample. ::: ::: @@ -28,8 +28,8 @@ Learn how to log metrics over time, set thresholds, and analyze model performanc ## Prerequisites - [x] {{< var link.login >}} -- [x] The model you want to log metrics over time for is registered in the inventory.[^1] -- [x] Metrics over time have already been logged via the {{< var validmind.developer >}} for your model.[^2] +- [x] The record you want to log metrics over time for is registered in the inventory.[^1] +- [x] Metrics over time have already been logged via the {{< var validmind.developer >}} for your record.[^2] - [x] You are a [{{< fa code >}} Developer]{.bubble} or assigned another role with sufficient permissions to perform the tasks in this guide.[^3] ::: @@ -38,9 +38,9 @@ Learn how to log metrics over time, set thresholds, and analyze model performanc 1. In the left sidebar, click **{{< fa cubes >}} Inventory**. -2. Select a model or find your model by applying a filter or searching for it.[^4] +2. Select a record or find your record by applying a filter or searching for it.[^4] -3. In the left sidebar that appears for your model, click **{{< fa file >}} Documents** and select the **Latest** tab.[^5] +3. In the left sidebar that appears for your record, click **{{< fa file >}} Documents** and select the **Latest** tab.[^5] 4. Select a Monitoring type file.[^6] @@ -69,15 +69,15 @@ Learn how to log metrics over time, set thresholds, and analyze model performanc ## Add integration metrics -You can add metrics from external integrations such as Amazon Bedrock directly to your monitoring documents. This feature enables you to track evaluation results from Bedrock agents alongside your other model metrics. +You can add metrics from external integrations such as Amazon Bedrock directly to your monitoring documents. This feature enables you to track evaluation results from Bedrock agents alongside your other metrics. ::: {.callout title="Integration metrics require a configured connection."} -Before you can add metrics from an integration, ensure that the connection is configured and the model is linked to the external system.[^11] +Before you can add metrics from an integration, ensure that the connection is configured and the record is linked to the external system.[^11] ::: 1. In the left sidebar, click **{{< fa cubes >}} Inventory**. -2. Select a model or find your model by applying a filter or searching for it.[^12] +2. Select a record or find your record by applying a filter or searching for it.[^12] 3. In the left sidebar, click **Monitoring** to open the monitoring report. @@ -113,7 +113,7 @@ When working with monitoring documents that contain multiple time-based metrics, - Relative presets (Last 7 days, Last 30 days, etc.) - **Custom range** — Specify exact start and end dates -The selected time range applies to all metrics in the document that support time-based filtering, providing a consistent view of your model's performance over the chosen period. This setting is only visible to you and does not affect other users. +The selected time range applies to all metrics in the document that support time-based filtering, providing a consistent view of your record's performance over the chosen period. This setting is only visible to you and does not affect other users. ## View metric over time metadata @@ -127,9 +127,9 @@ After you have added metrics over time to your document, you can view the follow 1. In the left sidebar, click **{{< fa cubes >}} Inventory**. -2. Select a model or find your model by applying a filter or searching for it.[^8] +2. Select a record or find your record by applying a filter or searching for it.[^8] -3. In the left sidebar that appears for your model, click **{{< fa file >}} Documents** and select the **Latest** tab.[^9] +3. In the left sidebar that appears for your record, click **{{< fa file >}} Documents** and select the **Latest** tab.[^9] 4. Select a Monitoring type file.[^10] diff --git a/site/guide/reporting/_create-a-visualization.qmd b/site/guide/reporting/_create-a-visualization.qmd index a485657a72..517c4b9deb 100644 --- a/site/guide/reporting/_create-a-visualization.qmd +++ b/site/guide/reporting/_create-a-visualization.qmd @@ -23,10 +23,10 @@ a. Select a **[visualization type]{.smallcaps}**: a. Select a **[dataset]{.smallcaps}**: - - **Artifact - Model Limitations** — Logged Model Limitation type artifacts only.^[[Working with artifacts](/guide/validation/working-with-artifacts.qmd)] + - **Artifact - Limitations** — Logged Limitation type artifacts only.^[[Working with artifacts](/guide/validation/working-with-artifacts.qmd)] - **Artifact - Policy Exceptions** — Logged Policy Exception type artifacts only.^[[Manage artifact types](/guide/validation/manage-artifact-types.qmd)] - **Artifact - Validation Issues** — Logged Validation Issue type artifacts only. - - **Inventory Models** — Models logged in your inventory.^[[Working with the inventory](/guide/inventory/working-with-the-inventory.qmd)] + - **Inventory Records** — Records logged in your inventory.^[[Working with the inventory](/guide/inventory/working-with-the-inventory.qmd)] a. Select the visualization configuration options to apply to the dataset.^[[Visualization configuration options](/guide/reporting/manage-custom-reports.qmd#visualization-configuration-options)] @@ -47,10 +47,10 @@ a. Select a **[visualization type]{.smallcaps}**: a. Select a **[dataset]{.smallcaps}**: - - **Artifact - Model Limitations** — Logged Model Limitation type [artifacts](/guide/validation/working-with-artifacts.qmd){target="_blank"}. only. + - **Artifact - Limitations** — Logged Limitation type [artifacts](/guide/validation/working-with-artifacts.qmd){target="_blank"}. only. - **Artifact - Policy Exceptions** — Logged Policy Exception type artifacts only. - **Artifact - Validation Issues** — Logged Validation Issue type artifacts only. - - **Inventory Models** — [Models logged in your inventory](/guide/inventory/working-with-the-inventory.qmd){target="_blank"}. + - **Inventory Records** — [Records logged in your inventory](/guide/inventory/working-with-the-inventory.qmd){target="_blank"}. a. Select the [visualization configuration options](/guide/reporting/manage-custom-reports.qmd#visualization-configuration-options){target="_blank"} to apply to the dataset. diff --git a/site/guide/reporting/_reports-grouping.qmd b/site/guide/reporting/_reports-grouping.qmd index b02089ffa2..236c5e39d5 100644 --- a/site/guide/reporting/_reports-grouping.qmd +++ b/site/guide/reporting/_reports-grouping.qmd @@ -3,4 +3,4 @@ Refer to the LICENSE file in the root of this repository for details. SPDX-License-Identifier: AGPL-3.0 AND ValidMind Commercial --> [grouping]{.smallcaps} -: The model or artifact fields that the visualization should group the data by. \ No newline at end of file +: The record or artifact fields that the visualization should group the data by. \ No newline at end of file diff --git a/site/guide/reporting/_reports-metrics.qmd b/site/guide/reporting/_reports-metrics.qmd index 96eba8441c..57a62bccc0 100644 --- a/site/guide/reporting/_reports-metrics.qmd +++ b/site/guide/reporting/_reports-metrics.qmd @@ -3,7 +3,7 @@ Refer to the LICENSE file in the root of this repository for details. SPDX-License-Identifier: AGPL-3.0 AND ValidMind Commercial --> [metrics]{.smallcaps} -: The model or artifact field that the visualization should retrieve data for: +: The record or artifact field that the visualization should retrieve data for: - For `string` type fields, you can retrieve a `Count` or `Count %`. - For `number` type fields, you can retrieve any of the following: `Count`, `Count %`, `Sum`, `Average`, `Min`, `Max` \ No newline at end of file diff --git a/site/guide/reporting/_view-report-data.qmd b/site/guide/reporting/_view-report-data.qmd index 0a773b634f..0f04b9f6ac 100644 --- a/site/guide/reporting/_view-report-data.qmd +++ b/site/guide/reporting/_view-report-data.qmd @@ -4,7 +4,7 @@ SPDX-License-Identifier: AGPL-3.0 AND ValidMind Commercial --> :::: {.content-visible unless-format="revealjs"} ::: {.callout} -Reports will not include data or artifacts from archived models.^[[Archive and delete models](/guide/inventory/archive-delete-records.qmd)] +Reports will not include data or artifacts from archived records.^[[Archive and delete records](/guide/inventory/archive-delete-records.qmd)] ::: 1. In the left sidebar, click **{{< fa square-poll-vertical >}} Analytics**. @@ -20,7 +20,7 @@ Reports will not include data or artifacts from archived models.^[[Archive and d c. On the page that opens, the applicable validation issues are listed: - Filter this list of validation issues^[[Filter and sort artifacts](/guide/validation/view-filter-artifacts.qmd#filter-and-sort-artifacts)] - Or customize column headers that display on this validation issue view^[[Customize artifact layouts](/guide/validation/customize-artifact-layouts.qmd)] - d. To get more details on the validation issue itself, click on the title of the validation issue to take you to the **{{< fa triangle-exclamation >}} Validation Issues** section for the specific model the issue applies to. + d. To get more details on the validation issue itself, click on the title of the validation issue to take you to the **{{< fa triangle-exclamation >}} Validation Issues** section for the specific record the issue applies to. :::: @@ -39,6 +39,6 @@ Reports will not include data or artifacts from archived models.^[[Archive and d c. On the page that opens, the applicable validation issues are listed: - [Filter this list of validation issues](/guide/validation/view-filter-artifacts.qmd#filter-and-sort-artifacts){target="_blank"} - Or customize column headers that display on this [validation issue view](/guide/validation/customize-artifact-layouts.qmd){target="_blank"} - d. To get more details on the validation issue itself, click on the title of the issue to take you to the **{{< fa triangle-exclamation >}} Validation Issues** section for the specific model the validation issue applies to. + d. To get more details on the validation issue itself, click on the title of the issue to take you to the **{{< fa triangle-exclamation >}} Validation Issues** section for the specific record the validation issue applies to. :::: \ No newline at end of file diff --git a/site/guide/reporting/export-analytics.qmd b/site/guide/reporting/export-analytics.qmd index b0e3519013..d90abd1788 100644 --- a/site/guide/reporting/export-analytics.qmd +++ b/site/guide/reporting/export-analytics.qmd @@ -13,8 +13,8 @@ Export analytics widgets as comma-delimited tables (`.csv`). ## Prerequisites - [x] {{< var link.login >}} -- [x] To export data on models, there must be models registered in the model inventory.[^1] -- [x] To export data on artifacts, there must be artifacts logged on any model.[^2] +- [x] To export data on records (models), there must be records registered in the inventory.[^1] +- [x] To export data on artifacts (findings), there must be artifacts logged on any record.[^2] ::: diff --git a/site/guide/reporting/export-artifacts.qmd b/site/guide/reporting/export-artifacts.qmd index 47580d321f..c763471885 100644 --- a/site/guide/reporting/export-artifacts.qmd +++ b/site/guide/reporting/export-artifacts.qmd @@ -8,7 +8,7 @@ aliases: - export-findings.html --- -Export tracked model artifacts as comma-delimited tables (`.csv`). +Export tracked artifacts (findings) as comma-delimited tables (`.csv`). ::: {.attn} @@ -16,7 +16,7 @@ Export tracked model artifacts as comma-delimited tables (`.csv`). - [x] {{< var link.login >}} - [x] There are records registered in the inventory.[^1] -- [x] There are artifacts logged on any model.[^2] +- [x] There are artifacts logged on any record (model).[^2] ::: @@ -26,7 +26,7 @@ Export tracked model artifacts as comma-delimited tables (`.csv`). #### Export validation issues -1. In the left sidebar, click **{{< fa triangle-exclamation >}} Validation Issues** for the list of validation issues across all models. +1. In the left sidebar, click **{{< fa triangle-exclamation >}} Validation Issues** for the list of validation issues across all records. (Optional) Narrow down and organize the list of validation issues and column headers you want to export by: @@ -39,12 +39,12 @@ Export tracked model artifacts as comma-delimited tables (`.csv`). #### Export all other artifact types -1. In the left sidebar, click **{{< fa border-all >}} Artifacts** to view lists of other artifact types across all models. +1. In the left sidebar, click **{{< fa border-all >}} Artifacts** to view lists of other artifact types across all records. 2. Select an artifact type[^5] to export from the tabs: - Policy Exception - - Model Limitation + - Limitation (Optional) Narrow down and organize the list of artifacts and column headers you want to export by: diff --git a/site/guide/reporting/export-documents.qmd b/site/guide/reporting/export-documents.qmd index fbe902a80e..4d698115a9 100644 --- a/site/guide/reporting/export-documents.qmd +++ b/site/guide/reporting/export-documents.qmd @@ -9,7 +9,7 @@ aliases: - /guide/documentation/export-documentation.html --- -Export model documents as XML-based Microsoft Word (`.docx`) or Adobe Portable Document Format `.pdf` files. +Export documents as XML-based Microsoft Word (`.docx`) or Adobe Portable Document Format `.pdf` files. ::: {.callout title="Microsoft Word compatibility"} Documents exported from {{< var vm.product >}} are compatible with Word 365, Word 2019, Word 2016, and Word 2013. @@ -22,7 +22,7 @@ Documents exported from {{< var vm.product >}} are compatible with Word 365, Wor - [x] {{< var link.login >}} - [x] There are records registered in the inventory.[^1] -- [x] Documents exist and are completed or are in progress for your model.[^2] +- [x] Documents exist and are completed or are in progress for your record (model).[^2] - [x] You are assigned a role with sufficient permissions to perform the tasks in this guide.[^3] ::: @@ -31,13 +31,13 @@ Documents exported from {{< var vm.product >}} are compatible with Word 365, Wor 1. In the left sidebar, click **{{< fa cubes >}} Inventory**. -2. Select a model or find your model by applying a filter or searching for it.[^4] +2. Select a record or find your record by applying a filter or searching for it.[^4] -3. In the left sidebar that appears for your model, click **{{< fa file >}} Documents** and select the **Latest** or **Saved Versions** tab.[^5] +3. In the left sidebar that appears for your record, click **{{< fa file >}} Documents** and select the **Latest** or **Saved Versions** tab.[^5] 4. Select a document to export. What information is available to export depends on the document type: - - **Development**[^6] — Model documentation + - **Development**[^6] — Development documentation - **Validation**[^7] — Validation reports - **Monitoring**[^8] — Ongoing monitoring reports - Other custom document types[^9] diff --git a/site/guide/reporting/export-inventory.qmd b/site/guide/reporting/export-inventory.qmd index 7abf57bc75..4ccb1def8d 100644 --- a/site/guide/reporting/export-inventory.qmd +++ b/site/guide/reporting/export-inventory.qmd @@ -6,7 +6,7 @@ title: "Export inventory" date: last-modified --- -Export lists of models tracked in the inventory as comma-delimited tables (`.csv`). +Export lists of records (models) tracked in the inventory as comma-delimited tables (`.csv`). ::: {.attn} @@ -17,16 +17,16 @@ Export lists of models tracked in the inventory as comma-delimited tables (`.csv ::: -## Export lists of models +## Export lists of records 1. In the left sidebar, click **{{< fa cubes >}} Inventory**. -2. (Optional) Narrow down and organize the list of models and column headers you want to export by: +2. (Optional) Narrow down and organize the list of records and column headers you want to export by: - - Filtering and sorting models[^2] + - Filtering and sorting records[^2] - Toggling visibility of and reordering column headers[^3] -3. Click **{{< fa download >}} Export** to export a list of your models with your adjustments applied. +3. Click **{{< fa download >}} Export** to export a list of your records with your adjustments applied. A `.csv` file will be automatically downloaded with your configuration. diff --git a/site/guide/reporting/generating-exports.qmd b/site/guide/reporting/generating-exports.qmd index d265051d9f..f3e0420e83 100644 --- a/site/guide/reporting/generating-exports.qmd +++ b/site/guide/reporting/generating-exports.qmd @@ -18,4 +18,4 @@ listing: - export-analytics.qmd --- -Export your documents, model inventory, artifacts, and analytics for use outside of the {{< var validmind.platform >}}. \ No newline at end of file +Export your documents, inventory, artifacts, and analytics for use outside of the {{< var validmind.platform >}}. \ No newline at end of file diff --git a/site/guide/reporting/view-report-data.qmd b/site/guide/reporting/view-report-data.qmd index e97980147b..39e1880976 100644 --- a/site/guide/reporting/view-report-data.qmd +++ b/site/guide/reporting/view-report-data.qmd @@ -6,7 +6,7 @@ title: "View report data" date: last-modified --- -Review detailed data on your models or artifacts within the {{< var validmind.platform >}}. +Review detailed data on your records (models) or artifacts (findings) within the {{< var validmind.platform >}}. ::: {.attn} @@ -14,7 +14,7 @@ Review detailed data on your models or artifacts within the {{< var validmind.pl - [x] {{< var link.login >}} - [x] There are records registered in the inventory.[^1] -- [x] There are artifacts logged on any model.[^2] +- [x] There are artifacts logged on any record.[^2] ::: diff --git a/site/guide/reporting/working-with-analytics.qmd b/site/guide/reporting/working-with-analytics.qmd index d5a7f4eb60..b1f644060c 100644 --- a/site/guide/reporting/working-with-analytics.qmd +++ b/site/guide/reporting/working-with-analytics.qmd @@ -20,17 +20,17 @@ listing: - export-analytics.qmd --- -Analytical reports allow you to gain quick insights into your model validation efforts. Reports detail critical artifacts, risk exposure, and compliance status to ensure effective oversight and management of model-related risks. +Analytical reports allow you to gain quick insights into your validation efforts. Reports detail critical artifacts, risk exposure, and compliance status to ensure effective oversight and management of risk. #### Typical uses for reports -- **Executive summaries** — Gain a high-level overview of model validation activities, focusing on critical artifacts and risk exposure. -- **Model tracking** — Get quick insights into each model's validation artifacts, highlighting any major issues or deficiencies. +- **Executive summaries** — Gain a high-level overview of validation activities, focusing on critical artifacts and risk exposure. +- **Record (model) tracking** — Get quick insights into each record's validation artifacts, highlighting any major issues or deficiencies. - **Compliance review** — Find information on compliance with relevant regulatory requirements and internal policies. - **Remediation tracking** — See the status of actions taken or proposed to address validation artifacts and mitigate identified risks. -- **Risk assessment** — Find summaries of the risks associated with each model, including model limitations, model stability and robustness issues, needed model adjustments, and much more. +- **Risk assessment** — Find summaries of the risks associated with each record, including limitations, stability and robustness issues, needed adjustments, and much more. -Reports are primarily designed for model validators or Chief Risk Officers (CROs) who require an understanding of artifacts related to model validation activities. +Reports are primarily designed for validators or Chief Risk Officers (CROs) who require an understanding of artifacts related to validation activities. ::: {.attn} @@ -38,7 +38,7 @@ Reports are primarily designed for model validators or Chief Risk Officers (CROs - [x] {{< var link.login >}} - [x] There are records registered in the inventory.[^1] -- [x] There are validation issue type artifacts logged on any model.[^2] +- [x] There are Validation Issues logged on any record.[^2] ::: @@ -72,6 +72,8 @@ By default, Models and Validation Issues analytics pages can only be modified by #### 1. Reports on models +Analytics for model-type records only. + Total Models : Count of your total models registered with {{< var vm.product >}}. @@ -79,27 +81,29 @@ Number of Models by Business Unit and Tier : Count of your model volume by business unit[^4] further broken down by risk tier. Assists you in pinpointing where in the organization resources should be concentrated on, along with the materiality of risk that unit may pose based on the models they are responsible for. Number of Models by Stage and Business Unit -: Count of your model volume by stage[^5] further broken down by business unit. Assists you in pinpointing where in the organization resources may need to be focused, based on the progress and severity of artifacts. +: Count of your model volume by stage[^5] further broken down by business unit. Assists you in pinpointing where in the organization resources may need to be focused, based on the progress and severity of artifacts. Avg. Number of Days Models Spend in a Stage -: Average count of days your models spend in a certain stage further broken down by business unit. Assists you in pinpointing what point of your model risk management process resources should be concentrated on, along with where in the organization your resources should be allotted to. +: Average count of days your models spend in a certain stage further broken down by business unit. Assists you in pinpointing what point of your risk management process resources should be concentrated on, along with where in the organization your resources should be allotted to. #### 2. Reports on validation issues +Analytics for Validation Issue artifacts only. + Total Validation Issues, Open Validation Issues, Closed Validation Issues, Past Due Validation Issues -: Count of your total validation issue type artifacts,[^6] currently open validation issues, currently past due validation issues, and previous validation issues now closed. +: Count of your total validation issues,[^6] currently open validation issues, currently past due validation issues, and previous validation issues now closed. Validation Issues by Risk Area -: Identifies issues based on the type of risk they pose to the organization, such as model limitations, data quality issues, regulatory compliance, and more. Helps in understanding the risk landscape for your models and risk areas requiring attention or improvement. +: Identifies issues based on the type of risk they pose to the organization, such as limitations, data quality issues, regulatory compliance, and more. Helps in understanding the risk landscape for your models and risk areas requiring attention or improvement. Validation Issues by Status -: Sorts validation issue type artifacts based on their resolution status, such as open, closed, or past due. Provides a snapshot of the validation process, highlighting which issues have been addressed, which are currently being worked on, and which are yet to be tackled. +: Sorts validation issues based on their resolution status, such as open, closed, or past due. Provides a snapshot of the validation process, highlighting which issues have been addressed, which are currently being worked on, and which are yet to be tackled. Validation Issues by Business Unit -: Segmented according to different divisions or departments within your organization, such as Finance, Marketing, and more. Assists you in pinpointing where in the organization model-related issues are occurring, facilitating more targeted resolution, if necessary. +: Segmented according to different divisions or departments within your organization, such as Finance, Marketing, and more. Assists you in pinpointing where in the organization issues are occurring, facilitating more targeted resolution, if necessary. Validation Issues by Model -: Specific to individual models, details the validation issue type artifacts for each model, including any errors, limitations, or assumptions that may affect the model's performance. This helps in assessing the reliability and accuracy of each model. +: Specific to individual models, details the validation issues for each model, including any errors, limitations, or assumptions that may affect the model's performance. This helps in assessing the reliability and accuracy of each model. ::: diff --git a/site/guide/workflows/_workflow-step-types.qmd b/site/guide/workflows/_workflow-step-types.qmd index dd5cffe0f8..17ea416006 100644 --- a/site/guide/workflows/_workflow-step-types.qmd +++ b/site/guide/workflows/_workflow-step-types.qmd @@ -187,7 +187,7 @@ Sends a HTTP request with optional additional conditions.^[[Conditional step req [^user-action]: - Select a field from the the drop-down to add it to your User Action step — click **Add All** to add all fields in that group. Users must have write permissions for the field in order to enter values. + Select a field from the the drop-down to add it to your User Action step — click **Add all** to add all fields in that group. Users must have write permissions for the field in order to enter values.

- [Manage inventory fields](/guide/inventory/manage-inventory-fields.qmd) diff --git a/site/training/administrator-fundamentals/customizing-your-inventory.qmd b/site/training/administrator-fundamentals/customizing-your-inventory.qmd index d7232152c3..7f08c86f45 100644 --- a/site/training/administrator-fundamentals/customizing-your-inventory.qmd +++ b/site/training/administrator-fundamentals/customizing-your-inventory.qmd @@ -38,7 +38,7 @@ aliases: # Learning objectives {.center} -_"As an **administrator** who has set up our organization's foundational units and granted users granulated access to the {{< var validmind.platform >}}, I want to configure our model inventory and customize our reusable resources."_ +_"As an **administrator** who has set up our organization's foundational units and granted users granulated access to the {{< var validmind.platform >}}, I want to configure our inventory and customize our reusable resources."_ ::: {.tc}
@@ -50,7 +50,7 @@ This second module is part of a four-part series: ## Module 2 — Contents {.center} ::: {.f2} -1. [The {{< var validmind.platform >}} model inventory](#the-validmind-inventory) +1. [The {{< var validmind.platform >}} inventory](#the-validmind-inventory) 2. [Configure your inventory](#configure-your-inventory) 3. [Customize reusable templates](#customize-reusable-templates) 4. [Configure reusable blocks](#configure-reusable-blocks) @@ -92,12 +92,12 @@ Already logged in and refreshed this module? Click [{{< fa chevron-right >}}]() :::: {.slideover--r} **Welcome to the {{< fa cubes >}} Inventory** -Use the model inventory to track comprehensive details for all your models throughout the model lifecycle. The model inventory is customizable and extensible, with a layout that can be configured to suit your needs. +Use the inventory to track comprehensive details for all your records (models) throughout their lifecycles. The inventory is customizable and extensible, with a layout that can be configured to suit your needs. ::: {.embed} -Typically, your development team will independently register models in your model inventory. +Typically, your development team will independently register records in your inventory. -For this training course, you'll register a dummy model via the **{{< fa plus >}} Register Model** modal to familiarize you with the {{< var validmind.platform >}} interface and the model details page. +For this training course, you'll register a dummy model-type record to familiarize you with the {{< var validmind.platform >}} interface and the record details page. ::: **Let's register a model together** on the next page. {{< fa hand-point-right >}} @@ -110,7 +110,7 @@ For this training course, you'll register a dummy model via the **{{< fa plus >} **Register a new model** 1. Enter in some details for your sample model, then click **Next {{< fa angle-right >}}**. -2. Select your own name under the **[model owner]{.smallcaps}** drop-down. +2. Select your own name under the **[record owner]{.smallcaps}** drop-down. 3. Click **Register Model** to add the model to your inventory. When you're done, click [{{< fa chevron-right >}}]() to continue. @@ -124,10 +124,10 @@ When you're done, click [{{< fa chevron-right >}}]() to continue. Once you've registered your model: -1. **Select the name of your model you registered** to open up the model details page. +1. **Select the name of your model you registered** to open up the record details page. 2. Scroll through the page to explore the default layout, and the fields available out-of-the-box. -When you're done, click [{{< fa chevron-right >}}]() to learn how to **customize your model inventory.** +When you're done, click [{{< fa chevron-right >}}]() to learn how to **customize your inventory.** :::: @@ -241,8 +241,9 @@ Aggregation ::: 1. Click **{{< fa plus >}} Add Field**. -2. Enter a **[title]{.smallcaps}** (name), select the field **[type]{.smallcaps}**, and indicate what **[inventory record types]{.smallcaps}** the field is available on. -3. Click **Save** to add your field. +2. Enter a **[title]{.smallcaps}** (name), select the field **[type]{.smallcaps}**. +3. Indicate what **[inventory record types]{.smallcaps}** the field is available on — choose `Model` so that we can review this field on our sample model. +4. Click **Save** to add your field. When you're done, click [{{< fa chevron-right >}}]() to continue. @@ -255,8 +256,8 @@ When you're done, click [{{< fa chevron-right >}}]() to continue. **View inventory record field** ::: -1. Select the name of your record you registered earlier to open up the record details page. -2. On the details page of your record, confirm that the new field that you added is present. +1. Select the name of your model you registered earlier to open up the record details page. +2. On the landing page of your model, confirm that the new field that you added is present. When you're done, click [{{< fa chevron-right >}}]() to continue. @@ -720,15 +721,15 @@ When you're done, click [{{< fa chevron-right >}}]() to continue. :::: {.columns} ::: {.column width="30%" .pr4 .f2} -Customizing your model inventory +Customizing your inventory ::: ::: {.column width="70%" .bl .pl4 .f3} In this second module, you learned how to: -- [x] Configure your model inventory's fields -- [x] Define granular access to model inventory fields +- [x] Configure your inventory record fields +- [x] Define granular access to inventory fields - [x] Customize and duplicate document templates - [x] Configure reusable blocks for use in templates diff --git a/site/training/administrator-fundamentals/onboarding-your-organization.qmd b/site/training/administrator-fundamentals/onboarding-your-organization.qmd index 140a167272..1ce0198bbd 100644 --- a/site/training/administrator-fundamentals/onboarding-your-organization.qmd +++ b/site/training/administrator-fundamentals/onboarding-your-organization.qmd @@ -90,11 +90,11 @@ Already logged in and refreshed this module? Click [{{< fa chevron-right >}}]() From here, you can: -- Keep track of all your organization's models and their progress throughout the model lifecycle in the customizable inventory ... +- Keep track of all your organization's records (models) and their progress throughout their lifecycles in the customizable inventory ... - Manage your organization's custom setup, including inventory fields and reusable templates and blocks ... - Manage users and user access to features ... -- Set up workflows and attestations to streamline the governance of your models ... -- View analytics on your models and artifacts (findings), and create custom reports ... +- Set up workflows and attestations to streamline the governance of your records ... +- View analytics on your records and artifacts (findings), and create custom reports ... - ... and much more! ::: {.embed} @@ -530,5 +530,5 @@ In this first module, you learned how to: Continue your {{< var vm.platform >}} administration journey with:

-[Customizing Your Model Inventory](customizing-your-inventory.html){.button target="_blank"} +[Customizing Your Inventory](customizing-your-inventory.html){.button target="_blank"} ::: \ No newline at end of file diff --git a/site/training/administrator-fundamentals/organizational-oversight-reporting.qmd b/site/training/administrator-fundamentals/organizational-oversight-reporting.qmd index ecf65ce2b2..3cc2be2641 100644 --- a/site/training/administrator-fundamentals/organizational-oversight-reporting.qmd +++ b/site/training/administrator-fundamentals/organizational-oversight-reporting.qmd @@ -36,7 +36,7 @@ skip_preview: true # Learning objectives {.center} -_"As an **administrator** who has configured the {{< var validmind.platform >}} to suit our organizational requirements, I want to use {{< var vm.product >}} to oversee how our teams interact with our models and customize reporting to provide insights tailored to our use cases."_ +_"As an **administrator** who has configured the {{< var validmind.platform >}} to suit our organizational requirements, I want to use {{< var vm.product >}} to oversee how our teams interact with our records (models) and customize reporting to provide insights tailored to our use cases."_ ::: {.tc}
@@ -49,7 +49,7 @@ This final module is part of a four-part series: ## Module 4 — Contents {.center} ::: {.f2} -1. [Audit record activity](#audit-model-activity) +1. [Audit record activity](#audit-record-activity) 2. [Audit workflows](#audit-workflows) 3. [Work with analytics](#work-with-analytics) @@ -192,8 +192,8 @@ Review workflows associated with your records or artifacts: **View active workflows** ::: -1. Select the name of your record you registered for this course to open up the record details page. -2. On the landing page of your record, locate the [active workflows]{.smallcaps} section. +1. Select the name of your model you registered for this course to open up the record details page. +2. On the landing page of your model, locate the [active workflows]{.smallcaps} section. 3. Click on the name of any active workflow to review that specific workflow's details. 4. Close the detail view for your workflow. 5. Click **{{< fa arrow-right-arrow-left >}} See All Workflows** to see records of workflows by status. @@ -209,10 +209,10 @@ When you're done, click [{{< fa chevron-right >}}]() to continue. :::: {.slideover--r .three-quarters .auto-collapse-10} **Welcome to {{< fa square-poll-vertical >}} Analytics** -Here, you can find executive summaries, track information on models, artifacts, and more. +Here, you can find executive summaries, track information on records (models), artifacts (findings), and more. ::: {.embed} -Default report widgets are provided in the **Models** and **Validation Issues** dashboards. +Default report widgets are provided in the **Models** (model-type records only) and **Validation Issues** (Validation Issue artifacts only) dashboards. You can supplement these with your own custom reports in additional dashboards to suit your organizational needs. @@ -235,7 +235,7 @@ View report data
Try it **live** on the next page. {{< fa hand-point-right >}} :::: {.f4 .pr3 .pl3 .mt4 .embed} -Reports will not include data or artifacts from [archived models](/guide/inventory/archive-delete-records.qmd){target="_blank"}. +Reports will not include data or artifacts from [archived records](/guide/inventory/archive-delete-records.qmd){target="_blank"}. ::: ::: diff --git a/site/training/administrator-fundamentals/using-validmind-for-risk-management.qmd b/site/training/administrator-fundamentals/using-validmind-for-risk-management.qmd index 20b484f621..8a94d74d75 100644 --- a/site/training/administrator-fundamentals/using-validmind-for-risk-management.qmd +++ b/site/training/administrator-fundamentals/using-validmind-for-risk-management.qmd @@ -39,7 +39,7 @@ aliases: # Learning objectives {.center} ::: {.f2} -_"As an **administrator** who has set up user access to the {{< var validmind.platform >}} and prepared our model inventory for tracking models, I want to maximize the potential of {{< var vm.product >}} as a centralized hub for risk management oversight by configuring workflows, regulations and policies, and attestations in accordance with our organization's requirements."_ +_"As an **administrator** who has set up user access to the {{< var validmind.platform >}} and prepared our inventory for tracking records (models), I want to maximize the potential of {{< var vm.product >}} as a centralized hub for risk management oversight by configuring workflows, regulations and policies, and attestations in accordance with our organization's requirements."_ ::: @@ -54,7 +54,7 @@ This third module is part of a four-part series: ## Module 3 — Contents {.center} ::: {.f2} -1. [{{< var vm.product >}} for risk management](#validmind-for-model-risk-management) +1. [{{< var vm.product >}} for risk management](#validmind-for-risk-management) 2. [Set up & manage custom workflows](#set-up-manage-workflows) 3. [Manage regulations & policies](#manage-regulations-policies) 4. [Configure attestations](#configure-attestations) @@ -85,7 +85,7 @@ To continue, you need to have been [onboarded](administrator-fundamentals-regist ::: ::: {.column width="50%"} ::: {.tc} -[Customizing Your Model Inventory](customizing-your-inventory.html){.button target="_blank"} +[Customizing Your Inventory](customizing-your-inventory.html){.button target="_blank"} ::: ::: @@ -110,9 +110,9 @@ Already logged in and refreshed this module? Click [{{< fa chevron-right >}}]() :::: {.slideover--r} **Welcome back to the {{< var validmind.platform >}}** -The {{< var validmind.platform >}} is more than just a model inventory where your model development and model validation teams collaborate seamlessly and more efficiently — it's also an indispensable tool for model risk management oversight: +The {{< var validmind.platform >}} is more than just an inventory where your development and validation teams collaborate seamlessly and more efficiently — it's also an indispensable tool for risk management oversight: -- Configure workflows to match your organizational needs for every part of your model lifecycles. +- Configure workflows to match your organizational needs for every part of your record (model) lifecycles. - Set up attestations to support compliance and governance processes. - Customize reporting to provide insights tailored to your use cases. @@ -350,10 +350,10 @@ Run workflows manually :::: {.slideover--b .auto-collapse-10} ::: {.tc} -**Initiate a model workflow** +**Initiate your model workflow** ::: -1. Select the name of your model you registered for this course to open up the model details page. +1. Select the name of your model you registered for this course to open up the record details page. 2. On the landing page of your model, click the **Workflows** tab. 3. Click **{{< fa arrow-right-arrow-left >}} See All Workflows**. 4. Select **{{< fa arrow-right >}} Run Workflow** for the workflow you configured earlier, then select **Run Now** under **[workflow start]{.smallcaps}**. @@ -404,10 +404,10 @@ As an administrator, you may need to reset or abort workflows as required: :::: {.slideover--b .auto-collapse-10} ::: {.tc} -**Reset a model workflow** +**Reset your model workflow** ::: -1. Select the name of your model you registered for this course to open up the model details page. +1. Select the name of your model you registered for this course to open up the record details page. 2. On the landing page of your model, locate the [active workflows]{.smallcaps} section. 3. Click on the name of the workflow you manually initiated earlier to open that workflow's details. 4. Click on the **{{< fa ellipsis-vertical >}}** in the top-right hand corner and select [**{{< fa ban >}} Abort Workflow**]{.red}. @@ -587,7 +587,7 @@ Attestation periods are typically scheduled quarterly or annually and should be #### Attestation fields -By default, attestations include the model name. Add additional model inventory fields as needed. +By default, attestations include the record name. Add additional inventory fields as needed. #### Questionnaires @@ -666,7 +666,7 @@ Click on the name of the attestation you added previously to configure it: 1. Click **{{< fa plus >}} Add Attestation Period** and add a period. - Enter the **[name]{.smallcaps}**, **[start date]{.smallcaps}**, and **[end date]{.smallcaps}**. - Click **Add Period**. -2. Under Relevant Attestation Fields, drag fields into the **Relevant Attestation Fields** column to display in model snapshots. +2. Under Relevant Attestation Fields, drag fields into the **Relevant Attestation Fields** column to display in record snapshots. 3. Under Questionnaire Template, click the template area to edit, then click **Save** to apply your changes. When you're done, click [{{< fa chevron-right >}}]() to continue. @@ -679,16 +679,16 @@ When you're done, click [{{< fa chevron-right >}}]() to continue. :::: {.columns} ::: {.column width="30%" .pr4 .f2} -{{< var vm.product >}} for model risk management +{{< var vm.product >}} for risk management ::: ::: {.column width="70%" .bl .pl4 .f3} In this third module, you learned how to: -- [x] Use {{< var vm.product >}} to oversee your entire MRM lifecycle -- [x] Set up model stages and custom workflows -- [x] Manage and review workflows on models +- [x] Use {{< var vm.product >}} to oversee your entire risk management lifecycle +- [x] Set up record stages and custom workflows +- [x] Manage and review workflows - [x] Set up and schedule attestations ::: diff --git a/site/training/ai-governance/managing-ai-use-cases.qmd b/site/training/ai-governance/managing-ai-use-cases.qmd index 34075584b6..b0bbdf59c5 100644 --- a/site/training/ai-governance/managing-ai-use-cases.qmd +++ b/site/training/ai-governance/managing-ai-use-cases.qmd @@ -108,8 +108,8 @@ Configure custom inventory fields for AI governance: The **Inventory** displays AI use cases with: - **Tier** — Risk classification level -- **Model owner** — Accountability assignment -- **Model stage** — Current lifecycle stage +- **Owner** — Accountability assignment +- **Stage** — Current lifecycle stage ::: :::: diff --git a/site/training/common-slides/_review-model-activity.qmd b/site/training/common-slides/_review-model-activity.qmd index 8f7b98133e..becf7518c5 100644 --- a/site/training/common-slides/_review-model-activity.qmd +++ b/site/training/common-slides/_review-model-activity.qmd @@ -9,7 +9,7 @@ SPDX-License-Identifier: AGPL-3.0 AND ValidMind Commercial --> **Review model activity** ::: -1. Select the name of your model you registered for this course to open up the model details page. +1. Select the name of your model you registered for this course to open up the record details page. 2. In the left sidebar that appears for your model, click **{{< fa wifi >}} Activity**. 3. Confirm activity logged on the model reflects your activities from this training course. 4. (Optional) Filter the activity by category or specific values to narrow down results. diff --git a/site/training/validator-fundamentals/using-validmind-for-validation.qmd b/site/training/validator-fundamentals/using-validmind-for-validation.qmd index 5606c4bf65..32506b258e 100644 --- a/site/training/validator-fundamentals/using-validmind-for-validation.qmd +++ b/site/training/validator-fundamentals/using-validmind-for-validation.qmd @@ -146,7 +146,7 @@ From here, you can: - Keep track of your records (models) in the customizable inventory ... - Review documentation submitted by developers ... - Assess the compliance of records by logging artifacts (findings) and test results as evidence on validation reports ... -- View analytics on your records and artifacts, and create custom reports ... +- View analytics on your records and artifacts (findings), and create custom reports ... - ... and much more! ::: {.embed} @@ -311,8 +311,8 @@ Typically, a champion is submitted to you for validation with completed document Here, review the empty sample documentation for the record you registered earlier to familiarize you with what the structure of documentation could look like when presented to you for review: -1. Select the name of your record you registered for this course to open up the record details page. -2. In the left sidebar that appears for your record, click **Development** under {{< fa file >}} Documents. +1. Select the name of your model you registered for this course to open up the record details page. +2. In the left sidebar that appears for your model, click **Development** under {{< fa file >}} Documents. ::: {.embed} The structure of the documentation reflects the documentation template applied to the record, just like the validation report template. From 8b254c1b71463403b33afa7a6ebe70ea4f2af397 Mon Sep 17 00:00:00 2001 From: Nik Richers Date: Tue, 2 Jun 2026 11:59:28 -0700 Subject: [PATCH 2/2] Preview for 26.06 release notes + `make get-source` (#1353) * Add temporary release-notes branch ref for 26.06 preview Points to nrichers/sc-16263/release-notes-for-26-06 branch in release-notes repo. * Run make get-source * Add subtitle to all-releases page listing * refreshing vm-library * cache invalidation * Update .github/workflows/validate-docs-site.yaml Co-authored-by: Nik Richers * REF FOR PREVIEW, REMOVE BEFORE MERGE * Update .github/workflows/validate-docs-site.yaml --------- Co-authored-by: Beck <164545837+validbeck@users.noreply.github.com> --- .github/workflows/deploy-docs-prod.yaml | 2 +- .github/workflows/deploy-docs-staging.yaml | 2 +- .github/workflows/validate-docs-site.yaml | 2 +- .../templates/_template-schema-generated.qmd | 2 +- site/notebooks.zip | Bin 61189993 -> 61200527 bytes .../configure_judge_llms.ipynb | 9 +- ...model_documentation_oidc_device_flow.ipynb | 889 ++++++++++++++++++ site/releases/all-releases.qmd | 2 +- 8 files changed, 899 insertions(+), 9 deletions(-) create mode 100644 site/notebooks/quickstart/quickstart_model_documentation_oidc_device_flow.ipynb diff --git a/.github/workflows/deploy-docs-prod.yaml b/.github/workflows/deploy-docs-prod.yaml index cce53a8fd1..29390293db 100644 --- a/.github/workflows/deploy-docs-prod.yaml +++ b/.github/workflows/deploy-docs-prod.yaml @@ -117,7 +117,7 @@ jobs: run: aws configure set aws_access_key_id ${{ secrets.AWS_ACCESS_KEY_ID_PROD }} && aws configure set aws_secret_access_key ${{ secrets.AWS_SECRET_ACCESS_KEY_PROD }} && aws configure set default.region us-east-1 - name: Deploy docs prod site - run: aws s3 sync site/_site s3://validmind-docs-prod/site --delete --exclude "installation/omnibus/*" --exclude "installation/helm-repo/*" --exclude "notebooks/EXECUTED/*" --exclude "llm/*" && aws cloudfront create-invalidation --distribution-id E2BGG3USKQTR9W --paths "/*" --no-cli-pager + run: aws s3 sync site/_site s3://validmind-docs-prod/site --delete --exclude "installation/omnibus/*" --exclude "installation/helm-repo/*" --exclude "notebooks/EXECUTED/*" --exclude "llm/*" --cache-control "no-cache, max-age=0, must-revalidate" && aws cloudfront create-invalidation --distribution-id E2BGG3USKQTR9W --paths "/*" --no-cli-pager # Release headroom and shrink before final lightweight steps & post-job - name: Release reserve & shrink diff --git a/.github/workflows/deploy-docs-staging.yaml b/.github/workflows/deploy-docs-staging.yaml index 158faaff7a..2ade3f3aa1 100644 --- a/.github/workflows/deploy-docs-staging.yaml +++ b/.github/workflows/deploy-docs-staging.yaml @@ -120,7 +120,7 @@ jobs: run: aws configure set aws_access_key_id ${{ secrets.AWS_ACCESS_KEY_ID_STAGING }} && aws configure set aws_secret_access_key ${{ secrets.AWS_SECRET_ACCESS_KEY_STAGING }} && aws configure set default.region us-west-2 - name: Deploy docs staging site - run: aws s3 sync site/_site s3://validmind-docs-staging/site --delete --exclude "installation/helm-repo/*" --exclude "pr_previews/*" --exclude "notebooks/EXECUTED/*" --exclude "llm/*" && aws cloudfront create-invalidation --distribution-id ESWVTZYFL873V --paths "/*" --no-cli-pager + run: aws s3 sync site/_site s3://validmind-docs-staging/site --delete --exclude "installation/helm-repo/*" --exclude "pr_previews/*" --exclude "notebooks/EXECUTED/*" --exclude "llm/*" --cache-control "no-cache, max-age=0, must-revalidate" && aws cloudfront create-invalidation --distribution-id ESWVTZYFL873V --paths "/*" --no-cli-pager # Release headroom and shrink before final lightweight steps & post-job - name: Release reserve & shrink diff --git a/.github/workflows/validate-docs-site.yaml b/.github/workflows/validate-docs-site.yaml index f9a7fc80f6..fe28a4d362 100644 --- a/.github/workflows/validate-docs-site.yaml +++ b/.github/workflows/validate-docs-site.yaml @@ -128,7 +128,7 @@ jobs: run: aws configure set aws_access_key_id ${{ secrets.AWS_ACCESS_KEY_ID_STAGING }} && aws configure set aws_secret_access_key ${{ secrets.AWS_SECRET_ACCESS_KEY_STAGING }} && aws configure set default.region us-east-1 - name: Deploy PR preview - run: aws s3 sync site/_site s3://validmind-docs-staging/site/pr_previews/${{ github.head_ref }} --delete --exclude "notebooks/EXECUTED/*" && aws cloudfront create-invalidation --distribution-id ESWVTZYFL873V --paths "/*" --no-cli-pager + run: aws s3 sync site/_site s3://validmind-docs-staging/site/pr_previews/${{ github.head_ref }} --delete --exclude "notebooks/EXECUTED/*" --cache-control "no-cache, max-age=0, must-revalidate" && aws cloudfront create-invalidation --distribution-id ESWVTZYFL873V --paths "/*" --no-cli-pager - name: Post comment with preview URL uses: actions/github-script@v6 diff --git a/site/guide/templates/_template-schema-generated.qmd b/site/guide/templates/_template-schema-generated.qmd index 3256ea27cc..2deebf3d2b 100644 --- a/site/guide/templates/_template-schema-generated.qmd +++ b/site/guide/templates/_template-schema-generated.qmd @@ -1137,7 +1137,7 @@ Source: backend/src/backend/templates/documentation/model_documentation/mdd_temp ``` diff --git a/site/notebooks.zip b/site/notebooks.zip index f505f25d9071b1551448dcb07a3fe1bb093d1bbb..d9486a687559396803ff63b98d3857a8dcdc113b 100644 GIT binary patch delta 29468 zcmZ^qWmKHYwyp{8ZV4{IU4uh#cXx-z9fCCOZjD25f@|Z1;7)=EcL?qf?6O!V*?Zi3 zE@M3Wc;8vG;-9a@;O+zzQaDt6V!#U^<9&{IQ2%Jk z)PJ;EU@r4NQW6;p?H?^kndLPs$)1JtkM_51(g2IrA2G!r4m(Uf(Dntp7wlhfc){@n zC$M~=Ge-PBsX+?eu=0PJfSWK~3f{iTQj4$}f8|yNmrs-!FVT7i;e2ekmQWDLlF$$$ zpl(Y<@?1xwA}C5k1PG}AnMjB~T!=q>h(AK3KO&3eYqEMvMBTs17+Tx~$zNrSNJOJQ za#%+s;tNPGAisd}0_qEB;EqUiT-Mj{**0{Uzv03tzG1>BelGyQ!YKZ;G5>_5%3!Ph zi2&=!U{_MUHbiv5=J+GOF4gN)KBf&nwz|E6=&N}H!T--xa6zX6L`a~@ul^K&qQlx8 zu)%E(IMZ&gO?agVb1j_%lt3kX}7H3nbN-({$qi^ z(7&HQeU(E9sYL(E6NFUH>96wlrvH)&z*PR=S2@R?`foG{14xSvez2!@9Dns31Ze(x zVJ!kQ;1&T|XpvXzs~@e*YbP(KDeR@$3v^%%AbmK(tDO~2FZS2o4W|e1hBLU}zFPXZ z41ZgKt8y7W(Y(s^gAD)9^PhxXgA6Y)g1rWrc&T3Preh|#zX=gen8FZFn864qEQKFl z?HX!Uo?1+^dB~%>Flhm;o_6*npVBe)!dz*yCXQlkD#51<6p6qCp5LXlSzhJoH!_-k9&S@dc$9l)+I~DrHQsvD}oZ+J9ohK$NO4sJ);L22pCXPQ8Yx-D>Fm z4Qaa72y434d_n64ZE(}Aj^e-X8Tdz?zk4Cf@=@mnU9jb&o`T!zcu}9gDKyc z^3l9LpVyzN|MTtr>+{Jq@+pEsMOIh)-`+^wA59hhbOOc|H7!1PP2R<1_IGLid7of~ zu`!#yVE%%|3zjcff!Ua?2en`0AH%Kx&hXz?rEP?@7yGNX*KW=Ir;Gn=LD3HDzY_=N zby)jTqW$x;^}mJ>3ZSLL1=Va=lm36k9=l=vpF{PqVU6nKpyPIna9;;sQ}I3uuz!`V{JiqmHieEM}^U%YEdiJ*LiJW~?MCtFKY)N_K(@Cb}HOd@;d$7sQ2}c&N zl6ta(?Jym7{1dlGLLr&8@VR|22P&)F8wW~iY%>~JVk?6xIurSpPf+Y++)%Kh0xnj< z83{;*J)+StP9onNQlb@MMT@@?2jOJ<7b5Y8reG_}i>E3>BLUg`O}at;vjb!VuMsio8T`-y zngNr(UxFu$6>rpq`6#2mLyff_M_4(bXBm* z0UV6g>%`&|VA_QINjf=!^*33=;lLh3&2aJIGvv1-m-b7trbSB{oMvlLTI4ASCWJC~ z^tciLg`k>{3PSo}XyOM6@m;(4W;cfXxi1+#XuAKc`q{UkkJYoj)pE%9tH868mky*WRl;78P+3T=Y}wv zGe&vfAaCY+2y@viLIZW9jw%_piTmGb5@hf6+xHSR%p6>$rb3{_M^+TYtNYNXeB)F~ zv%{rE+v!c}_^`5XcXIjrD%5Hh(MtTfpc+p} zWW}dw#I_O<&(uv!rDkB0k%9CPA8hAXGi6~)54o6!>E|iFjbO;i4^X7PgBVZOpVrsA z-w63|AlXuq+6$m@hK!0}JxJ?Za1Lp;X$%d&m18datrxquT6rb+C8(@CPF08AO4x3= zXFU3}+jJfc$j^UzLt-TaYr}Re%RI>cO%ArFyXFLqBrS(sEB7nV(AU8-W=*q1E6hmY zfiEfq9S^h@2Fq0$-m?@Bf6;&!laRoSVbt~}Z){eZx*aAev|TFaGJD(+uhHoTvK;Ek z&~?VCo1R(1PrcfAQ+6MY5!BEAVM7xUL`>>AyU zL~&SO4F5%bo;CxZP@0?&Km6~_jwczNPj*mnX~Tz4g9__Irv8!Z0rS^4VcnwY)^<-1 zPu%lAb@<637gAa`Vynm`q8l=5YG~c{f7bWt9O$C_Y}r+NWTb<+dBmyMqR);o!H?B5 zt>oGj@rWEJFb*HlcIJvOUF|o1-ygK260E?Iqh_(Qk&)R2Eb>C*kK(e=4JWma6mj^- zXr9kInu&Mr<^_)?Q+DD#87aJrkRzXlMCH~xG9QL~ibwv{n$S{xSkmUS1L|v0 zNxS#sa?tYnbq}R?b8(wZoAiRcJ8J|MIe&~p4}YvJ^f*x_>Gb4CtMo@$6AwB2enbfx zQjL%Mx);e&z?Pp;TLiqCR0L|qf{KDl89@?PuC&>2>&8R1bYN58cVM}4ZHV)j%piOI zc*kd#L}V})DA}$@xblcO@(tT>;@^P`o%-RoKI9cmCMjDhtqj4 zGxZQ?_t6(c?|Y?wO*n>rpX1bii!$e549&Ak1(6Su9Zo^3m&lnsi68Hhh9(Wb7?6CVI>U#tsRt*5Ic7Ncqc391lm>^=!0EO7DBUkH1kg6I&_L9-}TZ>biws-?!U z9X*+$s^wB*BNo~8O0QF63ZvWy>@qNSJCxl-MF*|}8DvvyV>Zfe(;KUawBbo}_90fE zI>-t_O+&tyGe(>oU?TMJ&2B8XJLggOzRfvjfyPj~~9*}pxCS{oX z2sH7FEf@IdivXR$6#W3_-D>1r1N3aZ@T~QXu{yL4{cbfNW=6rBTr)$F1|G*anN-D` zP~OD*DATq7lPJhIjzHH(KXq^N(`0`vgVkI�wOsqX zUCw;q8+6qOpU)_PPYvAfKI>QeS`{sQ1hSs@hepoHE|&A2E5b+@r#Q*lETzKUb>dIZ zYUr>szmsOoyxLh)bC!6*2NB<(U8A-?!oiMu+rW;>9^+KlnH`2HszFhSUC6hbX6UrU zl&?-}B?d2=c4^_4S)9dKcxScYT_OawbLp2)(QW5FZ{$aGU@AL)h9Sxq)$_Gd1&+^f zr#~1i7LnO#e23dEgz(iXlF@v0&E!R|GSK*Ld8OU|2EOd~5jr9n8}d%G`~gGN()q(8 z!uWgaO%3^COpC-{%-`0Wuz$N`XIWA=qu*gF6t#%kQ%&=pF}3*>ImGMPz?aBir=*8? zUj_RV;$ilL+%yTlZyk=<6D|`9uhr8 za+q{2P~9c8L*Hf6(4?s1fhcBVwiERR;ZakkbX9JL8hRyMj&w;S8PBqgVLuXk?SmSh zv}cAWiw9iMpd&2X21aU+-uExb&lYj+2#)KTLeTIuoZ`38XeTD`mHl{?fZu>uU3m*q zqxH#}`w2*6n!n0x6sn4e-mVqDE6Oif3hGFeZUfRk!#2V9@+(M4^yfzBX!tWfK`3!3 zQfO%c1v8h`K}`Z3-)u()e_zxo%-ttq*#3q`Zn&bdf9x-WXTMZNCu&o*W{8j{vT=Fp z&4}4dd~ijCy|}#qmCP)Kd&>cvN`U(dets1iD8Ko_;JR0aGdpZ-QUu2A<|%wCz@@^0 z7CRxCzusofHCr}}{JPCtN3F^`82^n(v6O>Z3(dB^gz4bY+ za*gfM&9j;B^8Ox;e`>`L%a;cfLyYDW;?nrg({UM~zY(0>LpB0R6SV}wNi1(0vK6mf zdL}HTH`kIQGe@Z=vD?)d0u1jtyIEMdZGvc9x#c7Qo^(5!UsatDe)?JA$Fvxv{S>fh zEj;{Idw#PXDO+YL!ku*nF-g3xst>1L^-b*O1g(?tNscTuI6_(X1fTy+!8Wz^UK{g<32U%OBBCbAz;B~1K26O= zAsPc^e{_xn84i5Y;WUZY7H8>lXSM|g!HL-gl64Py^P~X`er^N)^jDXYG&B7E#m}Cp zIO<6zBL;xiV!cU&acg{ly&2hJcZ=UCQ$uGN<4w1ppb78c0O+V$VdB}5CNmLf$1UPv93RI#@ee$x1C`Ykx~US zB@$#SEYa$Lf3F4vO)PZ>LGUQ-OA((k`Ca{x+#a8@3>Fkmyi^~Ec={@~ShwHZJ|Z+R zfX%H7LFW37+0}$9&AVuBR`I<=)Fz+s#=cmgzFL8OPIG)9&C)%|C=(U{*69{#xU9cuzbJw3d!q^e$=Xy) zQ>9M57cJC|5;l5T>Sulk#^6Yl&TQgfqzab6WzcVe$GY*+ClU~h(wLpH_1g!)Dhyys@B!hCr0r;FSf{Wp! zSL1y0F<+o!$KSVL%MZrlyLsl^6>g$E-oWQ(c0}n6uf)IEVSnvH1MiP*$8>YfB=%4%La%1KIOz+L0SD zLU%ux1v*M=&60&QqaSl28Snbo?7ZVz_WwZs$TD`#oW%fQkp$onqkE++LocBgaJptu zsvRbW>aSu@G}Tgv($)}(h+~6HB>|CeHP}Oq2eM0`OGyAdDfCXx4AW5&{f`vRQSy6} zEQXXU(W^)qAM?w?fe8}r1(1*w01rqXR)iG2jq5*r(1|>grtrPc0!1J1>YIHB(n^Ig z$%kX6%5bWC29^??HJke-UJ~6`j!Tk$9wa<_+MUJRG4r46M zlcjVOu(K_{7?u<(Xwt==_KqMC2?h_bP;;>nam&+bI1;|en^4GtMOrntmer7+*76EN zQQt2+p_hTyqyW?i##}{~D1fA@Um$d801+^X4fTM!LoEe@ku!;wnV{KwU`359hQQu! z62kKATcoC?)NN6&@$u+eF$AvfkxKs1>d8n4Db8=Crxt%hUHJ;=!$}0jTflmM{>GZ_ z)QRTe60aXn+caiTZ15#xf%CJGHFTtJZ|{tM)vuzJ>NcY|92t{2rE#e{%lG=n8+KKo z3TXfu5Z`;0^5fS+GeGuE@1CoG>HEV(&Rd-b3#@_!d}8|iU#aS2$!m-@&qC1ZAQpT? z8v;!p@?LGY>)>5JItixqqf(O-4vQ*zEhtQDqHG_GhZ4H763qIKscik^=ik^-4XTE@ zemSCHvOWUl-PHR;WZ3ke7!c!f3xc}TKJtY01IrNozf%WANA5g+yJfom=4P=eZr)(Q zUgKJEK|meY75Xgc*l$c$iRNKZgs@2&m1%9+Xs!G^_1?f?f^F4a7~QVf1ri7eM!lI_`c?N zwg}WB13;2Ozs~a*Hl_OQFNeoik&ceFn3Y2kY!w#>QxI4>j~6h<;>_Px<0pH>Xqwjc zN#{+Lp%y`8P43`s;b*WoIzxy1k90esd54-$AY@qpG12HPn&yoMcb?#?YgL!;?f&C( zCV15xDY3*1Qe6xr`4S?ZO5T*OqL`Nsp=@aRMWcPd6PogI^Ktp=s0|GqW{2xaf~tn5 zqNr3;&LisQ*?d>)aS{Lzzr3O!Z17ut_t#jD@5CM&pQiAHC)b?TXHcCifE^Zw-4;{H z4Rjz2V1y=bBXPY3~#=<)aZOiks#uFpQH z0KWWNwxMJR<@8_aKF z^KZ%Kr8WZ&H?eX2{2F`@W&5`#0hWfgEkSyfFVJlNt=7DhD!?xVv0;u^8AaUoZ_NX& zC2p&L_9~aD*-HQE5qzg+Yy9se@O5S|92(6_bu|+LLI(5(-IfAq=U`@TYiQSkwd z?`rI7?O<={U~OhZ}Q*k190<1f_MXc^#%e08vEsL_E2xp zc}XJCceT0{$MmC_gP|0cLopK;hqM_S?}qkoL= z!Yc3rbU0K&r}sOsSZsWxzdzcJ)Bnrx@WD_8!{ascNB#>n{9l$~8|;4YX9UlSf}O*~ z5#P!!P19F?pXpy#92vj~3as+Y6Wp$>*x&h7uPy?RU6siet`(v>{cIo0|p~(oG;^vA)>cL;6aOf-X0u)Xn>Km2&Wgba>2f3Amgz;Kc$255X4Q$F(;Dmk%Bl|p)J`%N);tOy-%GulFN#m2 zmByLhphXchze-rtKVwA*ik@pS3U{%&B)_Imw#lP zOmP2`Q?fBITX{{iL%V=aENj#N=<6k+akag$dO)bBi#1>oU%Yw+B09Tc82&LoA+Ry8 zcwnP_Xs54l$MA4zg}$pj^!DQF`df9#*PB)P_~LmZxL;J^Oa6UGr|~zxVkwm>&Lp{u zjnjqbwe$$R=!P2Z+q-PhNYo{CsLp$RT5m;jZjcS6KbqqH%p+K?0o7-;02y*k7G^yY z(TuVzp3S1uj!FnyY`uuRlPT9-Bjl4+IhdZI9$@%1(GC!aV5IZwoNXy?0Bv1q-v-G) z=S#=;V|?GV(5CxBQc;?R23GdrF8Aacnd_9oD3wYY4C1r}mPq>mKflEdaUe+85eUw7 zWtEJe?x&dAYgOc;@BqcM0cqMfFM_f_?d5UHx}6C;4#yH4woe6+o_Q=E)YwMW>3AR@ zW|n9y&Adl$ywYWQ5##A7&gMuu<4m)0Se1*^iB=rQ$*=(AJxj34DMi9U_F7O!4u|$O zBMhJ+HhQ1DM*Xdh)(QU}wT&Iy(+(j>iHy@R1CxLjY!?oxL&`}5e&aFWc6fx-#B1TB59q0` z3@IBWTU05X8CMv(QQ5MOc;q&Z-6Oyw*WL+bI4oDf<3rMK$U$#i&)-L#p9y0a`En+I z)~K&G-+DvaHT=Wp7&wEjV0xD4gx>MK@Co339MjJ2;BK*ciJL^vn>~kg?S5BMAGdLP zd8B)``S$7RvFz6I)7kaj?WI^&7anMLU}LIp==-*{W>!|z%`RwT0L>Cqvf;3sD;Pd9 z)nyP_f0M#f)30h&&~egvaB|&6d+>=i;M!1tdw%=w03>kN3Y1>+7gL{}y4-JT>6L~x zg^fUnqb?wUeQR8!kXVASKA6fK=)&v|jQUF3+g)CJA1nXEc0s#A$?`tCg5qChwoZ?8m7(XN~$0kbHbM`cK8@{>?{ zPpfFqPqH?CDly~&RIX77VB}~Eqff(Hm-l~@-E{#oDY1C*mf=iU{v#FYNwv z7J;t#K?WCSDpzMO=QQIQT@ z|H_>vO25ijG^RE6u3y4z@Ls9M?5u_y{-Z+Fo>e^$>zJ9bZUO;EWyePoSuZbWu=9Iu32y zC>j5`*hN)Mz6)}8^ljN;qmc~Uv9$u4y&|d!Y<~z~cIPB=1Rve2@CVmeK}rmkqaQEn z1l)23u$kW?E&sb6oA3EWzt?QG+5TI7GT5wP=S{J2w*+QYphS$U$pUNHp@MM`^i%_4 z*hAUp-nyWrajepl>)0iAxHjE5^;I~Db52%WbE}H_giTlc2sMnvDSlTmzigxm^j`cx z|C90J&_^ndN&s8C0`c7`UGJw)5bXG5;zTw(*Cf%BjLLx@Fk9%m80vUx86Tg% zg^@rdY<@M%n2h6B(+e{96w2)dYvbvs^!iRr>rsnrtQcs@F{o!R9J%*p&IbY`E$w`Q zfL3D3&E&|+rkX%Y+%mOvQ$-Ifzaf|%KV}-(^|;Q~6BTpYP1=TgBcNyA!_m zA|s7HTtK*@aOAs-@C=Lkk6d!OprecJ@AL5M7lGf@Vj2%g&`%$#kU!xfmN#hf9|v2c ztvk0>fkU)02bFH5t%mO#YO)3)i0RbKf%%q347O_Y6RoN=9du(ME}{jri4>qk6%4oN zu;5T9#Qg_8R~1%t7#&NkgEq|QyUzS&rqD?Q=mWC#0}0{lA81st(Vf4~wyh4fwalcl&d8t(4+a>$F_|u2%Fi9jl&H(<_5!jp| zTAQ+jL=?1dVt0`TY^~okaQnzP%Igu38-vS-^44z) z-T7w^Ft@=xZM*e_ z`#Zu_2}FoIwsuUXv*dP_8*>K;EFjrO3zvD3NW9y*QBzt8bbP#Tpt2Kr)bM%sAt(Ef zrp%4+XHi=!74BMSOD0&?w_LqFo~PM9(@e}x2qW)fJ_~c^kn^o9{=Bm{{1kW1_Zap& z8=%~=zo=m&Lbm`QiY#Dby|l{+I;7}Db4D#wjF7Mx39f*!?XZiL-W1D#a0QO2MJIjO zxIoJGrl?zo0PfrpDDb>fCER^LDn{*$HX*6Y?=2Wp=Qr*^f+?q@2KqPNO$R$BGBPyL z_eng!zd2#Q=`muWL=hZ>6NBaw82NC|8j&)jAZHJfa>iFhK9}tGNSltr&_nJ@{9<@c zYCdUMlFGH76NGJ!D{7@YVGP{O$3MLndV+Jn@|SRMgIoq=GgL$*O|M`>NI1Wf7l(qn zjF?nXNa)>d3e?|u>=9RLMuA^GEdb3lW;u8z~{2{Nm8_rv!tj(dTz8D_L-3I z`xW2aDdt;t0%RT+Yr+Gojo-RMmEhj;Xa>^2tMQ>k*Qv#tgCa`fj}gZ^*>|k$d(mJCx9%cAdOi6tNsABWux|hU`}4E`SyExP-vrbXv{IWN+oo}d|arS<#s%%TwQHVyKW$C+`g4W;lACCxujG-qp7>kUcx=<#pLrjq znCj)K0aWr2x-v?@?~UhUlsN1RR5BG^jF<_c-V6h%z0(brmez@pE|RTbG_EOq0y(8{ zlHTS|TVKDlW_Y=X4?CN$_onDhBdBD$R@63D%(TXr_2xkErGYw0=JONf9OeO)rgg>A z&7)@%MOx0qcwdlNFFB2rSgDJIr@wq#$t5Fkskcj)EzIBBoXYhdhEjmQFm`M};5TejukxyOx$_skI;Ch*u(r8@8OG6(BORMoDTiQrAu z9{zo#$7y41RAn!6NC-CjJD|JXgI^Gg;15XxbsAYaV3i$nhzWixsju+m^uj~|O-B02 zPjdFwVgHgPH@7gt@Q4=k8^r)Z`jAz~@FXRI@o73|pkJVx``qHfPXYYfI9YR&w86-_ z)dK`6lHYvJ9=mG^jYm&CHj6QgR?mis8I01tj^>gd_;trJNgEPkfp)Azs3#t1y(5|% zIZrbu_D5-fPv0XQVIvUUGoR{h`Y4V#w*6NT)4$$)iu((Y+ za{;qFOMc~j2KZ?0EdDN?h(4d1#-%PsyVjh?@04Rjqy^zItj9bKOUrLye+nbF6i3V} zYf`03J90+4^8s=Dw8hst<|`kreM-gz_NM3leZk zykRvZa}~xFR!MRV9GZ`;m=$jMhC*_YZH~?ucsTvj&TYM1LfyV8Pv|4h8Fv5l_00%b*%fDpiH#o8KM6RgRsn$<33-{+? zmE!j|p;cqU9d*^U*mxK;_w3YriQe7|i?}mR@|D_Znb4H`kyjHZ`|uSXE}u35ROcK< zS<6OVj^>Coy+1a#q3WKqnI>6_qOXC!r-^*%Fx6~y8%{K$pn^U;pFQrw+5S-6EELuF zG1$6X^`Yr7-JGYs|AtuO^({YR@IL7xj6%P)Ex$x*l%^zCAUuiuzs*#9ow0 zZgvZ!yP{q4tz#wnL0d-4s(++yF}BdFolYrJT+abUs(0BH`}wYA8^s_wyOkc9aAgaN z{!_V|*YD$vqnAIv%4gi!^?_Y9M4blPY*{N!6K3c z4m*c)ex>hm`T}+CDT$Qg^Oe@OKykwPkHJBNZ1q-T@-Yv#7^@#qI{5$&L5Yr|0c0+T zCk0J9uu7)FXhMiAF{`+ZWBSOIKj)D%n0_|G*B{y1U%7MLyHcio{WM57%BkG!mLGm^ z6#{i=*X?MHNKRPF$kjcG(>1J%$ys_QOpq>=WleCZ&bq()e9={t)5dj`0E7UO8>H=0 z?+)ayWOsJ;1|3AE>tO}DA90ep0R+0z&u^y|5I(!e2Xwp7{lFpC)*E#rU0p%RL|JF4 z=bl~bb)l(9^$?7QLQ0B7_h?C`HSP}Jfj#fhexg=&Ph z7K(MhcXY}`dsy_sR2avdtbi$0VWtGC3TNM@JCf!b3ib(GLi4Le$WQSG?8<(m^YzeW zENL{>CqgVKd|o$Zc>E+dn=>@mEG(GIk)B_SJ*Q9UdyC^pqTtl?^Gf3sXN8pf`RV1? znM~KGi-G7#DHzH643{1f#fuuszOTtTj(gu5?PtnH<&0pa+^uvnDcgVztO;)!7%>l` zTr{RwtsiKS3i-dRZ8;JeH9xB_`>iTDDKD<1KKo02h11lVjhw!n4DTerYbqVi2F{4t zQ`&C{Y}WRXSk=)7mI=Bg$?dFU4&?LQR6WtNc;F$WCn2RpXG%2hv3Lhk2RlKhNHG4I zjcDEcDr*{ZC}>f6NbUwa?0|M+8|DpoZslG$w)gXz#WksPY>^fH~WJBufZJck& zJBl%84AV6vHsL#VEuhM$E0=!M`YMKrOMM_DvWE<<_?H!tIjLHxf&N0{f(k1dJP&DN+E^^Al`13eio*SJH$o z@Z!kr*FOhZJDq+Sn@pTAsHYS6iI$(1g9MPX5Z&<87nkV)CG-7o1X(Qb)d>sFSGf}C z##!ak-sel_iJy%?!v##-NxAnz2;KHi~uXe9PB0su_{#wSZT{$LSVM4Un_pU8TD&1iKRV)n zxOKW(EvRV)(ph5_AAa4F*ZbFO~I@xp#EiS$juwj~|1?rl2gC>_8bkh*!hS4x`Ci&&@IT1NZk zdXFwqAP+3kWb4d$ty6Y1&lY11S>TZzPA#FKdCaE(9IkYEDz2(oK;T9vk93bvVH2l| z6G}4OoWoC)kNPnsba(1fBXFt)1Qu!Y&pR+~?WpQGo!s2O!j&WFn|GlJ-ah>9`c`Ny z@x_e~o;}~pxg}*IhVPc{GL`(@LC^Ei1+1COu{4wpH7FX4uLaJ3DQKCP-oJ9B;AoqE&tQudBr?0?%ip;|+br_HFL zBc0rkXsmDRBZcdx*W;u(k0K^RSgCyy`GrPX0}9G7d}XU@Pv4LtE|GfwA&lpgR7FRb zn-h`(cWB*lCpZ21(zolwDfIEN%u!8@S;<}t5QNS*@_mBm!SA9{=>Dr$u=9K_nbczp zY#D)++A>%D|f^I3akZ{<@A<$h0|(0t4>6n~ZU|ev})u z`Mk=oO|o~A{%*$Rrj_AZh<*DR*h;RIJb~xu<1Ws5&<5APKu4@8=JZk!%OHh^(zf@;Z|6POT$Fygg=d z_%YruaO+SkR(VKMT#7$-AA+XgS;XH;paZ#tWAl2HPEx#Y!c`j<(R7kTEcF&>BlB&p zb+ZxEhgr7zii6F9Ayp_t)AHjt)iPFc+7q#u!<9rQ1S$|{=fLa_4tI;7=MN{g7=9Z* z7cS@9D41s$`p*wBra8T%DrI9sFxVr?(gI0g>4ywyc$bC!?S2St7J<}{4lRytR zLuZ{4@4Pnyj_7}z78^kI!Jvc4wDW*JFs{;f};uZW6M8~6@ zI<0^E*_QI?a8wCoMl$Efx*RxdZd+(mLw+bwe_?KFQfC=&lw+&CP?a?sXaKB!xTcUA zPAaYxNEr}l;H`WzTe^B2MrGt=-&HUvoFF1jJH)k>=>?yV@>jcoXUzHM0d#dO5sgswmWK6OJD@gT^Aem{uMu zB^am~l7Q2a1Z*y}71#yiTU>sPy;~l2Rh3QjyO)ozP~EctOI!fs6b~%0ZME6Te093T z+$c*19yt;EMbH!} zJZ<=~tC+HWWJuIw8I$?e*P$@tQ37WIX1Y=O>~a@1V%qB}hmNne)sQ!T2vXW0PnVWG zbjyo`UB~vtbAAh93Fr6;1A`LQj_d1le@h?UsSl62 z7~j3$L#4rL**?(z`kp$^+I1BcvB`=PnpV*3Q+Zd0qbRc$(s(1!%Qo9a{0@_WrRWrp zKa%{UZW2hkm>gQoUUiG=$(Ar^L56&2Obh_iyY4YXLPEDiKjv9nG4MA52khXEd$ZlY z==Y=0mQW}Wk!%17LCEkY;h~nEF5ZEO&=K#!)ckjsXgpqEY9S&raWT1!fIVbXSAN6e zDhcY+fvCr^v@MU=FD1CM_;fF|=&IxY>xkYcXK;ruT;!P0`@-GIaQ%jV* zIVoA#bz`&i&7|9F$PNll2IE9oi8`Oox()e#Lp4ap(tfrr~1@8di zzy-sJ29&^kK`ufLQ*Qs7vZS>HxUymP5A+RDFlmCG6X`C(IQ`nz1A8%+_4R6BPfTvh zoCEJlRtDgz`}uQTPTR30`e?JH`M0eKguw<%6z-wozy%@*b>#*wWx*Bbx6@eKIA8LU z&US}gh~|~nVZO`Joeejlui3-t$<-WL0qxhT%1waPyqDsOFyC}JgIbHZG`;%X-|sA> zmlTJ&#d}D{HA(xiQhlh;=(XJsx$^KOv$e5i9YgqSz_%kBpCR_4c_NAs+ zwv!f+7sEi!B~b|M2C0;p7DY^&BsuX$dJ_29!iym;Ohv(yL#-jVJGk59xpWa^QO)aN zZeh6iPV(-S<>L*d?AD~Ti{!T1#g7!S7ETuh`1ff&i&*7HOBFl85GY6;lwJ{w7O_(k zKj(ggsjVl(6Z2I1+7Sp|WA}N-WM5O`#S*Uq-J;JHUC^5;`({~%nQwDMg>x!wSLVXRxn288UCdjZI&k?uGa}qW zZ`aGTNK~D>M~PZzE7^18mk-6gGoH+{v}oGWRv%U6fgPRmO~He42aJt#*=d!p?O zq^Uy+HDL{jimjM7HDX_~s9(0zS3uKe<>OndYix+j5gu)1E$3X#m!4ve-VEN_efe;C zMm-6rzmcf$X5yTeDQ|6xKtf|NzD<_D|30)S#UFiher}<-QZ6KVCII?i{V+%^PE>vP z{aJ^haW7{5(wNvZrRF(&9tgMYE7rYeo(R!{I+b~k2$3p7Z+%Pd(k*INS1xhDa z9Vvytjf!bJ2wE40(ka?K)h|^B@Zi9ZPOuX468tT2BEBdi?Ikyr7zbaoO{65e(jLRP2-u~RHc-iwOb(U zd+1i+iofVoJ$RZf5zIZYMiailLE8Ubn*H3mD1Lj5HQjbn*>oU5o;T zs7DGSh%S60x+#Y6zy{UF8Q;;eskM@aL?;6*O_-cC>{=UZU5DCP1?Rind3p8eo~%p0 z4*PPB9aZxFwpka>zQvKHs=cGyn%yj=Zsi3+Sbdmo-C!jGYB}`^$^xA3s!qE0Ai^54 zC=u-$NYz?`ad6=YGd=}3u588U9f!$IMy7AAOpH8+FdG(=!2JjX7a!6S8=E-ZVj%5f zRiau4Xt)*x{oY|&V?4V}iWbzOWNv<>*{zOPPldw#r54oM45eu;h`YED!%zqRLH}d+ zsFOEDy-Urzn`onTMUXhmV0ae{9C9hkNo`sVzoC+ok;12(TC)Jz(WT#TlVgtC%q?RH z-MP9u$TyCJKqn(v5hh;MeEg7zso?_gid(D!v~_3i0p(pd1C&9T0kZ&fIiE{*&MX4e z8`z3GXP&|In=PHbmTE{g_T#qWwR7QcJ)5{66%*+;(GsGTHTx#T=IJyHB8&b5S3K0y0VE;wt}SKo=_!}KK&5&mWjQhxv%r)%3xa&Y#F3nl z{_;`@>t9iQHi9t?V(Hw(-)x>5ZIy?jt!v%gQ>S8fzcA7m=HEY~ z$&TxhVGgQcft7ch%bu>c@yd|)vIJWiCEifP08jBH+d~j%9sQtZ+4bpgAcDRmlZMr* zd!C$*`_ou!&igq{W>}o-Tz2eHj-aq)RQH({XxQ00&10Az(9I z0WIv4AxYsXro1Ox+1kEmt5s{5Zes%=zxDO|qchN0m?V+@y53 zRoW4!Zn5I@!#Tgh;lN3f5S1*#jAnbu9;#VJsmVCgVr9;r5aR8dCf4MS%7Vwejy~wl zPseY`ORZ_P9?B*%M#hG_D0UHMqSe)vYk~arrCazA8TJj|4)T`|44Zn@vTAn-&B=-}`-@^PK0Ld+v=hPp+>520y6kcX63f_rbhB@1+%^QHGf=nzuV>0%nu_vJ02OGe{ez6F@tcy40qL=Jz>;B~Izfg+YBCJU(VFNzbKc#vP{&Pu@1=Xh%=`zk+>7 z1qT=T+C*)d`Mf0G_v)k_?)A?VP09`HD$MSn>r}h&PtE@D9gns?wZG7NaQI~Z?1!zl zT(I~2=lyh#C9mS&{V9M zz{u8yVPoqwG}^KfKn07ye0Mx1#5^jaya>#!J294LZfU`n|7{VNS2f}+kIT~HFBL2T zBh<-AkRNBYtO7&phe;XV%c$-Rb4)vOtBm?t>p6G zt?JUagLZ^@*y>eXl1FGQ<1VwR3tey;;YO_1Mp}rVbd4X&r56^fZD;8bPcJ)QC4=lI z1Fy_==C~7q948d%@@flF>DsEo-B@jXbFq&Mc#)NiJ0OQcKvqE0I< z_kpyFUh5;J2c&B)X9SNxy9#;_9sNH->00a2th+NWw1JlHsuXK`I8_t)slMc{Pn6cIpu0&EVa_nU zqxm&9TQzZB3pCG)AhmTUW{qe z^-1pRdHVjncTG-o;<)iWIj*O0E>>HM^;Aa~vO=%1)X!U?&v0VhIArL<%xR%$p2^j? zGDBY?T|Dsz_1!G5SstG+I;fw@vNW~JgL3o~t0~q{q#J{B)*cIB9akw{JJ?bdQoMF~ zNbx#~^%Mfd28xXon%d^vtP#mTBn&KFR zk>WVTHxwr*PEwqrI8AYeBA4PU#W{*^Db7<|ptwkJ$(WsX`MPug$K1ax%`!%D=l*4i zD-_=u@7%vyD6-L5mQ=;_0 zLG<-7NgpS;;J0{~;-#g=8U&PSLMv6(HtGUQPkmW;U(7SfS2UgabJD6&h83kf;r|qP zzrr+Dy3ldyin-aeu; zo`d50VsC_%ZwLO-tq#YsveICV4iXSL`q{yczp!kX67R7CfpDBK!xIJJg0a;QB3iY;uXbfiZ>J`6u(paLGdTWTZ(rSe^LBR@ejp&ivLjjOYwo? zBgLoG#(FCq1+WG@s01p5DxfN;2C4%aU<+yhJ5UqY0|(#;oPaZM0j|Ie)B?3Z9Z(l& zKs`_&Gyv|PA@Bf=fG21Snt-OD8E6h#fR>;Y@B*zt8{iGvf_9)i=m2~`N8k%OfzH4W z_=5n@1#|_Ufo`BX2n0PqPtXf|4tj$=pfBhL`hx*rAP543z+f;030kzk05d@(m<48oIUowm1@k~Om=6|! z7!V5a6^LLP*bXwm4v+;5ARFuiyTER+2kZs=z&=7clM!*v^22DUy&+#U=o-N zrhus+983e#!3+=qW`amC3(N*{Kopn@=7DH1A1nYdAQmhHi@+D)ORyNkfq1Y4B!EPa z1d_p0unc?!mV*@_1*`IMTA%}Zuo|oZ>0m8b2i5}tYycabPQ zg0tWp_!gW87r;eu30wwOz<1y(xCXul*TD^70(syj_yPO~@<9Rk3ETp=!O!3qa0e8E zU%_wSF1QDZzqf@H_Ye{0ZKIci=DZH~0s< z2mb;8f)C&$_+(ADKc4z)TPcW`WsY4u}GC z!8{NR=7R+w2E>AeU=jENd%e*-fDK?H*aR}bX0Qcp1tQo6wu4MQmu`Q2J@%ux*wZnK4`Dw9+ShK7 zn@w9jE%C#l_-jBGZ&|@Pw_BGizDq6kLj15SURseGEgsR8uoDz|yRf!BzkVf2Hzs)wDF3s>d5#~b)@s@y)xO;P3A z(+cNPRk?2{H$#={HPOJ&R^{?3H(He&G|9kEWI6e0Zc?s?s;(m~D?UM$Gg5ApD%Y0g zX__kc4du#>Z)*9moS~fhczcDH@9#S0q%q59WymxGKS{OUe9B4tm5*&aEeh_36~W7# z(G!#lV>!9be+JGn>g3#N%5_z>eM7lns@&8FoJ-|C?o)2KDmQwjfghpDnJ70%mFpI1 z;D@Smhbzc6nN_}SLj^h8*_eCf_?A&FSe5&yg1YEA<;VP-a${Lrd9E@x;_6R~GVori zUQLt>S5=RiYv4z!>nOKCl?$I|;A2#|kCam%%|d!-O8vWiE|!(pxPM{b=cwv(EID~z0>3oy{Zw`PC>Ns2wWQ^%)nh(FIcY3%f1TnC{O78+TPY`v zPp)emU;awUpq%=gw_jr57;oW>^3T=I7_AWmfd<r?_VBYYkEy%{mh;fwHCY`BC9Hj*g>z!9MH$dp5x+Maa`tfFB#g{q%>r z^4Vom{R}ViCw4~)fr*i|%o8DBJFBwMxY{0Z!b$_*OddIFyqj_we$t*uVXjlOhp_cd zjE(r3zF{?x>pbutEqs1$RSNw(i3=@m-&t{l&;6s(Ho&}N`84IEqVGq;+3OzSYuaq8 z>bC4}q3KcDEiTlo#IuW}X2SHL(T&UZox2Ks2N%V#af=^mvz6R8JG23GqBQ(FKGt$* zt;_dQ$ckb+#z{r)c5#*&WCy|y-r+${+w{g9$J0twpD7ML^L&CdfK~LZ&yx*6o_O{h zmf!z0K`hc4_=Xj|O}f(y8otW2(h@PAwxw>^-r3uv?C+}aQOr@N@!~ydAbl>&^WIfY zomz)YVlR3K-ltM*1bhFa`ii4x2Nv0*dukNjc}LCQxZa8byONtMjarx^B)3qsWP@rk zH~V^@0(vmu4aap<4C;8H7`58KcTt^<82T`nQkp4-!_M_x>P^G1cxbMoWA;!~u|dzP*<|u#4~`owa9lS<|NF;@ z3F!vDw>)SzKlh*ZiL#>xUaQsfQzKq04M=Rd*8D9^KJ)A?X}wVtSJR6c_QhI$@dk3% zPox*!*aH<6T^io&P}XG|>q9ETa~E~XAJ)#)9J^cei;RxZ-Of{SSapw``4)^Z4b# z;p@4EkLC2jM9(}o_OOYIyfb!vpUyk+_Ow4;-y^KJ{)*8Rlx=1&Ms1-Rn7C-Ofwxt4 zQQ5AFVGwe&r{k z=Zu^(J=@^{m)hnjm-leLRo?hwdR!{-bB^n)*u%#2rOi(p47`)*ybXQgqLhzu2+LKy z%?evqbd0&IB0{PNF28&3`s_nbmek{3uMgt<6uZ~CtP@MN)0dtaZ6-&@D8H|<>om1s zNSn}xR@}#RJl9jv^x0bR4YlmAI&~qLID+$EAI2H!EFPxK`l^x=R`SW_3QAhiFTuI2 zyeG5swqytT>AClsrN~D3m~^d*;^v5WRZG~&21JJp{s{>7|Szw#3izY{wiWuJ#$I}2`4Oitz^dDqgHkMr&Fgzc$X zS3K8ZXD*WOBT>81Q@!?bk@WD3`aUNe$1el&a4YddKgz$=o46bO_zYyPdx+a;v$^W( ze$GmQlBE(MWL%zIMVEN!9`tmyv`pMgn;lg1UcyQaYD;(4cCX@{&MuY3l+*B`$4g62 z(q?UW$c^xYcEpb}((AUQ&zf=*|Gi%pFKo}$xr#Z|&XygGo7jJ!fp?YrV7p1AoTgIv z6lMKc%C=Nh*4U5As*BB)-D%sO)u_%`rA~XP#v@kiBATeiLDi4d0raC+)YyvMkX2PK zW|a-Q8cKURNZUSg4V#1RK3lh2)6e#)6txs@`k18#Ui?UZv0KMWI@#(qREG?xTm>K0 RJL$iiV2=Bae(dD9{{r`kGL--T delta 19416 zcma*v2{=@5`#A8!P_pkt$u9dIS<*&!Ds8e$GL$SyDQgmCY0*$>C_|C0Vk}8gXt9(; zS&CLI+O?qm&wZwuVfwzm>;Jyj{kd|VXFKOS=ggUTX1?haq`4!5$LI1n+YvapG5FuM zz}mxnvNLtq#fiG?;w0}811SRJR;ti*6)|LONF9~aQKjdOuGGPckverdrAh7n++w_I zTxMYe)@F{j=&YyL$&8&TnqZ=vWTMN4f*l0`1qTXFs)_Clp$zDROGu6W-14N~yTTYlPmG91< z3EDD>t6ULGQn@08LYS&@MdV>B6gh4yx|&{s>S`-mqKH@I4i+__r}2Z0@5a)s+mp3B z*sz=BY?vABhB)b?`CJ|{*C7f)mYaE8leUh;Qn!x8J=lR)GqE|DSHq5B69f#?pHeLa zRg;)fjhg8h-^GsiU;SEa1HJ#GyFT&MyFLjs-0@9vRuZ%$14rKYT0+VkJT2%e!Pr%* zx37eEH=e#}EXg<_Nslh>Ly?GLKZ+z22dIxOCKqzxMemx=W%Nnn+teXRouDQk*TA<8 z5a%)uQ_{!h$tY4#97J)5`tf;cO9WnwZCK8QK5SAWhrB3t`>@<5Hm7aVh>C@D5eQ6^urt>B=K~2sxEW?mD6;o%4vEp)A3CLGWv|>lRRYfsU9*0 z+iv5VI$j%S(tBpiBH0lG6MB+4_UI7hOrU z=3@K?aH0Qba9jZoR_ZH&Ig(b^2S||xPg4jBr(F9QvE009Ha48 zs-xT)HPIJszZLd{c>mC-aWRZRknmHuMVe#5e6a=H*S_l_YdUqwuB<9rNkk zhi|?z+n2GKswm}aIv+nf`0MLmlxSxshC(6m|Bw4o<5FKoyc3v~;>&ys-r|inXl-H~ z8n~N?B40j(SG@{0fB?s)%$G5${5@1B%59S11 zRv*AP*VNqlfLKKVcx|0Bd{UgMqByC1I6(G)j6H2QfObitTOM^{IDn6K?eLBV1k?MY z#*YV>=G#U#Z)wix42^xt_*&Uq#Ev(R_8aH*Qh)1 zN7RHD_0M88zny#4DBtFc{iz5LJL$pou}lAX_#Qi+CB|nu9s4CbPrRwCCyY&G?Y%j# zuKs;ob7V<{OW$#^Qbm6W;__oDcj9z6kX}SoH_fS|c)N5*&lyo`OHU`{QVv=xLPySMn5LhQd^+SdztP4l zYApEX-6sLPu1?3I{MPHm@MV147B6qJtU31iz=41{KE~{W){gVDoR2GZPHbFr<3#A^ z8Jm}GE8y85{&IV)fr5l}Qm`b?@Sc57)Mk@;*V-qVkE)rl*+@k8tEq;bNYR<~GOs;7 zB-2a7ENwBjqG`Hzc<;HMX5pO21u3rE^9+;&xtl^aO>#wad>@P8zheF&W+)=q{m#sS zGINzczNVQk&ZuQ?^67n{VimAMFOC1!S)ip1NyDG?d*#SukC%3wKc`+ zqEBi_a=@6rOxsDT!P@*v-(^Oeav>9zSBWM*mu<8*t^UN7{X?SXjD3w-pn{`i(ho%m z#kI%xNAvFO|L}pM>3!1&+i10#K#C} z7JgoRahGuO9K*(^Qr9l3D(&U^x>o&&=thcv-$%KJo~qj~`SSCdJe27w@QGgjKyl(k z58=tLl~El>URQJLIZc>|EFILlcT0c2jV!TZeMN0Y6i?t{u}=P3zF!w#DvgqNbtAlH zGg_k4maCNi=cDK27ix7&XDG$JXzV)bqrW&t*5OWI^@@9~-nRMjm$_FvMRpeG4L{N$ ziTNB3HCtvSSTFtk{(4pF8ot>- z+@2&5F_HP{&*Q`G1%sX|v=u8V;xzPI9r^~VJY$x0&-T7pB)}u8sTTZg*~8-c^S#^= z9@cAPurzpnaORVuSOJpWDe91Eqd}QA?a6UrNH5b*0I5X@iC%#kFMvg#0pO(5Z(_=d0Qd_lRWTvm&zHr~XFr(ppXNrvpUw{^R^059^Gv{8z#<$sKGghx7XI@_0Isfsoq*Jqp z5?adV{f>-$`oW?zn)hM;2x%~S*S>7t{R!Sr<}K3szPWr+V8gFfG3`=9W$k4C#+)BJ z#W6Watq^`bQCFcoH6en^-r7=4a-Q-<5oZ^jOu6J}&m+!vE5q!p>8T+t;`76Bo*C% zod0vOtgx~&a_NN%^&K}k-J14G1{hX&t-e-!**~f1;X3ilVI4nzZgk-u9alfsy3VBg zMyFIwv+LG&h4-STjIIlk&OHwjZn?^L)z)BU8|47W%k2E!yCnuj3+;~1jfw0G>R(i0 z(Jz`XxX;zrVn>#d*dU)2CZGMKdhW4Ro-cF7;@U;bGIXdb-pnCY2+V$}y*#$dxV^|M z{o&K76?^T{BfkHtn_1WNKFMZmyTi_&2#1RU?|k+v@7p2uWdE*@y<9n_3&socNIUdn zd2ftn-8d*_;Jnyn-~DyNs`o-GC1P$Fg_qhF6<*(TYEQ1g%+xm{%?tH;PoZ@Vi|=u+wlU-|7a<_YP|Pac|xoFEAU; zX^7Z=QA6eGRkJ|FRwKd&|!>b}q`7*@*?|3cWF( zN{{b2cS25`G!I)Jpl#>$bH9S$rwbGGhs=vr-P;qp_pRHN<#|@PhL% zh#WvXsV}K0D|J)BUhBcff@`ZwpHQ~9R+ZdzNDew(y~27%@5k5en^zXdTL?w-&grr+ z=B*lMi;0<^w!ySlX|U8*v!>d2u3MP*P^8@a4Q^p0sW;}%Zhqsvr=&n(?aytm(#k$X znQV1%DRtAXmvFkMNHl6*cl}&^ptfE>ikQdHljD1y6>{2UT^G$wcQ+pWWfHjJ%Zsp_ zGj{?@+ho>TbvgWOwBu4$KI(kV_c~z-ss8ZQ`QtYyKE|w#x;!q-W#BIPQ(!E-am?$_ z$L+5^`HV~5>+Rw_doL}y)yiYOckv3H;8WOOUZLXYv&|hb$D+F`O=mR87*+kq|12jw z5Fy!a;>u%{CO)TUciBU&KUSN6-s;}ptvB#YNT&RC_}I(gpMqPc=Zt?pv-v#{5Nxot z&SN~qwkFZ%d1?%wSQ)8m-HzvE?j!c<&yGK=*5v!VSNHVl3zvO74379Lc-z)^{`sK^ z-E9V=^WM#k92elt`u>nNi{vJ?zOS{n?8Lbl_6J^`6do|F43%DIkyWAirK%;mLT6Rm zuyos)v4r~TarRwXYxN7)++OnMrlxSzlf0)(c}$ikRsZTN$T<)kbN_2&mEE$37tRRW z&c=RK+h&e_8dwCGiNALS7f zb9Y_znJrt%`JG{k`vP%#xT(hqL@u z%kC>koG5$n-R%^IgQ?y%2SZy+r<$2jnnxr4to_XwBy?a`&Y?qxEM}XRV4f3fxm!+K zm6!hBv!Je6YipEa;-lk-zrFH1Rr*~?q4%qh6NxqagEUTtRSa9G50S`POLAH`nr=}Aih4cC!*(}TGWZY@)bYU z2d|9dXuHc})UCguETm%#sr!>r(_FVDJ(=C(muf%P>zwX?D>)J+9x|S5F*;M-`)14& z`7=BI%-h)L_xkO}?K@w8|2{9WpuR!*SLilLbD3f8SHz#|c8$EPxiQN$Y1wgY-K2K@ zL$xt?ELPjsk9?A^(GT&pl=KiUR9^XbQ^SKrF9PGeeYQQEu{vBeEA7EYm%C-Lg~R(V zO{VBbh1Kg++T9K(JWBL^k=`I?D*Zg{6Eq{?)V)AA3>f!4i;`Lg4M{9m% zzkT!j!DG|By27&XGusr8h4$SEe(bxNZRTQ*_AAvf>&kk9ZaQpu{Y--Yh1d9wFE=_@ zDhO>0i)&I(l3m!Gv{ZKP<#WXmMlCN*PI0g9^0hsdtIzIGMY7p6nYb+RRPTfE?L+gOl+!)>Fh^2B>i$Tng(UyB4^hu@wIk+ry!?EtRRAL@Mr)Ja%^*ya z{O0X|i@{ciRWlbHt6-CChxH&@DkEbdlq77J_qs3jJi_`WQ)pnogB zeZqsoH?vk4b#h4ZeV7q(oyYI8)!ww4j+cD{D>;fs52wfnSlR5%|dt8S~pmcBOdczwfO=euoL3EB3Lc+1Y%8(qoP zb?2-G%tGWiS608sBYp^fWKveY-C=m>BO{#j(L;6M7^d?jdGxym0zMp-+{Z| z+D#o&iwJ4$N;RSmt!Y?z4POP zC1(4Zz{pT;TN%p=YsR_XVM4 zpL0|!`DdqAoqZ`ZbddekO)me#$A7%DX^nc3!sZ+Ewru;_Rmn;v$&1>LW>-Geb>5{e zQ7aVpfRmiYe>bxEdGEVXUBbxmu!Qs4r}X3uZ?EzbF^$#Z8w`D2UYN4$-5Qzf>pHoC zS9nhg?SGYbK3>vh{UtfhOM;^dc)VZj34AHFU?A35ep07xOgl#|DC=Owu9ew_YsR@A z)D0WQ*b>&I6h4h0*{&c;v5N_lHm@M2%v>*etMaW~IqCZfqI%5ofz#ckeUv+GnTB_- zkNM84?-eyeak@809HX6>psEZ#vWPI|F+ z=k2#wc`s%;-pk+2=PvQ>`nGzW!3!%Vj7GIe4k(Xn=oGp>4IJ+`nYVF&)ErM=N|WoM zxg=|iz;g%VT1UQ^y5zAh<)m!q-xJ&xOYJuG(tc%%l$4NawYR^hl+i@>SR3f2&gO(*0e-@TjV`bahnApv!_(>1{hY zG5=53L)ZFS|Dce)NAJ#FK5N0f6z5ou%!52{y93y4y83<*zU?)4e{LzCzIjGeUXf0A z$W?)`unhOOimJBF1u-5)CmtCt@eJTd*?Z$g_x#c~l)vK0Ubj&G#)g?=`(Bxk*u+E|f z(KEcOFa4@&xUIhTj8ZceQD}ILe?#$4w^VcT_k>CH_!e2OuZjlTO68*uDe~V&MrJ+8 z_B2O=z%}al*wBKI`oA?Sd;*zkI0t0_ZSx8g%=+ZD$6aQ#2lRy~Iz z+!(c(YpVo~os21p)=~eDwN?CcXL%Rhge$28>{=Cpbgw^QHaVk`Lf|EzOPRi?iI!Xc zW7(Bd))dQ}F3fdTe4ScrBzDR=HB1ruayLn2B$l}j$yjB@cY3*JJaZX~xsn>_6Yomj zS(cU5?uG=~ae!IETuHszpTM}1rmUn!UEIf9PWiuAQWL1jj6PXcQZt_?Gj{y{SV^tj zn6`#KY}S?3+0kjt^T%9CEg+};J2AhK({vf_Fb<KPFqMX zO!dE<_K7y*H&&-H*3YO#)oBK_^iW+I9iFII5joO>Cb4(|74qV#(j&5y0oILPP;~=Gna@fBGScN@Ijz8 zNNLg{vkV^6hui5Q^x-fklk%PP&Ganw=$-V1Gx2I2_tP1No~quJo<>_iz0#jfTmOS0 z$quA@E8wNkVxe>tD%rY3WiyCK^v4wUcd$%GZ03RYMO+uZDQA4aVf@rHm=$^BHK78g*ne546tiO@a za=8e#Y)^)yDBj8KjEq(E8dQbD8S%7x^ttj3#vO{vRgrO07%yGWl;OcB&2uAT2CWg% z-V6(Rn%V8n-V8r_HtAV!28zQdj-WV7eb#%d$`o%iM>NxkQS7s5X3}TT<0wcdvQT8B zIDz6MiX0TDP@G0VMv;pm55*Z2XHn#%pr9x~QHbIkit{LnP!yxMfT9FNDT*=_<PEiFvgBmF!nfUZDs{^ZRW+DviPRX%8Se$F{`?mWL9+v#bp$gD5_9UQBJ472wqHTZTYlSDN;Y#=Iy*-~2Gb`xZRe!T7EXb?3>BM%ouI zOe?0ekx|j%N}IrFHY$;V-gWT)k1Kk`yAI~fw&-04m3BpM66j=JRjmS@oU}B09WbQN z6XqmS)$3%;ZR$b2&JDET<6XxjbpEfdm)LjG&Nf_f=!jG;!1vEJp_B1t3T7XT37w4P zaH__>&Qxx^@RhW`T|*l(>76#Jc$!$<$+(dw)mC?+xQF6CiY^onP;{esh@uBYFN#Mf z9;4_((T`#P#UP3&D4wDiLNScu8H(p9UZ8l1;uVV5C`M4cLGc#FI~4CxjH38};viFPKrxBpH;O+8YpdDPQ2^`!0pI{Qff>L|fD4!f za05I5FTe-z0|I~`AOr{lB7i6$28aU^fFv**kOHKEIlx>%2ABuP0&;*npa3WWN`Nw; z0?Y?g0X0A!&;T?6EkGO40dxU9Kp!vw41ooJ5nv1~1WW)^zzi@476BH(Vqgid6j%l< z2P^?AUL|{LV1RMa8ffV2%a0o~R(tvaz12_yE0geL4fK1>xKmxLWY~TcN z6378g0jB{nkPGAiXMnRnK0pBqfI{FLa2_ZEih&D22~Y}@0p&mia1po!Tm~wEDu4=9 z12sS`PzPKA>Vd0318@y!1g-;3zzyIg&A2rvc~0w#bdUPb1D1dl zumZ3KYyexp4zLFt07t+Ha0XTatAN#j3$O-Q3#4yMZWR53m=A24a9% zAP$HJ5`cX`BCsDw0uBJlKnidWI0U2uX+S!V0UQR707rpiKqhb;AOTrGHgEzs3FH8$ zfYSgO$OZC%Gr(COAD{pQKp}7rI1dy7#lQui1Skc{fO4P$xCmSVE(4W76+i{5ff}F| zr~|G5^}to20k{S<0@r~i;0ACLXa-t)<02}}(FawwgZ~?ObZh!~i1^57d zKmZT~gaBbc1P}$p0C7M9kOXD}Qh+ot2bc@U0P_G@Kn{=x6aYm)2~Y-9fcbzbpa!S| z8h|FC1!x00fG(g1=mQ3TA+P{20*rx$fC*p0;~bn0_%YFzy`n-*a)}*?tlm233vhCfDhmcYy$j% z&453!1qc9$Kp+qVYz4Lf!N7JP1PBGffN)?35CQB2B7t4NZXgQS1MCH&ffyhbhy&t* z1YjSK2kOCY84gsk^8jucT0EdAiz)|2BkO>?INI({l4V(Z@0y)4b;50x6 za)CVH450zt{s?p8;5{Ju(ZYNJpzwke-Ld%ud_G&Xdjzsw}?4Wj7N zAlh4N*bs&OYNma^8oljRBD?nd&mTjOPZj4A7#~hHd_aGh9>$y0G^Y?4Z*uWZ;~I)k z>Y)|BHDMox!1$hf0~P%xga2ZI+?z-tXu;2c$hzn!82E<-%gi$J`tLgA~G2mU4Gc&Wk1IfPSE}jwRQOa{eqir_}!#AIhy<}7vh zqugqi+&Idyp3%so6oLy&xlxo`!;ja^WR+Ef8?}||648(Jl3#nMMQ%CVN4mG$qKV^Pi(ugg4sjo0C5OJq|B8Z52)p&acRV3vN5a%)-3ouCkQ zu;eCCE`lX@?Bu_rIlLTxDDVMK z`K1bG_;U0;R}LQ8iBHVZnEo z;^C26oNRp9F<`msmkIt7uM!s*qvu=m^NbvVJ2z& z(ChJ_+N_hv9_6IjM+xMo=pXVVZH^!j`Y!3hg^UmQyz^u3asv#@9K!oz>!$yh4<&e^92aKK{Nz&fM8M5#%p8u^pBH2ElF-j?%V%Pk z$h6L#_Xc6xg2VlMf`bFYv{zX!S?*w|HKi}+IT5%7b4f8Sh#Xf#A&4bzxZyzC5(1%yww6f%Kl+|JBY0B3PrrbPWjt=qeo!S%r4FoA>ZG2z$ zQlTH-ld*qGmZQJ5@iNSlh(FH~B(BlQ$dWD4^DXO@GW8eW=70AWZfp5*+ZpxX*Wd~) z{g1%2GJD zIz*wIG~sDlGOc6MxcOX_X{Q#S8Y_G{1xq40u#*@k)n??LPCA;+HNvVJBJWD!TVEFG#!uzVlOG zlcX6t53flpX|?28j#j}d`_>Q<469v&e$4;30_|0|fL38HIURL2mst(pr*@Q+X5V)Z zAIV%YR|AE>$1H*m<%0O>y;A626~~2Pded62PN!8P#fqlSB4t#k3I2OH%212a?2C^6 zEy;H!P5R$Ls@I@U#am``!t_GSgLVQhWHE^s5+}t(q>GTBqTG~g0Uxizcu&bKbnWlU z!mtg~#+y5>u0tdJy(2z%X|Ih7S2$~EVA!We7`A3w-3i%r(xLODI=nZbL!hL_VQ zVm8K4B6Ow8(2pH+zq}n^6s=IeumfZaTQP0FW?!CK`(uB)GQBoFX#A*5os5cqYttT( zhI-P;Ah5r`Juh)nXk^EoiMM84qlew;d|9B{pOO8D263kVVLE#c1csK zl25myYl3{b<==~E0Oh0!DqCnBkp)pfO_u%hL^c4`SXW;E^Kl*HeeUR#7$%Qq}y`col(M`C^z2E)vzwXv9k+}H-ssOZ}< z(=S`RV&}OB9h=ajmfB|4M=eu1yrKp3kl__}ZxwZXxd6k)##wt3G9|CEv`&1xZ5a|4m7Z>~?i(PXTxv>+yB9k2TI(f*!_x^oxN%>~~#R8p?2zS<-#{MaF z_36W7-TmbXe>;G$1(Chb<2mcAS6cUsMLFmUTFA4pypK9PwJzy>8u~9MrP_CL|J#*d z7u-{+uFS5cUy1nP>gL_BBMkjFoRZO3v>d0MaG8zFinGZbs4-=hS*G3ddWN@tHTs8)D*E`GxFeDW iW8?USo&-72OC9EH$2kLzbpiUn>?-u6=uSaTcK;9YlkqkH diff --git a/site/notebooks/how_to/tests/run_tests/configure_tests/configure_judge_llms.ipynb b/site/notebooks/how_to/tests/run_tests/configure_tests/configure_judge_llms.ipynb index 3e8d27bcd1..bde5200a41 100644 --- a/site/notebooks/how_to/tests/run_tests/configure_tests/configure_judge_llms.ipynb +++ b/site/notebooks/how_to/tests/run_tests/configure_tests/configure_judge_llms.ipynb @@ -158,6 +158,7 @@ }, { "cell_type": "markdown", + "id": "322b05cc", "metadata": {}, "source": [ "\n", @@ -266,11 +267,11 @@ "import validmind as vm\n", "\n", "vm.init(\n", - " api_host=\"http://localhost:5000/api/v1/tracking\",\n", - " api_key=\"..\",\n", - " api_secret=\"..\",\n", + " api_host=\"https://app.prod.validmind.ai/api/v1/tracking\",\n", + " api_key=\"...\",\n", + " api_secret=\"...\",\n", " document=\"documentation\", # requires library >=2.12.0\n", - " model=\"..\",\n", + " model=\"...\",\n", ")" ] }, diff --git a/site/notebooks/quickstart/quickstart_model_documentation_oidc_device_flow.ipynb b/site/notebooks/quickstart/quickstart_model_documentation_oidc_device_flow.ipynb new file mode 100644 index 0000000000..9fff9205a5 --- /dev/null +++ b/site/notebooks/quickstart/quickstart_model_documentation_oidc_device_flow.ipynb @@ -0,0 +1,889 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "686bdc65", + "metadata": {}, + "source": [ + "# Quickstart for model documentation (OIDC device flow)\n", + "\n", + "Learn the basics of using ValidMind to document models as part of a model development workflow. **This variant authenticates with the OIDC device authorization flow** ([RFC 8628](https://datatracker.ietf.org/doc/html/rfc8628)) instead of storing API keys in the notebook: you sign in through your organization's identity provider when `vm.init()` runs. After setup, generate a draft of documentation using ValidMind tests for a binary classification model.\n", + "\n", + "To document a model with the ValidMind Library, we'll:\n", + "\n", + "1. Import a sample dataset and preprocess it\n", + "2. Split the datasets and initialize them for use with ValidMind\n", + "3. Initialize a model object for use with testing\n", + "4. Run a full suite of tests as defined by our documentation template, which will send the results of those tests to the ValidMind Platform" + ] + }, + { + "cell_type": "markdown", + "id": "17e1b850", + "metadata": {}, + "source": [ + "::: {.content-hidden when-format=\"html\"}\n", + "## Contents \n", + "- [Introduction](#toc1__) \n", + "- [About ValidMind](#toc2__) \n", + " - [Before you begin](#toc2_1__) \n", + " - [New to ValidMind?](#toc2_2__) \n", + " - [Key concepts](#toc2_3__) \n", + "- [Setting up](#toc3__) \n", + " - [Install the ValidMind Library](#toc3_1__) \n", + " - [Initialize the ValidMind Library](#toc3_2__) \n", + " - [Register sample model](#toc3_2_1__) \n", + " - [Apply documentation template](#toc3_2_2__) \n", + " - [Get your code snippet](#toc3_2_3__) \n", + " - [Initialize the Python environment](#toc3_3__) \n", + "- [Getting to know ValidMind](#toc4__) \n", + " - [Preview the documentation template](#toc4_1__) \n", + " - [View model documentation in the ValidMind Platform](#toc4_2__) \n", + "- [Import the sample dataset](#toc5__) \n", + "- [Preprocessing the raw dataset](#toc6__) \n", + " - [Split the dataset](#toc6_1__) \n", + " - [Separate features and targets](#toc6_2__) \n", + "- [Training an XGBoost classifier model](#toc7__) \n", + " - [Set evaluation metrics](#toc7_1__) \n", + " - [Fit the model](#toc7_2__) \n", + "- [Initialize the ValidMind datasets](#toc8__) \n", + "- [Initialize a model object](#toc9__) \n", + " - [Assign predictions](#toc9_1__) \n", + "- [Run the full suite of tests](#toc10__) \n", + "- [In summary](#toc11__) \n", + "- [Next steps](#toc12__) \n", + " - [Work with your model documentation](#toc12_1__) \n", + " - [Discover more learning resources](#toc12_2__) \n", + "- [Upgrade ValidMind](#toc13__) \n", + "\n", + ":::\n", + "\n", + "" + ] + }, + { + "cell_type": "markdown", + "id": "16993535", + "metadata": {}, + "source": [ + "\n", + "\n", + "## Introduction\n", + "\n", + "Model development aims to produce a fit-for-purpose *champion model* by conducting thorough testing and analysis, supporting the capabilities of the model with evidence in the form of documentation and test results. Model documentation should be clear and comprehensive, ideally following a structure or template covering all aspects of compliance with model risk regulation.\n", + "\n", + "A *binary classification model* is a type of predictive model used in churn analysis to identify customers who are likely to leave a service or subscription by analyzing various behavioral, transactional, and demographic factors.\n", + "\n", + "- This model helps businesses take proactive measures to retain at-risk customers by offering personalized incentives, improving customer service, or adjusting pricing strategies.\n", + "- Effective validation of a churn prediction model ensures that businesses can accurately identify potential churners, optimize retention efforts, and enhance overall customer satisfaction while minimizing revenue loss." + ] + }, + { + "cell_type": "markdown", + "id": "3b8e33f0", + "metadata": {}, + "source": [ + "\n", + "\n", + "## About ValidMind\n", + "\n", + "ValidMind is a suite of tools for managing model risk, including risk associated with AI and statistical models. \n", + "\n", + "You use the ValidMind Library to automate documentation and validation tests, and then use the ValidMind Platform to collaborate on model documentation. Together, these products simplify model risk management, facilitate compliance with regulations and institutional standards, and enhance collaboration between yourself and model validators." + ] + }, + { + "cell_type": "markdown", + "id": "24a5e851", + "metadata": {}, + "source": [ + "\n", + "\n", + "### Before you begin\n", + "\n", + "This notebook assumes you have basic familiarity with Python, including an understanding of how functions work. If you are new to Python, you can still run the notebook but we recommend further familiarizing yourself with the language. \n", + "\n", + "If you encounter errors due to missing modules in your Python environment, install the modules with `pip install`, and then re-run the notebook. For more help, refer to [Installing Python Modules](https://docs.python.org/3/installing/index.html)." + ] + }, + { + "cell_type": "markdown", + "id": "87f8ed22", + "metadata": {}, + "source": [ + "\n", + "\n", + "### New to ValidMind?\n", + "\n", + "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models and running tests, as well as find code samples and our Python Library API reference.\n", + "\n", + "
For access to all features available in this notebook, you'll need access to a ValidMind account.\n", + "

\n", + "Register with ValidMind
" + ] + }, + { + "cell_type": "markdown", + "id": "61497ac2", + "metadata": {}, + "source": [ + "\n", + "\n", + "### Key concepts\n", + "\n", + "**Model documentation**: A structured and detailed record pertaining to a model, encompassing key components such as its underlying assumptions, methodologies, data sources, inputs, performance metrics, evaluations, limitations, and intended uses. It serves to ensure transparency, adherence to regulatory requirements, and a clear understanding of potential risks associated with the model’s application.\n", + "\n", + "**Documentation template**: Functions as a test suite and lays out the structure of model documentation, segmented into various sections and sub-sections. Documentation templates define the structure of your model documentation, specifying the tests that should be run, and how the results should be displayed.\n", + "\n", + "**Tests**: A function contained in the ValidMind Library, designed to run a specific quantitative test on the dataset or model. Tests are the building blocks of ValidMind, used to evaluate and document models and datasets, and can be run individually or as part of a suite defined by your model documentation template.\n", + "\n", + "**Metrics**: A subset of tests that do not have thresholds. In the context of this notebook, metrics and tests can be thought of as interchangeable concepts.\n", + "\n", + "**Custom metrics**: Custom metrics are functions that you define to evaluate your model or dataset. These functions can be registered with the ValidMind Library to be used in the ValidMind Platform.\n", + "\n", + "**Inputs**: Objects to be evaluated and documented in the ValidMind Library. They can be any of the following:\n", + "\n", + " - **model**: A single model that has been initialized in ValidMind with [`vm.init_model()`](https://docs.validmind.ai/validmind/validmind.html#init_model).\n", + " - **dataset**: Single dataset that has been initialized in ValidMind with [`vm.init_dataset()`](https://docs.validmind.ai/validmind/validmind.html#init_dataset).\n", + " - **models**: A list of ValidMind models - usually this is used when you want to compare multiple models in your custom metric.\n", + " - **datasets**: A list of ValidMind datasets - usually this is used when you want to compare multiple datasets in your custom metric. (Learn more: [Run tests with multiple datasets](https://docs.validmind.ai/notebooks/how_to/tests/run_tests/configure_tests/run_tests_that_require_multiple_datasets.html))\n", + "\n", + "**Parameters**: Additional arguments that can be passed when running a ValidMind test, used to pass additional information to a metric, customize its behavior, or provide additional context.\n", + "\n", + "**Outputs**: Custom metrics can return elements like tables or plots. Tables may be a list of dictionaries (each representing a row) or a pandas DataFrame. Plots may be matplotlib or plotly figures.\n", + "\n", + "**Test suites**: Collections of tests designed to run together to automate and generate model documentation end-to-end for specific use-cases.\n", + "\n", + "Example: the [`classifier_full_suite`](https://docs.validmind.ai/validmind/validmind/test_suites/classifier.html#ClassifierFullSuite) test suite runs tests from the [`tabular_dataset`](https://docs.validmind.ai/validmind/validmind/test_suites/tabular_datasets.html) and [`classifier`](https://docs.validmind.ai/validmind/validmind/test_suites/classifier.html) test suites to fully document the data and model sections for binary classification model use-cases." + ] + }, + { + "cell_type": "markdown", + "id": "5c4158bd", + "metadata": {}, + "source": [ + "\n", + "\n", + "## Setting up" + ] + }, + { + "cell_type": "markdown", + "id": "fd4a5481", + "metadata": {}, + "source": [ + "\n", + "\n", + "### Install the ValidMind Library\n", + "\n", + "
Recommended Python versions\n", + "

\n", + "Python 3.8 <= x <= 3.14
\n", + "\n", + "To install the library:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d1f6dbed", + "metadata": {}, + "outputs": [], + "source": [ + "%pip install -q validmind" + ] + }, + { + "cell_type": "markdown", + "id": "d32e4d62", + "metadata": {}, + "source": [ + "\n", + "\n", + "### Initialize the ValidMind Library\n", + "\n", + "Use [`vm.init()`](https://docs.validmind.ai/validmind/validmind.html) with **`issuer`** and **`client_id`** (your IdP’s OAuth public client for device flow), plus **`model`** and **`api_host`** (or environment variables `VM_API_MODEL` / `VM_API_HOST`). Do not pass `api_key` / `api_secret` here; those are for the API-key quickstart. Optional **`audience`** (or `VM_OIDC_AUDIENCE`) and **`scope`** are often needed so tokens are accepted by the ValidMind API; your ValidMind or identity administrator can supply values that match your deployment. For general library usage, see the [ValidMind Library documentation](https://docs.validmind.ai/developer/validmind-library.html).\n", + "\n", + "When you run the cell below, the library may print a **verification URL** and **user code**: open the URL, enter the code, and complete sign-in; the notebook waits until the flow finishes. Valid tokens are cached under `~/.validmind/credentials.json`." + ] + }, + { + "cell_type": "markdown", + "id": "70757032", + "metadata": {}, + "source": [ + "\n", + "\n", + "#### Register sample model\n", + "\n", + "Let's first register a sample model for use with this notebook:\n", + "\n", + "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n", + "\n", + "2. In the left sidebar, navigate to **Inventory** and click **+ Register Model**.\n", + "\n", + "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", + "\n", + "4. Select your own name under the **MODEL OWNER** drop-down.\n", + "\n", + "5. Click **Register Model** to add the model to your inventory." + ] + }, + { + "cell_type": "markdown", + "id": "445ac2ce", + "metadata": {}, + "source": [ + "\n", + "\n", + "#### Apply documentation template\n", + "\n", + "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", + "\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", + "\n", + "2. Under **TEMPLATE**, select `Binary classification`.\n", + "\n", + "3. Click **Use Template** to apply the template." + ] + }, + { + "cell_type": "markdown", + "id": "c30a58b5", + "metadata": {}, + "source": [ + "\n", + "\n", + "#### Configure `vm.init()` for OIDC\n", + "\n", + "Point the library at the correct **model** and **tracking API**, and supply OIDC **issuer** / **client_id** values your organization registered for device flow. If you see 401/403 responses after signing in at your IdP, verify **audience** and **scope** with your administrator.\n", + "\n", + "1. In the ValidMind Platform, open your model → **Getting Started** → **DOCUMENT** `Development` so you can copy the **model CUID** (same identifier as in the API-key snippet).\n", + "2. Fill in `issuer`, `client_id`, `model`, and `api_host` below. Use optional `audience` / `scope` if your platform team specified them.\n", + "3. Run the cell. Complete browser sign-in if prompted; when polling succeeds, continue with the rest of the notebook.\n", + "\n", + "You can optionally load `VM_API_HOST`, `VM_API_MODEL`, or `VM_OIDC_AUDIENCE` from a `.env` file instead of passing them inline—see [Store credentials in an env file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) for the pattern (environment variables still apply when arguments are omitted)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e2c1dd22", + "metadata": {}, + "outputs": [], + "source": [ + "# Optional: load VM_API_HOST, VM_API_MODEL, VM_OIDC_AUDIENCE from `.env`\n", + "# %load_ext dotenv\n", + "# %dotenv .env\n", + "\n", + "import validmind as vm\n", + "\n", + "vm.init(\n", + " issuer=\"https://login.microsoftonline.com//v2.0\", # your IdP issuer (OpenID discovery)\n", + " client_id=\"\",\n", + " model=\"\",\n", + " api_host=\"https:///api/v1/tracking/\",\n", + " # audience=\"\", # often required; or set VM_OIDC_AUDIENCE\n", + " # scope=\"openid profile email\", # optional; default if omitted\n", + " document=\"documentation\",\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "d1dcb3d0", + "metadata": {}, + "source": [ + "\n", + "\n", + "### Initialize the Python environment\n", + "\n", + "Then, let's import the necessary libraries and set up your Python environment for data analysis:\n", + "\n", + "- Import **Extreme Gradient Boosting** (XGBoost) with an alias so that we can reference its functions in later calls. XGBoost is a powerful machine learning library designed for speed and performance, especially in handling structured or tabular data.\n", + "- Enable **`matplotlib`**, a plotting library used for visualizing data. Ensures that any plots you generate will render inline in our notebook output rather than opening in a separate window." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "62d7c2c1", + "metadata": {}, + "outputs": [], + "source": [ + "import xgboost as xgb\n", + "\n", + "%matplotlib inline" + ] + }, + { + "cell_type": "markdown", + "id": "7bbc5e0c", + "metadata": {}, + "source": [ + "\n", + "\n", + "## Getting to know ValidMind" + ] + }, + { + "cell_type": "markdown", + "id": "b07067d5", + "metadata": {}, + "source": [ + "\n", + "\n", + "### Preview the documentation template\n", + "\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model.\n", + "\n", + "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b2bce375", + "metadata": {}, + "outputs": [], + "source": [ + "vm.preview_template()" + ] + }, + { + "cell_type": "markdown", + "id": "1336b875", + "metadata": {}, + "source": [ + "\n", + "\n", + "### View model documentation in the ValidMind Platform\n", + "\n", + "Next, let's head to the ValidMind Platform to see the template in action:\n", + "\n", + "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n", + "\n", + "2. In the left sidebar, navigate to **Inventory** and select the model you registered for this notebook.\n", + "\n", + "3. Click **Development** under Documents for your model and note how the structure of the documentation matches our preview above." + ] + }, + { + "cell_type": "markdown", + "id": "e2281cec", + "metadata": {}, + "source": [ + "\n", + "\n", + "## Import the sample dataset\n", + "\n", + "First, let's import the public [Bank Customer Churn Prediction](https://www.kaggle.com/datasets/shantanudhakadd/bank-customer-churn-prediction) dataset from Kaggle so that we have something to work with.\n", + "\n", + "In our below example, note that: \n", + "\n", + "- The target column, `Exited` has a value of `1` when a customer has churned and `0` otherwise.\n", + "- The ValidMind Library provides a wrapper to automatically load the dataset as a [Pandas DataFrame](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html) object. A Pandas Dataframe is a two-dimensional tabular data structure that makes use of rows and columns." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "58d1c94b", + "metadata": {}, + "outputs": [], + "source": [ + "from validmind.datasets.classification import customer_churn\n", + "\n", + "print(\n", + " f\"Loaded demo dataset with: \\n\\n\\t• Target column: '{customer_churn.target_column}' \\n\\t• Class labels: {customer_churn.class_labels}\"\n", + ")\n", + "\n", + "raw_df = customer_churn.load_data()\n", + "raw_df.head()" + ] + }, + { + "cell_type": "markdown", + "id": "0aafde18", + "metadata": {}, + "source": [ + "\n", + "\n", + "## Preprocessing the raw dataset\n", + "\n", + "Before running tests with ValidMind, we'll need to preprocess our imported dataset. This involves splitting the data and separating the features (inputs) from the targets (outputs)." + ] + }, + { + "cell_type": "markdown", + "id": "dcd9848d", + "metadata": {}, + "source": [ + "\n", + "\n", + "### Split the dataset\n", + "\n", + "Splitting our dataset helps assess how well the model generalizes to unseen data.\n", + "\n", + "Use [`preprocess()`](https://docs.validmind.ai/validmind/validmind/datasets/classification/customer_churn.html#preprocess) to split our dataset into three subsets:\n", + "\n", + "1. **train_df** — Used to train the model.\n", + "2. **validation_df** — Used to evaluate the model's performance during training.\n", + "3. **test_df** — Used later on to asses the model's performance on new, unseen data." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "418cb5aa", + "metadata": {}, + "outputs": [], + "source": [ + "train_df, validation_df, test_df = customer_churn.preprocess(raw_df)" + ] + }, + { + "cell_type": "markdown", + "id": "0ed6cb75", + "metadata": {}, + "source": [ + "\n", + "\n", + "### Separate features and targets\n", + "\n", + "To train the model, we need to provide it with:\n", + "\n", + "1. **Inputs** — Features such as customer age, usage, etc.\n", + "2. **Outputs (Expected answers/labels)** — in our case, we would like to know whether the customer churned or not.\n", + "\n", + "Here, we'll use `x_train` and `x_val` to hold the input data (features), and `y_train` and `y_val` to hold the answers (the target we want to predict):" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "6fd365fd", + "metadata": {}, + "outputs": [], + "source": [ + "x_train = train_df.drop(customer_churn.target_column, axis=1)\n", + "y_train = train_df[customer_churn.target_column]\n", + "x_val = validation_df.drop(customer_churn.target_column, axis=1)\n", + "y_val = validation_df[customer_churn.target_column]" + ] + }, + { + "cell_type": "markdown", + "id": "3e1d226e", + "metadata": {}, + "source": [ + "\n", + "\n", + "## Training an XGBoost classifier model\n", + "\n", + "Next, let's create an XGBoost classifier model that will automatically stop training if it doesn’t improve after 10 tries.\n", + "\n", + "Setting a threshold avoids wasting time and helps prevent overfitting by stopping training when further improvement isn’t happening." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "3296cac6", + "metadata": {}, + "outputs": [], + "source": [ + "model = xgb.XGBClassifier(early_stopping_rounds=10)" + ] + }, + { + "cell_type": "markdown", + "id": "d641a3f1", + "metadata": {}, + "source": [ + "\n", + "\n", + "### Set evaluation metrics\n", + "\n", + "Then, we'll set the evaluation metrics, which tells the model to use three different ways to measure its performance:\n", + "\n", + "1. **error** — Measures how often the model makes incorrect predictions.\n", + "2. **logloss** — Indicates how confident the predictions are.\n", + "3. **auc** — Evaluates how well the model distinguishes between churn and not churn.\n", + "\n", + "Using multiple metrics gives a more complete picture of how good (or bad) the model is." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "32d3c3f4", + "metadata": {}, + "outputs": [], + "source": [ + "model.set_params(\n", + " eval_metric=[\"error\", \"logloss\", \"auc\"],\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "ec0de49c", + "metadata": {}, + "source": [ + "\n", + "\n", + "### Fit the model\n", + "\n", + "Finally, our actual training step — where the model learns patterns from the data, so it can make predictions later:\n", + "\n", + "- The model is trained on `x_train` and `y_train`, and evaluates its performance using `x_val` and `y_val` to check if it’s learning well.\n", + "- To turn off printed output while training, we'll set `verbose` to `False`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "3fb95ce4", + "metadata": {}, + "outputs": [], + "source": [ + "model.fit(\n", + " x_train,\n", + " y_train,\n", + " eval_set=[(x_val, y_val)],\n", + " verbose=False,\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "833a5047", + "metadata": {}, + "source": [ + "\n", + "\n", + "## Initialize the ValidMind datasets\n", + "\n", + "Before you can run tests with your preprocessed datasets, you must first initialize a ValidMind `Dataset` object using the [`init_dataset`](https://docs.validmind.ai/validmind/validmind.html#init_dataset) function from the ValidMind (`vm`) module. **This step is always necessary every time you want to connect a dataset to documentation and produce test results through ValidMind,** but you only need to do it once per dataset.\n", + "\n", + "For this example, we'll pass in the following arguments:\n", + "\n", + "- **`dataset`** — The raw dataset that you want to provide as input to tests.\n", + "- **`input_id`** — A unique identifier that allows tracking what inputs are used when running each individual test.\n", + "- **`target_column`** — A required argument if tests require access to true values. This is the name of the target column in the dataset.\n", + "- **`class_labels`** — An optional value to map predicted classes to class labels." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "bb6ad06a", + "metadata": {}, + "outputs": [], + "source": [ + "# Initialize the raw dataset\n", + "vm_raw_dataset = vm.init_dataset(\n", + " dataset=raw_df,\n", + " input_id=\"raw_dataset\",\n", + " target_column=customer_churn.target_column,\n", + " class_labels=customer_churn.class_labels,\n", + ")\n", + "\n", + "# Initialize the training dataset\n", + "vm_train_ds = vm.init_dataset(\n", + " dataset=train_df,\n", + " input_id=\"train_dataset\",\n", + " target_column=customer_churn.target_column,\n", + ")\n", + "\n", + "# Initialize the testing dataset\n", + "vm_test_ds = vm.init_dataset(\n", + " dataset=test_df,\n", + " input_id=\"test_dataset\",\n", + " target_column=customer_churn.target_column\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "316ae030", + "metadata": {}, + "source": [ + "\n", + "\n", + "## Initialize a model object\n", + "\n", + "You'll also need to initialize a ValidMind model object (`vm_model`) that can be passed to other functions for analysis and tests on the data for our model.\n", + "\n", + "You simply initialize this model object with [`vm.init_model()`](https://docs.validmind.ai/validmind/validmind.html#init_model):" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "0e44eebd", + "metadata": {}, + "outputs": [], + "source": [ + "vm_model = vm.init_model(\n", + " model,\n", + " input_id=\"model\",\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "3002517f", + "metadata": {}, + "source": [ + "\n", + "\n", + "### Assign predictions\n", + "\n", + "Once the model has been registered, you can assign model predictions to the training and testing datasets.\n", + "\n", + "- The [`assign_predictions()` method](https://docs.validmind.ai/validmind/validmind/vm_models.html#assign_predictions) from the `Dataset` object can link existing predictions to any number of models.\n", + "- This method links the model's class prediction values and probabilities to our `vm_train_ds` and `vm_test_ds` datasets.\n", + "\n", + "If no prediction values are passed, the method will compute predictions automatically:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "62bd94fc", + "metadata": {}, + "outputs": [], + "source": [ + "vm_train_ds.assign_predictions(\n", + " model=vm_model,\n", + ")\n", + "\n", + "vm_test_ds.assign_predictions(\n", + " model=vm_model,\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "123d94f9", + "metadata": {}, + "source": [ + "\n", + "\n", + "## Run the full suite of tests\n", + "\n", + "This is where it all comes together — you are now ready to **run the documentation tests for the model as defined by the documentation template** you looked at earlier.\n", + "\n", + "The [`vm.run_documentation_tests`](https://docs.validmind.ai/validmind/validmind.html#run_documentation_tests) function finds and runs every test specified in the template and then uploads all the documentation and test artifacts that get generated to the ValidMind Platform:\n", + "\n", + "- The function requires information about the inputs to use on every test. These inputs can be passed as an `inputs` argument if we want to use the same inputs for all tests. \n", + "- It's also possible to pass a `config` argument that has information about the `params` and `inputs` that each test requires. The `config` parameter is a dictionary with the following structure:\n", + "\n", + " ```python\n", + " config = {\n", + " \"\": {\n", + " \"params\": {\n", + " \"param1\": \"value1\",\n", + " \"param2\": \"value2\",\n", + " ...\n", + " },\n", + " \"inputs\": {\n", + " \"input1\": \"value1\",\n", + " \"input2\": \"value2\",\n", + " ...\n", + " }\n", + " },\n", + " ...\n", + " }\n", + " ```\n", + "\n", + " Each `` above corresponds to the test driven block identifiers shown by `vm.preview_template()`. For this model, we will use the default parameters for all tests, but we'll need to specify the input configuration for each one. The method `get_demo_test_config()` below constructs the default input configuration for our demo." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b3d6741b", + "metadata": {}, + "outputs": [], + "source": [ + "from validmind.utils import preview_test_config\n", + "\n", + "test_config = customer_churn.get_demo_test_config()\n", + "preview_test_config(test_config)" + ] + }, + { + "cell_type": "markdown", + "id": "323c9246", + "metadata": {}, + "source": [ + "Now we can pass the input configuration to `vm.run_documentation_tests()` and run the full suite of tests.\n", + "\n", + "The variable `full_suite` then holds the result of these tests:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "ae3accf7", + "metadata": {}, + "outputs": [], + "source": [ + "full_suite = vm.run_documentation_tests(config=test_config)" + ] + }, + { + "cell_type": "markdown", + "id": "5235c139", + "metadata": {}, + "source": [ + "\n", + "\n", + "## In summary\n", + "\n", + "In this notebook, you learned how to:\n", + "\n", + "- [x] Register a model within the ValidMind Platform\n", + "- [x] Install and initialize the ValidMind Library\n", + "- [x] Preview the documentation template for your model\n", + "- [x] Import a sample dataset\n", + "- [x] Initialize ValidMind datasets and model objects\n", + "- [x] Assign model predictions to your ValidMind model objects\n", + "- [x] Run a full suite of documentation tests" + ] + }, + { + "cell_type": "markdown", + "id": "2c84651a", + "metadata": {}, + "source": [ + "\n", + "\n", + "## Next steps\n", + "\n", + "You can look at the output produced by the ValidMind Library right in the notebook where you ran the code, as you would expect. But there is a better way — use the ValidMind Platform to work with your model documentation." + ] + }, + { + "cell_type": "markdown", + "id": "946e40b2", + "metadata": {}, + "source": [ + "\n", + "\n", + "### Work with your model documentation\n", + "\n", + "1. From the **Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", + "\n", + "2. In the left sidebar that appears for your model, click **Development** under Documents.\n", + "\n", + "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/working-with-model-documentation.html)" + ] + }, + { + "cell_type": "markdown", + "id": "0b83397d", + "metadata": {}, + "source": [ + "\n", + "\n", + "### Discover more learning resources\n", + "\n", + "For a more in-depth introduction to using the ValidMind Library for development, check out our introductory development series and the accompanying interactive training:\n", + "\n", + "- **[ValidMind for model development](https://docs.validmind.ai/developer/validmind-library.html#for-model-development)**\n", + "- **[Developer Fundamentals](https://docs.validmind.ai/training/developer-fundamentals/developer-fundamentals-register.html)**\n", + "\n", + "We also offer many interactive notebooks to help you document models:\n", + "\n", + "- [Run tests & test suites](https://docs.validmind.ai/developer/how-to/testing-overview.html)\n", + "- [Use ValidMind Library features](https://docs.validmind.ai/developer/how-to/feature-overview.html)\n", + "- [Code samples by use case](https://docs.validmind.ai/guide/samples-jupyter-notebooks.html)\n", + "\n", + "Or, visit our [documentation](https://docs.validmind.ai/) to learn more about ValidMind." + ] + }, + { + "cell_type": "markdown", + "id": "9c78dfe5", + "metadata": {}, + "source": [ + "\n", + "\n", + "## Upgrade ValidMind\n", + "\n", + "
After installing ValidMind, you’ll want to periodically make sure you are on the latest version to access any new features and other enhancements.
\n", + "\n", + "Retrieve the information for the currently installed version of ValidMind:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "35955b6b", + "metadata": {}, + "outputs": [], + "source": [ + "%pip show validmind" + ] + }, + { + "cell_type": "markdown", + "id": "f6061b3d", + "metadata": {}, + "source": [ + "If the version returned is lower than the version indicated in our [production open-source code](https://github.com/validmind/validmind-library/blob/prod/validmind/__version__.py), restart your notebook and run:\n", + "\n", + "```bash\n", + "%pip install --upgrade validmind\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "f7216e8c", + "metadata": {}, + "source": [ + "You may need to restart your kernel after running the upgrade package for changes to be applied." + ] + }, + { + "cell_type": "markdown", + "id": "copyright-e26871efeffc48e386d68e90db1838e7", + "metadata": {}, + "source": [ + "\n", + "\n", + "\n", + "\n", + "***\n", + "\n", + "Copyright © 2023-2026 ValidMind Inc. All rights reserved.
\n", + "Refer to [LICENSE](https://github.com/validmind/validmind-library/blob/main/LICENSE) for details.
\n", + "SPDX-License-Identifier: AGPL-3.0 AND ValidMind Commercial" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "ValidMind Library", + "language": "python", + "name": "validmind" + }, + "language_info": { + "name": "python", + "version": "3.10.13" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/site/releases/all-releases.qmd b/site/releases/all-releases.qmd index da609e01fd..3da385153b 100644 --- a/site/releases/all-releases.qmd +++ b/site/releases/all-releases.qmd @@ -44,7 +44,7 @@ listing: - "hidden" sort-ui: false filter-ui: false - fields: [date, title] + fields: [date, title, subtitle,] contents: - ../releases/frontend/**/pr*.qmd - ../releases/documentation/**/pr*.qmd