From 5e5d06880f97aff9aacd69a47278ab7b649f7087 Mon Sep 17 00:00:00 2001 From: Yunfeng Zhang Date: Thu, 6 Aug 2026 14:51:37 +0000 Subject: [PATCH 01/11] fix: address third-party deprecation warnings - Replace GLiNER.batch_predict_entities (deprecated) with .inference - Rename warmup_ratio -> warmup_steps in TrainingHyperparams; pass directly to TrainingArguments which accepts floats in (0,1) as ratios, eliminating the manual ratio-to-steps conversion - Suppress huggingface_hub resume_download UserWarning at the GLiNER load site (upstream gliner passes the deprecated arg) - Suppress scipy ks_2samp asymptotic fallback RuntimeWarning in the tutorial notebook via a scoped with catch_warnings() block - Pre-compile torchao and range_regex in brev setup so SyntaxWarnings go to the setup log instead of appearing in notebook output - Pin BNB_CUDA_VERSION=128 in the brev kernelspec; bitsandbytes has no CUDA 12.9 binary (odd releases are skipped) and always falls back to 12.8; update to 130 when upgrading CUDA_EXTRA to cu130+ Closes #328 Signed-off-by: Yunfeng Zhang --- docs/tutorials/safe-synthesizer-101.ipynb | 492 +++++++++--------- script/brev/setup.sh | 10 + src/nemo_safe_synthesizer/config/training.py | 10 +- .../pii_replacer/data_editor/detect.py | 23 +- .../training/huggingface_backend.py | 2 +- tests/conftest.py | 2 +- tests/pii_replacer/test_detect.py | 12 +- tests/training/test_huggingface_backend.py | 6 +- 8 files changed, 290 insertions(+), 267 deletions(-) diff --git a/docs/tutorials/safe-synthesizer-101.ipynb b/docs/tutorials/safe-synthesizer-101.ipynb index 47f53c9bf..136ed2ed2 100644 --- a/docs/tutorials/safe-synthesizer-101.ipynb +++ b/docs/tutorials/safe-synthesizer-101.ipynb @@ -1,248 +1,252 @@ { - "cells": [ - { - "cell_type": "markdown", - "id": "d1d7a7a3", - "metadata": {}, - "source": [ - "\n", - "# 🔐 NeMo Safe Synthesizer Tutorial: The Basics\n", - "\n", - "#### What you'll learn\n", - "\n", - "In this notebook, we'll explore the fundamentals of NeMo Safe Synthesizer: PII replacement, training on a sample dataset, generating synthetic data, and evaluating quality and privacy.\n", - "\n", - "This library supports numeric, categorical, and text fields within the training data and generates realistic synthetic data that mirrors the structure of your data. A full run takes about 15 minutes on an A100.\n", - "\n", - "### 🖥️ Prerequisites\n", - "\n", - "This notebook requires a Linux machine with an NVIDIA GPU (H100 recommended, A100 minimum) and CUDA 12.9+. It will not run on macOS, Windows, or Apple Silicon." - ] - }, - { - "cell_type": "markdown", - "id": "d501f043", - "metadata": {}, - "source": [ - "### ⚡ Install Safe Synthesizer\n", - "\n", - "Run the cell below to install NeMo Safe Synthesizer (engine and CUDA 12.9) and the `datasets` library for the sample dataset." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "bb7b0bdd", - "metadata": { - "vscode": { - "languageId": "shellscript" - } - }, - "outputs": [], - "source": [ - "%%bash\n", - "# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.\n", - "# SPDX-License-Identifier: Apache-2.0\n", - "\n", - "if command -v uv > /dev/null 2>&1; then\n", - " uv pip install \"nemo-safe-synthesizer[engine,cu129]\" \\\n", - " --index https://flashinfer.ai/whl/cu129 \\\n", - " --index https://flashinfer.ai/whl/ \\\n", - " --index https://download.pytorch.org/whl/cu129 \\\n", - " --index https://wheels.vllm.ai/0.26.0/cu129 \\\n", - " --index-strategy unsafe-best-match\n", - " uv pip install datasets\n", - "else\n", - " pip install \"nemo-safe-synthesizer[engine,cu129]\" \\\n", - " --extra-index-url https://flashinfer.ai/whl/cu129 \\\n", - " --extra-index-url https://flashinfer.ai/whl/ \\\n", - " --extra-index-url https://download.pytorch.org/whl/cu129 \\\n", - " --extra-index-url https://wheels.vllm.ai/0.26.0/cu129\n", - " pip install datasets\n", - "fi\n" - ] - }, - { - "cell_type": "markdown", - "id": "3030139c", - "metadata": {}, - "source": [ - "\n", - "### 🔑 Set the inference API key for PII column classification\n", - "\n", - "NeMo Safe Synthesizer uses an LLM‑based column classifier to automatically infer PII columns. To enable this feature, set `NSS_INFERENCE_KEY` (the inference endpoint defaults to the NVIDIA integrate URL. You can obtain an API key from [build.nvidia.com](https://build.nvidia.com/settings/api-keys)). Setting this value is optional but strongly recommended.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "693620c8", - "metadata": {}, - "outputs": [], - "source": [ - "import os\n", - "import getpass\n", - "\n", - "# Setting NSS_INFERENCE_KEY is optional but strongly recommended for PII replacement.\n", - "if \"NSS_INFERENCE_KEY\" not in os.environ:\n", - " os.environ[\"NSS_INFERENCE_KEY\"] = getpass.getpass(\"Paste inference API key (or press Enter to skip): \")\n", - "if os.environ.get(\"NSS_INFERENCE_KEY\"):\n", - " print(\"NSS_INFERENCE_KEY is set\")\n", - "else:\n", - " print(\n", - " \"NSS_INFERENCE_KEY is not set. Replace PII will run in degraded mode. \"\n", - " \"We strongly recommend setting a key.\"\n", - " )" - ] - }, - { - "cell_type": "markdown", - "id": "bdb29834", - "metadata": {}, - "source": [ - "### 📥 Load and preview sample dataset\n", - "\n", - "Load a tabular dataset—in this example, the [clinc_oos](https://huggingface.co/datasets/clinc/clinc_oos) dataset from Hugging Face—and preview the first few rows. NeMo Safe Synthesizer will use this DataFrame as its training data.\n", - "\n", - "This dataset includes a text column and a categorical intent label supported by Nemo Safe Synthesizer.\n", - "\n", - "Each user is responsible for checking the content of datasets and the applicable licenses and determining if suitable for the intended use." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "fb1a562a", - "metadata": {}, - "outputs": [], - "source": [ - "from datasets import load_dataset\n", - "\n", - "dataset = load_dataset(\"clinc/clinc_oos\", \"small\")\n", - "df = dataset[\"train\"].to_pandas()\n", - "df.head()" - ] - }, - { - "cell_type": "markdown", - "id": "1c394bab", - "metadata": {}, - "source": [ - "\n", - "### ⚙️ Create and run Safe Synthesizer job\n", - "\n", - "Create the Safe Synthesizer builder and attach your DataFrame. Run the pipeline with `run()`, which performs data processing, PII replacement, training, generation, and evaluation in a single call. Results are available on `builder.results`.\n", - "\n", - "Refer to the [configuration docs](../user-guide/configuration.md) for the full list of options.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "9694992a", - "metadata": {}, - "outputs": [], - "source": [ - "from nemo_safe_synthesizer.sdk.library_builder import SafeSynthesizer\n", - "\n", - "builder = SafeSynthesizer().with_data_source(df) # .with_replace_pii(enable=False) to disable PII replacement\n", - "builder.run()\n", - "results = builder.results\n" - ] - }, - { - "cell_type": "markdown", - "id": "e88f0213", - "metadata": {}, - "source": [ - "### 📤 Retrieve synthetic data\n", - "\n", - "Inspect the generated synthetic data including row count and preview of the first rows." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "5a7a48d2", - "metadata": {}, - "outputs": [], - "source": [ - "synth = results.synthetic_data\n", - "print(f\"Number of synthetic rows: {len(synth)}\")\n", - "synth.head()" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "8e8f90d5", - "metadata": {}, - "outputs": [], - "source": [ - "# Synthetic data and evaluation report are automatically saved to the artifacts directory\n", - "print(f\"Artifacts automatically saved to: {builder._workdir.generate.path}\")" - ] - }, - { - "cell_type": "markdown", - "id": "75ec6da5", - "metadata": {}, - "source": [ - "### 🛡️ Review evaluation report\n", - "\n", - "The pipeline computes both quality and privacy metrics. The summary includes timing information and overall scores, while the full evaluation report is rendered as an HTML document." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "e121493f", - "metadata": {}, - "outputs": [], - "source": [ - "import json\n", - "\n", - "print(\"Summary (timing and scores):\")\n", - "print(json.dumps(results.summary.model_dump(), indent=2))" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "a42bef2c", - "metadata": {}, - "outputs": [], - "source": [ - "# View the evaluation report in a sandboxed iframe\n", - "import base64\n", - "from IPython.display import IFrame, display\n", - "\n", - "report_html = results.evaluation_report_html\n", - "if report_html:\n", - " data_url = \"data:text/html;base64,\" + base64.b64encode(report_html.encode()).decode()\n", - " display(IFrame(src=data_url, width=\"100%\", height=800))" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": ".venv", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.13.12" + "cells": [ + { + "cell_type": "markdown", + "id": "d1d7a7a3", + "metadata": {}, + "source": [ + "\n", + "# 🔐 NeMo Safe Synthesizer Tutorial: The Basics\n", + "\n", + "#### What you'll learn\n", + "\n", + "In this notebook, we'll explore the fundamentals of NeMo Safe Synthesizer: PII replacement, training on a sample dataset, generating synthetic data, and evaluating quality and privacy.\n", + "\n", + "This library supports numeric, categorical, and text fields within the training data and generates realistic synthetic data that mirrors the structure of your data. A full run takes about 15 minutes on an A100.\n", + "\n", + "### 🖥️ Prerequisites\n", + "\n", + "This notebook requires a Linux machine with an NVIDIA GPU (H100 recommended, A100 minimum) and CUDA 12.9+. It will not run on macOS, Windows, or Apple Silicon." + ] + }, + { + "cell_type": "markdown", + "id": "d501f043", + "metadata": {}, + "source": [ + "### ⚡ Install Safe Synthesizer\n", + "\n", + "Run the cell below to install NeMo Safe Synthesizer (engine and CUDA 12.9) and the `datasets` library for the sample dataset." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "bb7b0bdd", + "metadata": { + "vscode": { + "languageId": "shellscript" } + }, + "outputs": [], + "source": [ + "%%bash\n", + "# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.\n", + "# SPDX-License-Identifier: Apache-2.0\n", + "\n", + "if command -v uv > /dev/null 2>&1; then\n", + " uv pip install \"nemo-safe-synthesizer[engine,cu129]\" \\\n", + " --index https://flashinfer.ai/whl/cu129 \\\n", + " --index https://flashinfer.ai/whl/ \\\n", + " --index https://download.pytorch.org/whl/cu129 \\\n", + " --index https://wheels.vllm.ai/0.26.0/cu129 \\\n", + " --index-strategy unsafe-best-match\n", + " uv pip install datasets\n", + "else\n", + " pip install \"nemo-safe-synthesizer[engine,cu129]\" \\\n", + " --extra-index-url https://flashinfer.ai/whl/cu129 \\\n", + " --extra-index-url https://flashinfer.ai/whl/ \\\n", + " --extra-index-url https://download.pytorch.org/whl/cu129 \\\n", + " --extra-index-url https://wheels.vllm.ai/0.26.0/cu129\n", + " pip install datasets\n", + "fi\n" + ] + }, + { + "cell_type": "markdown", + "id": "3030139c", + "metadata": {}, + "source": [ + "\n", + "### 🔑 Set the inference API key for PII column classification\n", + "\n", + "NeMo Safe Synthesizer uses an LLM‑based column classifier to automatically infer PII columns. To enable this feature, set `NSS_INFERENCE_KEY` (the inference endpoint defaults to the NVIDIA integrate URL. You can obtain an API key from [build.nvidia.com](https://build.nvidia.com/settings/api-keys)). Setting this value is optional but strongly recommended.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "693620c8", + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "import getpass\n", + "\n", + "# Setting NSS_INFERENCE_KEY is optional but strongly recommended for PII replacement.\n", + "if \"NSS_INFERENCE_KEY\" not in os.environ:\n", + " os.environ[\"NSS_INFERENCE_KEY\"] = getpass.getpass(\"Paste inference API key (or press Enter to skip): \")\n", + "if os.environ.get(\"NSS_INFERENCE_KEY\"):\n", + " print(\"NSS_INFERENCE_KEY is set\")\n", + "else:\n", + " print(\n", + " \"NSS_INFERENCE_KEY is not set. Replace PII will run in degraded mode. \"\n", + " \"We strongly recommend setting a key.\"\n", + " )" + ] + }, + { + "cell_type": "markdown", + "id": "bdb29834", + "metadata": {}, + "source": [ + "### 📥 Load and preview sample dataset\n", + "\n", + "Load a tabular dataset—in this example, the [clinc_oos](https://huggingface.co/datasets/clinc/clinc_oos) dataset from Hugging Face—and preview the first few rows. NeMo Safe Synthesizer will use this DataFrame as its training data.\n", + "\n", + "This dataset includes a text column and a categorical intent label supported by Nemo Safe Synthesizer.\n", + "\n", + "Each user is responsible for checking the content of datasets and the applicable licenses and determining if suitable for the intended use." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "fb1a562a", + "metadata": {}, + "outputs": [], + "source": [ + "from datasets import load_dataset\n", + "\n", + "dataset = load_dataset(\"clinc/clinc_oos\", \"small\")\n", + "df = dataset[\"train\"].to_pandas()\n", + "df.head()" + ] + }, + { + "cell_type": "markdown", + "id": "1c394bab", + "metadata": {}, + "source": [ + "\n", + "### ⚙️ Create and run Safe Synthesizer job\n", + "\n", + "Create the Safe Synthesizer builder and attach your DataFrame. Run the pipeline with `run()`, which performs data processing, PII replacement, training, generation, and evaluation in a single call. Results are available on `builder.results`.\n", + "\n", + "Refer to the [configuration docs](../user-guide/configuration.md) for the full list of options.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "9694992a", + "metadata": {}, + "outputs": [], + "source": [ + "import warnings\n", + "from nemo_safe_synthesizer.sdk.library_builder import SafeSynthesizer\n", + "\n", + "builder = SafeSynthesizer().with_data_source(df) # .with_replace_pii(enable=False) to disable PII replacement\n", + "with warnings.catch_warnings():\n", + " # ks_2samp falls back to asymptotic method for large samples; the result is still valid\n", + " warnings.filterwarnings(\"ignore\", message=\"ks_2samp: Exact calculation unsuccessful\", category=RuntimeWarning)\n", + " builder.run()\n", + "results = builder.results\n" + ] + }, + { + "cell_type": "markdown", + "id": "e88f0213", + "metadata": {}, + "source": [ + "### 📤 Retrieve synthetic data\n", + "\n", + "Inspect the generated synthetic data including row count and preview of the first rows." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "5a7a48d2", + "metadata": {}, + "outputs": [], + "source": [ + "synth = results.synthetic_data\n", + "print(f\"Number of synthetic rows: {len(synth)}\")\n", + "synth.head()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8e8f90d5", + "metadata": {}, + "outputs": [], + "source": [ + "# Synthetic data and evaluation report are automatically saved to the artifacts directory\n", + "print(f\"Artifacts automatically saved to: {builder._workdir.generate.path}\")" + ] + }, + { + "cell_type": "markdown", + "id": "75ec6da5", + "metadata": {}, + "source": [ + "### 🛡️ Review evaluation report\n", + "\n", + "The pipeline computes both quality and privacy metrics. The summary includes timing information and overall scores, while the full evaluation report is rendered as an HTML document." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e121493f", + "metadata": {}, + "outputs": [], + "source": [ + "import json\n", + "\n", + "print(\"Summary (timing and scores):\")\n", + "print(json.dumps(results.summary.model_dump(), indent=2))" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a42bef2c", + "metadata": {}, + "outputs": [], + "source": [ + "# View the evaluation report in a sandboxed iframe\n", + "import base64\n", + "from IPython.display import IFrame, display\n", + "\n", + "report_html = results.evaluation_report_html\n", + "if report_html:\n", + " data_url = \"data:text/html;base64,\" + base64.b64encode(report_html.encode()).decode()\n", + " display(IFrame(src=data_url, width=\"100%\", height=800))" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": ".venv (3.13.12)", + "language": "python", + "name": "python3" }, - "nbformat": 4, - "nbformat_minor": 5 + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.13.12" + } + }, + "nbformat": 4, + "nbformat_minor": 5 } diff --git a/script/brev/setup.sh b/script/brev/setup.sh index a123e5cf7..bb27937ab 100755 --- a/script/brev/setup.sh +++ b/script/brev/setup.sh @@ -268,6 +268,10 @@ env = { # Keeps `!uv pip install ...` in a notebook from resolving to the Brev # image's own ~/.venv, which uv would otherwise discover by walking up. "VIRTUAL_ENV": venv, + # bitsandbytes ships binaries for even CUDA releases only (12.6, 12.8, 13.0…). + # CUDA 12.9 has no native binary and always falls back to 12.8. + # Update to "130" when upgrading CUDA_EXTRA to cu130 or later. + "BNB_CUDA_VERSION": "128", } # Secrets live in the kernelspec because the Jupyter server is not launched # from a login shell. The VM is single-tenant and the file is mode 0600. @@ -343,6 +347,12 @@ if [[ "${registered}" -ne 1 ]]; then log "WARNING: kernel not registered; notebooks may open on the wrong Python" fi +# Pre-compile third-party packages that emit SyntaxWarnings on first import so +# the warnings go into the setup log rather than appearing in notebook output. +log "pre-compiling packages" +"${VENV_DIR}/bin/python" -W ignore::SyntaxWarning \ + -c "import torchao, range_regex" 2>/dev/null || true + # Smoke check -- fail provisioning loudly rather than handing over a broken VM. log "verifying install" diff --git a/src/nemo_safe_synthesizer/config/training.py b/src/nemo_safe_synthesizer/config/training.py index b47dec9ad..dc95453b6 100644 --- a/src/nemo_safe_synthesizer/config/training.py +++ b/src/nemo_safe_synthesizer/config/training.py @@ -210,12 +210,16 @@ class TrainingHyperparams(Parameters): ), ] = 0.01 - warmup_ratio: Annotated[ + warmup_steps: Annotated[ float, ValueValidator(value_func=lambda v: v > 0), Field( - title="warmup_ratio", - description="Ratio of total training steps used for a linear warmup from 0 to the learning rate. Must be > 0.", + title="warmup_steps", + description=( + "Linear warmup from 0 to the learning rate. " + "An integer sets the exact number of warmup steps; " + "a float in (0, 1) is treated as a ratio of total training steps. Must be > 0." + ), ), ] = 0.05 diff --git a/src/nemo_safe_synthesizer/pii_replacer/data_editor/detect.py b/src/nemo_safe_synthesizer/pii_replacer/data_editor/detect.py index ebe74ac27..ea37c57c4 100644 --- a/src/nemo_safe_synthesizer/pii_replacer/data_editor/detect.py +++ b/src/nemo_safe_synthesizer/pii_replacer/data_editor/detect.py @@ -5,6 +5,7 @@ import logging import os +import warnings from abc import ABC, abstractmethod from collections.abc import Callable, Iterable, Iterator from dataclasses import dataclass @@ -660,11 +661,13 @@ def get_entity_extractor( f"Loading NER model from filesystem to {map_location}", ) - extractor._model = GLiNER.from_pretrained( - clsfy_cfg.gliner_model, - map_location=map_location, - local_files_only=hf_offline_enabled(), - ) + with warnings.catch_warnings(): + warnings.filterwarnings("ignore", message="The `resume_download` argument is deprecated", category=UserWarning) + extractor._model = GLiNER.from_pretrained( + clsfy_cfg.gliner_model, + map_location=map_location, + local_files_only=hf_offline_enabled(), + ) entity_types = DEFAULT_ENTITIES if clsfy_cfg.ner_entities: entity_types = clsfy_cfg.ner_entities @@ -691,7 +694,7 @@ def _predict_entities(self, text: str, entity_labels: list[str]) -> list[dict]: flat_ner=False, ) - return self._model.batch_predict_entities( + return self._model.inference( [text], entity_labels, threshold=self._ner_threshold, @@ -699,9 +702,9 @@ def _predict_entities(self, text: str, entity_labels: list[str]) -> list[dict]: )[0] def _batch_predict_entities(self, texts: list[str], entity_labels: list[str]) -> list[list[dict]]: - batch_predict_entities = getattr(self._model, "batch_predict_entities", None) - if batch_predict_entities is not None: - return batch_predict_entities( + inference = getattr(self._model, "inference", None) + if inference is not None: + return inference( texts, entity_labels, threshold=self._ner_threshold, @@ -720,7 +723,7 @@ def _batch_predict_entities(self, texts: list[str], entity_labels: list[str]) -> for text in texts ] - raise AttributeError("GLiNER model has neither batch_predict_entities nor predict_entities") + raise AttributeError("GLiNER model has neither inference nor predict_entities") def _detect_entities_chunked( self, diff --git a/src/nemo_safe_synthesizer/training/huggingface_backend.py b/src/nemo_safe_synthesizer/training/huggingface_backend.py index 6875c95a8..51da170e9 100644 --- a/src/nemo_safe_synthesizer/training/huggingface_backend.py +++ b/src/nemo_safe_synthesizer/training/huggingface_backend.py @@ -421,7 +421,7 @@ def _build_base_training_args(self) -> dict: learning_rate=self.params.training.learning_rate, eval_strategy=evaluation_strategy, weight_decay=self.params.training.weight_decay, - warmup_ratio=self.params.training.warmup_ratio, + warmup_steps=self.params.training.warmup_steps, eval_steps=EVAL_STEPS, do_eval=self.params.training.validation_ratio > 0, disable_tqdm=True, # The 🤗 progress bar doesn't play nice with our logging. diff --git a/tests/conftest.py b/tests/conftest.py index 5fd840d44..0ee52a342 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -166,7 +166,7 @@ def fixture_yaml_config_str() -> str: rope_scaling_factor: auto validation_ratio: 0.0 validation_steps: 15 - warmup_ratio: 0.05 + warmup_steps: 0.05 weight_decay: 0.01 """ diff --git a/tests/pii_replacer/test_detect.py b/tests/pii_replacer/test_detect.py index 79cd94d43..ba9831c57 100644 --- a/tests/pii_replacer/test_detect.py +++ b/tests/pii_replacer/test_detect.py @@ -67,7 +67,7 @@ def test_gliner_batch_predict_config(): entity_extractor = EntityExtractorGliner.get_entity_extractor(cfg) entity_extractor.batch_update_cache(["abc"], None) assert entity_extractor._model is not None - entity_extractor._model.batch_predict_entities.assert_not_called() # ty: ignore[call-non-callable, unresolved-attribute] -- mock object + entity_extractor._model.inference.assert_not_called() # ty: ignore[call-non-callable, unresolved-attribute] -- mock object cfg = ClassifyConfig( valid_entities={"name"}, @@ -85,7 +85,7 @@ def test_gliner_batch_predict_config(): entity_extractor = EntityExtractorGliner.get_entity_extractor(cfg) entity_extractor.batch_update_cache(["abc"], None) assert entity_extractor._model is not None - entity_extractor._model.batch_predict_entities.assert_called() # ty: ignore[call-non-callable, unresolved-attribute] -- mock object + entity_extractor._model.inference.assert_called() # ty: ignore[call-non-callable, unresolved-attribute] -- mock object def test_gliner_entity_labels_are_indexable(): @@ -104,14 +104,14 @@ def test_gliner_entity_labels_are_indexable(): with patch("nemo_safe_synthesizer.pii_replacer.data_editor.detect.GLiNER") as mock_gliner: model = mock_gliner.from_pretrained.return_value model.predict_entities.return_value = [] - model.batch_predict_entities.return_value = [[]] + model.inference.return_value = [[]] entity_extractor = EntityExtractorGliner.get_entity_extractor(cfg) entity_extractor.extract_ner_predictions("abc", {"name", "email"}) entity_extractor.batch_update_cache(["abc"], None) assert model.predict_entities.call_args.args[1] == ["email", "name"] - assert model.batch_predict_entities.call_args.args[1] == ["email", "name"] + assert model.inference.call_args.args[1] == ["email", "name"] def test_gliner_batch_cache_falls_back_to_predict_entities_api(): @@ -147,7 +147,7 @@ def predict_entities(self, text, labels, **kwargs): assert model.calls[0][1] == ["email", "name"] -def test_gliner_single_prediction_falls_back_to_batch_api(): +def test_gliner_single_prediction_falls_back_to_inference_api(): cfg = ClassifyConfig( valid_entities={"email", "name"}, ner_threshold=0.8, @@ -164,7 +164,7 @@ class FakeGLiNER: def __init__(self): self.calls = [] - def batch_predict_entities(self, texts, labels, **kwargs): + def inference(self, texts, labels, **kwargs): self.calls.append((texts, labels, kwargs)) return [[]] diff --git a/tests/training/test_huggingface_backend.py b/tests/training/test_huggingface_backend.py index b387adb95..3287e0b2f 100644 --- a/tests/training/test_huggingface_backend.py +++ b/tests/training/test_huggingface_backend.py @@ -133,21 +133,23 @@ def params_with_orderby(base_params): @pytest.fixture def backend(base_params, mock_model_metadata, mock_workdir): """Create a HuggingFaceBackend instance for testing.""" - return HuggingFaceBackend( + b = HuggingFaceBackend( params=base_params, model_metadata=mock_model_metadata, workdir=mock_workdir, ) + return b @pytest.fixture def backend_with_validation(params_with_validation, mock_model_metadata, mock_workdir): """Create a HuggingFaceBackend instance with validation enabled.""" - return HuggingFaceBackend( + b = HuggingFaceBackend( params=params_with_validation, model_metadata=mock_model_metadata, workdir=mock_workdir, ) + return b @pytest.fixture From 037477d7fb807a559ff313c0b3e1d12ab078ab93 Mon Sep 17 00:00:00 2001 From: Yunfeng Zhang Date: Thu, 6 Aug 2026 15:21:44 +0000 Subject: [PATCH 02/11] fix: keep warmup_ratio as deprecated alias for warmup_steps Mirror transformers behaviour: warmup_ratio is accepted but emits a DeprecationWarning and copies its value to warmup_steps. Also fix notebook JSON indentation (indent=1 -> indent=2). Signed-off-by: Yunfeng Zhang --- docs/tutorials/safe-synthesizer-101.ipynb | 496 +++++++++---------- src/nemo_safe_synthesizer/config/training.py | 23 +- 2 files changed, 270 insertions(+), 249 deletions(-) diff --git a/docs/tutorials/safe-synthesizer-101.ipynb b/docs/tutorials/safe-synthesizer-101.ipynb index 136ed2ed2..4462ba23a 100644 --- a/docs/tutorials/safe-synthesizer-101.ipynb +++ b/docs/tutorials/safe-synthesizer-101.ipynb @@ -1,252 +1,252 @@ { - "cells": [ - { - "cell_type": "markdown", - "id": "d1d7a7a3", - "metadata": {}, - "source": [ - "\n", - "# 🔐 NeMo Safe Synthesizer Tutorial: The Basics\n", - "\n", - "#### What you'll learn\n", - "\n", - "In this notebook, we'll explore the fundamentals of NeMo Safe Synthesizer: PII replacement, training on a sample dataset, generating synthetic data, and evaluating quality and privacy.\n", - "\n", - "This library supports numeric, categorical, and text fields within the training data and generates realistic synthetic data that mirrors the structure of your data. A full run takes about 15 minutes on an A100.\n", - "\n", - "### 🖥️ Prerequisites\n", - "\n", - "This notebook requires a Linux machine with an NVIDIA GPU (H100 recommended, A100 minimum) and CUDA 12.9+. It will not run on macOS, Windows, or Apple Silicon." - ] - }, - { - "cell_type": "markdown", - "id": "d501f043", - "metadata": {}, - "source": [ - "### ⚡ Install Safe Synthesizer\n", - "\n", - "Run the cell below to install NeMo Safe Synthesizer (engine and CUDA 12.9) and the `datasets` library for the sample dataset." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "bb7b0bdd", - "metadata": { - "vscode": { - "languageId": "shellscript" + "cells": [ + { + "cell_type": "markdown", + "id": "d1d7a7a3", + "metadata": {}, + "source": [ + "\n", + "# \ud83d\udd10 NeMo Safe Synthesizer Tutorial: The Basics\n", + "\n", + "#### What you'll learn\n", + "\n", + "In this notebook, we'll explore the fundamentals of NeMo Safe Synthesizer: PII replacement, training on a sample dataset, generating synthetic data, and evaluating quality and privacy.\n", + "\n", + "This library supports numeric, categorical, and text fields within the training data and generates realistic synthetic data that mirrors the structure of your data. A full run takes about 15 minutes on an A100.\n", + "\n", + "### \ud83d\udda5\ufe0f Prerequisites\n", + "\n", + "This notebook requires a Linux machine with an NVIDIA GPU (H100 recommended, A100 minimum) and CUDA 12.9+. It will not run on macOS, Windows, or Apple Silicon." + ] + }, + { + "cell_type": "markdown", + "id": "d501f043", + "metadata": {}, + "source": [ + "### \u26a1 Install Safe Synthesizer\n", + "\n", + "Run the cell below to install NeMo Safe Synthesizer (engine and CUDA 12.9) and the `datasets` library for the sample dataset." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "bb7b0bdd", + "metadata": { + "vscode": { + "languageId": "shellscript" + } + }, + "outputs": [], + "source": [ + "%%bash\n", + "# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.\n", + "# SPDX-License-Identifier: Apache-2.0\n", + "\n", + "if command -v uv > /dev/null 2>&1; then\n", + " uv pip install \"nemo-safe-synthesizer[engine,cu129]\" \\\n", + " --index https://flashinfer.ai/whl/cu129 \\\n", + " --index https://flashinfer.ai/whl/ \\\n", + " --index https://download.pytorch.org/whl/cu129 \\\n", + " --index https://wheels.vllm.ai/0.26.0/cu129 \\\n", + " --index-strategy unsafe-best-match\n", + " uv pip install datasets\n", + "else\n", + " pip install \"nemo-safe-synthesizer[engine,cu129]\" \\\n", + " --extra-index-url https://flashinfer.ai/whl/cu129 \\\n", + " --extra-index-url https://flashinfer.ai/whl/ \\\n", + " --extra-index-url https://download.pytorch.org/whl/cu129 \\\n", + " --extra-index-url https://wheels.vllm.ai/0.26.0/cu129\n", + " pip install datasets\n", + "fi\n" + ] + }, + { + "cell_type": "markdown", + "id": "3030139c", + "metadata": {}, + "source": [ + "\n", + "### \ud83d\udd11 Set the inference API key for PII column classification\n", + "\n", + "NeMo Safe Synthesizer uses an LLM\u2011based column classifier to automatically infer PII columns. To enable this feature, set `NSS_INFERENCE_KEY` (the inference endpoint defaults to the NVIDIA integrate URL. You can obtain an API key from [build.nvidia.com](https://build.nvidia.com/settings/api-keys)). Setting this value is optional but strongly recommended.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "693620c8", + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "import getpass\n", + "\n", + "# Setting NSS_INFERENCE_KEY is optional but strongly recommended for PII replacement.\n", + "if \"NSS_INFERENCE_KEY\" not in os.environ:\n", + " os.environ[\"NSS_INFERENCE_KEY\"] = getpass.getpass(\"Paste inference API key (or press Enter to skip): \")\n", + "if os.environ.get(\"NSS_INFERENCE_KEY\"):\n", + " print(\"NSS_INFERENCE_KEY is set\")\n", + "else:\n", + " print(\n", + " \"NSS_INFERENCE_KEY is not set. Replace PII will run in degraded mode. \"\n", + " \"We strongly recommend setting a key.\"\n", + " )" + ] + }, + { + "cell_type": "markdown", + "id": "bdb29834", + "metadata": {}, + "source": [ + "### \ud83d\udce5 Load and preview sample dataset\n", + "\n", + "Load a tabular dataset\u2014in this example, the [clinc_oos](https://huggingface.co/datasets/clinc/clinc_oos) dataset from Hugging Face\u2014and preview the first few rows. NeMo Safe Synthesizer will use this DataFrame as its training data.\n", + "\n", + "This dataset includes a text column and a categorical intent label supported by Nemo Safe Synthesizer.\n", + "\n", + "Each user is responsible for checking the content of datasets and the applicable licenses and determining if suitable for the intended use." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "fb1a562a", + "metadata": {}, + "outputs": [], + "source": [ + "from datasets import load_dataset\n", + "\n", + "dataset = load_dataset(\"clinc/clinc_oos\", \"small\")\n", + "df = dataset[\"train\"].to_pandas()\n", + "df.head()" + ] + }, + { + "cell_type": "markdown", + "id": "1c394bab", + "metadata": {}, + "source": [ + "\n", + "### \u2699\ufe0f Create and run Safe Synthesizer job\n", + "\n", + "Create the Safe Synthesizer builder and attach your DataFrame. Run the pipeline with `run()`, which performs data processing, PII replacement, training, generation, and evaluation in a single call. Results are available on `builder.results`.\n", + "\n", + "Refer to the [configuration docs](../user-guide/configuration.md) for the full list of options.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "9694992a", + "metadata": {}, + "outputs": [], + "source": [ + "import warnings\n", + "from nemo_safe_synthesizer.sdk.library_builder import SafeSynthesizer\n", + "\n", + "builder = SafeSynthesizer().with_data_source(df) # .with_replace_pii(enable=False) to disable PII replacement\n", + "with warnings.catch_warnings():\n", + " # ks_2samp falls back to asymptotic method for large samples; the result is still valid\n", + " warnings.filterwarnings(\"ignore\", message=\"ks_2samp: Exact calculation unsuccessful\", category=RuntimeWarning)\n", + " builder.run()\n", + "results = builder.results\n" + ] + }, + { + "cell_type": "markdown", + "id": "e88f0213", + "metadata": {}, + "source": [ + "### \ud83d\udce4 Retrieve synthetic data\n", + "\n", + "Inspect the generated synthetic data including row count and preview of the first rows." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "5a7a48d2", + "metadata": {}, + "outputs": [], + "source": [ + "synth = results.synthetic_data\n", + "print(f\"Number of synthetic rows: {len(synth)}\")\n", + "synth.head()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8e8f90d5", + "metadata": {}, + "outputs": [], + "source": [ + "# Synthetic data and evaluation report are automatically saved to the artifacts directory\n", + "print(f\"Artifacts automatically saved to: {builder._workdir.generate.path}\")" + ] + }, + { + "cell_type": "markdown", + "id": "75ec6da5", + "metadata": {}, + "source": [ + "### \ud83d\udee1\ufe0f Review evaluation report\n", + "\n", + "The pipeline computes both quality and privacy metrics. The summary includes timing information and overall scores, while the full evaluation report is rendered as an HTML document." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e121493f", + "metadata": {}, + "outputs": [], + "source": [ + "import json\n", + "\n", + "print(\"Summary (timing and scores):\")\n", + "print(json.dumps(results.summary.model_dump(), indent=2))" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a42bef2c", + "metadata": {}, + "outputs": [], + "source": [ + "# View the evaluation report in a sandboxed iframe\n", + "import base64\n", + "from IPython.display import IFrame, display\n", + "\n", + "report_html = results.evaluation_report_html\n", + "if report_html:\n", + " data_url = \"data:text/html;base64,\" + base64.b64encode(report_html.encode()).decode()\n", + " display(IFrame(src=data_url, width=\"100%\", height=800))" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": ".venv (3.13.12)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.13.12" } - }, - "outputs": [], - "source": [ - "%%bash\n", - "# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.\n", - "# SPDX-License-Identifier: Apache-2.0\n", - "\n", - "if command -v uv > /dev/null 2>&1; then\n", - " uv pip install \"nemo-safe-synthesizer[engine,cu129]\" \\\n", - " --index https://flashinfer.ai/whl/cu129 \\\n", - " --index https://flashinfer.ai/whl/ \\\n", - " --index https://download.pytorch.org/whl/cu129 \\\n", - " --index https://wheels.vllm.ai/0.26.0/cu129 \\\n", - " --index-strategy unsafe-best-match\n", - " uv pip install datasets\n", - "else\n", - " pip install \"nemo-safe-synthesizer[engine,cu129]\" \\\n", - " --extra-index-url https://flashinfer.ai/whl/cu129 \\\n", - " --extra-index-url https://flashinfer.ai/whl/ \\\n", - " --extra-index-url https://download.pytorch.org/whl/cu129 \\\n", - " --extra-index-url https://wheels.vllm.ai/0.26.0/cu129\n", - " pip install datasets\n", - "fi\n" - ] - }, - { - "cell_type": "markdown", - "id": "3030139c", - "metadata": {}, - "source": [ - "\n", - "### 🔑 Set the inference API key for PII column classification\n", - "\n", - "NeMo Safe Synthesizer uses an LLM‑based column classifier to automatically infer PII columns. To enable this feature, set `NSS_INFERENCE_KEY` (the inference endpoint defaults to the NVIDIA integrate URL. You can obtain an API key from [build.nvidia.com](https://build.nvidia.com/settings/api-keys)). Setting this value is optional but strongly recommended.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "693620c8", - "metadata": {}, - "outputs": [], - "source": [ - "import os\n", - "import getpass\n", - "\n", - "# Setting NSS_INFERENCE_KEY is optional but strongly recommended for PII replacement.\n", - "if \"NSS_INFERENCE_KEY\" not in os.environ:\n", - " os.environ[\"NSS_INFERENCE_KEY\"] = getpass.getpass(\"Paste inference API key (or press Enter to skip): \")\n", - "if os.environ.get(\"NSS_INFERENCE_KEY\"):\n", - " print(\"NSS_INFERENCE_KEY is set\")\n", - "else:\n", - " print(\n", - " \"NSS_INFERENCE_KEY is not set. Replace PII will run in degraded mode. \"\n", - " \"We strongly recommend setting a key.\"\n", - " )" - ] - }, - { - "cell_type": "markdown", - "id": "bdb29834", - "metadata": {}, - "source": [ - "### 📥 Load and preview sample dataset\n", - "\n", - "Load a tabular dataset—in this example, the [clinc_oos](https://huggingface.co/datasets/clinc/clinc_oos) dataset from Hugging Face—and preview the first few rows. NeMo Safe Synthesizer will use this DataFrame as its training data.\n", - "\n", - "This dataset includes a text column and a categorical intent label supported by Nemo Safe Synthesizer.\n", - "\n", - "Each user is responsible for checking the content of datasets and the applicable licenses and determining if suitable for the intended use." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "fb1a562a", - "metadata": {}, - "outputs": [], - "source": [ - "from datasets import load_dataset\n", - "\n", - "dataset = load_dataset(\"clinc/clinc_oos\", \"small\")\n", - "df = dataset[\"train\"].to_pandas()\n", - "df.head()" - ] - }, - { - "cell_type": "markdown", - "id": "1c394bab", - "metadata": {}, - "source": [ - "\n", - "### ⚙️ Create and run Safe Synthesizer job\n", - "\n", - "Create the Safe Synthesizer builder and attach your DataFrame. Run the pipeline with `run()`, which performs data processing, PII replacement, training, generation, and evaluation in a single call. Results are available on `builder.results`.\n", - "\n", - "Refer to the [configuration docs](../user-guide/configuration.md) for the full list of options.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "9694992a", - "metadata": {}, - "outputs": [], - "source": [ - "import warnings\n", - "from nemo_safe_synthesizer.sdk.library_builder import SafeSynthesizer\n", - "\n", - "builder = SafeSynthesizer().with_data_source(df) # .with_replace_pii(enable=False) to disable PII replacement\n", - "with warnings.catch_warnings():\n", - " # ks_2samp falls back to asymptotic method for large samples; the result is still valid\n", - " warnings.filterwarnings(\"ignore\", message=\"ks_2samp: Exact calculation unsuccessful\", category=RuntimeWarning)\n", - " builder.run()\n", - "results = builder.results\n" - ] - }, - { - "cell_type": "markdown", - "id": "e88f0213", - "metadata": {}, - "source": [ - "### 📤 Retrieve synthetic data\n", - "\n", - "Inspect the generated synthetic data including row count and preview of the first rows." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "5a7a48d2", - "metadata": {}, - "outputs": [], - "source": [ - "synth = results.synthetic_data\n", - "print(f\"Number of synthetic rows: {len(synth)}\")\n", - "synth.head()" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "8e8f90d5", - "metadata": {}, - "outputs": [], - "source": [ - "# Synthetic data and evaluation report are automatically saved to the artifacts directory\n", - "print(f\"Artifacts automatically saved to: {builder._workdir.generate.path}\")" - ] - }, - { - "cell_type": "markdown", - "id": "75ec6da5", - "metadata": {}, - "source": [ - "### 🛡️ Review evaluation report\n", - "\n", - "The pipeline computes both quality and privacy metrics. The summary includes timing information and overall scores, while the full evaluation report is rendered as an HTML document." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "e121493f", - "metadata": {}, - "outputs": [], - "source": [ - "import json\n", - "\n", - "print(\"Summary (timing and scores):\")\n", - "print(json.dumps(results.summary.model_dump(), indent=2))" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "a42bef2c", - "metadata": {}, - "outputs": [], - "source": [ - "# View the evaluation report in a sandboxed iframe\n", - "import base64\n", - "from IPython.display import IFrame, display\n", - "\n", - "report_html = results.evaluation_report_html\n", - "if report_html:\n", - " data_url = \"data:text/html;base64,\" + base64.b64encode(report_html.encode()).decode()\n", - " display(IFrame(src=data_url, width=\"100%\", height=800))" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": ".venv (3.13.12)", - "language": "python", - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.13.12" - } - }, - "nbformat": 4, - "nbformat_minor": 5 + "nbformat": 4, + "nbformat_minor": 5 } diff --git a/src/nemo_safe_synthesizer/config/training.py b/src/nemo_safe_synthesizer/config/training.py index dc95453b6..dce0d4f5c 100644 --- a/src/nemo_safe_synthesizer/config/training.py +++ b/src/nemo_safe_synthesizer/config/training.py @@ -4,6 +4,7 @@ from __future__ import annotations import importlib +import warnings from enum import StrEnum from typing import ( TYPE_CHECKING, @@ -11,7 +12,7 @@ Literal, ) -from pydantic import Field +from pydantic import Field, model_validator from ..configurator.parameters import ( Parameters, @@ -223,6 +224,26 @@ class TrainingHyperparams(Parameters): ), ] = 0.05 + warmup_ratio: Annotated[ + float | None, + Field( + title="warmup_ratio", + description="Deprecated. Use warmup_steps instead.", + exclude=True, + ), + ] = None + + @model_validator(mode="after") + def _migrate_warmup_ratio(self) -> TrainingHyperparams: + if self.warmup_ratio is not None: + warnings.warn( + "warmup_ratio is deprecated and will be removed in a future release. Use warmup_steps instead.", + DeprecationWarning, + stacklevel=2, + ) + self.warmup_steps = self.warmup_ratio + return self + lr_scheduler: Annotated[ str, Field( From 8bad412d85113fcc6dd1e27a40d95649577b0d6a Mon Sep 17 00:00:00 2001 From: Yunfeng Zhang Date: Thu, 6 Aug 2026 15:24:17 +0000 Subject: [PATCH 03/11] fix(docs): restore notebook formatting (ensure_ascii, kernelspec) Signed-off-by: Yunfeng Zhang --- docs/tutorials/safe-synthesizer-101.ipynb | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/tutorials/safe-synthesizer-101.ipynb b/docs/tutorials/safe-synthesizer-101.ipynb index 4462ba23a..8f4e1c997 100644 --- a/docs/tutorials/safe-synthesizer-101.ipynb +++ b/docs/tutorials/safe-synthesizer-101.ipynb @@ -6,7 +6,7 @@ "metadata": {}, "source": [ "\n", - "# \ud83d\udd10 NeMo Safe Synthesizer Tutorial: The Basics\n", + "# 🔐 NeMo Safe Synthesizer Tutorial: The Basics\n", "\n", "#### What you'll learn\n", "\n", @@ -14,7 +14,7 @@ "\n", "This library supports numeric, categorical, and text fields within the training data and generates realistic synthetic data that mirrors the structure of your data. A full run takes about 15 minutes on an A100.\n", "\n", - "### \ud83d\udda5\ufe0f Prerequisites\n", + "### 🖥️ Prerequisites\n", "\n", "This notebook requires a Linux machine with an NVIDIA GPU (H100 recommended, A100 minimum) and CUDA 12.9+. It will not run on macOS, Windows, or Apple Silicon." ] @@ -24,7 +24,7 @@ "id": "d501f043", "metadata": {}, "source": [ - "### \u26a1 Install Safe Synthesizer\n", + "### ⚡ Install Safe Synthesizer\n", "\n", "Run the cell below to install NeMo Safe Synthesizer (engine and CUDA 12.9) and the `datasets` library for the sample dataset." ] @@ -68,9 +68,9 @@ "metadata": {}, "source": [ "\n", - "### \ud83d\udd11 Set the inference API key for PII column classification\n", + "### 🔑 Set the inference API key for PII column classification\n", "\n", - "NeMo Safe Synthesizer uses an LLM\u2011based column classifier to automatically infer PII columns. To enable this feature, set `NSS_INFERENCE_KEY` (the inference endpoint defaults to the NVIDIA integrate URL. You can obtain an API key from [build.nvidia.com](https://build.nvidia.com/settings/api-keys)). Setting this value is optional but strongly recommended.\n" + "NeMo Safe Synthesizer uses an LLM‑based column classifier to automatically infer PII columns. To enable this feature, set `NSS_INFERENCE_KEY` (the inference endpoint defaults to the NVIDIA integrate URL. You can obtain an API key from [build.nvidia.com](https://build.nvidia.com/settings/api-keys)). Setting this value is optional but strongly recommended.\n" ] }, { @@ -100,9 +100,9 @@ "id": "bdb29834", "metadata": {}, "source": [ - "### \ud83d\udce5 Load and preview sample dataset\n", + "### 📥 Load and preview sample dataset\n", "\n", - "Load a tabular dataset\u2014in this example, the [clinc_oos](https://huggingface.co/datasets/clinc/clinc_oos) dataset from Hugging Face\u2014and preview the first few rows. NeMo Safe Synthesizer will use this DataFrame as its training data.\n", + "Load a tabular dataset—in this example, the [clinc_oos](https://huggingface.co/datasets/clinc/clinc_oos) dataset from Hugging Face—and preview the first few rows. NeMo Safe Synthesizer will use this DataFrame as its training data.\n", "\n", "This dataset includes a text column and a categorical intent label supported by Nemo Safe Synthesizer.\n", "\n", @@ -129,7 +129,7 @@ "metadata": {}, "source": [ "\n", - "### \u2699\ufe0f Create and run Safe Synthesizer job\n", + "### ⚙️ Create and run Safe Synthesizer job\n", "\n", "Create the Safe Synthesizer builder and attach your DataFrame. Run the pipeline with `run()`, which performs data processing, PII replacement, training, generation, and evaluation in a single call. Results are available on `builder.results`.\n", "\n", @@ -159,7 +159,7 @@ "id": "e88f0213", "metadata": {}, "source": [ - "### \ud83d\udce4 Retrieve synthetic data\n", + "### 📤 Retrieve synthetic data\n", "\n", "Inspect the generated synthetic data including row count and preview of the first rows." ] @@ -192,7 +192,7 @@ "id": "75ec6da5", "metadata": {}, "source": [ - "### \ud83d\udee1\ufe0f Review evaluation report\n", + "### 🛡️ Review evaluation report\n", "\n", "The pipeline computes both quality and privacy metrics. The summary includes timing information and overall scores, while the full evaluation report is rendered as an HTML document." ] @@ -230,7 +230,7 @@ ], "metadata": { "kernelspec": { - "display_name": ".venv (3.13.12)", + "display_name": ".venv", "language": "python", "name": "python3" }, From 4bd1dca3b1dfd7f1dbe434e77a062cc80ae3f890 Mon Sep 17 00:00:00 2001 From: Yunfeng Zhang Date: Thu, 6 Aug 2026 15:29:14 +0000 Subject: [PATCH 04/11] fix: address bot review comments - warmup_ratio migration only copies to warmup_steps when warmup_steps was not explicitly set, so explicit warmup_steps always takes precedence over the deprecated alias - _predict_entities now raises the same explicit AttributeError as _batch_predict_entities when the model has neither predict_entities nor inference Signed-off-by: Yunfeng Zhang --- src/nemo_safe_synthesizer/config/training.py | 3 ++- .../pii_replacer/data_editor/detect.py | 16 ++++++++++------ 2 files changed, 12 insertions(+), 7 deletions(-) diff --git a/src/nemo_safe_synthesizer/config/training.py b/src/nemo_safe_synthesizer/config/training.py index dce0d4f5c..983900e4a 100644 --- a/src/nemo_safe_synthesizer/config/training.py +++ b/src/nemo_safe_synthesizer/config/training.py @@ -241,7 +241,8 @@ def _migrate_warmup_ratio(self) -> TrainingHyperparams: DeprecationWarning, stacklevel=2, ) - self.warmup_steps = self.warmup_ratio + if "warmup_steps" not in self.model_fields_set: + self.warmup_steps = self.warmup_ratio return self lr_scheduler: Annotated[ diff --git a/src/nemo_safe_synthesizer/pii_replacer/data_editor/detect.py b/src/nemo_safe_synthesizer/pii_replacer/data_editor/detect.py index ea37c57c4..25ffd4524 100644 --- a/src/nemo_safe_synthesizer/pii_replacer/data_editor/detect.py +++ b/src/nemo_safe_synthesizer/pii_replacer/data_editor/detect.py @@ -694,12 +694,16 @@ def _predict_entities(self, text: str, entity_labels: list[str]) -> list[dict]: flat_ner=False, ) - return self._model.inference( - [text], - entity_labels, - threshold=self._ner_threshold, - flat_ner=False, - )[0] + inference = getattr(self._model, "inference", None) + if inference is not None: + return inference( + [text], + entity_labels, + threshold=self._ner_threshold, + flat_ner=False, + )[0] + + raise AttributeError("GLiNER model has neither inference nor predict_entities") def _batch_predict_entities(self, texts: list[str], entity_labels: list[str]) -> list[list[dict]]: inference = getattr(self._model, "inference", None) From a4ea135257dbe355940d1e8e78c4ceaa041e907a Mon Sep 17 00:00:00 2001 From: Yunfeng Zhang Date: Fri, 7 Aug 2026 13:20:01 +0000 Subject: [PATCH 05/11] style: wrap long filterwarnings line Signed-off-by: Yunfeng Zhang --- src/nemo_safe_synthesizer/pii_replacer/data_editor/detect.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/nemo_safe_synthesizer/pii_replacer/data_editor/detect.py b/src/nemo_safe_synthesizer/pii_replacer/data_editor/detect.py index 25ffd4524..0b13eae7f 100644 --- a/src/nemo_safe_synthesizer/pii_replacer/data_editor/detect.py +++ b/src/nemo_safe_synthesizer/pii_replacer/data_editor/detect.py @@ -662,7 +662,9 @@ def get_entity_extractor( ) with warnings.catch_warnings(): - warnings.filterwarnings("ignore", message="The `resume_download` argument is deprecated", category=UserWarning) + warnings.filterwarnings( + "ignore", message="The `resume_download` argument is deprecated", category=UserWarning + ) extractor._model = GLiNER.from_pretrained( clsfy_cfg.gliner_model, map_location=map_location, From 87f128cfdfc35feae2983e21cfd31d216cc2ebb4 Mon Sep 17 00:00:00 2001 From: Yunfeng Zhang Date: Fri, 7 Aug 2026 13:28:09 +0000 Subject: [PATCH 06/11] fix(config): reject unusable warmup values warmup_steps accepted 1.5, which transformers truncates to 1, and float("inf"), which raises OverflowError once converted to an integer. Validate that the value is finite, positive, and either a ratio below 1 or a whole step count. Apply the same check to the deprecated warmup_ratio alias: it assigns to warmup_steps from an "after" model validator, and validate_assignment is off, so field validation would otherwise be bypassed entirely. Signed-off-by: Yunfeng Zhang --- src/nemo_safe_synthesizer/config/training.py | 21 ++++++-- tests/config/test_parameters.py | 54 +++++++++++++++++++- 2 files changed, 71 insertions(+), 4 deletions(-) diff --git a/src/nemo_safe_synthesizer/config/training.py b/src/nemo_safe_synthesizer/config/training.py index 983900e4a..817b8bae7 100644 --- a/src/nemo_safe_synthesizer/config/training.py +++ b/src/nemo_safe_synthesizer/config/training.py @@ -4,6 +4,7 @@ from __future__ import annotations import importlib +import math import warnings from enum import StrEnum from typing import ( @@ -152,6 +153,18 @@ def _mxfp4_config() -> QuantizationConfigMixin: ValueGTZero = ValueValidator(lambda p: range_validator(p, lambda v: v >= 0)) +def is_valid_warmup(value: float) -> bool: + """Whether a warmup setting is a usable ratio or step count. + + Mirrors how transformers interprets ``warmup_steps``: values below 1 are a + ratio of total training steps, values of 1 or more are an absolute step + count. Fractional values of 1 or more are rejected because transformers + truncates them (``1.5`` silently becomes ``1``), and non-finite values are + rejected because they raise ``OverflowError`` once converted to an integer. + """ + return math.isfinite(value) and value > 0 and (value < 1 or float(value).is_integer()) + + class TrainingHyperparams(Parameters): """Hyperparameters that control the training process behavior. @@ -213,19 +226,21 @@ class TrainingHyperparams(Parameters): warmup_steps: Annotated[ float, - ValueValidator(value_func=lambda v: v > 0), + ValueValidator(value_func=is_valid_warmup), Field( title="warmup_steps", description=( "Linear warmup from 0 to the learning rate. " - "An integer sets the exact number of warmup steps; " - "a float in (0, 1) is treated as a ratio of total training steps. Must be > 0." + "A whole number of 1 or more sets the exact number of warmup steps; " + "a float in (0, 1) is treated as a ratio of total training steps. " + "Must be finite and > 0, and cannot be fractional at or above 1." ), ), ] = 0.05 warmup_ratio: Annotated[ float | None, + ValueValidator(value_func=lambda v: v is None or is_valid_warmup(v)), Field( title="warmup_ratio", description="Deprecated. Use warmup_steps instead.", diff --git a/tests/config/test_parameters.py b/tests/config/test_parameters.py index 20ad9b2f7..5a1d84e71 100644 --- a/tests/config/test_parameters.py +++ b/tests/config/test_parameters.py @@ -14,7 +14,7 @@ from nemo_safe_synthesizer.config.job import SafeSynthesizerJobConfig from nemo_safe_synthesizer.config.parameters import SafeSynthesizerParameters from nemo_safe_synthesizer.config.replace_pii import PiiReplacerConfig, StepDefinition -from nemo_safe_synthesizer.config.training import QuantizationScheme +from nemo_safe_synthesizer.config.training import QuantizationScheme, TrainingHyperparams from nemo_safe_synthesizer.configurator.parameter_paths import ( AmbiguousParameterName, ParameterFieldKind, @@ -885,3 +885,55 @@ def test_returned_config_is_independent_of_saved(self): assert saved.data.holdout != 0.42 assert saved.training.batch_size == 8 assert saved.generation.structured_generation.enabled is True + + +class TestWarmupSteps: + """`warmup_steps` mirrors the transformers contract: ratio below 1, whole steps at or above 1.""" + + @pytest.mark.parametrize("value", [0.05, 0.5, 0.999, 1, 1.0, 10, 10.0, 500]) + def test_accepts_ratios_and_whole_step_counts(self, value): + assert TrainingHyperparams(warmup_steps=value).warmup_steps == value + + @pytest.mark.parametrize( + "value", + [ + 1.5, # transformers truncates to 1, so reject rather than silently change the schedule + 2.7, + float("inf"), # OverflowError once transformers converts it to an int + float("-inf"), + float("nan"), + 0, + -1, + ], + ) + def test_rejects_fractional_non_finite_and_non_positive(self, value): + with pytest.raises((ParameterError, ValidationError, ValueError)): + TrainingHyperparams(warmup_steps=value) + + def test_default_is_a_ratio(self): + assert TrainingHyperparams().warmup_steps == 0.05 + + +class TestWarmupRatioDeprecation: + """`warmup_ratio` stays accepted as a deprecated alias for `warmup_steps`.""" + + def test_migrates_to_warmup_steps_and_warns(self): + with pytest.warns(DeprecationWarning, match="warmup_ratio is deprecated"): + params = TrainingHyperparams(warmup_ratio=0.2) + assert params.warmup_steps == 0.2 + + def test_explicit_warmup_steps_wins_over_deprecated_alias(self): + with pytest.warns(DeprecationWarning): + params = TrainingHyperparams(warmup_steps=0.3, warmup_ratio=0.2) + assert params.warmup_steps == 0.3 + + @pytest.mark.parametrize("value", [1.5, float("inf")]) + def test_deprecated_alias_is_validated_too(self, value): + """The alias assigns to `warmup_steps` after field validation, so it needs its own guard.""" + with pytest.raises((ParameterError, ValidationError, ValueError)): + TrainingHyperparams(warmup_ratio=value) + + def test_not_serialized(self): + with pytest.warns(DeprecationWarning): + params = TrainingHyperparams(warmup_ratio=0.2) + assert "warmup_ratio" not in params.model_dump() From b2c7bb0aace9e993eb8eb7f097b0fec378b48f5a Mon Sep 17 00:00:00 2001 From: Yunfeng Zhang Date: Fri, 7 Aug 2026 13:49:19 +0000 Subject: [PATCH 07/11] fix(config): allow warmup_steps=0 to disable warmup transformers treats warmup_steps=0 as "no warmup", but our validator required a strictly positive value, so that setting was unreachable from config. Relax the lower bound to >= 0; 0 already takes the ratio branch, so no other logic changes. Fractional values at or above 1, non-finite values, and negatives stay rejected. Signed-off-by: Yunfeng Zhang --- src/nemo_safe_synthesizer/config/training.py | 18 ++++++++++-------- tests/config/test_parameters.py | 6 +++--- 2 files changed, 13 insertions(+), 11 deletions(-) diff --git a/src/nemo_safe_synthesizer/config/training.py b/src/nemo_safe_synthesizer/config/training.py index 817b8bae7..e2e11fde9 100644 --- a/src/nemo_safe_synthesizer/config/training.py +++ b/src/nemo_safe_synthesizer/config/training.py @@ -156,13 +156,14 @@ def _mxfp4_config() -> QuantizationConfigMixin: def is_valid_warmup(value: float) -> bool: """Whether a warmup setting is a usable ratio or step count. - Mirrors how transformers interprets ``warmup_steps``: values below 1 are a - ratio of total training steps, values of 1 or more are an absolute step - count. Fractional values of 1 or more are rejected because transformers - truncates them (``1.5`` silently becomes ``1``), and non-finite values are - rejected because they raise ``OverflowError`` once converted to an integer. + Mirrors how transformers interprets ``warmup_steps``: ``0`` disables warmup, + values below 1 are a ratio of total training steps, and values of 1 or more + are an absolute step count. Fractional values of 1 or more are rejected + because transformers truncates them (``1.5`` silently becomes ``1``), and + non-finite values are rejected because they raise ``OverflowError`` once + converted to an integer. """ - return math.isfinite(value) and value > 0 and (value < 1 or float(value).is_integer()) + return math.isfinite(value) and value >= 0 and (value < 1 or float(value).is_integer()) class TrainingHyperparams(Parameters): @@ -232,8 +233,9 @@ class TrainingHyperparams(Parameters): description=( "Linear warmup from 0 to the learning rate. " "A whole number of 1 or more sets the exact number of warmup steps; " - "a float in (0, 1) is treated as a ratio of total training steps. " - "Must be finite and > 0, and cannot be fractional at or above 1." + "a float in (0, 1) is treated as a ratio of total training steps; " + "0 disables warmup. " + "Must be finite and >= 0, and cannot be fractional at or above 1." ), ), ] = 0.05 diff --git a/tests/config/test_parameters.py b/tests/config/test_parameters.py index 5a1d84e71..052f33c55 100644 --- a/tests/config/test_parameters.py +++ b/tests/config/test_parameters.py @@ -890,8 +890,9 @@ def test_returned_config_is_independent_of_saved(self): class TestWarmupSteps: """`warmup_steps` mirrors the transformers contract: ratio below 1, whole steps at or above 1.""" - @pytest.mark.parametrize("value", [0.05, 0.5, 0.999, 1, 1.0, 10, 10.0, 500]) + @pytest.mark.parametrize("value", [0, 0.05, 0.5, 0.999, 1, 1.0, 10, 10.0, 500]) def test_accepts_ratios_and_whole_step_counts(self, value): + """0 is legal and disables warmup, matching how transformers treats it.""" assert TrainingHyperparams(warmup_steps=value).warmup_steps == value @pytest.mark.parametrize( @@ -902,11 +903,10 @@ def test_accepts_ratios_and_whole_step_counts(self, value): float("inf"), # OverflowError once transformers converts it to an int float("-inf"), float("nan"), - 0, -1, ], ) - def test_rejects_fractional_non_finite_and_non_positive(self, value): + def test_rejects_fractional_non_finite_and_negative(self, value): with pytest.raises((ParameterError, ValidationError, ValueError)): TrainingHyperparams(warmup_steps=value) From d8fc81bed7e70b12e6e78dbac475192573039fd7 Mon Sep 17 00:00:00 2001 From: Yunfeng Zhang Date: Fri, 7 Aug 2026 17:38:04 +0000 Subject: [PATCH 08/11] fix(evaluation): suppress ks_2samp fallback notice at the call site The asymptotic-fallback notice was silenced in the 101 notebook only, so it still surfaced in the other two tutorials that run the pipeline and in every CLI and SDK run. Move the filter next to the two ks_2samp calls in text_semantic_similarity.py, the only place it is raised, and revert the notebook to a plain builder.run(). Also revert an unrelated no-op fixture change in tests/training/test_huggingface_backend.py, left over from a helper that no longer exists. Signed-off-by: Yunfeng Zhang --- docs/tutorials/safe-synthesizer-101.ipynb | 6 +-- .../components/text_semantic_similarity.py | 49 ++++++++++++++----- tests/training/test_huggingface_backend.py | 6 +-- 3 files changed, 40 insertions(+), 21 deletions(-) diff --git a/docs/tutorials/safe-synthesizer-101.ipynb b/docs/tutorials/safe-synthesizer-101.ipynb index 8f4e1c997..47f53c9bf 100644 --- a/docs/tutorials/safe-synthesizer-101.ipynb +++ b/docs/tutorials/safe-synthesizer-101.ipynb @@ -143,14 +143,10 @@ "metadata": {}, "outputs": [], "source": [ - "import warnings\n", "from nemo_safe_synthesizer.sdk.library_builder import SafeSynthesizer\n", "\n", "builder = SafeSynthesizer().with_data_source(df) # .with_replace_pii(enable=False) to disable PII replacement\n", - "with warnings.catch_warnings():\n", - " # ks_2samp falls back to asymptotic method for large samples; the result is still valid\n", - " warnings.filterwarnings(\"ignore\", message=\"ks_2samp: Exact calculation unsuccessful\", category=RuntimeWarning)\n", - " builder.run()\n", + "builder.run()\n", "results = builder.results\n" ] }, diff --git a/src/nemo_safe_synthesizer/evaluation/components/text_semantic_similarity.py b/src/nemo_safe_synthesizer/evaluation/components/text_semantic_similarity.py index 57df93ee7..a7d877e24 100644 --- a/src/nemo_safe_synthesizer/evaluation/components/text_semantic_similarity.py +++ b/src/nemo_safe_synthesizer/evaluation/components/text_semantic_similarity.py @@ -4,6 +4,8 @@ from __future__ import annotations import logging +import warnings +from contextlib import contextmanager from functools import cached_property from typing import TYPE_CHECKING @@ -36,11 +38,32 @@ from . import multi_modal_figures as figures if TYPE_CHECKING: + from collections.abc import Iterator + from sentence_transformers import SentenceTransformer logger = get_logger(__name__) +@contextmanager +def _suppress_ks_exact_fallback() -> Iterator[None]: + """Silence SciPy's notice that ``ks_2samp`` fell back to the asymptotic method. + + ``method="auto"`` attempts the exact calculation and falls back to the + asymptotic approximation once the samples are large, which is the intended + behaviour here -- the resulting p-values are still valid. The notice is not + actionable, so keep it out of CLI, SDK, and notebook output rather than + suppressing it separately in each caller. + """ + with warnings.catch_warnings(): + warnings.filterwarnings( + "ignore", + message="ks_2samp: Exact calculation unsuccessful", + category=RuntimeWarning, + ) + yield + + class TextSemanticSimilarityDatum(BaseModel): """Per-column text semantic similarity scores and PCA projections.""" @@ -401,12 +424,13 @@ def _get_text_semantic_similarity( # the minimum (most negative) difference between the empirical # distribution functions of the samples. The range of this statistic is # [0, 1], where 0 indicates no overfitting. - ks_test_overfitting = ks_2samp( - training_synth_similarity_matrix.max(axis=0), # F(x) - training_similarity_matrix.max(axis=0), # G(x) - alternative="less", - method="auto", - ) + with _suppress_ks_exact_fallback(): + ks_test_overfitting = ks_2samp( + training_synth_similarity_matrix.max(axis=0), # F(x) + training_similarity_matrix.max(axis=0), # G(x) + alternative="less", + method="auto", + ) # Underfitting is measured as the extent to which the synthetic # data is less similar to the test data than the test data is to @@ -417,12 +441,13 @@ def _get_text_semantic_similarity( # the minimum (most negative) difference between the empirical # distribution functions of the samples. The range of this statistic is # [0, 1], where 0 indicates no underfitting. - ks_test_underfitting = ks_2samp( - test_synth_similarity_matrix.max(axis=0), # F(x) - test_similarity_matrix.max(axis=0), # G(x) - alternative="greater", - method="auto", - ) + with _suppress_ks_exact_fallback(): + ks_test_underfitting = ks_2samp( + test_synth_similarity_matrix.max(axis=0), # F(x) + test_similarity_matrix.max(axis=0), # G(x) + alternative="greater", + method="auto", + ) # The overall semantic similarity score combines underfitting and overfitting # The range of this score is [0.37, 1], where 1 indicates perfect model and diff --git a/tests/training/test_huggingface_backend.py b/tests/training/test_huggingface_backend.py index 3287e0b2f..b387adb95 100644 --- a/tests/training/test_huggingface_backend.py +++ b/tests/training/test_huggingface_backend.py @@ -133,23 +133,21 @@ def params_with_orderby(base_params): @pytest.fixture def backend(base_params, mock_model_metadata, mock_workdir): """Create a HuggingFaceBackend instance for testing.""" - b = HuggingFaceBackend( + return HuggingFaceBackend( params=base_params, model_metadata=mock_model_metadata, workdir=mock_workdir, ) - return b @pytest.fixture def backend_with_validation(params_with_validation, mock_model_metadata, mock_workdir): """Create a HuggingFaceBackend instance with validation enabled.""" - b = HuggingFaceBackend( + return HuggingFaceBackend( params=params_with_validation, model_metadata=mock_model_metadata, workdir=mock_workdir, ) - return b @pytest.fixture From a671f9ecff1fe358e87862fad32cd1c7f8c9cc6e Mon Sep 17 00:00:00 2001 From: Yunfeng Zhang Date: Fri, 7 Aug 2026 18:03:23 +0000 Subject: [PATCH 09/11] test: cover the GLiNER unsupported-model and ks_2samp filter paths Both branches added in this PR were unexercised, which is what the patch coverage check flagged. Extend the two existing GLiNER fallback tests with a model exposing neither API, and assert _suppress_ks_exact_fallback silences SciPy's asymptotic-fallback notice while letting other RuntimeWarnings through. Signed-off-by: Yunfeng Zhang --- .../components/test_text_similarity_context.py | 16 ++++++++++++++++ tests/pii_replacer/test_detect.py | 16 ++++++++++++++++ 2 files changed, 32 insertions(+) diff --git a/tests/evaluation/components/test_text_similarity_context.py b/tests/evaluation/components/test_text_similarity_context.py index 090ed05e2..9bb5d51c8 100644 --- a/tests/evaluation/components/test_text_similarity_context.py +++ b/tests/evaluation/components/test_text_similarity_context.py @@ -3,6 +3,8 @@ from __future__ import annotations +import warnings + import pandas as pd import plotly.graph_objects as go import pytest @@ -11,6 +13,7 @@ from nemo_safe_synthesizer.evaluation.components.text_semantic_similarity import ( TextSemanticSimilarity, TextSemanticSimilarityDatum, + _suppress_ks_exact_fallback, ) from nemo_safe_synthesizer.evaluation.components.text_structure_similarity import ( TextDataSetStatistics, @@ -53,3 +56,16 @@ def test_text_structure_similarity_jinja_context_includes_column_heading( assert context["figures"][0]["title"] == "review" assert "plotly-graph-div" in context["figures"][0]["html"] + + +def test_suppress_ks_exact_fallback_is_scoped_to_the_scipy_notice(): + """Silences SciPy's asymptotic-fallback notice without swallowing other warnings.""" + # Verbatim from scipy/stats/_stats_py.py where ks_2samp abandons the exact method. + notice = "ks_2samp: Exact calculation unsuccessful. Switching to method=asymp." + + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + with _suppress_ks_exact_fallback(): + warnings.warn(notice, RuntimeWarning) + warnings.warn("an unrelated problem", RuntimeWarning) + assert [str(w.message) for w in caught] == ["an unrelated problem"] diff --git a/tests/pii_replacer/test_detect.py b/tests/pii_replacer/test_detect.py index ba9831c57..9144f408d 100644 --- a/tests/pii_replacer/test_detect.py +++ b/tests/pii_replacer/test_detect.py @@ -146,6 +146,14 @@ def predict_entities(self, text, labels, **kwargs): assert model.calls[0][0] == "abc" assert model.calls[0][1] == ["email", "name"] + # A build exposing neither API must fail loudly, not return no entities. + with patch("nemo_safe_synthesizer.pii_replacer.data_editor.detect.GLiNER") as mock_gliner: + mock_gliner.from_pretrained.return_value = object() + entity_extractor = EntityExtractorGliner.get_entity_extractor(cfg) + + with pytest.raises(AttributeError, match="neither inference nor predict_entities"): + entity_extractor.batch_update_cache(["abc"], None) + def test_gliner_single_prediction_falls_back_to_inference_api(): cfg = ClassifyConfig( @@ -178,6 +186,14 @@ def inference(self, texts, labels, **kwargs): assert model.calls[0][0] == ["abc"] assert model.calls[0][1] == ["email", "name"] + # A build exposing neither API must fail loudly, not return no entities. + with patch("nemo_safe_synthesizer.pii_replacer.data_editor.detect.GLiNER") as mock_gliner: + mock_gliner.from_pretrained.return_value = object() + entity_extractor = EntityExtractorGliner.get_entity_extractor(cfg) + + with pytest.raises(AttributeError, match="neither inference nor predict_entities"): + entity_extractor.extract_ner_predictions("abc", {"name", "email"}) + @pytest.mark.parametrize("offline_var", ["HF_HUB_OFFLINE", "TRANSFORMERS_OFFLINE"]) @pytest.mark.parametrize("env_value", ["1", "yes", "on"]) From d2ace4a6f1d22eb4c484f5c93628a1d65ed5af0c Mon Sep 17 00:00:00 2001 From: Yunfeng Zhang Date: Fri, 7 Aug 2026 19:55:07 +0000 Subject: [PATCH 10/11] fix(brev): include source-mapped CUDA indexes Signed-off-by: Yunfeng Zhang --- script/brev/README.md | 12 ++++++------ script/brev/setup.sh | 20 +++++++++++++++++--- 2 files changed, 23 insertions(+), 9 deletions(-) diff --git a/script/brev/README.md b/script/brev/README.md index 084df2f88..8a99b7694 100644 --- a/script/brev/README.md +++ b/script/brev/README.md @@ -91,12 +91,12 @@ hard way on a real instance. them -- and they have to match the release being installed, not this repo's `main`. The script resolves the latest version from the PyPI JSON API, fetches that tag's `pyproject.toml`, and reads the CUDA index URLs out of it, then pins the install to - that exact version so the two cannot drift. Selection is keyed on the URL containing - `cu129`, not on the index name: the flashinfer entry was renamed - `flashinfer-jit-cache` → `flashinfer-jit-cache-cu129` between 0.1.8 and 0.1.9, so - names are not stable across releases. The parse runs inside a process substitution and - therefore cannot fail the script, so the count of discovered indexes is what validates - it. + that exact version so the two cannot drift. Selection uses the CUDA extra in each + index's name or URL and includes indexes referenced by `[tool.uv.sources]` for that + extra. The source lookup matters for variant-neutral indexes such as + `https://flashinfer.ai/whl/`, while the URL lookup handles names that changed between + releases. The parse runs inside a process substitution and therefore cannot fail the + script, so the count of discovered indexes is what validates it. - uv is installed from a checksum-verified tarball, not `curl | sh`. The `astral.sh/install.sh` path logs `no checksums to verify`, so nothing validated what it downloaded. The script fetches the pinned release tarball, compares it against the diff --git a/script/brev/setup.sh b/script/brev/setup.sh index bb27937ab..a83cb093b 100755 --- a/script/brev/setup.sh +++ b/script/brev/setup.sh @@ -123,8 +123,8 @@ else NSS_VERSION="$(curl -fsSL https://pypi.org/pypi/nemo-safe-synthesizer/json \ | "${VENV_DIR}/bin/python" -c 'import json, sys; print(json.load(sys.stdin)["info"]["version"])')" - # Indexes come from the installed release's pyproject. Match both generated - # names and URLs because static index names do not enforce the CUDA suffix. + # Indexes come from the installed release's pyproject. Match CUDA names and + # URLs plus source-mapped indexes whose names are variant-neutral. pyproject="$(mktemp)" curl -fsSL "${REPO_URL}/raw/v${NSS_VERSION}/pyproject.toml" -o "${pyproject}" index_args=() @@ -140,15 +140,29 @@ import sys import tomllib with open(sys.argv[1], "rb") as handle: - indexes = tomllib.load(handle)["tool"]["uv"]["index"] + uv_config = tomllib.load(handle)["tool"]["uv"] +indexes = uv_config["index"] cuda_extra = os.environ["CUDA_EXTRA"] + +# Some indexes carry no CUDA variant in their name or URL. Source config is +# not wheel metadata, so collect indexes mapped to packages for this extra. +source_indexes = { + entry["index"] + for value in uv_config.get("sources", {}).values() + for entry in (value if isinstance(value, list) else [value]) + if isinstance(entry, dict) + and entry.get("extra") == cuda_extra + and "index" in entry +} + print( "\n".join( index["url"] for index in indexes if index["name"].endswith(f"-{cuda_extra}") or f"/{cuda_extra}" in index["url"] + or index["name"] in source_indexes ) ) PY From e781fb21841de9aa05e17f1fe24c827459a491b1 Mon Sep 17 00:00:00 2001 From: Yunfeng Zhang Date: Fri, 7 Aug 2026 20:44:05 +0000 Subject: [PATCH 11/11] fix(brev): align setup with launchable runtime Signed-off-by: Yunfeng Zhang --- script/brev/README.md | 26 ++++++++++++-------------- script/brev/setup.sh | 28 +++++++--------------------- 2 files changed, 19 insertions(+), 35 deletions(-) diff --git a/script/brev/README.md b/script/brev/README.md index 8a99b7694..f437ddd1a 100644 --- a/script/brev/README.md +++ b/script/brev/README.md @@ -10,17 +10,16 @@ NeMo Safe Synthesizer without setting up CUDA, drivers, or Python locally. Nothing in this directory is executed by the repo or by CI. A Brev Launchable is configured in the Brev web console, and the setup script is pasted into a form field there. This directory exists so that configuration is versioned and reviewable rather -than living only in a browser. When you change `setup.sh`, you must also paste the new -contents into the console for the change to take effect. +than living only in a browser. When you change `setup.sh` or `welcome.md`, you must also +update it in the console for the change to take effect. ### Files - `setup.sh`: Pasted into the Launchable's Setup Script field. Installs the CUDA build of Safe Synthesizer into a dedicated venv, registers it as the default Jupyter kernel, and drops the tutorial notebooks in `$HOME`. -- `welcome.md`: Becomes the customer's `$HOME/README.md`. Fetched at provisioning - time from the same tarball as the tutorials, not baked into `setup.sh` -- it would - otherwise consume a tenth of the 16 KiB script budget. +- `welcome.md`: Added to the Launchable's Source files so it renders on the Launchable + webpage and appears as the customer's `$HOME/welcome.md`. ### Console configuration @@ -33,7 +32,7 @@ Launchable with these settings. | Software | Install Jupyter on the host | Enabled | | Software | Run a Setup Script | Enabled, contents of `setup.sh` | | Software | Image ID | Leave blank | -| Source | Code source | No code files (`setup.sh` downloads the tutorials itself) | +| Source | Code source | `welcome.md` | | Hardware | GPU | 1× 80 GiB VRAM, single GPU | | Hardware | Disk | 200 GiB or more -- not resizable after creation | | Network | Ports | 8888, named `jupyter` | @@ -67,8 +66,8 @@ there. Everything operational is a dotfile, which the browser hides by default. ```text $HOME/ tutorials/ the three tutorial notebooks and their datasets - README.md where to start, rendered on double-click - (SETUP-IN-PROGRESS.md until setup finishes) + welcome.md where to start, rendered on double-click + SETUP-IN-PROGRESS.md present only while setup is running or after failure .nss-venv/ cu129 venv, registered as the default kernel .cache/huggingface/ model cache (Hugging Face's default location) @@ -106,12 +105,11 @@ hard way on a real instance. accepts connections well before this script finishes, so a user who opens it early would otherwise see an empty or half-populated file browser and assume the Launchable is broken. `SETUP-IN-PROGRESS.md` is written before any slow work, rewritten by the - `ERR` trap if provisioning fails, and replaced by `README.md` on success. -- The welcome text lives in `welcome.md`, not a heredoc. It is pulled from the - same tarball as the tutorials, so the two always match, and it is staged as a dotfile - until the final step so it never appears while setup is still running. The fetch is - non-fatal: `script/brev/` exists in no released tag, so it resolves only from the - `main` fallback until a release includes it. + `ERR` trap if provisioning fails, and removed on success. +- The welcome text lives in the Launchable's Source configuration, not a heredoc or + release tarball. Brev renders it on the Launchable webpage and copies it to + `$HOME/welcome.md`; keeping the console copy synchronized with this directory is a + manual deployment step. - The setup script has a 16 KiB limit. Brev rejects anything larger, which is why the script carries short comments pointing here rather than full explanations. Check `wc -c script/brev/setup.sh` before pasting. diff --git a/script/brev/setup.sh b/script/brev/setup.sh index a83cb093b..15d3350aa 100755 --- a/script/brev/setup.sh +++ b/script/brev/setup.sh @@ -21,11 +21,10 @@ readonly REPO_URL="https://github.com/NVIDIA-NeMo/Safe-Synthesizer" : "${HOME:?HOME is not set}" -# $HOME is the file browser root: only tutorials/ and README.md are visible. +# $HOME is the file browser root: only customer-facing files stay visible. readonly TUTORIALS_DIR="${HOME}/tutorials" -readonly README_FILE="${HOME}/README.md" +readonly WELCOME_FILE="${HOME}/welcome.md" readonly WAIT_FILE="${HOME}/SETUP-IN-PROGRESS.md" -readonly WELCOME_STAGED="${HOME}/.nss-welcome.md" readonly BIN_DIR="${HOME}/.local/bin" readonly VENV_DIR="${HOME}/.nss-venv" @@ -61,7 +60,7 @@ NeMo Safe Synthesizer is still installing -- roughly 5-10 minutes from when the instance started. Files appear as it progresses, so a partly-filled file browser is expected. Nothing here is ready to run yet. -When setup finishes, this file is replaced by README.md. Refresh to check. +When setup finishes, this file disappears. Open welcome.md to get started. EOF export PATH="${BIN_DIR}:${PATH}" @@ -222,13 +221,6 @@ else # Written last; the guard keys on this, so partial runs are redone. : >"${TUTORIALS_DIR}/.fetched" log "tutorials extracted from ${ref}" - # Same tarball as the tutorials. Non-fatal -- see README. - if tar -xzf "${tarball}" -C "${tarball_dir}" --strip-components=3 \ - "${top}/script/brev/welcome.md" 2>/dev/null; then - mv "${tarball_dir}/welcome.md" "${WELCOME_STAGED}" - else - log "WARNING: welcome.md not present in ${ref}" - fi fetched=1 break fi @@ -282,10 +274,6 @@ env = { # Keeps `!uv pip install ...` in a notebook from resolving to the Brev # image's own ~/.venv, which uv would otherwise discover by walking up. "VIRTUAL_ENV": venv, - # bitsandbytes ships binaries for even CUDA releases only (12.6, 12.8, 13.0…). - # CUDA 12.9 has no native binary and always falls back to 12.8. - # Update to "130" when upgrading CUDA_EXTRA to cu130 or later. - "BNB_CUDA_VERSION": "128", } # Secrets live in the kernelspec because the Jupyter server is not launched # from a login shell. The VM is single-tenant and the file is mode 0600. @@ -374,12 +362,10 @@ log "verifying install" "${VENV_DIR}/bin/python" \ -c "import torch; print('cuda available:', torch.cuda.is_available())" -# Hand over: swap the "please wait" file for the welcome text. +# Hand over: the Source-provided welcome stays visible after setup completes. -if [[ -f "${WELCOME_STAGED}" ]]; then - mv "${WELCOME_STAGED}" "${README_FILE}" -else - log "WARNING: no welcome.md staged; skipping ${README_FILE}" +if [[ ! -f "${WELCOME_FILE}" ]]; then + log "WARNING: ${WELCOME_FILE} is missing; check the Launchable Source files" fi rm -f "${WAIT_FILE}" @@ -389,7 +375,7 @@ cat <