Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 18 additions & 20 deletions script/brev/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,17 +10,16 @@ NeMo Safe Synthesizer without setting up CUDA, drivers, or Python locally.
Nothing in this directory is executed by the repo or by CI. A Brev Launchable is
configured in the Brev web console, and the setup script is pasted into a form field
there. This directory exists so that configuration is versioned and reviewable rather
than living only in a browser. When you change `setup.sh`, you must also paste the new
contents into the console for the change to take effect.
than living only in a browser. When you change `setup.sh` or `welcome.md`, you must also
update it in the console for the change to take effect.

### Files

- `setup.sh`: Pasted into the Launchable's Setup Script field. Installs the CUDA
build of Safe Synthesizer into a dedicated venv, registers it as the default Jupyter
kernel, and drops the tutorial notebooks in `$HOME`.
- `welcome.md`: Becomes the customer's `$HOME/README.md`. Fetched at provisioning
time from the same tarball as the tutorials, not baked into `setup.sh` -- it would
otherwise consume a tenth of the 16 KiB script budget.
- `welcome.md`: Added to the Launchable's Source files so it renders on the Launchable
webpage and appears as the customer's `$HOME/welcome.md`.

### Console configuration

Expand All @@ -33,7 +32,7 @@ Launchable with these settings.
| Software | Install Jupyter on the host | Enabled |
| Software | Run a Setup Script | Enabled, contents of `setup.sh` |
| Software | Image ID | Leave blank |
| Source | Code source | No code files (`setup.sh` downloads the tutorials itself) |
| Source | Code source | `welcome.md` |
| Hardware | GPU | 1× 80 GiB VRAM, single GPU |
| Hardware | Disk | 200 GiB or more -- not resizable after creation |
| Network | Ports | 8888, named `jupyter` |
Expand Down Expand Up @@ -67,8 +66,8 @@ there. Everything operational is a dotfile, which the browser hides by default.
```text
$HOME/
tutorials/ the three tutorial notebooks and their datasets
README.md where to start, rendered on double-click
(SETUP-IN-PROGRESS.md until setup finishes)
welcome.md where to start, rendered on double-click
SETUP-IN-PROGRESS.md present only while setup is running or after failure

.nss-venv/ cu129 venv, registered as the default kernel
.cache/huggingface/ model cache (Hugging Face's default location)
Expand All @@ -91,12 +90,12 @@ hard way on a real instance.
them -- and they have to match the release being installed, not this repo's `main`.
The script resolves the latest version from the PyPI JSON API, fetches that tag's
`pyproject.toml`, and reads the CUDA index URLs out of it, then pins the install to
that exact version so the two cannot drift. Selection is keyed on the URL containing
`cu129`, not on the index name: the flashinfer entry was renamed
`flashinfer-jit-cache` → `flashinfer-jit-cache-cu129` between 0.1.8 and 0.1.9, so
names are not stable across releases. The parse runs inside a process substitution and
therefore cannot fail the script, so the count of discovered indexes is what validates
it.
that exact version so the two cannot drift. Selection uses the CUDA extra in each
index's name or URL and includes indexes referenced by `[tool.uv.sources]` for that
extra. The source lookup matters for variant-neutral indexes such as
`https://flashinfer.ai/whl/`, while the URL lookup handles names that changed between
releases. The parse runs inside a process substitution and therefore cannot fail the
script, so the count of discovered indexes is what validates it.
- uv is installed from a checksum-verified tarball, not `curl | sh`. The
`astral.sh/install.sh` path logs `no checksums to verify`, so nothing validated what
it downloaded. The script fetches the pinned release tarball, compares it against the
Expand All @@ -106,12 +105,11 @@ hard way on a real instance.
accepts connections well before this script finishes, so a user who opens it early
would otherwise see an empty or half-populated file browser and assume the Launchable
is broken. `SETUP-IN-PROGRESS.md` is written before any slow work, rewritten by the
`ERR` trap if provisioning fails, and replaced by `README.md` on success.
- The welcome text lives in `welcome.md`, not a heredoc. It is pulled from the
same tarball as the tutorials, so the two always match, and it is staged as a dotfile
until the final step so it never appears while setup is still running. The fetch is
non-fatal: `script/brev/` exists in no released tag, so it resolves only from the
`main` fallback until a release includes it.
`ERR` trap if provisioning fails, and removed on success.
- The welcome text lives in the Launchable's Source configuration, not a heredoc or
release tarball. Brev renders it on the Launchable webpage and copies it to
`$HOME/welcome.md`; keeping the console copy synchronized with this directory is a
manual deployment step.
- The setup script has a 16 KiB limit. Brev rejects anything larger, which is why
the script carries short comments pointing here rather than full explanations. Check
`wc -c script/brev/setup.sh` before pasting.
Expand Down
50 changes: 30 additions & 20 deletions script/brev/setup.sh
Original file line number Diff line number Diff line change
Expand Up @@ -21,11 +21,10 @@ readonly REPO_URL="https://github.com/NVIDIA-NeMo/Safe-Synthesizer"

: "${HOME:?HOME is not set}"

# $HOME is the file browser root: only tutorials/ and README.md are visible.
# $HOME is the file browser root: only customer-facing files stay visible.
readonly TUTORIALS_DIR="${HOME}/tutorials"
readonly README_FILE="${HOME}/README.md"
readonly WELCOME_FILE="${HOME}/welcome.md"
readonly WAIT_FILE="${HOME}/SETUP-IN-PROGRESS.md"
readonly WELCOME_STAGED="${HOME}/.nss-welcome.md"

readonly BIN_DIR="${HOME}/.local/bin"
readonly VENV_DIR="${HOME}/.nss-venv"
Expand Down Expand Up @@ -61,7 +60,7 @@ NeMo Safe Synthesizer is still installing -- roughly 5-10 minutes from when
the instance started. Files appear as it progresses, so a partly-filled file
browser is expected. Nothing here is ready to run yet.

When setup finishes, this file is replaced by README.md. Refresh to check.
When setup finishes, this file disappears. Open welcome.md to get started.
EOF

export PATH="${BIN_DIR}:${PATH}"
Expand Down Expand Up @@ -123,8 +122,8 @@ else
NSS_VERSION="$(curl -fsSL https://pypi.org/pypi/nemo-safe-synthesizer/json \
| "${VENV_DIR}/bin/python" -c 'import json, sys; print(json.load(sys.stdin)["info"]["version"])')"

# Indexes come from the installed release's pyproject. Match both generated
# names and URLs because static index names do not enforce the CUDA suffix.
# Indexes come from the installed release's pyproject. Match CUDA names and
# URLs plus source-mapped indexes whose names are variant-neutral.
pyproject="$(mktemp)"
curl -fsSL "${REPO_URL}/raw/v${NSS_VERSION}/pyproject.toml" -o "${pyproject}"
index_args=()
Expand All @@ -140,15 +139,29 @@ import sys
import tomllib

with open(sys.argv[1], "rb") as handle:
indexes = tomllib.load(handle)["tool"]["uv"]["index"]
uv_config = tomllib.load(handle)["tool"]["uv"]

indexes = uv_config["index"]
cuda_extra = os.environ["CUDA_EXTRA"]

# Some indexes carry no CUDA variant in their name or URL. Source config is
# not wheel metadata, so collect indexes mapped to packages for this extra.
source_indexes = {
entry["index"]
for value in uv_config.get("sources", {}).values()
for entry in (value if isinstance(value, list) else [value])
if isinstance(entry, dict)
and entry.get("extra") == cuda_extra
and "index" in entry
}

print(
"\n".join(
index["url"]
for index in indexes
if index["name"].endswith(f"-{cuda_extra}")
or f"/{cuda_extra}" in index["url"]
or index["name"] in source_indexes
)
)
PY
Expand Down Expand Up @@ -208,13 +221,6 @@ else
# Written last; the guard keys on this, so partial runs are redone.
: >"${TUTORIALS_DIR}/.fetched"
log "tutorials extracted from ${ref}"
# Same tarball as the tutorials. Non-fatal -- see README.
if tar -xzf "${tarball}" -C "${tarball_dir}" --strip-components=3 \
"${top}/script/brev/welcome.md" 2>/dev/null; then
mv "${tarball_dir}/welcome.md" "${WELCOME_STAGED}"
else
log "WARNING: welcome.md not present in ${ref}"
fi
fetched=1
break
fi
Expand Down Expand Up @@ -343,19 +349,23 @@ if [[ "${registered}" -ne 1 ]]; then
log "WARNING: kernel not registered; notebooks may open on the wrong Python"
fi

# Pre-compile third-party packages that emit SyntaxWarnings on first import so
# the warnings go into the setup log rather than appearing in notebook output.
log "pre-compiling packages"
"${VENV_DIR}/bin/python" -W ignore::SyntaxWarning \
-c "import torchao, range_regex" 2>/dev/null || true
Comment thread
zywind marked this conversation as resolved.

Comment thread
zywind marked this conversation as resolved.
# Smoke check -- fail provisioning loudly rather than handing over a broken VM.

log "verifying install"
"${VENV_DIR}/bin/safe-synthesizer" --version
"${VENV_DIR}/bin/python" \
-c "import torch; print('cuda available:', torch.cuda.is_available())"

# Hand over: swap the "please wait" file for the welcome text.
# Hand over: the Source-provided welcome stays visible after setup completes.

if [[ -f "${WELCOME_STAGED}" ]]; then
mv "${WELCOME_STAGED}" "${README_FILE}"
else
log "WARNING: no welcome.md staged; skipping ${README_FILE}"
if [[ ! -f "${WELCOME_FILE}" ]]; then
log "WARNING: ${WELCOME_FILE} is missing; check the Launchable Source files"
fi
rm -f "${WAIT_FILE}"

Expand All @@ -365,7 +375,7 @@ cat <<EOF

Safe Synthesizer is ready.

Start here : ${README_FILE}
Start here : ${WELCOME_FILE}
Tutorials : ${TUTORIALS_DIR}/
Kernel : "Safe Synthesizer" -- already the default
Setup log : ${LOG_FILE}
Expand Down
53 changes: 48 additions & 5 deletions src/nemo_safe_synthesizer/config/training.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,16 @@
from __future__ import annotations

import importlib
import math
import warnings
from enum import StrEnum
from typing import (
TYPE_CHECKING,
Annotated,
Literal,
)

from pydantic import Field
from pydantic import Field, model_validator

from ..configurator.parameters import (
Parameters,
Expand Down Expand Up @@ -151,6 +153,19 @@ def _mxfp4_config() -> QuantizationConfigMixin:
ValueGTZero = ValueValidator(lambda p: range_validator(p, lambda v: v >= 0))


def is_valid_warmup(value: float) -> bool:
"""Whether a warmup setting is a usable ratio or step count.

Mirrors how transformers interprets ``warmup_steps``: ``0`` disables warmup,
values below 1 are a ratio of total training steps, and values of 1 or more
are an absolute step count. Fractional values of 1 or more are rejected
because transformers truncates them (``1.5`` silently becomes ``1``), and
non-finite values are rejected because they raise ``OverflowError`` once
converted to an integer.
"""
return math.isfinite(value) and value >= 0 and (value < 1 or float(value).is_integer())


class TrainingHyperparams(Parameters):
"""Hyperparameters that control the training process behavior.

Expand Down Expand Up @@ -210,15 +225,43 @@ class TrainingHyperparams(Parameters):
),
] = 0.01

warmup_ratio: Annotated[
warmup_steps: Annotated[
Comment thread
zywind marked this conversation as resolved.
float,
ValueValidator(value_func=lambda v: v > 0),
ValueValidator(value_func=is_valid_warmup),
Field(
title="warmup_ratio",
description="Ratio of total training steps used for a linear warmup from 0 to the learning rate. Must be > 0.",
title="warmup_steps",
description=(
"Linear warmup from 0 to the learning rate. "
"A whole number of 1 or more sets the exact number of warmup steps; "
"a float in (0, 1) is treated as a ratio of total training steps; "
"0 disables warmup. "
"Must be finite and >= 0, and cannot be fractional at or above 1."
),
Comment thread
zywind marked this conversation as resolved.
),
] = 0.05

warmup_ratio: Annotated[
float | None,
ValueValidator(value_func=lambda v: v is None or is_valid_warmup(v)),
Field(
title="warmup_ratio",
description="Deprecated. Use warmup_steps instead.",
exclude=True,
),
] = None

@model_validator(mode="after")
def _migrate_warmup_ratio(self) -> TrainingHyperparams:
if self.warmup_ratio is not None:
warnings.warn(
"warmup_ratio is deprecated and will be removed in a future release. Use warmup_steps instead.",
DeprecationWarning,
stacklevel=2,
)
Comment thread
zywind marked this conversation as resolved.
if "warmup_steps" not in self.model_fields_set:
self.warmup_steps = self.warmup_ratio
Comment thread
zywind marked this conversation as resolved.
return self
Comment thread
coderabbitai[bot] marked this conversation as resolved.

lr_scheduler: Annotated[
str,
Field(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@
from __future__ import annotations

import logging
import warnings
from contextlib import contextmanager
from functools import cached_property
from typing import TYPE_CHECKING

Expand Down Expand Up @@ -36,11 +38,32 @@
from . import multi_modal_figures as figures

if TYPE_CHECKING:
from collections.abc import Iterator

from sentence_transformers import SentenceTransformer

logger = get_logger(__name__)


@contextmanager
def _suppress_ks_exact_fallback() -> Iterator[None]:
"""Silence SciPy's notice that ``ks_2samp`` fell back to the asymptotic method.

``method="auto"`` attempts the exact calculation and falls back to the
asymptotic approximation once the samples are large, which is the intended
behaviour here -- the resulting p-values are still valid. The notice is not
actionable, so keep it out of CLI, SDK, and notebook output rather than
suppressing it separately in each caller.
"""
with warnings.catch_warnings():
warnings.filterwarnings(
"ignore",
message="ks_2samp: Exact calculation unsuccessful",
category=RuntimeWarning,
)
yield


class TextSemanticSimilarityDatum(BaseModel):
"""Per-column text semantic similarity scores and PCA projections."""

Expand Down Expand Up @@ -401,12 +424,13 @@ def _get_text_semantic_similarity(
# the minimum (most negative) difference between the empirical
# distribution functions of the samples. The range of this statistic is
# [0, 1], where 0 indicates no overfitting.
ks_test_overfitting = ks_2samp(
training_synth_similarity_matrix.max(axis=0), # F(x)
training_similarity_matrix.max(axis=0), # G(x)
alternative="less",
method="auto",
)
with _suppress_ks_exact_fallback():
ks_test_overfitting = ks_2samp(
training_synth_similarity_matrix.max(axis=0), # F(x)
training_similarity_matrix.max(axis=0), # G(x)
alternative="less",
method="auto",
)

# Underfitting is measured as the extent to which the synthetic
# data is less similar to the test data than the test data is to
Expand All @@ -417,12 +441,13 @@ def _get_text_semantic_similarity(
# the minimum (most negative) difference between the empirical
# distribution functions of the samples. The range of this statistic is
# [0, 1], where 0 indicates no underfitting.
ks_test_underfitting = ks_2samp(
test_synth_similarity_matrix.max(axis=0), # F(x)
test_similarity_matrix.max(axis=0), # G(x)
alternative="greater",
method="auto",
)
with _suppress_ks_exact_fallback():
ks_test_underfitting = ks_2samp(
test_synth_similarity_matrix.max(axis=0), # F(x)
test_similarity_matrix.max(axis=0), # G(x)
alternative="greater",
method="auto",
)

# The overall semantic similarity score combines underfitting and overfitting
# The range of this score is [0.37, 1], where 1 indicates perfect model and
Expand Down
Loading
Loading