From 40fb7b3f9751ea360adc2800af42251544c7c6a4 Mon Sep 17 00:00:00 2001 From: gabrielfruet Date: Thu, 25 Jun 2026 13:44:44 -0300 Subject: [PATCH 1/4] docs: add model instability debugging page --- docs/source/debugging/model_instability.md | 84 ++++++++++++++++++++++ docs/source/index.md | 1 + docs/source/settings/train_settings.md | 6 +- 3 files changed, 89 insertions(+), 2 deletions(-) create mode 100644 docs/source/debugging/model_instability.md diff --git a/docs/source/debugging/model_instability.md b/docs/source/debugging/model_instability.md new file mode 100644 index 000000000..d406be03f --- /dev/null +++ b/docs/source/debugging/model_instability.md @@ -0,0 +1,84 @@ +(model-instability-debugging)= + +# Model Instability Debugging + +Training instabilities — such as exploding or vanishing gradients, sudden loss spikes, +or numerical collapse to `NaN`/`inf` — can derail a run silently or abruptly. This page +collects the tools LightlyTrain provides to detect and diagnose these issues. + +> [!NOTE] This section is growing. More debugging tools will be documented here as they +> are added. The first available tool is gradient norm logging. + +## What Instability Looks Like + +Common symptoms of an unstable run: + +- The training loss spikes sharply or collapses to `NaN`/`inf`. +- The loss plateaus at a high value and never improves. +- The model stops learning partway through training (validation metrics flatten or + regress). +- Training crashes with a numerical error during the forward or backward pass. + +Not all of these mean instability — a high plateau can also be caused by a too low +learning rate or a data issue. Use the tools below to distinguish between them. + +## Gradient Norm Logging + +The total gradient norm is the single most useful signal for spotting exploding and +vanishing gradients. LightlyTrain logs it for every training step: + +- `gradient_norm`: Total gradient norm computed after backpropagation, before the + optimizer step. If gradient clipping is enabled (`gradient_clip_val > 0`) this is the + pre-clipping norm; otherwise it is computed via an unbounded (`inf`) norm. It is also + shown in the console progress line as `grad_norm`. + +It is written to all configured loggers (`metrics.jsonl`, TensorBoard, MLflow, Weights & +Biases) at the cadence set by +[`log_every_num_steps`](../settings/train_settings.md#log_every_num_steps). + +### How to View the Gradient Norm + +- **Console:** The progress line shows `grad_norm` for each logged training step. + +- **TensorBoard:** Plot `gradient_norm` over training steps: + + ```bash + tensorboard --logdir out/my_experiment + ``` + +- **MLflow / Weights & Biases:** The `gradient_norm` metric is available under the same + key. See [](../settings/train_settings.md) for how to enable these loggers. + +### How to Interpret the Trend + +Interpret the gradient norm as a trend over steps, not as an isolated value. Its +absolute scale depends on the model, dataset, and batch size, so there is no universal +"good" value. What matters is the shape: + +- **Stable:** The norm fluctuates within a steady band across training. +- **Exploding gradients:** The norm grows rapidly, often by several orders of magnitude, + and may precede a loss spike or a `NaN` collapse. +- **Vanishing gradients:** The norm shrinks toward zero and stays there, often + accompanying a loss that no longer decreases. + +A short-lived spike during warmup or learning-rate scheduling is usually normal. A +persistent upward or downward drift is the signal to act on. + +### Common Next Actions + +- **Exploding gradients:** + - Lower the learning rate with [`model_args.lr`](../settings/train_settings.md). + - Switch to a more stable precision, e.g. `precision="bf16-mixed"` or + `precision="32-true"` (see [](../settings/train_settings.md)). + - Resume from the last good checkpoint using + [`resume_interrupted`](../settings/train_settings.md) after changing settings. +- **Vanishing gradients:** + - Increase the learning rate, especially for small models (~10M parameters or fewer). + - Check that the input normalization in `transform_args` matches your data + distribution. +- **NaN/inf collapse:** Re-run from the latest checkpoint. If it reproduces, switch to + `precision="32-true"` to isolate whether the instability is caused by + reduced-precision arithmetic. + +See the FAQ entry on [improving model performance](../faq.md) for broader guidance on +stable training. diff --git a/docs/source/index.md b/docs/source/index.md index e9f8da8da..f494275ad 100644 --- a/docs/source/index.md +++ b/docs/source/index.md @@ -282,6 +282,7 @@ embed Settings data/index performance/index +Debugging docker tutorials/index python_api/index diff --git a/docs/source/settings/train_settings.md b/docs/source/settings/train_settings.md index 0261c1f28..9616bb7f6 100644 --- a/docs/source/settings/train_settings.md +++ b/docs/source/settings/train_settings.md @@ -601,9 +601,11 @@ training steps, at the cadence set by [`log_every_num_steps`](#log_every_num_ste - `gradient_norm`: Total gradient norm computed after backpropagation, before the optimizer step. If gradient clipping is enabled (`gradient_clip_val > 0`) this is the - pre-clipping norm; otherwise it is the total gradient norm computed without applying +pre-clipping norm; otherwise it is the total gradient norm computed without applying gradient clipping. Use it to spot exploding or vanishing gradients during training. It - is also shown in the console progress line as `grad_norm`. + is also shown in the console progress line as `grad_norm`. See + [Model Instability Debugging](../debugging/model_instability.md) for how to interpret + this value and diagnose unstable training. - `learning_rate`: Current learning rate after scheduler scaling. Both are written to all configured loggers (`metrics.jsonl`, TensorBoard, MLflow, From 8a428731c4d87a94e1c0da04939b8e00aee99507 Mon Sep 17 00:00:00 2001 From: gabrielfruet Date: Thu, 25 Jun 2026 15:56:53 -0300 Subject: [PATCH 2/4] formatting and fixes --- docs/source/debugging/model_instability.md | 6 ++---- docs/source/settings/train_settings.md | 2 +- 2 files changed, 3 insertions(+), 5 deletions(-) diff --git a/docs/source/debugging/model_instability.md b/docs/source/debugging/model_instability.md index d406be03f..fa4b90409 100644 --- a/docs/source/debugging/model_instability.md +++ b/docs/source/debugging/model_instability.md @@ -6,8 +6,8 @@ Training instabilities — such as exploding or vanishing gradients, sudden loss or numerical collapse to `NaN`/`inf` — can derail a run silently or abruptly. This page collects the tools LightlyTrain provides to detect and diagnose these issues. -> [!NOTE] This section is growing. More debugging tools will be documented here as they -> are added. The first available tool is gradient norm logging. +:::\{note} This section is growing. More debugging tools will be documented here as they +are added. The first available tool is gradient norm logging. ::: ## What Instability Looks Like @@ -70,8 +70,6 @@ persistent upward or downward drift is the signal to act on. - Lower the learning rate with [`model_args.lr`](../settings/train_settings.md). - Switch to a more stable precision, e.g. `precision="bf16-mixed"` or `precision="32-true"` (see [](../settings/train_settings.md)). - - Resume from the last good checkpoint using - [`resume_interrupted`](../settings/train_settings.md) after changing settings. - **Vanishing gradients:** - Increase the learning rate, especially for small models (~10M parameters or fewer). - Check that the input normalization in `transform_args` matches your data diff --git a/docs/source/settings/train_settings.md b/docs/source/settings/train_settings.md index 9616bb7f6..c7cf427da 100644 --- a/docs/source/settings/train_settings.md +++ b/docs/source/settings/train_settings.md @@ -601,7 +601,7 @@ training steps, at the cadence set by [`log_every_num_steps`](#log_every_num_ste - `gradient_norm`: Total gradient norm computed after backpropagation, before the optimizer step. If gradient clipping is enabled (`gradient_clip_val > 0`) this is the -pre-clipping norm; otherwise it is the total gradient norm computed without applying + pre-clipping norm; otherwise it is the total gradient norm computed without applying gradient clipping. Use it to spot exploding or vanishing gradients during training. It is also shown in the console progress line as `grad_norm`. See [Model Instability Debugging](../debugging/model_instability.md) for how to interpret From 59740ecd9974c76a7782277445ae0d2a7e496732 Mon Sep 17 00:00:00 2001 From: fruet <90271657+gabrielfruet@users.noreply.github.com> Date: Mon, 29 Jun 2026 12:08:30 -0300 Subject: [PATCH 3/4] Fix gradient norm calculation description Corrected the description of the gradient norm calculation method in the documentation. --- docs/source/debugging/model_instability.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/debugging/model_instability.md b/docs/source/debugging/model_instability.md index fa4b90409..ace6d3c92 100644 --- a/docs/source/debugging/model_instability.md +++ b/docs/source/debugging/model_instability.md @@ -29,7 +29,7 @@ vanishing gradients. LightlyTrain logs it for every training step: - `gradient_norm`: Total gradient norm computed after backpropagation, before the optimizer step. If gradient clipping is enabled (`gradient_clip_val > 0`) this is the - pre-clipping norm; otherwise it is computed via an unbounded (`inf`) norm. It is also + pre-clipping norm; otherwise it is computed via an L2 norm. It is also shown in the console progress line as `grad_norm`. It is written to all configured loggers (`metrics.jsonl`, TensorBoard, MLflow, Weights & From 3593d75f10b851416ffe45cde2206773d6dd1833 Mon Sep 17 00:00:00 2001 From: gabrielfruet Date: Mon, 29 Jun 2026 12:12:22 -0300 Subject: [PATCH 4/4] format --- docs/source/debugging/model_instability.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/debugging/model_instability.md b/docs/source/debugging/model_instability.md index ace6d3c92..b746571e4 100644 --- a/docs/source/debugging/model_instability.md +++ b/docs/source/debugging/model_instability.md @@ -29,8 +29,8 @@ vanishing gradients. LightlyTrain logs it for every training step: - `gradient_norm`: Total gradient norm computed after backpropagation, before the optimizer step. If gradient clipping is enabled (`gradient_clip_val > 0`) this is the - pre-clipping norm; otherwise it is computed via an L2 norm. It is also - shown in the console progress line as `grad_norm`. + pre-clipping norm; otherwise it is computed via an L2 norm. It is also shown in the + console progress line as `grad_norm`. It is written to all configured loggers (`metrics.jsonl`, TensorBoard, MLflow, Weights & Biases) at the cadence set by