[5726458] [Experimental] Add NVFP4 projection-output-quantizer recipe and HF embedding ONNX export example (#1981)

### What does this PR do?

Type of change: new example

Without output-side quantization, TensorRT's quantized GEMMs emit FP16
activations: every quantized GEMM input adds a low-precision copy *on
top of* the FP16 tensors instead of replacing them, so FP8/FP4 engines
can use as much or more activation memory than an unquantized FP16
engine ([5726458]). Quantizing the projection-Linear outputs makes the
engine carry inter-layer activations in the low-precision format. This
PR ships that as recipes for the Llama-Nemotron embedding/reranking
family (NVFP4 and FP8 variants) plus an end-to-end example.

Measured on RTX PRO 6000 Blackwell with TensorRT 10.16 (strongly-typed
engines, 5 dynamic-shape profiles up to 32x512), activation memory per
profile:

| Model | FP16 | `fp8` preset | **fp8 recipe** | `nvfp4` preset |
**nvfp4 recipe** |

|-------|-----:|-------------:|---------------:|---------------:|-----------------:|
| llama-nemotron-embed-1b-v2 | 1040 MiB | 1392 MiB | **1096 MiB** | 1040
MiB | **516 MiB** |
| llama-nemotron-rerank-1b-v2 | 1040 MiB | 1392 MiB | **1096 MiB** | 520
MiB | **331 MiB** |

Engine sizes (dominated by weights): FP16 ≈ 2374 MiB, FP8 ≈ 1453 MiB,
NVFP4 ≈ 1050–1075 MiB. The presets only shrink weights — their
activation memory matches (or exceeds, for FP8) the FP16 engine because
every quantized GEMM still emits FP16; the output-quantizer recipes are
what reduce activation memory (fp8: −21% vs its preset; nvfp4: −50% vs
its preset and 2x below FP16).

-
**`modelopt_recipes/huggingface/nemotron_llama/ptq/nvfp4_output_quant_proj.yaml`**
— the general `nvfp4` preset plus dynamic NVFP4 output quantizers scoped
to the projection Linears (`*_proj.output_quantizer`). Scoping matters:
a `DynamicQuantize` on non-GEMM outputs (embedding lookup, pooling)
fails to compile in TensorRT. The sequence-classification `score` head
is kept unquantized: final heads stay in high precision like `lm_head`,
and its `[1, hidden]` weight cannot be packed by the NVFP4 exporter.
-
**`modelopt_recipes/huggingface/nemotron_llama/ptq/fp8_output_quant_proj.yaml`**
— the FP8 twin: per-tensor FP8 output quantizers on the projection
Linears, switching the engine from FP16-out GEMMs
(`e4m3f16..._bias_f16`) to FP8-out GEMMs (`e4m3e4m3_e4m3`).
- **`examples/torch_onnx/hf_embedding_quant_to_onnx.py`** — minimal
end-to-end recipe-driven quantize → ONNX export for HF bidirectional
Llama embedding and reranking encoders (auto-detected from the model
architecture; embedding models export mean-pooled L2-normalized
embeddings, rerankers export relevance logits), with the export shims
needed for a TensorRT-fusable graph (bidirectional sdpa symbolic with
single-precision attention constants; static blocked-axis extents for
`Reshape → TRT_FP4DynamicQuantize`).
- **`modelopt/torch/quantization/export_onnx.py`** —
`configure_linear_module_onnx_quantizers` now types output quantizers as
`"dynamic"` (they previously fell to the static path, which the NVFP4
weight exporter rejects on activations), and the sdpa symbolic's
`JitScalarType` import is fixed for torch >= 2.11.
- **`examples/torch_onnx/torch_quant_to_onnx.py`** — replaces the
`mtq.*_CFG` module-constant table with YAML recipe loading and adds a
`--recipe` flag (preset basename or `QuantizeConfig` YAML path);
`--auto_quantization_formats` values switch from config-constant names
to preset basenames.
- **`tests/examples/torch_onnx/test_hf_embedding_quant_to_onnx.py`** —
end-to-end example test running both model kinds through quantize →
export with tiny random-weight stand-ins (plain Llama encoder and
`LlamaForSequenceClassification`, sized to the NVFP4 block size).
- README section for the new example, including the results table and
trtexec engine-build steps; CHANGELOG entry.

### Usage

```bash
# Embedding model (default recipe: nvfp4 + projection output quantizers)
python examples/torch_onnx/hf_embedding_quant_to_onnx.py \
    --model_path=nvidia/llama-nemotron-embed-1b-v2 \
    --trust_remote_code \
    --onnx_save_path=llama_nemotron_embed_nvfp4.onnx

# Reranking model (auto-detected), FP8 variant of the recipe
python examples/torch_onnx/hf_embedding_quant_to_onnx.py \
    --model_path=nvidia/llama-nemotron-rerank-1b-v2 \
    --trust_remote_code \
    --recipe=huggingface/nemotron_llama/ptq/fp8_output_quant_proj \
    --onnx_save_path=llama_nemotron_rerank_fp8.onnx

# Build a strongly-typed TensorRT engine (Blackwell, TensorRT >= 10.11)
trtexec --onnx=llama_nemotron_embed_nvfp4.onnx --stronglyTyped \
    --saveEngine=llama_nemotron_embed_nvfp4.plan \
    --minShapes=input_ids:1x2,attention_mask:1x2 \
    --optShapes=input_ids:32x128,attention_mask:32x128 \
    --maxShapes=input_ids:32x512,attention_mask:32x512
```

### Testing

- New end-to-end example test `test_hf_embedding_quant_to_onnx.py` runs
both model kinds (embedding + reranking) through quantize → ONNX export
on tiny random-weight checkpoints; `test_torch_onnx_recipe_flag` covers
the `--recipe` flag.
- Ran the example end-to-end on GPU for both real models and both
recipes; each graph parses with TensorRT strongly-typed mode (NVFP4
graphs carry 112 input-side + 112 output-side `TRT_FP4DynamicQuantize`;
FP8 graphs carry the matching static Q/DQ placement).
- Built strongly-typed 5-profile engines on RTX PRO 6000 Blackwell with
TensorRT 10.16 for FP16 and both presets/recipes on both models;
compared per-profile activation memory via
`ICudaEngine.get_device_memory_size_for_profile_v2` (table above) and
verified kernel selection (FP8 recipe → `e4m3e4m3_e4m3` FP8-out GEMMs;
NVFP4 → block-scaled GEMMs).
- Recipes pass `tools/precommit/check_modelopt_recipes.py`; `pre-commit`
green on all changed files.

### Before your PR is "*Ready for review*"

Make sure you read and follow [Contributor
guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md)
and your commits are signed (`git commit -s -S`).

Make sure you read and follow the [Security Best
Practices](https://github.com/NVIDIA/Model-Optimizer/blob/main/SECURITY.md#security-coding-practices-for-contributors)
(e.g. avoiding hardcoded `trust_remote_code=True`, `torch.load(...,
weights_only=False)`, `pickle`, etc.).

- Is this change backward compatible?: ❌ — `torch_quant_to_onnx.py`'s
`--auto_quantization_formats` values are renamed from config-constant
names (e.g. `NVFP4_AWQ_LITE_CFG`) to preset basenames (e.g.
`nvfp4_awq_lite`); the loaded configs are identical. Core APIs are
backward compatible (the output-quantizer export typing is additive).
- If you copied code from any other sources or added a new PIP
dependency, did you follow guidance in `CONTRIBUTING.md`: N/A
- Did you write any new necessary tests?: ✅
(`test_hf_embedding_quant_to_onnx` for both model kinds,
`test_torch_onnx_recipe_flag`)
- Did you update
[Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?:
✅

### Additional Information

The target model family requires remote modeling code, so the usage
examples opt in explicitly with `--trust_remote_code`. Accuracy parity
(embedding quality / reranking scores) of the output-quantized recipes
has not been evaluated yet and should be validated before recommending
them as defaults.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
> 🤖 _Generated by Claude (AI agent)._

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added NVFP4 and FP8 projection-output PTQ recipes for Llama-Nemotron
embedding/reranking.
* Added an end-to-end Hugging Face “quantize-to-ONNX” CLI workflow for
TensorRT export.
* Enhanced Torch→ONNX quantization with YAML-driven recipes via a new
`--recipe` flag and improved auto-quantization format handling.
* **Bug Fixes**
* Improved ONNX export and TensorRT DynamicQuantize compatibility for
scaled dot-product attention and quantizer export behavior.
* **Documentation**
* Updated Torch→ONNX example docs and PTQ recipe guidance (including
Nemotron Llama).
* **Tests**
* Strengthened ONNX graph checks for recipe and Hugging Face export
flows.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

> 🤖 _Generated by Codex (AI agent)._

---------

Signed-off-by: ajrasane <131806219+ajrasane@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>

Author: 3 people

CHANGELOG.rst

@@ -69,6 +69,11 @@ python torch_quant_to_onnx.py \
     --onnx_save_path=<path to save the exported ONNX model>
 ```
 
+Quantization configs are loaded from the YAML preset recipes under
+`modelopt_recipes/configs/ptq/presets/model/`, selected by `--quantize_mode`. Pass
+`--recipe=<preset basename or path to a QuantizeConfig YAML>` to use a different
+recipe (e.g. `--recipe=nvfp4_awq_lite` or `--recipe=/path/to/my_quant_cfg.yaml`).
+
 ### Conv2d Quantization Override
 
 TensorRT only supports FP8 and INT8 for convolution operations. When quantizing models with Conv2d layers (like SwinTransformer), the script automatically applies the following overrides:
@@ -93,6 +98,72 @@ python ../onnx_ptq/evaluate.py \
     --model_name=<timm model name>
 ```
 
+## HF Embedding and Reranking Models
+
+> **Experimental:** Accuracy has not yet been validated for this example.
+
+`hf_embedding_quant_to_onnx.py` quantizes an HF text-embedding or reranking
+model (bidirectional Llama encoders such as
+[nvidia/llama-nemotron-embed-1b-v2](https://huggingface.co/nvidia/llama-nemotron-embed-1b-v2)
+and
+[nvidia/llama-nemotron-rerank-1b-v2](https://huggingface.co/nvidia/llama-nemotron-rerank-1b-v2))
+with a PTQ recipe and exports it to ONNX. Embedding models are exported with
+mean pooling and L2 normalization on top of the encoder; reranking
+(sequence-classification) models are exported to their relevance logits. Both
+graphs take `input_ids` and `attention_mask` with dynamic batch/sequence axes.
+
+The default recipe
+(`modelopt_recipes/huggingface/nemotron_llama/ptq/nvfp4_output_quant_proj.yaml`)
+quantizes weights and activations to NVFP4 and additionally quantizes the
+projection-Linear outputs. Without output-side quantization, quantized GEMMs
+emit FP16 activations, so FP8/FP4 engines can use as much or more activation
+memory than an unquantized FP16 engine; quantizing the projection outputs keeps
+inter-layer activations in the low-precision format. An FP8 twin of the recipe
+(`fp8_output_quant_proj.yaml`, pass it via `--recipe`) applies the same idea to
+the FP8 preset. With TensorRT 10.16 on RTX PRO 6000 Blackwell (strongly-typed
+engines, 5 dynamic-shape profiles up to 32x512), engine activation memory:
+
+| Model | FP16 | `fp8` preset | fp8 recipe | `nvfp4` preset | nvfp4 recipe |
+|-------|-----:|-------------:|-----------:|---------------:|-------------:|
+| llama-nemotron-embed-1b-v2 | 1040 MiB | 1392 MiB | 1096 MiB | 1040 MiB | 516 MiB |
+| llama-nemotron-rerank-1b-v2 | 1040 MiB | 1392 MiB | 1096 MiB | 520 MiB | 331 MiB |
+
+### Usage
+
+```bash
+python hf_embedding_quant_to_onnx.py \
+    --model_path=nvidia/llama-nemotron-embed-1b-v2 \
+    --trust_remote_code \
+    --recipe=huggingface/nemotron_llama/ptq/nvfp4_output_quant_proj \
+    --onnx_save_path=llama_nemotron_embed_nvfp4.onnx
+
+# Reranking variant (auto-detected from the model architecture)
+python hf_embedding_quant_to_onnx.py \
+    --model_path=nvidia/llama-nemotron-rerank-1b-v2 \
+    --trust_remote_code \
+    --onnx_save_path=llama_nemotron_rerank_nvfp4.onnx
+```
+
+### Building a TensorRT engine with trtexec
+
+NVFP4 requires a Blackwell GPU (SM100+) and TensorRT 10.11 or later. Build a
+strongly-typed engine with dynamic shapes (add optimization profiles matching
+your serving batch sizes and sequence lengths):
+
+```bash
+trtexec --onnx=llama_nemotron_embed_nvfp4.onnx \
+    --stronglyTyped \
+    --saveEngine=llama_nemotron_embed_nvfp4.plan \
+    --minShapes=input_ids:1x2,attention_mask:1x2 \
+    --optShapes=input_ids:32x128,attention_mask:32x128 \
+    --maxShapes=input_ids:32x512,attention_mask:32x512
+```
+
+The exported `.onnx` references a sibling weights file (`<name>.onnx_data`);
+keep the two files in the same directory when building. To inspect the chosen
+kernels and per-profile activation memory, add
+`--profilingVerbosity=detailed --exportLayerInfo=<path>.json --verbose`.
+
 ## LLM Quantization and Export with TensorRT-Edge-LLM
 
 [TensorRT-Edge-LLM](https://github.com/NVIDIA/TensorRT-Edge-LLM) provides a complete pipeline for quantizing LLMs and VLMs using NVIDIA ModelOpt and exporting them to optimized ONNX for deployment on edge platforms such as NVIDIA Jetson and DRIVE.
@@ -283,7 +354,7 @@ The `auto` mode enables mixed precision quantization by searching for the optima
 python torch_quant_to_onnx.py \
     --timm_model_name=vit_base_patch16_224 \
     --quantize_mode=auto \
-    --auto_quantization_formats NVFP4_AWQ_LITE_CFG FP8_DEFAULT_CFG \
+    --auto_quantization_formats nvfp4_awq_lite fp8 \
     --effective_bits=4.8 \
     --num_score_steps=128 \
     --calibration_data_size=512 \