docs(eval skill): vLLM backend env vars + SLURM HF-cache/cpu_partition guidance (#1625)

### What does this PR do?

Type of change: documentation

Hardens the **evaluation** skill with operational guidance discovered
while running AA-Index evals (NVFP4 Nemotron-3-Nano) on SLURM. Two
themes:

**1. vLLM backend env vars (original commits).** Some models need a
model-card backend toggle that is an env var, not a CLI flag — e.g.
NVFP4 MoE models like
[NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4)
need `VLLM_USE_FLASHINFER_MOE_FP4=1` +
`VLLM_FLASHINFER_MOE_BACKEND=throughput`. These go in
`deployment.env_vars` (with the `lit:` prefix), not the `vllm serve`
command.

**2. SLURM deploy/eval operational lessons (new commits).**
- **`mount_home: false` always.** Some internal cluster templates
default it `true`, which mounts the host `~/.cache`; where that is a
symlink into a shared/networked filesystem, it dangles in the container
and the vLLM `trust-remote-code` deploy dies with `FileNotFoundError:
/root/.cache/huggingface` — **invisible to `--dry-run`**.
- **HF cache:** mount the realpath of `~/.cache/huggingface` to
`/hf-cache` and set `HF_HOME: lit:/hf-cache` for both stages.
- **`execution.cpu_partition`:** on split GPU/CPU-partition clusters,
the CPU-only MLflow auto-export job is rejected by the GPU partition and
marks the whole task FAILED despite `EVAL_EXIT_CODE=0`. Set
`cpu_partition` to route it correctly.
- **Top-level `env_vars:`** for vars both stages need (`HF_TOKEN`,
`HF_HOME`); `execution.env_vars` is unsupported and hard-errors.
- **`lcr.md`:** `LOG_LEVEL=WARNING` to skip logging AA-LCR's ~120K-token
inputs.

### Usage

```yaml
execution:
  cpu_partition: <cpu-partition>   # CPU-only auto-export job
  mounts:
    mount_home: false
    deployment: { <shared-fs>/<user>/.cache/huggingface: /hf-cache }
    evaluation: { <shared-fs>/<user>/.cache/huggingface: /hf-cache }
env_vars:                          # shared by both stages
  HF_TOKEN: host:HF_TOKEN
  HF_HOME: lit:/hf-cache
deployment:
  env_vars:
    VLLM_USE_FLASHINFER_MOE_FP4: lit:1
    VLLM_FLASHINFER_MOE_BACKEND: lit:throughput
```

### Testing

Docs/skill-only. Validated end-to-end by running the full AA suite (GPQA
+ SciCode + AA-LCR) across 5 NVFP4 checkpoints on SLURM: dry-run +
canary + full runs all SUCCESS with these settings. Pre-commit (yamlfmt,
markdownlint) passed.

### Before your PR is "*Ready for review*"

- Is this change backward compatible?: ✅
- If you copied code from any other sources or added a new PIP
dependency, did you follow guidance in `CONTRIBUTING.md`: N/A
- Did you write any new necessary tests?: N/A
- Did you update Changelog?: N/A
- Did you get Claude approval on this PR?: ❌

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Documentation**
* Clarified where backend-related environment variables must be
configured (deployment-level) and not embedded in execution commands.
* Added guidance to mount the real HuggingFace cache path and set
home-mounting to false for containerized SLURM runs.
* Documented routing CPU-only jobs via CPU partitions and added a
task-level LOG_LEVEL setting to reduce verbose long-context logging.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Signed-off-by: Chenjie Luo <chenjiel@nvidia.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

Author: cjluo-nv

.agents/skills/evaluation/SKILL.md

@@ -84,7 +84,7 @@ nel skills build-config --execution <...> --deployment <...> --model_type <...>
 - **vLLM:** no `--quantization` flag by default — vLLM auto-detects from `quantization_config` / `hf_quant_config.json`. Add only when the card, vLLM version, or dry-run error requires it.
 - **SGLang:** may need `--quantization modelopt_fp8` / `modelopt_fp4` / `modelopt` — verify against installed version.
 
-Some models need extra env vars (e.g. `VLLM_NVFP4_GEMM_BACKEND=marlin` for Nemotron Super) — discovered via model card research.
+Some models need extra vLLM backend env vars (model-card research) — e.g. `VLLM_NVFP4_GEMM_BACKEND=marlin` (Nemotron Super), or `VLLM_USE_FLASHINFER_MOE_FP4=1` + `VLLM_FLASHINFER_MOE_BACKEND=throughput` (NVFP4 MoE, e.g. NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4). Put them in `deployment.env_vars` (**not** `command`) with the `lit:` prefix (`VLLM_USE_FLASHINFER_MOE_FP4: lit:1`); see `example_eval.yaml` and Step 5's prefix rule.
 
 **Auto-detect from `config.json`:**
 
@@ -223,6 +223,8 @@ for f in "$PKG"/configs/execution/internal/slurm/*.yaml; do \
 
 Hostname match → set `defaults: - execution: internal/slurm/<cluster>`, drop the redundant `execution.hostname` (keep account/output_dir/walltime), verify with `--dry-run`. Else keep `slurm/default` and fill hostname/account/output_dir manually.
 
+On SLURM, several deploy/eval failures are invisible to `--dry-run` and only surface at canary (`mount_home`, HF cache, `cpu_partition`, top-level vs per-stage `env_vars`) — read `references/slurm.md`.
+
 - Find every `???` left. Ask the user only for what can't be inferred (SLURM hostname/account/output_dir, MLflow tracking URI, etc.). Don't propose defaults; let them give plain text.
 - **`parallelism`** — size it yourself from the run shape (total requests = `dataset_size × repeats` vs GPU serving capacity), and set `--max-num-seqs` to match. Read `references/parallelism.md` for the decision rule and worked examples; only ask the user if a non-GPU cap (e.g. judge rate limit) is unknown.
 - Ask about other defaults they may want to change (partition, walltime, MLflow tags).
@@ -295,12 +297,12 @@ Default images:
 | Framework | Image | Registry |
 | --- | --- | --- |
 | vLLM | `vllm/vllm-openai:v0.19.1` (bump per recipe; never `:latest`) | DockerHub |
-| vLLM (NVFP4 on Blackwell) | `vllm/vllm-openai:v0.19.1-cu130` (bump to `cu130-nightly-<arch>` for new archs) | DockerHub |
+| vLLM (NVFP4 on B300/GB300) | `vllm/vllm-openai:v0.19.1-cu130` (bump to `cu130-nightly-<arch>` for new archs) | DockerHub |
 | SGLang | `lmsysorg/sglang:latest` | DockerHub |
 | TRT-LLM | `nvcr.io/nvidia/tensorrt-llm/release:...` | NGC |
 | Eval tasks | `nvcr.io/nvidia/eval-factory/*:26.03` | NGC |
 
-> NVFP4 checkpoints on Blackwell (sm_100/sm_103) need the `cu130-nightly` image — cu129/v0.19.1 lack sm_103 FP4 kernels (see the "NVFP4 on Blackwell" note in Step 3).
+> NVFP4 checkpoints on B300/GB300 (sm_103) need the `cu130` image — cu129/v0.19.1 lack sm_103 FP4 kernels (see the "NVFP4 on Blackwell" note in Step 3).
 
 Public images → submit without preflight. Private/restricted → check credentials:
 

.agents/skills/evaluation/recipes/examples/example_eval.yaml

@@ -56,13 +56,24 @@ execution:
   # On the slurm/default fallback it's gpu:8 — set to the node's GPU count or sbatch
   # fails "Requested node configuration is not available" (4-GPU GB300 -> gpu:4).
   mounts:
+    # mount_home ALWAYS false: true mounts host ~/.cache, which (if a shared-fs symlink) dangles
+    # in-container -> vLLM deploy dies "FileNotFoundError /root/.cache/huggingface" (unseen by
+    # --dry-run). Instead mount the real HF cache to /hf-cache + set HF_HOME below. See SKILL Step 4.
     mount_home: false
+    # deployment: { <shared-fs>/<user>/.cache/huggingface: /hf-cache }
+    # evaluation: { <shared-fs>/<user>/.cache/huggingface: /hf-cache }
   auto_export:          # REQUIRED trigger for auto-export. Without this, the
     destinations:       # export.mlflow block below is ignored and the run is
       - mlflow          # NOT uploaded — you'd have to `nel export` it by hand.
 deployment:
   env_vars:
     HF_TOKEN: host:HF_TOKEN
+    HF_HOME: lit:/hf-cache    # with the /hf-cache mount above
+    # vLLM backend env vars go HERE, not in the command below. Some models need
+    # a card-documented toggle — e.g. NVFP4 MoE (NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4)
+    # needs FlashInfer FP4 kernels. Uncomment per card (`lit:` prefix, Step 5):
+    #   VLLM_USE_FLASHINFER_MOE_FP4: lit:1
+    #   VLLM_FLASHINFER_MOE_BACKEND: lit:throughput
   checkpoint_path: ???
   hf_model_handle:
   served_model_name: ???
@@ -96,6 +107,7 @@ deployment:
 evaluation:
   env_vars:
     HF_TOKEN: host:HF_TOKEN
+    HF_HOME: lit:/hf-cache    # with the /hf-cache mount above
     # nemo-skills tasks (ns_*) hard-require the served-endpoint api key env var
     # (api_key_name below) to be set WITH A VALUE inside the eval container.
     # A shell `export DUMMY_API_KEY=dummy` does NOT reach the SLURM container —

.agents/skills/evaluation/recipes/tasks/aa/lcr.md

@@ -42,6 +42,7 @@ block. Per SKILL.md Step 3, the deployment flag must live inside
   container: nvcr.io/nvidia/eval-factory/nemo-skills:26.03
   env_vars:
     INFERENCE_API_KEY: host:INFERENCE_API_KEY
+    LOG_LEVEL: lit:WARNING # Skip logging the long context inputs.
   nemo_evaluator_config:
     target:
       api_endpoint:

.agents/skills/evaluation/references/slurm.md

@@ -0,0 +1,39 @@
+# SLURM gotchas (invisible to `--dry-run`; surface at canary)
+
+Use this reference when filling `execution` for a SLURM run (Step 4). These
+deploy/eval failures pass `--dry-run` clean and only show up once a real job is
+scheduled (canary), so resolve them up front.
+
+## `mount_home: false`
+
+`mount_home: true` mounts the host `~/.cache` into the container. A shared-fs
+symlink there (common for `~/.cache/huggingface`) dangles in-container → deploy
+dies with `FileNotFoundError /root/.cache/huggingface`. Mount the real cache to
+`/hf-cache` and point `HF_HOME` at it instead (see the snippet below).
+
+## `cpu_partition: <cpu-partition>`
+
+**Required for MLflow `auto_export` to work** on clusters with separate GPU/CPU
+partitions. If not specified, NEL runs the export (a CPU-only job) on the GPU
+partition — it does not auto-route — which gets rejected
+(`Cannot find GPU specification`) and fails the task despite `EVAL_EXIT_CODE=0`.
+Set it to the CPU partition (e.g. `cpu`).
+
+## Shared vs stage-specific env vars
+
+Shared env vars can go top-level `env_vars:` (merges into both the deployment and
+evaluation stages) or per-stage as the example shows; `execution.env_vars`
+hard-errors. Stage-specific vars stay under `deployment.env_vars` /
+`evaluation.env_vars`.
+
+## Snippet
+
+```yaml
+execution:
+  cpu_partition: <cpu-partition>
+  mounts:
+    mount_home: false
+    deployment: { <realpath ~/.cache/huggingface>: /hf-cache }   # ssh <host> realpath ~/.cache/huggingface
+    evaluation: { <realpath ~/.cache/huggingface>: /hf-cache }
+env_vars: { HF_TOKEN: host:HF_TOKEN, HF_HOME: lit:/hf-cache }   # both stages
+```