[skill] evaluation: stop GDPVal leaking secrets to the log; fix the fake canary

Two findings from review, both confirmed against a real run.

1. Secret leak. The task `command:` opens with `set -ex`, and `set -x` traces
   commands AFTER expansion -- so the `ng_prepare_benchmark` trace line wrote
   $HF_TOKEN, $INFERENCE_API_KEY and $TAVILY_API_KEY in plaintext into the eval
   log, which lives on shared FS and is uploaded to MLflow when
   export.mlflow.log_logs is true. Verified in a completed run's client log: the
   traced line carries the expanded token, not `$HF_TOKEN`.
   Wrap that one call in `set +x` / `set -x`. The rollout call below it was
   already safe -- it goes through the quoted heredoc, so its secrets stay
   literal until a shell that never sets -x expands them; this call just wasn't
   covered by the same pattern.

2. `limit_samples` is inert on the gym path -- the gym does its own data prep and
   rollout collection, so the launcher-level limiter is ignored and you get the
   full 220-task run. This was observed during the validated run but the finding
   was lost in the doc-compression pass while the instruction depending on it
   survived in three places, so following the documented canary launches the
   heaviest benchmark in the suite believing it is a two-task smoke test.
   Replaced with the real procedure: launch, then treat the first ~20-30 minutes
   as the canary (SIF-sandbox line, judge auth, rollouts flowing), with the
   grep commands to check; note stage 1 of multistage as a natural checkpoint.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Chenjie Luo <chenjiel@nvidia.com>

Author: cjluo-nv

.agents/skills/evaluation/SKILL.md

@@ -70,7 +70,10 @@ GDPVal:
    unsandboxed. Verify with `gdpval-sif.sh --check`. `.env` needs `HF_TOKEN`, `INFERENCE_API_KEY`, `TAVILY_API_KEY`,
    `INFERENCE_JUDGE_URL`, `GDPVAL_SIF_DIR`, and `NEMO_EVALUATOR_TRUST_PRE_CMD=1` (the
    config has a `pre_cmd`). Thinking mode is mandatory (non-thinking loses ~86%).
-4. Dry-run → canary (`limit_samples=2`, verify the SIF sandbox + judge) → full.
+4. Dry-run → launch. **`limit_samples` is inert on the gym path** (the gym runs all
+   220 tasks regardless), so there is no cheap canary: watch the real run's first
+   ~20–30 min for the SIF-sandbox line and judge auth, and cancel if wrong. See the
+   recipe's Canary section.
 
 ---
 

.agents/skills/evaluation/recipes/examples/gym_gdpval/example_gym_gdpval.yaml

@@ -39,9 +39,11 @@
 #      search), INFERENCE_JUDGE_URL (judge host), and GDPVAL_SIF_DIR (item 1). See
 #      recipes/env.example.
 #
-# Canary (validates SIF sandbox + judge + gym plumbing on a couple of tasks):
-#   nel run --config example_gym_gdpval.yaml --env-file .env \
-#     -o ++evaluation.nemo_evaluator_config.config.params.limit_samples=2
+# NOTE: `limit_samples` is INERT on the gym path — the gym runs all 220 tasks no
+# matter what, so there is no cheap smoke test. Launch the real run and watch its
+# first ~20-30 min (SIF-sandbox line, judge auth, rollouts appearing); see the
+# recipe's Canary section.
+#   nel run --config example_gym_gdpval.yaml --env-file .env
 # =============================================================================
 defaults:
   # slurm/default works anywhere; if your install ships a predefined
@@ -283,7 +285,15 @@ evaluation:
                   # Writable staging dir for ref files (bind-mounted, see execution).
                   mkdir -p /gdpval_ref_files
 
+                  # SECRETS: `set -x` traces commands AFTER expansion, and these param blobs
+                  # carry $HF_TOKEN / $INFERENCE_API_KEY / $TAVILY_API_KEY. Without this the
+                  # trace line writes all three in plaintext into the eval log — which lives on
+                  # shared FS and is uploaded to MLflow when export.mlflow.log_logs is true.
+                  # (The rollout call below is already safe: it goes through the quoted heredoc,
+                  # so its secrets stay literal until a shell that never sets -x expands them.)
+                  set +x
                   ng_prepare_benchmark {{config.params.extra.nemo_gym.data_prep_params}} {{config.params.extra.nemo_gym.common_params}}
+                  set -x
                   # The rollout command is written to a script and run from it, rather than passed
                   # to `bash -c '...'`. Gym params legitimately CONTAIN single quotes — comparison
                   # mode's ++multistage.stages='[{num_tasks: 45}, ...]' and ++...judge_panel='[{...}]'

.agents/skills/evaluation/recipes/tasks/aa_gym/gdpval.md

@@ -48,18 +48,26 @@ recipes/examples/gym_gdpval/example_gym_gdpval.yaml   # SLURM + single-node vLLM
 `num_repeats=1` — already set by the template via `++num_repeats=1`; both
 current goldens use it. A full 220-task run of a large MoE typically needs multi-node.
 
-## Canary
+## Canary — `limit_samples` does NOT work here
 
-Validate the SIF sandbox + judge + gym plumbing on a couple of tasks before the
-full run:
+**`++…params.limit_samples=N` is inert on the gym path.** The gym does its own data
+prep and rollout collection, so the launcher-level limiter is ignored: you get the
+full 220-task run. Do not use it believing you launched a two-task smoke test — this
+is the heaviest benchmark in the suite.
+
+There is no cheap sample-limited canary. Instead, **launch the real run and treat its
+first ~20–30 minutes as the canary**, cancelling if any of these is wrong:
 
 ```bash
-nel run --config example_gym_gdpval.yaml --env-file .env \
-  -o ++evaluation.nemo_evaluator_config.config.params.limit_samples=2
+RD=<output_dir>/<run>/nemo_gym.0
+grep -c "Using Apptainer container"  $RD/logs/client-*.log          # sandbox actually used
+grep -c "falling back\|not a git repo" $RD/logs/client-*.log        # unsandboxed / inert pin
+grep -ciE " 401 | 403 |Internal Server Error" $RD/artifacts/nemo_gym_logs/gdpval_judge_model.log
+wc -l $RD/artifacts/evaluator_rollouts.jsonl                        # rollouts flowing
 ```
 
-Inspect logs for the SIF fallback warning, judge auth/429s, and Ray/gym shutdown
-hangs (see `references/gym-gdpval.md` → failure modes).
+In comparison mode stage 1 (45 tasks) is a natural early checkpoint — an ELO estimate
+appears before the full 220-task stage 2 starts.
 
 ## Score Extraction