Merge remote-tracking branch 'origin/main' into naren/plugin-evaluation-all-tiers

Signed-off-by: Narendran Raghavan <nraghavan@nvidia.com>

# Conflicts:
#	CHANGELOG.md
#	src/skillevaluator/cli.py
#	tests/validators/test_policy.py

Author: rng1995

CHANGELOG.md

@@ -37,6 +37,11 @@ All notable changes to SkillEvaluator are documented in this file.
   `CLAUDE.md -> AGENTS.md` public compatibility alias; Windows reads now use
   native handle verification with the same fail-closed identity checks as
   POSIX.
+- Secure Docker exec redaction now ignores environment values shorter than eight
+  characters, matching the exact secret length floor used elsewhere. Short
+  flags such as `CLAUDE_CODE_DISABLE_POLICY_SKILLS=1` no longer rewrite digits
+  in `docker exec` output, which had broken NVIDIA Build bridge loopback
+  origins during Tier 3 preflight.
 - Isolated NVIDIA Build bridge credentials from vendor CLI processes using a
   transient, root-managed, container-only key handoff with cleanup on failure.
 - Removed NVIDIA Build secrets from Harbor and Docker exec arguments using a
@@ -53,6 +58,10 @@ All notable changes to SkillEvaluator are documented in this file.
 
 ### Changed
 
+- Added explicit `--block-on-dedup` / `--no-block-on-dedup` and
+  `--block-on-agent-eval` / `--no-block-on-agent-eval` controls with
+  backward-compatible defaults, Tier 3 source preflight, and consistent gating
+  metadata across CLI, JSON, Markdown, and HTML reports.
 - Reduced pull-request runner use for changes confined to `docs/**` and
   `fern/**`: DCO, Gitleaks, and pinned Fern validation still run, while mixed
   and non-docs changes retain the complete Linux, macOS, Windows, packaging,

README.md

@@ -128,8 +128,10 @@ avoids managed sandbox charges, not hosted model charges. It is experimental
 and only for trusted skills and workspaces; use Docker or cloud for untrusted
 code. Start with one agent and a small dataset.
 See the [Tier 3 guide](https://docs.nvidia.com/skills/skillevaluator/tier3-live-evaluation#plan-for-cost)
-before scaling a run. Tier 3 results are advisory within `validate`; Tier 1 and
-Tier 2 determine its exit status.
+before scaling a run. Tier 1 always gates `validate`. Tier 2 gates by default;
+`--no-block-on-dedup` keeps its scan and reports but makes its findings
+advisory. Tier 3 is advisory by default; `--block-on-agent-eval` promotes its
+findings, including invalid task-source evidence, into the exit gate.
 
 ## Documentation
 

docs/ci-integration.mdx

@@ -16,7 +16,7 @@ Turn SkillEvaluator into a merge gate for your skill repository. The recommended
 | `2` | Configuration error — bad flags or an unreadable target | Fix the pipeline, not the skill |
 | `3` | Runtime error — an unexpected internal failure | Re-run; the skill was never judged. File an issue if it persists |
 
-Tier 1 checks gate the exit code — and so do Tier 2 deduplication findings when dedup runs (the recommended gate below disables it with `--no-dedup`). Tier 3 live-evaluation results are always advisory inside `validate`: they appear in the reports but never flip a passing run to `1`. That means you can layer Tier 3 on later without destabilizing the gate.
+Tier 1 checks always gate the exit code. Tier 2 deduplication findings gate when dedup runs unless you pass `--no-block-on-dedup`; the recommended keyless gate below instead disables Tier 2 with `--no-dedup`. Tier 3 live-evaluation results are advisory inside `validate` unless you pass `--block-on-agent-eval`. Reports record the effective choice for each tier.
 
 ## The recommended gate
 
@@ -156,11 +156,11 @@ You do not have to turn on the strictest gate on day one. A path that works well
 
   ### Add LLM-backed checks
 
-  Add a provider key as a masked CI secret, set `SKILL_EVAL_LLM_PROVIDER`, then add `--llm` (and `--llm-verify` to suppress false positives) and drop `--no-dedup` so Tier 2 overlap checks run too. Note that once dedup runs, its findings gate the exit code alongside Tier 1. See [Providers & Credentials](configuration.mdx) for provider setup.
+  Add a provider key as a masked CI secret, set `SKILL_EVAL_LLM_PROVIDER`, then add `--llm` (and `--llm-verify` to suppress false positives) and drop `--no-dedup` so Tier 2 overlap checks run too. Once dedup runs, its findings gate alongside Tier 1 by default; add `--no-block-on-dedup` if you want evidence without enforcing it yet. See [Providers & Credentials](configuration.mdx) for provider setup.
 
   ### Add an advisory Tier 3 job
 
-  Run live agent evaluation as a separate, non-blocking job — either `skillevaluator tier3 evaluate` on its own or `validate --tier3` (`--agent-eval` is a supported compatibility alias and is not currently deprecated; `validate --full` runs Tier 1+2+3 in one shot). Tier 3 results never gate `validate`'s exit code; the standalone `tier3 evaluate` exits non-zero only when the run itself fails to complete. Expose agent credentials as CI secrets in the job environment — the `env:` block of the workflow step — because SkillEvaluator reads operator credentials from the host environment only. With the `nv_build` provider and `--env-mode docker`, a single `NVIDIA_API_KEY` secret covers the evaluator and all three agents. See [Tier 3: Live Evaluation](tier3-live-evaluation.mdx).
+  Run live agent evaluation as a separate job — either `skillevaluator tier3 evaluate` on its own or `validate --tier3` (`--agent-eval` is a supported compatibility alias and is not currently deprecated; `validate --full` runs Tier 1+2+3 in one shot). Attached Tier 3 is advisory by default; add `--block-on-agent-eval` when the team is ready for its findings or invalid source evidence to fail the merge gate. The standalone `tier3 evaluate` exits non-zero when the run itself fails to complete. Expose agent credentials as CI secrets in the job environment — the `env:` block of the workflow step — because SkillEvaluator reads operator credentials from the host environment only. With the `nv_build` provider and `--env-mode docker`, a single `NVIDIA_API_KEY` secret covers the evaluator and all three agents. See [Tier 3: Live Evaluation](tier3-live-evaluation.mdx).
 </Steps>
 
 <Note>

docs/cli-reference.mdx

@@ -17,7 +17,7 @@ commands referenced below.
 
 | Command | Purpose | Needs |
 | --- | --- | --- |
-| [validate](#validate) | Run Tier 1 checks plus Tier 2 dedup by default (both gate the exit code), optional advisory Tier 3 | Nothing by default |
+| [validate](#validate) | Run Tier 1 checks plus Tier 2 dedup by default, with explicit Tier 2 and Tier 3 gating controls | Nothing by default |
 | [quality-check](#quality-check) | Score skill quality across four weighted categories | Nothing |
 | [rubric-eval](#rubric-eval) | LLM-as-judge rubric evaluation | LLM provider key |
 | [security-scan](#security-scan) | Scan for security vulnerabilities | Nothing (key only with `--llm`/`--llm-verify`) |
@@ -89,15 +89,18 @@ skillevaluator tier3 validate ./my-skill --strict
 
 ## validate
 
-Validate a skill, rule, workflow, or plugin. Tier 1 static, security, and quality checks gate the exit code — and so do Tier 2 deduplication findings when that pass runs (on by default, degrading gracefully without embedding access); Tier 3 live evaluation attaches as an advisory pass when enabled with `--tier3`, `--autopilot`, `--full`, or the supported `--agent-eval` compatibility alias.
+Validate a skill, rule, workflow, or plugin. Tier 1 static, security, and quality checks always gate the exit code. Tier 2 deduplication runs and gates by default, but `--no-block-on-dedup` keeps the scan while making its findings advisory. Tier 3 live evaluation attaches as an advisory pass when enabled with `--tier3`, `--autopilot`, `--full`, or the supported `--agent-eval` compatibility alias; add `--block-on-agent-eval` to make its findings gate.
 
 ```bash title="Common invocations"
 skillevaluator validate ./my-skill                        # Tier 1 + Tier 2
 skillevaluator validate ./my-skill --llm                  # add LLM security analysis
 skillevaluator validate ./my-skill -r cli,json,html       # multiple report formats
 skillevaluator validate ./my-skill --no-dedup             # skip Tier 2 dedup
+skillevaluator validate ./my-skill --no-block-on-dedup    # run Tier 2 as advisory
 skillevaluator validate ./my-skill --external             # strict publication profile
 skillevaluator validate ./my-skill --tier3 -a codex       # add advisory Tier 3 eval
+skillevaluator validate ./my-skill --tier3 -a codex \
+    --block-on-agent-eval                                  # gate on Tier 3 too
 skillevaluator validate ./my-skill --autopilot            # Tier 3, generating evals if missing
 skillevaluator validate ./my-skill --full -a codex        # everything, one shot
 skillevaluator validate ./my-plugin --type plugin --tier3 \
@@ -136,7 +139,7 @@ Static checks; LLM-free by default. Tier 1 gates the exit code and always runs.
 
 | Flag | Default | Effect |
 | --- | --- | --- |
-| `--checks, --tier1-checks TEXT` | all applicable | Comma-separated subset of Tier 1 checks. Choices: `schema`, `security`, `pii`, `license`, `code-integrity`, `unicode`, `quality`, `lint`; opt-in (not run by default): `version`, `dependency`. `quality`/`lint`/`version` are skill-only and skipped for rules and workflows. |
+| `--checks, --tier1-checks TEXT` | all applicable | Comma-separated subset of Tier 1 checks. Default choices: `schema`, `version`, `security`, `pii`, `license`, `code-integrity`, `unicode`, `quality`, `lint`; opt-in (not run by default): `dependency`. `quality`/`lint`/`version` are skill-only and skipped for rules and workflows. |
 | `--fail-fast` | off | Stop on the first failing check instead of collecting all issues. |
 | `-c, --continue-on-failure` | off | Run the full pipeline without stopping early; record all issues in the reports. Overrides `--fail-fast`, and for folder validation keeps scanning every skill past a CRITICAL finding. |
 | `--llm, --tier1-llm / --no-llm, --no-tier1-llm` | `no-llm` | Enable LLM-backed security analysis (requires a configured public provider — see [Providers & Credentials](configuration.mdx)). |
@@ -150,6 +153,7 @@ Embedding + LLM dedup; on by default, skips gracefully without a provider key.
 | Flag | Default | Effect |
 | --- | --- | --- |
 | `--dedup, --tier2 / --no-dedup, --no-tier2` | `dedup` | Run Tier 2 intra-skill semantic-overlap checks. On by default; skipped gracefully without public embedding access. Use `--no-tier2` (or `--no-dedup`) to disable. |
+| `--block-on-dedup / --no-block-on-dedup` | blocking | Choose whether Tier 2 findings participate in the `validate` exit gate. The scan and its report entries still run with `--no-block-on-dedup`. |
 
 ### Tier 3 · Live Agent Evaluation
 
@@ -159,7 +163,8 @@ The following flags are forwarded to the live-eval engine **only when Tier 3 is
 
 | Flag | Default | Effect |
 | --- | --- | --- |
-| `--tier3, --agent-eval` | off | Also run Tier 3 live agent evaluation (requires `evals/evals.json`). Advisory: it never changes the exit code. `--tier3` is the primary name; `--agent-eval` is a supported compatibility alias and is not currently deprecated. |
+| `--tier3, --agent-eval` | off | Also run Tier 3 live agent evaluation. Accepts a supported YAML/JSON dataset or native Harbor source and validates it before execution. Tier 3 is advisory by default. `--tier3` is the primary name; `--agent-eval` is a supported compatibility alias and is not currently deprecated. |
+| `--block-on-agent-eval / --no-block-on-agent-eval` | advisory | Choose whether Tier 3 findings, including missing or invalid selected task-source evidence, participate in the `validate` exit gate. This flag does not enable Tier 3 by itself. |
 | `--autopilot` | off | Generate an evaluation source automatically when missing, then run Tier 3 (implies `--tier3`). |
 | `-a, --agents TEXT` | `codex` | Comma-separated Harbor agents to evaluate. |
 | `--env-mode` | `docker` | Harbor environment backend (full list under [tier3 evaluate](#tier3-evaluate)). |

docs/environment-variables.mdx

@@ -79,7 +79,7 @@ These variables tune the static and security checks. The check-by-check guide is
 | `SKILLEVALUATOR_SKILLSPECTOR_PATH` | auto-discovered | Absolute path to the SkillSpector executable. |
 | `SKILLEVALUATOR_SCHEMA_ALLOWED_DIRS` | — | Extra allowed skill-root directory names (comma- or whitespace-separated), e.g. `data,fixtures`. Additive only — names extend the built-in allowlist and never replace it. |
 | `SKILLEVALUATOR_SUBMITTER` | auto-detected | Explicit submitter identity for the home-path PII check. It is combined with `GITHUB_ACTOR`, `USER`, `LOGNAME`, `USERNAME`, the OS login name, and the skill's declared author — every identity that resolves is protected. If none resolves, the home-path check is disabled with a warning. |
-| `SKILLEVALUATOR_PREVIOUS_VERSION` | — | The previously published skill version that the opt-in version check (`--checks version`) compares against. |
+| `SKILLEVALUATOR_PREVIOUS_VERSION` | — | The previously published skill version that the default version check compares against. Without this value or `--previous-version`, the check validates the current version's format but does not enforce an increase. |
 
 <Warning>
 The scanner path overrides fail closed: a value that is not an absolute, executable file is treated as a configuration error rather than silently falling back to another same-named binary on `PATH`. This prevents a planted executable from hijacking a security scan.

docs/quickstart.mdx

@@ -92,7 +92,7 @@ Get your first evaluation result in about two minutes, with no API key. This pag
 | Gate skill quality and security | `validate` | [Tier 1: Validation](tier1-validation.mdx) | Exit code `0`, no CRITICAL findings |
 | Find overlapping or duplicate skills | `similarity-check` | [Tier 2: Deduplication](tier2-deduplication.mdx) | No `EXACT_DUPLICATE` or `HIGH_SIMILARITY` findings |
 | Prove the skill helps a real agent | `tier3 evaluate` | [Tier 3: Live Evaluation](tier3-live-evaluation.mdx) | Positive Skill Lift across the five dimensions |
-| Run all three tiers in one shot | `validate --full` | [Tier 3: Live Evaluation](tier3-live-evaluation.mdx) | Exit code `0` plus an auto-generated dataset and advisory Tier 3 results |
+| Run all three tiers in one shot | `validate --full` | [Tier 3: Live Evaluation](tier3-live-evaluation.mdx) | Exit code `0` plus an auto-generated dataset; Tier 3 is advisory unless you add `--block-on-agent-eval` |
 | Block bad skills at merge time | `validate` in CI | [Gate Your CI](ci-integration.mdx) | A red check on failing PRs |
 | Look up every flag and default | — | [CLI Reference](cli-reference.mdx) | — |
 

docs/reports.mdx

@@ -280,7 +280,9 @@ For CI and tooling, two JSON entry points matter:
 - **`validate --tier3` runs** — the combined
   `skillevaluator-output-<timestamp>.json` report embeds the Tier 3 payload
   alongside the Tier 1 and Tier 2 results, so one file covers all three
-  tiers. Tier 3 findings are advisory there: they never change the exit code.
+  tiers. Each result carries finalized gating metadata. Tier 3 is advisory by
+  default and becomes blocking with `--block-on-agent-eval`; Tier 2 is blocking
+  by default and becomes advisory with `--no-block-on-dedup`.
 
 Both paths speak the same dialect: standalone `tier3 evaluate` and
 `validate --tier3` share one HTML renderer for `report.html`, and both embed

docs/tier1-validation.mdx

@@ -19,7 +19,7 @@ Tier 1 is exposed through one umbrella command and five standalone commands. Eac
 | `pii-scan` | Detect PII, credentials, secrets, and local identifiers | No (only with `--llm-verify`) |
 | `lint-scripts` | Run advisory AST-based checks on Python scripts | No |
 
-`validate` is the umbrella command: Tier 1 checks gate the exit code (non-zero on failure or incomplete required scanner evidence), Tier 2 dedup runs by default and degrades gracefully without embedding access, and Tier 3 can be attached as an advisory pass with `--tier3`, `--autopilot`, `--full`, or an explicit tier selection such as `--tiers 1,3` (`--agent-eval` remains a supported compatibility alias and is not currently deprecated).
+`validate` is the umbrella command: Tier 1 checks always gate the exit code (non-zero on failure or incomplete required scanner evidence), Tier 2 dedup runs and gates by default while degrading gracefully without embedding access, and Tier 3 can be attached as an advisory pass with `--tier3`, `--autopilot`, `--full`, or an explicit tier selection such as `--tiers 1,3` (`--agent-eval` remains a supported compatibility alias and is not currently deprecated). Use `--no-block-on-dedup` to keep Tier 2 advisory or `--block-on-agent-eval` to promote Tier 3 into the gate.
 
 ## Run a validation
 
@@ -297,7 +297,7 @@ Bundled Python scanners resolve next to the SkillEvaluator interpreter before `P
 | `2` | Configuration error — invalid flags, arguments, or target path |
 | `3` | Runtime error — an unexpected internal failure |
 
-Tier 1 checks — and the Tier 2 dedup pass, when it runs — gate the exit code inside `validate`; Tier 3 findings are advisory and never change it. The `quality` check passes when the composite score reaches `--min-score` (default `70`).
+Tier 1 checks always gate the exit code inside `validate`. Tier 2 gates when it runs unless `--no-block-on-dedup` is set. Tier 3 is advisory unless `--block-on-agent-eval` is set. The `quality` check passes when the composite score reaches `--min-score` (default `70`). Reports record the effective gate membership for every tier so saved evidence and process status agree.
 
 `--fail-fast` stops after the first failed check. `-c`/`--continue-on-failure` overrides it, records the full pipeline, and keeps scanning a collection past a CRITICAL finding. Wiring these exit codes into a merge gate — including a complete GitHub Actions recipe — is covered in [Gate Your CI](ci-integration.mdx).
 

docs/tier2-deduplication.mdx

@@ -227,6 +227,11 @@ produce a non-zero exit code; MEDIUM and LOW findings are advisory. Exit
 behavior is identical in every output mode, so these commands work as CI gates
 — see [Gate Your CI](ci-integration.mdx).
 
+Inside the combined `validate` pipeline, Tier 2 also gates by default. Pass
+`--no-block-on-dedup` to keep the scan and its findings in every report without
+letting those findings change `validate`'s exit code. Use `--no-dedup` only when
+you intend to skip Tier 2 entirely.
+
 ## Defaults and safety limits
 
 | Setting | Default | Purpose |

docs/tier3-live-evaluation.mdx

@@ -286,13 +286,20 @@ cloud backend for untrusted code.
 
 | Use case | Command | When to use |
 | --- | --- | --- |
-| Full validation | `skillevaluator validate ./my-skill --tier3` | Add advisory Tier 3 results after the standard validation stages |
+| Full validation | `skillevaluator validate ./my-skill --tier3` | Add Tier 3 after the standard stages; advisory by default, optionally blocking |
 | Focused evaluation | `skillevaluator tier3 evaluate ./my-skill --agents codex` | Iterate on datasets, agents, environments, and grading settings |
 
-Attached to `validate`, Tier 3 is advisory: its results appear in the combined
-reports but never change the exit code. `--full` is the one-shot variant
-— Tier 1+2+3 with `--autopilot` dataset generation — and `--autopilot` on its
-own enables Tier 3 while generating an eval case when none exists.
+Attached to `validate`, Tier 3 is advisory by default: its results appear in the
+combined reports without changing the exit code. Add `--block-on-agent-eval`
+to make its findings gate, including evidence that the selected YAML/JSON
+dataset or native Harbor source is missing or invalid. `--full` is the one-shot
+variant — Tier 1+2+3 with `--autopilot` dataset generation — and `--autopilot`
+on its own enables Tier 3 while generating an eval case when none exists.
+
+```bash title="Make attached Tier 3 blocking"
+skillevaluator validate ./my-skill --tier3 --agents codex \
+  --block-on-agent-eval
+```
 
 <Note>
 `validate` forwards a dedicated Tier 3 flag group (`--agents`, `--env-mode`,