Skip to content

Validate severity flag values instead of silently changing what runs - #166

Open
gesposito wants to merge 1 commit into
vercel-labs:mainfrom
gesposito:fix/validate-severity-flags
Open

Validate severity flag values instead of silently changing what runs#166
gesposito wants to merge 1 commit into
vercel-labs:mainfrom
gesposito:fix/validate-severity-flags

Conversation

@gesposito

@gesposito gesposito commented Aug 24, 2026

Copy link
Copy Markdown

What changed

The six options that take a severity — --severity, --min-severity, --only-severity — are now declared with .addOption(new Option(...).choices(SEVERITIES)), so Commander rejects an unrecognized value instead of passing it through to be cast to Severity and used as a filter key. export --only-severity already validated by hand, and that now-duplicated check is removed.

Why

Only one of the six severity options validated its input. The other five cast the raw string straight to Severity, and because the value ends up as a lookup key or an equality operand, a typo doesn't fail — it quietly changes which findings the command operates on, then reports success and exits 0.

The failure is not even consistent, because each call site compares differently:

flag comparison a bad value means
triage --severity finding.severity !== severity matches nothing — prints Triage complete., exits 0, triages zero findings
revalidate --min-severity ORDER[f] > ORDER[bad] n > undefined is false, so the filter is disabled and everything is revalidated
enrich --min-severity ORDER[f] <= ORDER[bad] n <= undefined is false, so nothing qualifies and nothing is enriched
export --min-severity ORDER[f] > ORDER[bad] filter disabled — every severity is exported
metrics --min-severity ORDER[bad] ?? 2 silently means MEDIUM
export --only-severity validated at export.ts:354 correctly rejected

So the same typo under-includes on two commands, over-includes on two others, and silently picks a tier on a fifth. What we hit in practice: deepsec triage --severity CRTICAL echoed the typo back three times, triaged nothing, printed a green Triage complete. and exited 0. Nothing in the run record showed it either — toTriage.length === 0 returns before createRunMeta, so no run is written at all. Separately, deepsec export --min-severity NONSENSE --only-true-positive exported all 474 findings rather than erroring.

--only-severity proves the intent: unrecognized severities were always meant to be an error, and the check simply wasn't applied to its five siblings.

Notes for reviewer

.choices() is Commander's own mechanism for this, so there is no hand-written parser to keep in step with the ladder. It also renders the allowed values into --help, which lets four descriptions drop the parenthetical list they were maintaining by hand — one of which had already drifted: revalidate --min-severity documented five of the six severities, omitting LOW even though it has always been accepted. Using .choices() means that list can no longer go stale.

.choices() requires .addOption(new Option(...)) rather than .option(...), which is a new shape in this file. It seemed worth it for six options that were all getting the same treatment; happy to switch to a shared parseArg function if you'd rather keep .option() throughout.

The error is Commander's standard one, naming the offending flag: error: option '--min-severity <sev>' argument 'NONSENSE' is invalid. Allowed choices are CRITICAL, HIGH, MEDIUM, HIGH_BUG, BUG, LOW.

SEVERITIES is declared in cli.ts rather than in @deepsec/core, to keep this a validation-only change. Hoisting a canonical ladder into core is #48's job, and it can't ride along here: the six local SEVERITY_ORDER maps have drifted (packages/processor/src/enrich.ts:156 and packages/deepsec/src/commands/export.ts:9 rank HIGH_BUG above MEDIUM; packages/deepsec/src/sandbox/partitioner.ts:8 omits LOW entirely), so unifying them changes what export --min-severity MEDIUM returns. That is user-visible and belongs in its own PR.

Two small things came along because they are the same lines: export.ts's hand-rolled check is now unreachable, since Commander rejects the value before exportCommand runs; and triage --help no longer prints its default twice (Severity to triage (default: MEDIUM) (default: "MEDIUM")), because the description no longer restates what Commander already renders.

The test is an e2e one rather than a unit test, since with .choices() there is no unit left to test — the regression worth guarding is the wiring. It exercises all six flags through the built bundle; reverting any single option to a bare .option() fails it with that flag named.

Not addressed here, and arguably the real ergonomic gap: triage takes only an exact tier, so covering the ladder means six sequential runs. Adding --min-severity to triage is a feature, not this bug, so it is left out.

Verification

  • pnpm test passes — 2411 passed, 1 skipped, 63 files
  • pnpm lint passes
  • pnpm knip passes
  • pnpm -r build passes
  • pnpm test:bundle passes — 30 tests
  • pnpm typecheck:deepsec passes
  • N/A — no matcher added

@vercel

vercel Bot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

@gesposito is attempting to deploy a commit to the Vercel Labs Team on Vercel.

A member of the Team first needs to authorize it.

@gesposito
gesposito force-pushed the fix/validate-severity-flags branch from ff19e9c to 7623506 Compare August 24, 2026 19:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant