fix(cli): exit 3 when incomplete-policy=error fails - #941
Conversation
|
Thanks @RohithPariki. This lands in an area that is actively being built out - the scan-completeness work under #907 (@luojiyin1987's #908 is merged, with #936 and #937 open now). A dedicated --incomplete-policy flag and exit code is a design decision that needs to fit that series, and #937 already covers part of the incomplete-scan behavior. Rather than land a competing design, I would like to let #907 settle first and keep the completeness UX coherent, so I will hold this for now, and once #936/#937 are in we can revisit whether the policy flag still fits on top. Appreciate the initiative. |
|
Thanks for the clarification. I understand that the scan completeness behavior is still being consolidated under #907 and related PRs. It makes sense to keep the UX and exit code semantics consistent instead of introducing a parallel approach. I'll keep an eye on #936/#937 and revisit this if the policy flag still fits after those changes land. Thanks for the guidance. |
sonukapoor
left a comment
There was a problem hiding this comment.
Thanks for taking this on @RohithPariki, and sorry for the slow turn. The direction is right, but a few things need sorting before it can land, plus one bigger coordination point.
The main thing: --incomplete-policy and the exit-code behavior for incomplete scans really belong to the scan-completeness work @luojiyin1987 has in flight (#907, with #937 still open). We should settle the semantics inside that series so the two approaches don't diverge - as it stands, this and #937 would treat incomplete scans differently. Let's hold this until #907 lands and align on the design there.
A few concrete issues in the current diff:
-
The exit-3 check gates on
!completeness.complete, which fires on any diagnostic - including remediation-only gaps (packument/remediation/chain-resolution failures). Both #898 and #937 are explicit that remediation-only gaps shouldn't fail the scan; only detection gaps should. There's already agetCompletenessImpact()helper on main returninghasDetectionGap/hasRemediationGap- gate on that instead ofcomplete. -
The description says unit + e2e tests were added asserting
--incomplete-policy erroryields exit 3, but I'm not finding them in the diff - the test-file changes are the USERPROFILE/harness bits, and there's noincomplete-policyassertion anywhere undertests/. Could you add the real coverage: single- and multi-folder exit-3, plus a test that a remediation-only gap does not trigger it? -
Exit code 3 is currently documented as "tool error" (
src/types.ts), so overloading it for "incomplete scan" means a CI consumer can't tell a crash from an incomplete-but-otherwise-fine scan. Worth deciding deliberately (and updating the docstring) as part of the #907 alignment. -
The Windows/test-harness changes (
shell: true, USERPROFILE, backslash normalization) look unrelated to this feature - could you split those into their own PR so this one stays tightly scoped?
Smaller: incompletePolicy?: string should be the 'warn' | 'error' union, and the validation strings in args.ts are duplicated across the two branches. It'll also need --help / cli-reference / README once the approach is settled.
Really appreciate the effort here - this just needs to land inside the completeness series rather than alongside it.
…nd add unit tests
Summary
Adds a configurable failure policy for incomplete scans using the
--incomplete-policy errorflag. When scans are incomplete due to missing/failed OSV details, the CLI now correctly triggers a process exit with code3(EXIT_ERROR).Problem
Previously, if a scan failed to fetch full vulnerability details from OSV (e.g. network timeout) but managed to complete the scan, it would silently return
EXIT_FINDINGSorEXIT_OK. Users running the tool in CI environments needed a strict mode to fail the build (Issue #898) when a scan was flagged asincomplete.Root Cause
The
EXIT_ERROR(3) logic wasn't integrated to check theScanCompletenessresult in the main command orchestrators (index.tsandmulti-folder-scan.ts) when an expliciterrorpolicy was provided.Solution
--incomplete-policy <warn|error>argument to the CLI parsing configuration.index.ts(single folder) andsrc/scan/multi-folder-scan.ts(multi-folder) to check if the scan result isScanCompleteness.INCOMPLETEand if the options flag is set toerror.EXIT_ERROR(3) before terminating.Testing & Verification
tests/e2e/commands-and-exit-codes.test.tsto assert that executing the CLI with--incomplete-policy errorcorrectly yields a non-zero exit code (3) when OSV data fails.References
Closes #898