Skip to content

fix(cli): exit 3 when incomplete-policy=error fails - #941

Open
RohithPariki wants to merge 3 commits into
OWASP:mainfrom
RohithPariki:fix-898
Open

fix(cli): exit 3 when incomplete-policy=error fails#941
RohithPariki wants to merge 3 commits into
OWASP:mainfrom
RohithPariki:fix-898

Conversation

@RohithPariki

Copy link
Copy Markdown

Summary

Adds a configurable failure policy for incomplete scans using the --incomplete-policy error flag. When scans are incomplete due to missing/failed OSV details, the CLI now correctly triggers a process exit with code 3 (EXIT_ERROR).

Problem

Previously, if a scan failed to fetch full vulnerability details from OSV (e.g. network timeout) but managed to complete the scan, it would silently return EXIT_FINDINGS or EXIT_OK. Users running the tool in CI environments needed a strict mode to fail the build (Issue #898) when a scan was flagged as incomplete.

Root Cause

The EXIT_ERROR (3) logic wasn't integrated to check the ScanCompleteness result in the main command orchestrators (index.ts and multi-folder-scan.ts) when an explicit error policy was provided.

Solution

  • Added the --incomplete-policy <warn|error> argument to the CLI parsing configuration.
  • Modified index.ts (single folder) and src/scan/multi-folder-scan.ts (multi-folder) to check if the scan result is ScanCompleteness.INCOMPLETE and if the options flag is set to error.
  • Upon matching, overridden the return code to EXIT_ERROR (3) before terminating.

Testing & Verification

  • Unit and E2E tests added in tests/e2e/commands-and-exit-codes.test.ts to assert that executing the CLI with --incomplete-policy error correctly yields a non-zero exit code (3) when OSV data fails.
  • All tests pass locally.

References

Closes #898

@sonukapoor

Copy link
Copy Markdown
Collaborator

Thanks @RohithPariki. This lands in an area that is actively being built out - the scan-completeness work under #907 (@luojiyin1987's #908 is merged, with #936 and #937 open now). A dedicated --incomplete-policy flag and exit code is a design decision that needs to fit that series, and #937 already covers part of the incomplete-scan behavior. Rather than land a competing design, I would like to let #907 settle first and keep the completeness UX coherent, so I will hold this for now, and once #936/#937 are in we can revisit whether the policy flag still fits on top. Appreciate the initiative.

@luojiyin1987

Copy link
Copy Markdown
Collaborator

Thanks for the clarification.

I understand that the scan completeness behavior is still being consolidated under #907 and related PRs. It makes sense to keep the UX and exit code semantics consistent instead of introducing a parallel approach.

I'll keep an eye on #936/#937 and revisit this if the policy flag still fits after those changes land.

Thanks for the guidance.

@sonukapoor sonukapoor left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking this on @RohithPariki, and sorry for the slow turn. The direction is right, but a few things need sorting before it can land, plus one bigger coordination point.

The main thing: --incomplete-policy and the exit-code behavior for incomplete scans really belong to the scan-completeness work @luojiyin1987 has in flight (#907, with #937 still open). We should settle the semantics inside that series so the two approaches don't diverge - as it stands, this and #937 would treat incomplete scans differently. Let's hold this until #907 lands and align on the design there.

A few concrete issues in the current diff:

  1. The exit-3 check gates on !completeness.complete, which fires on any diagnostic - including remediation-only gaps (packument/remediation/chain-resolution failures). Both #898 and #937 are explicit that remediation-only gaps shouldn't fail the scan; only detection gaps should. There's already a getCompletenessImpact() helper on main returning hasDetectionGap / hasRemediationGap - gate on that instead of complete.

  2. The description says unit + e2e tests were added asserting --incomplete-policy error yields exit 3, but I'm not finding them in the diff - the test-file changes are the USERPROFILE/harness bits, and there's no incomplete-policy assertion anywhere under tests/. Could you add the real coverage: single- and multi-folder exit-3, plus a test that a remediation-only gap does not trigger it?

  3. Exit code 3 is currently documented as "tool error" (src/types.ts), so overloading it for "incomplete scan" means a CI consumer can't tell a crash from an incomplete-but-otherwise-fine scan. Worth deciding deliberately (and updating the docstring) as part of the #907 alignment.

  4. The Windows/test-harness changes (shell: true, USERPROFILE, backslash normalization) look unrelated to this feature - could you split those into their own PR so this one stays tightly scoped?

Smaller: incompletePolicy?: string should be the 'warn' | 'error' union, and the validation strings in args.ts are duplicated across the two branches. It'll also need --help / cli-reference / README once the approach is settled.

Really appreciate the effort here - this just needs to land inside the completeness series rather than alongside it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] add a configurable failure policy for incomplete scans

3 participants