Retain empty group for npm to perform strict search (#193)

* Retain empty group for npm to perform strict search

Signed-off-by: Prabhu Subramanian <[email protected]>

* Fix tests

Signed-off-by: Prabhu Subramanian <[email protected]>

---------

Signed-off-by: Prabhu Subramanian <[email protected]>

Author: prabhu

README.md

@@ -9,12 +9,12 @@ OWASP dep-scan is a next-generation security and risk audit tool based on known
 
 ## Features
 
-- Scan most application code - local repos, Linux container images, Kubernetes manifests, and OS - to identify known CVEs with prioritization
-- Perform advanced reachability analysis for multiple languages (See reachability analysis)
-- Package vulnerability scanning is performed locally and is quite fast. No server is used!
-- Generate Software Bill-of-Materials (SBOM) with Vulnerability Disclosure Report (VDR) information
-- Generate a Common Security Advisory Framework (CSAF) 2.0 VEX document (check out the [CSAF Readme](contrib/CSAF_README.md))
-- Perform deep packages risk audit for dependency confusion attacks and maintenance risks (See risk audit)
+-   Scan most application code - local repos, Linux container images, Kubernetes manifests, and OS - to identify known CVEs with prioritization
+-   Perform advanced reachability analysis for multiple languages (See reachability analysis)
+-   Package vulnerability scanning is performed locally and is quite fast. No server is used!
+-   Generate Software Bill-of-Materials (SBOM) with Vulnerability Disclosure Report (VDR) information
+-   Generate a Common Security Advisory Framework (CSAF) 2.0 VEX document (check out the [CSAF Readme](contrib/CSAF_README.md))
+-   Perform deep packages risk audit for dependency confusion attacks and maintenance risks (See risk audit)
 
 ![Reachable Flows](docs/depscan-flows.png)
 
@@ -24,26 +24,26 @@ OWASP dep-scan is a next-generation security and risk audit tool based on known
 
 ### Vulnerability Data sources
 
-- OSV
-- NVD
-- GitHub
-- NPM
-- Linux [vuln-list](https://github.com/appthreat/vuln-list)
+-   OSV
+-   NVD
+-   GitHub
+-   NPM
+-   Linux [vuln-list](https://github.com/appthreat/vuln-list)
 
 ### Linux distros
 
-- AlmaLinux
-- Debian
-- Alpine
-- Amazon Linux
-- Arch Linux
-- RHEL/CentOS
-- Rocky Linux
-- Ubuntu
-- OpenSUSE/SLES
-- Photon
-- Chainguard
-- Wolfi OS
+-   AlmaLinux
+-   Debian
+-   Alpine
+-   Amazon Linux
+-   Arch Linux
+-   RHEL/CentOS
+-   Rocky Linux
+-   Ubuntu
+-   OpenSUSE/SLES
+-   Photon
+-   Chainguard
+-   Wolfi OS
 
 Application vulnerabilities would be reported for all Linux distros and Windows. To download the full vulnerability database suitable for scanning OS, invoke dep-scan with `--cache` for the first time. dep-scan would also download the appropriate database based on project type automatically.
 
@@ -66,9 +66,9 @@ oras pull ghcr.io/owasp-dep-scan/depscan:v4 -o $VDB_HOME
 
 Download the executable binary for your operating system from the [releases page](https://github.com/owasp-dep-scan/depscan-bin/releases). These binary bundle the following:
 
-- dep-scan with Python 3.10
-- cdxgen with Node.js 18
-- cdxgen binary plugins
+-   dep-scan with Python 3.10
+-   cdxgen with Node.js 18
+-   cdxgen binary plugins
 
 ```bash
 curl -LO https://github.com/owasp-dep-scan/depscan-bin/releases/latest/download/depscan-linux-amd64
@@ -108,25 +108,25 @@ Use the `/scan` endpoint to perform scans.
 > [!NOTE]
 > The `type` parameter is mandatory in server mode.
 
-- Scanning a local directory.
+-   Scanning a local directory.
 
 ```bash
 curl --json '{"path": "/tmp/vulnerable-aws-koa-app", "type": "js"}' http://0.0.0.0:7070/scan
 ```
 
-- Scanning a SBOM file (present locally).
+-   Scanning a SBOM file (present locally).
 
 ```bash
 curl --json '{"path": "/tmp/vulnerable-aws-koa-app/sbom_file.json", "type": "js"}' http://0.0.0.0:7070/scan
 ```
 
-- Scanning a GitHub repo.
+-   Scanning a GitHub repo.
 
 ```bash
 curl --json '{"url": "https://github.com/HooliCorp/vulnerable-aws-koa-app", "type": "js"}' http://0.0.0.0:7070/scan -o app.vdr.json
 ```
 
-- Uploading a SBOM file and generating results based on it.
+-   Uploading a SBOM file and generating results based on it.
 
 ```bash
 curl -X POST -H 'Content-Type: multipart/form-data' -F 'file=@/tmp/app/sbom_file.json' http://0.0.0.0:7070/scan?type=js
@@ -315,14 +315,14 @@ depscan --profile research -t js -i <source directory> --reports-dir <reports di
 
 The following environment variables can be used to customise the behaviour.
 
-- VDB_HOME - Directory to use for caching database. For docker based execution, this directory should get mounted as a volume from the host
+-   VDB_HOME - Directory to use for caching database. For docker based execution, this directory should get mounted as a volume from the host
 
 ## GitHub Security Advisory
 
 To download security advisories from GitHub, a personal access token with minimal permissions is necessary.
 
-- Fine-grained token: Grant no permissions and select the following for repository access: `Public Repositories (read-only)`
-- Token (classic): Grant no permissions
+-   Fine-grained token: Grant no permissions and select the following for repository access: `Public Repositories (read-only)`
+-   Token (classic): Grant no permissions
 
 ```bash
 export GITHUB_TOKEN="<PAT token>"
@@ -444,19 +444,32 @@ Severity counts:
 * Unspecified: {{ summary.UNSPECIFIED }}
 ```
 
-The objects available are taken from the CycloneDX *.vdr.json BOM file generated, just have a look to the file for its full structure:
+The objects available are taken from the CycloneDX \*.vdr.json BOM file generated, just have a look to the file for its full structure:
 
-* `metadata`
-* `vulnerabilities`
-* `components`
-* `dependencies`
-* `services`
+-   `metadata`
+-   `vulnerabilities`
+-   `components`
+-   `dependencies`
+-   `services`
 
 `summary` is a dictionary type with vulnerability severity quantities as shown in the example above.
 Furthermore insights are imaginably to be made available to the template, please reach out or contribute on demand.
 
 We appreciate if you like to contribute your report templates as examples, please add/find them [here](contrib/report-templates/).
 
+## Performance tuning
+
+### Use nydus to speed up the initial vdb download
+
+vdb v5 is published in RAFS (Registry Accelerated File System) format with better de-duplication and packing. depscan would automatically use this image if `nydus-static` binary is available in the PATH.
+
+```shell
+curl -LO https://github.com/dragonflyoss/nydus/releases/download/v2.2.4/nydus-static-v2.2.4-linux-amd64.tgz
+tar -xvf nydus-static-v2.2.4-linux-amd64.tgz
+chmod +x nydus-static/*
+mv nydus-static/* /usr/local/bin/
+```
+
 ## Discord support
 
 The developers could be reached via the [discord](https://discord.gg/DCNxzaeUpd) channel for enterprise support.

depscan/lib/normalize.py

@@ -54,6 +54,11 @@ def create_pkg_variations(pkg_dict):
             if purl_obj:
                 pkg_type = purl_obj.get("type")
                 qualifiers = purl_obj.get("qualifiers", {})
+                namespace = purl_obj.get("namespace")
+                # npm is known for packages with no group
+                # To reduce false positives we retain such empty groups here
+                if pkg_type in ("npm",) and namespace is None:
+                    vendor_aliases.add("")
                 if qualifiers and qualifiers.get("distro_name"):
                     os_distro_name = qualifiers.get("distro_name")
                     name_aliases.add(f"""{os_distro_name}/{name}""")
@@ -62,7 +67,9 @@ def create_pkg_variations(pkg_dict):
                     name_aliases.add(f"""{os_distro}/{name}""")
                     # almalinux-9.2 becomes almalinux-9
                     if "-" in os_distro and "." in os_distro:
-                        name_aliases.add(f"""{os_distro.rsplit(".", 1)[0]}/{name}""")
+                        name_aliases.add(
+                            f"""{os_distro.rsplit(".", 1)[0]}/{name}"""
+                        )
         except Exception:
             tmp_parts = purl.split(":")
             if tmp_parts and len(tmp_parts) > 1:
@@ -94,7 +101,6 @@ def create_pkg_variations(pkg_dict):
         ]:
             vendor_aliases.add("golang")
     if pkg_type not in config.OS_PKG_TYPES:
-        name_aliases.add("package_" + name)
         if purl.startswith("pkg:composer"):
             vendor_aliases.add("get" + name)
             vendor_aliases.add(name + "_project")
@@ -118,8 +124,6 @@ def create_pkg_variations(pkg_dict):
         vendor_aliases.add("python-" + name)
         vendor_aliases.add(name + "project")
     elif purl.startswith("pkg:npm"):
-        if not name.startswith("node-"):
-            name_aliases.add("node-" + name)
         # pg-promise CVE is filed as pg
         if name.endswith("-promise"):
             name_aliases.add(name.replace("-promise", ""))
@@ -158,7 +162,11 @@ def create_pkg_variations(pkg_dict):
         # The below aliasing is resulting in several false positives for npm
         if pkg_type not in ("npm",):
             for k, v in config.package_alias.items():
-                if name.startswith(k) or k.startswith(name) or v.startswith(name):
+                if (
+                    name.startswith(k)
+                    or k.startswith(name)
+                    or v.startswith(name)
+                ):
                     name_aliases.add(k)
                     name_aliases.add(v)
     if pkg_type in config.OS_PKG_TYPES:
@@ -170,14 +178,20 @@ def create_pkg_variations(pkg_dict):
             name_aliases.add(name + "-bin")
     else:
         # Filter vendor aliases that are also name aliases
-        vendor_aliases = [x for x in vendor_aliases if x not in name_aliases or x == vendor]
-    if len(vendor_aliases) > 0:
+        vendor_aliases = [
+            x for x in vendor_aliases if x not in name_aliases or x == vendor
+        ]
+    if len(vendor_aliases) > 1:
         for vvar in list(vendor_aliases):
             for nvar in list(name_aliases):
                 pkg_list.append(
-                    {"vendor": vvar, "name": nvar, "version": pkg_dict["version"]}
+                    {
+                        "vendor": vvar,
+                        "name": nvar,
+                        "version": pkg_dict["version"],
+                    }
                 )
-    else:
+    elif len(name_aliases) > 1:
         for nvar in list(name_aliases):
             pkg_list.append(
                 {

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "owasp-depscan"
-version = "5.1.1"
+version = "5.1.2"
 description = "Fully open-source security audit for project dependencies based on known vulnerabilities and advisories."
 authors = [
     {name = "Team AppThreat", email = "[email protected]"},

test/test_norm.py

@@ -31,15 +31,27 @@ def test_pkg_variations():
     )
     assert len(pkg_list) > 1
     pkg_list = create_pkg_variations(
-        {"vendor": "io.undertow", "name": "undertow-core", "version": "2.0.27.Final"}
+        {
+            "vendor": "io.undertow",
+            "name": "undertow-core",
+            "version": "2.0.27.Final",
+        }
     )
     assert len(pkg_list) > 1
     pkg_list = create_pkg_variations(
-        {"vendor": "io.undertow", "name": "undertow-core", "version": "2.0.27.Final"}
+        {
+            "vendor": "io.undertow",
+            "name": "undertow-core",
+            "version": "2.0.27.Final",
+        }
     )
     assert len(pkg_list) > 1
     pkg_list = create_pkg_variations(
-        {"vendor": "org.apache.logging.log4j", "name": "log4j-api", "version": "2.12.1"}
+        {
+            "vendor": "org.apache.logging.log4j",
+            "name": "log4j-api",
+            "version": "2.12.1",
+        }
     )
     assert len(pkg_list) > 1
     pkg_list = create_pkg_variations(
@@ -59,7 +71,11 @@ def test_pkg_variations():
     )
     assert len(pkg_list) > 1
     pkg_list = create_pkg_variations(
-        {"vendor": "github.com/go-sql-driver", "name": "mysql", "version": "v1.4.1"}
+        {
+            "vendor": "github.com/go-sql-driver",
+            "name": "mysql",
+            "version": "v1.4.1",
+        }
     )
     assert len(pkg_list) > 1
     pkg_list = create_pkg_variations(
@@ -69,44 +85,44 @@ def test_pkg_variations():
             "version": "0.0.0-20200220183623-bac4c82f6975",
         }
     )
-    assert len(pkg_list) > 1
+    assert pkg_list
     pkg_list = create_pkg_variations(
         {
             "vendor": "github.com/mitchellh",
             "name": "cli",
             "version": "6.14.1",
         }
     )
-    assert len(pkg_list) > 1
+    assert pkg_list
     pkg_list = create_pkg_variations(
         {
             "vendor": "github.com/jacobsa",
             "name": "crypto",
             "version": "6.14.1",
         }
     )
-    assert len(pkg_list) > 1
+    assert pkg_list
     pkg_list = create_pkg_variations(
         {
             "vendor": "org.hibernate",
             "name": "hibernate-core",
             "version": "5.4.18.Final",
         }
     )
-    assert len(pkg_list) > 1
+    assert pkg_list
     pkg_list = create_pkg_variations(
         {
             "vendor": "org.springframework.security",
             "name": "spring-security-crypto",
             "version": "5.3.3.RELEASE",
         }
     )
-    assert len(pkg_list) > 1
+    assert pkg_list
     pkg_list = create_pkg_variations(
         {
             "vendor": "deb",
             "name": "gnome-accessibility-themes",
             "version": "3.28-1ubuntu3",
         }
     )
-    assert len(pkg_list) > 1
+    assert pkg_list