[pull] master from owasp-dep-scan:master

.editorconfig

@@ -6,5 +6,5 @@ indent_size = 4
 insert_final_newline = true
 trim_trailing_whitespace = true
 
-[*.{yml,yaml,nix}]
+[*.{yml,yaml,nix,ts,js}]
 indent_size = 2

.github/workflows/repotests.yml

@@ -260,23 +260,27 @@ jobs:
         shell: bash
         env:
           DEPSCAN_SOURCE_IMAGE: ghcr.io/owasp-dep-scan/dep-scan:master
+          CDXGEN_TIMEOUT_MS: 3600000
       - uses: actions/checkout@v4
         with:
           repository: 'aboutcode-org/dejacode'
           path: 'repotests/dejacode'
       - name: semantic analysis dejacode
         run: |
-          mkdir -p ${GITHUB_WORKSPACE}/depscan_reports/dejacode ${GITHUB_WORKSPACE}/depscan_reports/dejacode2
+          mkdir -p ${GITHUB_WORKSPACE}/depscan_reports ${GITHUB_WORKSPACE}/repotests/dejacode/reports
           cd ${GITHUB_WORKSPACE}/repotests/dejacode
           docker build -t dejacode:latest -f Dockerfile .
           pip install --find-links=thirdparty/dist/ --no-index --no-cache-dir .
           cd ${GITHUB_WORKSPACE}
-          uv run depscan --src ${GITHUB_WORKSPACE}/repotests/dejacode --reports-dir ${GITHUB_WORKSPACE}/depscan_reports/dejacode -t python --bom-engine CdxgenGenerator --reachability-analyzer SemanticReachability --explain
-          uv run depscan --src ${GITHUB_WORKSPACE}/repotests/dejacode --bom-dir ${GITHUB_WORKSPACE}/depscan_reports/dejacode --reports-dir ${GITHUB_WORKSPACE}/depscan_reports/dejacode2 --reachability-analyzer SemanticReachability --explain --explanation-mode NonReachables
-          ls -lh ${GITHUB_WORKSPACE}/depscan_reports/dejacode ${GITHUB_WORKSPACE}/depscan_reports/dejacode2
+          uv run depscan --src ${GITHUB_WORKSPACE}/repotests/dejacode --reports-dir ${GITHUB_WORKSPACE}/repotests/dejacode/reports -t python --bom-engine CdxgenGenerator --reachability-analyzer SemanticReachability --explain
+          cp -rf ${GITHUB_WORKSPACE}/repotests/dejacode/reports ${GITHUB_WORKSPACE}/depscan_reports/dejacode
+          uv run depscan --src ${GITHUB_WORKSPACE}/repotests/dejacode --bom-dir ${GITHUB_WORKSPACE}/repotests/dejacode/reports --reports-dir ${GITHUB_WORKSPACE}/repotests/dejacode/reports --reachability-analyzer SemanticReachability --explain --explanation-mode NonReachables
+          cp -rf ${GITHUB_WORKSPACE}/repotests/dejacode/reports ${GITHUB_WORKSPACE}/depscan_reports/dejacode2
+          ls -lh ${GITHUB_WORKSPACE}/repotests/dejacode/reports
         shell: bash
         env:
           DEPSCAN_SOURCE_IMAGE: dejacode:latest
+          CDXGEN_TIMEOUT_MS: 3600000
       - uses: actions/upload-artifact@v4
         with:
           name: dejacode_reports

README.md

@@ -91,8 +91,6 @@ Always stay a step ahead with advanced vulnerability and exploit prediction.
 - Chainguard
 - Wolfi OS
 
-Application vulnerabilities would be reported for all Linux distros and Windows. To download the full vulnerability database suitable for scanning OS, invoke dep-scan with `` for the first time. dep-scan would also download the appropriate database based on project type automatically.
-
 ## Quick Start
 
 dep-scan is ideal for use during continuous integration (CI) and as a local development tool.

depscan/lib/bom.py

@@ -15,7 +15,6 @@
     CdxgenImageBasedGenerator,
     CdxgenServerGenerator,
 )
-
 from depscan.lib.logger import LOG, SPINNER, console
 from depscan.lib.utils import cleanup_license_string
 from typing import Dict, Optional
@@ -418,7 +417,12 @@ def create_lifecycle_boms(cdxgen_lib, src_dir, options):
             )
         status.update("Preparing blint for post-build BOM generation.")
     # post-build BOM with blint
-    coptions = {**options, "deep": False, "use_blintdb": False, "lifecycles": ["post-build"]}
+    coptions = {
+        **options,
+        "deep": False,
+        "use_blintdb": False,
+        "lifecycles": ["post-build"],
+    }
     # What if the build directory is different to the source
     build_dir = os.getenv("DEPSCAN_BUILD_DIR") or options.get("build_dir") or src_dir
     res = create_blint_bom(postbuild_bom_file, build_dir, options=coptions)

documentation/docs/Develop/debugging.mdx

@@ -52,15 +52,15 @@ Use the below launch.json configuration.
             "console": "integratedTerminal",
             "args": [
                 "--src",
-                "<source directory>",
+                "source directory",
                 "--reports-dir",
-                "<reports directory>",
+                "reports directory",
                 "-t",
-                "<project type>"
+                "project type"
             ]
         }
     ]
 }
 ```
 
-Replace "<source directory>", "<reports directory>", and "<project type>" with valid values.
+Replace "source directory", "reports directory", and "project type" with valid values.

documentation/docs/Lessons/01-java-semantic-analysis.mdx

@@ -0,0 +1,62 @@
+---
+position: 1
+title: Lesson 1 - Java Semantic Reachability Analysis
+---
+
+# Perform Semantic Reachability Analysis for a Java project
+
+## Learning Objective
+
+In this lesson, we will learn about performing a comprehensive semantic analysis for a Java project, namely dependency-track, with OWASP depscan.
+
+## Pre-requisites
+
+Ensure the following tools are installed:
+
+- Java >= 21
+- Maven
+- Python > 3.10
+- Node.js > 20
+- Docker Desktop, podman, or nerdctl
+
+
+## Getting started
+
+Install cdxgen and depscan.
+
+```shell
+sudo npm install -g @cyclonedx/cdxgen
+python -m pip install owasp-depscan[all]
+```
+
+Clone and compile dependency track
+
+```shell
+git clone https://github.com/DependencyTrack/dependency-track
+cd dependency-track
+mvn clean compile -P clean-exclude-wars -P enhance -P embedded-jetty -DskipTests
+```
+
+Pull the appropriate container image tag
+
+```shell
+docker pull dependencytrack/bundled:latest
+```
+
+## Invoke depscan
+
+```shell
+# Let depscan know the name of the container image
+export DEPSCAN_SOURCE_IMAGE=dependencytrack/bundled:latest
+
+# Perform semantic analysis with local cdxgen engine and detailed explanation
+depscan --src path/to/dependency-track --reports-dir path/to/dependency-track/reports -t java --bom-engine CdxgenGenerator --reachability-analyzer SemanticReachability --explain
+```
+
+Additional learning. Leaving out the `--bom-engine` argument would make depscan use the `CdxgenImageBasedGenerator`, which uses a container image-based BOM generation for better compatibility. On Windows, or in environments without Docker, `CdxgenGenerator` would be used by default.
+
+```shell
+depscan --src path/to/dependency-track --reports-dir path/to/dependency-track/reports -t java --reachability-analyzer SemanticReachability --explain
+```
+
+Unsetting the environment variable `DEPSCAN_SOURCE_IMAGE` would make depscan analyze only the source code without considering the container layer.

documentation/docs/dep-scan.mdx

@@ -39,12 +39,6 @@ OWASP dep-scan is a next-generation security and risk audit tool based on known
 - Chainguard
 - Wolfi OS
 
-:::tip
-
-Application vulnerabilities would be reported for all Linux distros and Windows. To download the full vulnerability database suitable for scanning OS, invoke dep-scan with `` for the first time. dep-scan would also download the appropriate database based on project type automatically.
-
-:::
-
 ## Funding
 
 This project is funded through [NGI Zero Core](https://nlnet.nl/core), a fund established by [NLnet](https://nlnet.nl) with financial support from the European Commission's [Next Generation Internet](https://ngi.eu) program. Learn more at the [NLnet project page](https://nlnet.nl/project/OWASP-dep-scan).

documentation/docs/reachability-analysis.mdx

@@ -107,3 +107,40 @@ depscan --bom-dir <bom directory> --reports-dir <reports directory> --reachabili
 ```
 
 Use the environment variable `MAX_REACHABLE_EXPLANATIONS` to customize the number of data-flow explanations.
+
+## Troubleshooting
+
+### atom file or slices are not generated under the reports directory
+
+Generating atom (intermediate representation) and performing static slicing for reachability and usages is a memory-intensive operation. For large projects, memory upwards of 32GB is the minimum. For example, for DejaCode (a Python application), we need a minimum of 40GB RAM. In resource-constrained environments, atom might crash with the log messages usually suppressed unless depscan is invoked with the environment variable `SCAN_DEBUG_MODE=debug`.
+
+To troubleshoot step by step, try generating the atom file directly using the instructions from the atom [documentation](https://atom-docs.appthreat.dev/).
+
+### No reachable flows were identified
+
+Computing reachable flows using atom requires an accurate SBOM generated by cdxgen with certain arguments. These are typically `--profile research` or `--deep` depending on the project type. For some project types such as Python or JavaScript, additional semantic tags may be required to correctly identify reachable flows. Look for files matching the pattern `*-reachables.slices.json` in the reports directory. If the file is empty or small, then no reachable flows were identified. In certain cases, there could be a file with contents that lack any value for the "purls" attribute. This usually indicates that the SBOM was lacking suitable information for PURL identification.
+
+Consider starting a discussion or filing a bug if the issue could be reproduced with a public project.
+
+### Container SBOMs were not generated
+
+For simple container scans, pass the image name via the `--src` argument. For lifecycle or semantic reachability analysis, use `--src` to pass the source directory and pass the container image name using the environment variable `DEPSCAN_SOURCE_IMAGE`.
+
+### Post-build binary SBOMs were not generated
+
+Ensure the application is first built, since depscan cannot automatically build arbitrary projects and container images. To customize the build directory, use the environment variable `DEPSCAN_BUILD_DIR`.
+
+### PDF reports were not generated
+
+Install the `wkhtmltopdf` [package](https://wkhtmltopdf.org) based on your operating system. Ensure depscan was installed including the group "ext" or "all"
+
+```shell
+pip install owasp-depscan[all]
+```
+
+Alternatively, use the official depscan container image.
+
+```shell
+docker pull ghcr.io/owasp-dep-scan/dep-scan
+# podman pull ghcr.io/owasp-dep-scan/dep-scan
+```