CDS extractor : Implement retry for CDS compilation tasks #209

data-douser · 2025-07-30T03:50:41Z

WARNING

This PR contains changes to the CDS extractor code and the CDL.qll library which are NOT backwards compatible with any CodeQL database created from a previous extractor version.

Dependency Installation Changes:

Replaced installDependencies with cacheInstallDependencies in cds-extractor.ts to improve dependency installation performance and caching behavior. [1] [2]

Compilation Process Improvements:

Updated the working directory for spawned processes during compilation to use the project base directory instead of the source root, ensuring consistent path generation. [1] [2] [3]
Adjusted output paths and relative path calculations to align with the project base directory for better clarity and consistency. [1] [2]
Added comments and TODOs for future enhancements, such as session-specific file suffixes for generated files. [1] [2]

Retry Mechanism Enhancements:

Introduced a retry orchestration phase, including methods like orchestrateRetryAttempts and identifyTasksRequiringRetry, to handle failed compilation tasks more robustly. [1] [2] [3]
Deferred diagnostics for failed tasks until after the retry phase to implement a "Silent Success" mechanism, reducing noise from transient errors.

Centralized Status Updates:

Centralized status updates for the dependency graph, ensuring accurate tracking of the compilation process across initial and retry phases.

Miscellaneous Updates:

Added a new build:tests script to the package.json file, ensuring TypeScript checks are run before tests.
Minor improvements to comments for clarity and consistency. [1] [2] [3]

This commit updates the implementation of the CDS extractor in order to attempt to retry failed compilation tasks using the full set of node dependencies specified by the `package.json` config file of the associated CDS project. The new implementation continues to use a minimal cache of `@sap/cds` and `@sap/cds-dk` dependencies as the default/initial installation behavior as a performance optimization. When this does not work, the CDS extractor should now attempt to install all node dependencies for the associated (CAP) project before re-attempting the failed compilation tasks(s) for that project. This commit also updates the `package.json` config for the CDS extractor itself in order to ensure that we build test code before running unit tests, which addresses a bug in the previous testing implementation. The test code is now automatically compiled as part of running commands like `npm run test` or `npm run test:coverage`.

Copilot

Pull Request Overview

This PR implements a comprehensive retry mechanism for CDS compilation tasks that fail initially. The retry system includes dependency installation, output validation, and centralized status tracking to improve compilation success rates.

Introduces retry orchestration with systematic validation and recovery for failed compilation tasks
Adds full dependency installation capabilities for projects requiring broader dependency access during retries
Implements centralized status tracking and validation to ensure accurate reporting across compilation phases

Reviewed Changes

Copilot reviewed 26 out of 28 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
cds-extractor.ts	Updates to use new `cacheInstallDependencies` function
packageManager/	Refactors dependency installation with separate cache and project installation utilities
cds/compiler/	Adds comprehensive retry orchestration, validation, and status tracking infrastructure
test/	Extensive test coverage for new retry mechanisms and validation utilities
package.json	Adds TypeScript validation to test scripts
logging/statusReport.ts	Enhances status reporting with retry-specific metrics

Comments suppressed due to low confidence (2)

extractors/cds/tools/src/cds/compiler/compile.ts:21

The function name parseCommandForSpawn is misleading. It returns baseArgs which suggests arguments that come after the executable, but the function actually splits a command string. Consider renaming to parseCommandString and the return property to args for clarity.

  return { executable, baseArgs };

extractors/cds/tools/src/cds/compiler/retry.ts:1

The substr() method is deprecated. Use substring() or slice() instead: Math.random().toString(36).substring(2, 7).

/** Main retry orchestration logic for CDS compilation failures. */

extractors/cds/tools/src/cds/compiler/compile.ts

extractors/cds/tools/src/cds/compiler/retry.ts

extractors/cds/tools/src/cds/compiler/graph.ts

Reworks the CDS extractor such that it only does project-level compilation, which allows for consistently generating a single model.cds.json file in the base/root directory of its associated project, which allows for matching file paths relative to the project base directory. Updates the `CDL.qll` library file to use location/path values that are relative to the project base directory rather than relative to the source root directory.

lcartey

Thanks @data-douser! I've provided a suggestion on the .qll which I believe should fix the query side. The other comments are suggestions, and could be completed in a separate PR.

javascript/frameworks/cap/lib/advanced_security/javascript/frameworks/cap/CDL.qll

extractors/cds/tools/src/cds/compiler/compile.ts

extractors/cds/tools/src/cds/parser/functions.ts

Implements special handling in the CDS extractor for the use case where an `index.cds` file exists in the base directory of a given CAP project, in which case we just compile the `index.cds` file instead of all the subdirectories of the project.

For CDS compilation tasks that need to be retried because the initial attempt failed to produce a `model.cds.json` file, this commit enforces that the `npx cds compile` command will require a compatible, non-deprecated version of the `@sap/cds-dk` dependency by using the `--yes --package @sap/cds-dk@^8` arguments with the `npx cds compile` command. This addresses a persistent challenge with reliably performing CDS compilation tasks via some form of the `npx cds` command when run from the base directory of a given project. If the `npx` command does not enforce the use of a compatible `--package @sap/cds-dk@*`, then we sometimes encounter errors in environments where the project inherits/uses a non-compatible version of `@sap/cds-dk` from the `node_modules` of some parent directory of the project.

Updates the `CodeQL - Run Unit Tests (javascript` actions workflow to align with recent changes to CDS extractor behavior. Adds the `cds-compilation-for-actions.test.sh` script as well as a note in the CDS extractor's `README.md` documentation in order to encourage improved synchronization between actual CDS extractor behavior versus what the unit tests actions workflow actually compiles.

Updates the `run-codeql-unit-tests-javascript.yml` actions workflow file in order to set the `cache-dependency-path` option as an attempt to resolve a runtime error for that actions workflow.

This commit replaces many separate string literals for `model.cds.json` with an exported constant of the same string in an effort to improve the readability and maintainability of CDS extractor code.

Fixes a regression affecting CodeQL queries for CAP / CDS by ensuring the `getImplementation` predicate of the `CdlService` uses `getAbsolutePath` instead of `getRelativePath`, where the `getRelativePath` approach is problematic in (unit test) cases where the source root directory and project directory are the same.

data-douser · 2025-08-01T20:56:40Z

The changes from this PR should be ready for review.

This PR contains changes to the CDS extractor code and the CDL.qll library which are NOT backwards compatible with any CodeQL database created from a previous extractor version.

data-douser added 2 commits July 28, 2025 12:49

Enhancements for CDS compilation retry

9c01e2b

data-douser requested a review from Copilot July 30, 2025 03:50

data-douser self-assigned this Jul 30, 2025

This comment was marked as outdated.

Sign in to view

Fixes for CDS extractor compilation task retry

08961e9

data-douser requested a review from Copilot July 30, 2025 17:48

Copilot AI reviewed Jul 30, 2025

View reviewed changes

data-douser added 2 commits July 30, 2025 13:20

More fixes for CDS compilation task retry

9bb6169

lcartey requested changes Jul 31, 2025

View reviewed changes

javascript/frameworks/cap/lib/advanced_security/javascript/frameworks/cap/CDL.qll Outdated Show resolved Hide resolved

extractors/cds/tools/src/cds/compiler/compile.ts Outdated Show resolved Hide resolved

extractors/cds/tools/src/cds/parser/functions.ts Outdated Show resolved Hide resolved

data-douser added 6 commits July 31, 2025 07:17

Use absolute path for CdlObject in CDL.qll

ce36be3

CDS extractor special case for index.cds

b3520de

Implements special handling in the CDS extractor for the use case where an `index.cds` file exists in the base directory of a given CAP project, in which case we just compile the `index.cds` file instead of all the subdirectories of the project.

Set cache-dependency-path for actions/setup-node@v4

3058ac2

Updates the `run-codeql-unit-tests-javascript.yml` actions workflow file in order to set the `cache-dependency-path` option as an attempt to resolve a runtime error for that actions workflow.

CDS extractor constant for model.cds.json filename

6c4a8fa

This commit replaces many separate string literals for `model.cds.json` with an exported constant of the same string in an effort to improve the readability and maintainability of CDS extractor code.

data-douser marked this pull request as ready for review August 1, 2025 19:11

lcartey approved these changes Aug 1, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

CDS extractor : Implement retry for CDS compilation tasks #209

CDS extractor : Implement retry for CDS compilation tasks #209

Uh oh!

data-douser commented Jul 30, 2025 •

edited

Loading

Uh oh!

This comment was marked as outdated.

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lcartey left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

data-douser commented Aug 1, 2025

Uh oh!

Uh oh!

CDS extractor : Implement retry for CDS compilation tasks #209

Are you sure you want to change the base?

CDS extractor : Implement retry for CDS compilation tasks #209

Uh oh!

Conversation

data-douser commented Jul 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

WARNING

Dependency Installation Changes:

Compilation Process Improvements:

Retry Mechanism Enhancements:

Centralized Status Updates:

Miscellaneous Updates:

Uh oh!

This comment was marked as outdated.

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lcartey left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

data-douser commented Aug 1, 2025

Uh oh!

Uh oh!

data-douser commented Jul 30, 2025 •

edited

Loading