Improve "Delegate Test to Gradle" to a default-level test experience (coverage, discoverability, latency)

[wenytang-ms]

Summary

Improve Delegate Test to Gradle into a complete and reliable test experience for BSP Gradle projects. The first step remains an explicit, non-default Delegate Test to Gradle (Coverage) profile so users can opt in and we can evaluate adoption and reliability before considering any project-level default change.

Motivation

Two vscode-java-test issues share the same execution-path problem:

• [BUG] The extension uses raw data. vscode-java-test#1880 — tests read raw src/main/resources instead of Gradle-processed resources.
• [BUG] IllegalAccessException: module java.base does not open java.lang to unnamed module vscode-java-test#1878 — the test JVM does not inherit Gradle Test task JVM arguments such as --add-opens.

The default Java Test runner launches a standalone test JVM from the JDT model. It does not execute Gradle's Test task, so Gradle task dependencies, processed resources, JVM arguments, environment and other task configuration may be missing.

Delegate Test avoids this by keeping Gradle as the execution authority:

vscode-java-test
    -> vscode-gradle delegate runner
    -> BSP buildTarget/test
    -> Gradle TestLauncher / Test task

This is why delegated Run and Debug already avoid #1880 and #1878. Coverage should preserve the same execution path.

Design principles

1. Gradle owns test execution. Coverage must continue to use BSP buildTarget/test and the real Gradle Test task.
2. Java Test owns coverage presentation and analysis. Source/class mapping, JaCoCo analysis and VS Code Coverage objects should remain in vscode-java-test/JDTLS.
3. Exchange artifacts, not implementation APIs. The Gradle runner should produce isolated JaCoCo execution data; it should not call an exposed Java Test coverage-analysis API.
4. Do not modify user report tasks. Delegated coverage must not reconfigure or execute the user's jacocoTestReport task.
5. Start opt-in. Add an explicit user-selectable profile and observe usage/reliability before changing defaults.

Proposed architecture

Execution and artifact ownership

vscode-java-test creates a unique execution-data directory for each Coverage run and passes it through the existing external TestRunner context using an optional coverage artifact descriptor, for example:

interface IRunTestContext {
    coverage?: {
        format: "jacoco-exec";
        outputDirectory: vscode.Uri;
    };
}

The exact API shape is subject to review, but the ownership should be:

vscode-java-test
    creates run-specific exec directory
        -> vscode-gradle
           runs BSP buildTarget/test
           configures Gradle Test tasks to write isolated *.exec files
        -> vscode-java-test/JDTLS
           loads all *.exec files
           maps class outputs to source roots from IJavaProject
           publishes VS Code FileCoverage
        -> vscode-java-test cleans the run directory

Gradle runner responsibilities

• Execute tests through BSP buildTarget/test; retain the task-server path only as fallback when BSP is unavailable.
• Use a per-run init script only to attach/configure JaCoCo on the forked Gradle Test JVM and redirect execution data into the provided directory.
• Keep execution data isolated by run, Gradle project and Test task.
• Wait for Gradle completion before signaling that coverage artifacts are ready.
• Avoid generating JaCoCo XML and avoid configuring, finalizing or explicitly running jacocoTestReport.

Java Test/JDTLS responsibilities

Extend the existing default Coverage implementation instead of creating a second analyzer:

• JavaTestCoverageProvider remains internal to vscode-java-test; it does not need to be exposed to vscode-gradle.
• After an external Coverage runner finishes, the Java Test controller invokes its own provider.
• CoverageHandler accepts an execution-data directory or file list and repeatedly calls ExecFileLoader.load(...) to merge all .exec files.
• Continue using the imported IJavaProject model (rawClasspath, source entries, output locations and required projects) for class/source mapping.
• Keep JaCoCo Agent and Analyzer versions compatible and owned by the Java Test integration contract.
• Publish line, branch and method coverage through the existing VS Code Coverage API.

This reuses the default Coverage path without reusing the default test launcher. Tests still execute through Gradle, so the design does not reintroduce #1880 or #1878.

Why not generate JaCoCo XML in vscode-gradle?

Generating XML in Gradle requires configuring and executing a report task, then duplicating source resolution and XML-to-VS-Code conversion in TypeScript. It also risks changing user build behavior when a build already defines custom jacocoTestReport dependencies, actions or outputs.

Using .exec as the boundary:

• avoids modifying user report tasks;
• reuses the existing JDTLS coverage analyzer;
• removes duplicate XML parsing and source-root matching;
• preserves method coverage;
• may remove the current Gradle 6.1 report-DSL requirement, subject to compatibility testing.

UX and onboarding

P0 should add explicit profiles:

• Delegate Test to Gradle
• Delegate Test to Gradle (Debug)
• Delegate Test to Gradle (Coverage)

Do not automatically retry a failed standard run with Gradle. A failure followed by an unexplained automatic rerun is surprising and makes execution behavior difficult to understand. Users should deliberately select the Gradle profile.

Existing java.test.runTests telemetry includes the selected profile label and can measure opt-in adoption. Reliability telemetry should additionally distinguish BSP execution, task-server fallback, cancellation and coverage-analysis failures.

Phased plan

P0 — Opt-in delegated Coverage

vscode-java-test

• Extend the external TestRunner contract with a run-scoped Coverage artifact location.
• Invoke the internal Coverage provider after a delegated Coverage run completes.
• Allow CoverageHandler to load multiple .exec files.
• Own artifact cleanup and detailed Coverage registration.

vscode-gradle

• Register Delegate Test to Gradle (Coverage).
• Keep BSP-first Gradle Test execution.
• Produce isolated .exec files through the Gradle Test tasks.
• Do not generate XML or interact with user JaCoCo report tasks.

Validation

• Standard single-project Java build.
• Multi-project and multiple Test tasks.
• Existing user JaCoCo configuration.
• Parallel forks and failed tests with partial execution data.
• BSP and task-server fallback.
• BSP/Buildship IJavaProject output locations matching Gradle class outputs.
• Cancellation and cleanup while preventing reads of files still being written.

P1 — Reliability and performance

• Thread BSP originId through test callbacks so runs can be correlated without global serialization.
• Reuse long-lived build-server/Gradle daemon state and avoid unnecessary clean tasks.
• Improve debug handling for multiple Test tasks/forks.
• Validate custom source sets, generated sources and composite builds.

P2 — Evaluate default behavior

Use adoption, reliability, latency and failure telemetry from the explicit profiles to decide whether BSP Gradle projects should use delegation by default. Preserve a clearly named standard-JVM profile as an escape hatch if the default changes.

Current gaps

Gap	Impact
Coverage artifact handoff is not part of the TestRunner contract	Java Test cannot yet automatically analyze delegated execution data
CoverageHandler expects one fixed jacoco.exec	It cannot safely consume per-project/per-task isolated files
BSP callbacks do not include a usable run identifier	Delegated runs must currently be serialized
Complex Gradle source/output models need validation	Coverage may be incomplete for custom source sets or composite builds
First-run Gradle latency	Slower feedback than the standalone JDT launcher for small test loops

Cross-repository work

• vscode-java-test: Coverage artifact contract, multi-exec analysis and controller-owned Coverage lifecycle.
• vscode-gradle: BSP execution and isolated execution-data production.
• build-server-for-gradle: no Coverage-specific endpoint is required for P0; originId propagation remains a useful follow-up for concurrency and routing.

---

This issue tracks the complete delegated test experience motivated by microsoft/vscode-java-test#1880 and #1878. PR #1893 is the initial Coverage implementation and should align with the artifact-based design above.

[github-actions]

Hi wenyt (@wenytang-ms), I'm an AI Support assistant here to help with your issue. While the team reviews your request, I wanted to provide some possible tips and documentation that might help you in the meantime.

Suggestions for next steps:

• Coverage support under Gradle delegate:
  • Check Code coverage report ? vscode-java-test#387 for discussion on adding a Coverage-kind profile and generating coverage files.
  Code coverage report ? vscode-java-test#387 (comment)

• Mapping java.test.config into the Gradle delegate init script:
  • Refer to Improve the test configuration vscode-java-test#1199 which proposes unifying test-configuration schema and passing all fields through to the Debugger.
  Improve the test configuration vscode-java-test#1199

Other references with low confidence

• microsoft/build-server-for-gradle#141 — Tracks adding missing BSP requests (BuildTargetTest, DebugSessionStart, etc.) which may be a cross-repo dependency when enhancing the Gradle build server for test delegation. https://github.com/microsoft/build-server-for-gradle/issues/141

The team will respond to your issue shortly. I hope these suggestions are helpful in the meantime. If this comment helped you, please give it a 👍. If the suggestion was not helpful or incorrect, please give it a 👎. Your feedback helps us improve!

[wenytang-ms]

Delegate Test → Coverage: current status & convergence roadmap

Where P0 stands

Client (vscode-gradle #1893)

• Registers a Coverage-kind run profile; coverage runs are routed to a JaCoCo/XML path.
• Coverage is produced by a task-server Gradle invocation (cleanTest test --tests … jacocoTestReport) with a generated --init-script that applies JaCoCo (0.8.15) and points the jacocoTestReport XML at a per-run temp dir; the client parses that XML back into VS Code coverage objects.
• A failure classifier (escalation.ts) exists to detect delegation-fixable failures (module-access / stale-resource signatures).

Server (build-server-for-gradle #226)

• Adds the BSP-standard buildTarget/jvmTestEnvironment / buildTarget/jvmRunEnvironment requests (+ capabilities), returning a faithful JvmEnvironmentItem (runtime classpath, test JVM options, working directory).

Known gaps vs this design

1. Coverage bypasses the BSP execution path. Normal delegate runs go through BSP buildTarget/test (Tooling API TestLauncher, streamed per-test events). Coverage runs do not — they shell out a plain gradle build via the task server and reconstruct results from JUnit XML. This two-path divergence loses TestLauncher''s structured streaming and re-introduces the per-run cleanTest cost that P1 wants to remove on the BSP path.
2. Escalation is not wired. The P0 "Retry with Gradle" action is currently inert — the classifier exists but the one-click retry isn''t surfaced/invoked.
3. jvmTestEnvironment has no consumer yet. Bump @types/node from 13.7.1 to 13.9.1 #226 exposes the capability, but no client path calls it; the coverage impl took the script route instead.

Why coverage can''t just "go through BSP" today

The root constraint is which Tooling API launcher the server uses:

• TestLauncher (used by BSP buildTarget/test): purpose-built for test selection + structured per-test events; it has no forTasks, so it cannot run jacocoTestReport.
• BuildLauncher.forTasks(...): runs arbitrary tasks (test, jacocoTestReport) — which is why the task-server coverage path can produce a report, at the cost of test-result fidelity.

Coverage inherently needs a report task that TestLauncher won''t run, so any BSP-native solution must bridge the two launchers.

Three convergence options

A. Client-orchestrated (no protocol change). Client runs coverage in two phases: (1) BSP buildTarget/test with a JaCoCo --init-script → faithful streamed test execution + .exec; (2) task-server jacocoTestReport (report-only, no tests) → XML. Small change; client still shells out gradle for phase 2.

B. BSP-native coverage via a data-kind extension (protocol change; we own the protocol). Extend TestParams with a custom dataKind (gradle-test-with-coverage) + a negotiated coverageProvider capability. Keep the server tool-agnostic: the client passes { initScript, reportTasks, reportOutputDir }; the server runs the TestLauncher test phase (init-script → .exec + streamed events) then a BuildLauncher report-only phase, returning the report URI via TestResult.data. JaCoCo specifics stay client-side. Removes the client''s direct gradle call and fixes gap #1, at the cost of protocol/versioning work.

C. Client-launched JVM via jvmTestEnvironment (#226). Client queries jvmTestEnvironment for classpath + JVM args + working dir, then launches its own java -javaagent:jacocoagent.jar …. Most flexible, sidesteps the report-task problem entirely, but the client must build the launch command (main class / JUnit runner) and loses BSP''s streamed test events. This is the model #226 currently enables.

Recommendation

• Near term (this P0): land coverage as-is (script path) to close the user-facing gap, but finish wiring the escalation action (gap Tests #2) — it''s an explicit P0 deliverable — and add a code comment documenting the coverage/BSP divergence.
• Target state: converge on B (BSP-native, tool-agnostic gradle-test-with-coverage data-kind): removes the two-path divergence, keeps execution on the faithful TestLauncher path, and aligns with P1''s "avoid redundant cleanTest on the BSP path". Ship as a follow-up server+client pair behind capability negotiation, with the current script path as the fallback for older servers.
• Keep Bump @types/node from 13.7.1 to 13.9.1 #226 / option C as an independent capability — useful on its own (run/debug environments), but not required for coverage if B lands.