From 953db5284c51acc6009c80851298b58d1c2780ce Mon Sep 17 00:00:00 2001 From: Olya Gupalo Date: Mon, 16 Feb 2026 18:07:15 +0200 Subject: [PATCH] Add Testing GraalVM: Understanding CI and Running Tests Locally doc --- ci/ci_common/run-spec.libsonnet | 2 +- ci/run-spec.md | 18 ++++ docs/ci/run-spec.md | 20 ----- vm/TESTING.md | 149 ++++++++++++++++++++++++++++++++ 4 files changed, 168 insertions(+), 21 deletions(-) create mode 100644 ci/run-spec.md delete mode 100644 docs/ci/run-spec.md create mode 100644 vm/TESTING.md diff --git a/ci/ci_common/run-spec.libsonnet b/ci/ci_common/run-spec.libsonnet index d75288d1d3ca..6d39edc3d5ed 100644 --- a/ci/ci_common/run-spec.libsonnet +++ b/ci/ci_common/run-spec.libsonnet @@ -1,6 +1,6 @@ // run-spec: a library for declaratively describing CI jobs // -// See `docs/ci/run-spec.md` +// See `ci/run-spec.md` // local _impl = import "run-spec-impl.libsonnet"; local examples = (import "run-spec-examples.libsonnet").examples; diff --git a/ci/run-spec.md b/ci/run-spec.md new file mode 100644 index 000000000000..83f34006c6f6 --- /dev/null +++ b/ci/run-spec.md @@ -0,0 +1,18 @@ +# CI Run Specification + +The CI run specification is a Jsonnet library used to declaratively define on which _platform_ (OS/architecture/JDK) and at which _frequency_ (gate/daily/weekly/etc.) a job should run. +The main idea is to have a single place that describes _what a task does_ (for example, running `mx unittest`), while keeping it logically decoupled from +_where_ and _when_ it runs. + +The entry point of the library is [run-spec.jsonnet](../../ci/ci_common/run-spec.jsonnet). +Examples and further details can be found in [run-spec-demo.jsonnet](../../ci/ci_common/run-spec-demo.jsonnet) and [run-spec-examples.jsonnet](../../ci/ci_common/run-spec-examples.jsonnet). + +## Terminology + +* A `platform` is defined by an operating system, an architecture, and a JDK version. +* A `task` defines something to execute, independent of the platform. +* `variants` of a task generate multiple tasks based on small user modifications. +* `batches` of a task split a large task into N smaller tasks. +* A `frequency` defines how often a task must be executed: `gate`, `postmerge`, `daily`, `weekly`, `monthly`, `ondemand`. +* A `job` is a runnable CI unit created by combining a task, a platform, and a frequency. +* A `build` is the result of executing one or more jobs within a CI system. Each build is identified by a unique number. \ No newline at end of file diff --git a/docs/ci/run-spec.md b/docs/ci/run-spec.md deleted file mode 100644 index 230ed93f7527..000000000000 --- a/docs/ci/run-spec.md +++ /dev/null @@ -1,20 +0,0 @@ -# CI Run Specification - -The CI run specification is a jsonnet library to declaratively express on which _platform_ (os/arch/jdk) -and at which _frequency_ (gate/daily/weekly/etc) to run a certain job. -The main idea is to have a single point that describes _what a task does_ (e.g. run `mx unittest`) that is decoupled (logically but not spatially) from -_where_ and _when_ to execute it. - -The entry point of the library is [`run-spec.jsonnet`](../../ci/ci_common/run-spec.jsonnet). -Examples and further details can be found in [`run-spec-demo.jsonnet`](../../ci/ci_common/run-spec-demo.jsonnet) -and [`run-spec-examples.jsonnet`](../../ci/ci_common/run-spec-examples.jsonnet). - -## Terminology - -* A `platform` is defined by an os, an architecture and a jdk. -* A `task` is the definition of something we want to execute without consideration of the platform. -* `variants` of task generate a set of tasks based on user slight alterations of a task. -* `batches` of a task splits a large task into N smaller tasks. -* A `frequency` defines how often a task must be executed: `gate`, `postmerge`, `daily`, `weekly`, `monthly`, `ondemand`. -* A `job` is the complete definition of a runnable CI job. It is defined by a task, a platform and a frequency. -* A `build` is the result of job execution within a specific CI system. It is identified with a unique number. diff --git a/vm/TESTING.md b/vm/TESTING.md new file mode 100644 index 000000000000..45be747092f9 --- /dev/null +++ b/vm/TESTING.md @@ -0,0 +1,149 @@ +# Testing GraalVM: Understanding CI and Running Tests Locally + +GraalVM is tested across a defined set of platforms and configurations using Oracle internal infrastructure and GitHub Actions. +**GitHub Actions test the Community Edition (CE) parts of GraalVM** on the following platforms: `linux-amd64`, `linux-aarch64`, `darwin-aarch64` (macOS). + +Tests are executed using `mx`, the build and project management tool [available on GitHub](https://github.com/graalvm/mx). + +This document explains where those tests are defined, how contributors can trigger CI workflows on GitHub, and reproduce testing locally. +Test failures help catch issues early in the development cycle. + +## CI Workflows + +GraalVM CE is tested by running various GitHub Actions workflows. +All workflows are defined in [.github/workflows](https://github.com/oracle/graal/tree/master/.github/workflows). +You can view and trigger them from the **Actions** tab in the repository. + +Several workflows exist, but two are particularly relevant for contributors. + +### Gate Workflow + +**GraalVM Gate** is the primary workflow defined in [**main.yml**](https://github.com/oracle/graal/blob/master/.github/workflows/main.yml). +This workflow runs on pushes and pull requests and includes the steps that build GraalVM and execute tests. + +At a high level, the workflow performs the following steps: +- Checking out the repository +- Setting up the required JDK +- Building the project +- Running the full test suite +- Uploading logs + +The `mx` commands executed by the workflow determine what is actually built and tested. +In most cases, the workflow runs a command equivalent to: +```bash +mx gate --strict-mode --tags +``` +with a specific primary suite selected, for example: +```bash +mx --primary-suite-path compiler gate --strict-mode --tags style,fullbuild,test +``` + +This means CI is running the gate pipeline for a particular GraalVM component (such as the compiler), which typically includes building the component, running unit tests, and performing additional validation checks. + +The parameters passed to `mx` are derived from environment variables defined in the [workflow matrix](https://github.com/oracle/graal/blob/master/.github/workflows/main.yml#L99). For example: + +```yml +# /compiler +- env: + JDK_VERSION: "latest" + TOOLS_JDK_VERSION: "21" + GATE_TAGS: "style,fullbuild,test" + PRIMARY: "compiler" +``` +These variables configure `mx gate` to run using the compiler suite as the primary suite, on the latest JDK, and with specific gate tags. + +The exact configuration of the workflow (platforms, JDK versions, and matrix parameters) may change over time. + +### Build Workflow + +The **Build GraalVM** workflow is defined in [**build-graalvm.yml**](https://github.com/oracle/graal/blob/master/.github/workflows/build-graalvm.yml) and builds GraalVM CE from source. +It can be triggered on demand: under the **Actions** tab, select the **Build GraalVM** pipeline, expand the **Run workflow** menu, and confirm the action. + +This workflow provides three fields for passing custom build arguments to `mx` (comma-separated, no spaces): + +- Value for `mx --dy` option + + With `--dy` you can dynamically import more static dependencies (suites) to the base GraalVM CE package. + Typical examples are `/substratevm`, `/compiler`, `/truffle`. + Dynamic imports are declared in _ci/common.jsonnet_. + Build dependencies of each component are specified in _mx.\/suite.py_ of the corresponding repository. + +- Value for `mx --native-images` option + + This option specifies which native images to build as part of the run. Search the repository for where native images are defined. + Typical examples are `native-image`, `native-image-configure`, `lib:native-image-agent`, `lib:jvmcicompiler`. + +- Value for `mx --components` option + + This option defines which GraalVM components are included in a build. + You can list all components with `mx graalvm-components`. + You can also inspect component definitions in the _sdk/mx.sdk/suite.py_ and _vm/mx.vm/suite.py_ files. + Typical examples are `Native Image`, `Native Image Configure Tool`, `LibGraal`. + +If you do not provide any other values, this workflow builds the base distribution. +When the build succeeds, the workflow attaches the binary as an artifact, named _graalvm.zip_. + +## Troubleshooting a Test Failure + +If a test fails, the most reliable way to understand or reproduce it is: + +1. Open the workflow run in GitHub Actions. +2. Locate the step that runs the gate command. +3. Check the logged `mx` command to see the exact parameters used. + +When you open a particular pipeline run, you can trace any step in the logs back to the exact definition in the workflow file. + +## Local Test Environment + +You can run the same core test commands locally that are used in GraalVM CI. +Start by preparing your local environment. + +1. Ensure your development environment meets the prerequisites: [prepare the environment](https://github.com/oracle/graal/blob/master/vm/README.md#prepare-the-environment). +2. Install `mx` as described [here](README.md#setting-mx). +3. Clone your fork of the Graal repository and enter the component directory you want to test (for example, `compiler`, `substratevm`, or `truffle`). +4. Make sure you are using a JDK version supported by CI (for example, `jdk21`, `jdk25`, or `latest`). +5. Run tests using `mx`. For example: + ```bash + mx unittest + ``` + +This command runs the JUnit-based tests for the current suite (component). + +If you want to test a full GraalVM build similar to CI artifacts, you can also run the [Build GraalVM workflow](#build-workflow) in GitHub Actions and download the produced build artifact. + +## Reproducing a CI Test Locally + +When a test fails in CI, the most reliable way to reproduce it is to run the same `mx gate` command locally. + +1. Open the failed workflow run in GitHub Actions. +2. Locate the step that runs the gate command. +3. Copy the exact `mx` command from the logs. + +A typical example looks like this: +```bash +mx --primary-suite-path compiler gate --strict-mode --tags style,fullbuild,test +``` +Running this command locally executes the same gate tasks that CI runs for that suite. + +## Local Testing Configuration + +When running locally, you can limit testing to specific components using tags or suite settings. +The following environment variables values translate directly into `mx` command arguments: + +- `GATE_TAGS` defines which gate tasks will run. Common tags include: `style`, `build`, `fullbuild`, and `test`. More tags may exist depending on the component (suite). Tags are defined in the component `mx` suite file: _mx.\/suite.py_. +- `PRIMARY` defines which suite is used as the primary suite for the gate run. Common suites include: `compiler`, `substratevm`, `truffle`, and `vm`. + +You can inspect all tasks and their associated tags by running: +```bash +mx -v gate --dry-run +``` + +If you are reproducing a failure, check the workflow logs to see which platform, JDK version, environment variables, and gate tags were used in that run. +Matching those settings locally usually reproduces CI behavior more closely. + +## Related Documentation + +- Learn more about suites in the [`mx` project documentation](https://github.com/graalvm/mx/blob/master/README.md). +- See also how to [build GraalVM Community Edition from source](vm/README.md) manually. + +We hope this document will help users test changes in a setup that mirrors GraalVM CI, thus improving the quality of external contributions. \ No newline at end of file