CI: Select Spark PR matrix by changed version #16946
Open
ajantha-bhat wants to merge 2 commits into
Open
Conversation
ajantha-bhat
force-pushed
the
codex/ci-spark-pr-version-matrix
branch
2 times, most recently
from
d07a80c to
350a854
Compare
Generated-by: GPT-5 Codex
Generated-by: GPT-5 Codex
ajantha-bhat
force-pushed
the
codex/ci-spark-pr-version-matrix
branch
from
350a854 to
29459b7
Compare
manuzhang
reviewed
Jun 25, 2026
| cancel-in-progress: ${{ github.event_name == 'pull_request' }} | ||
|
|
||
| jobs: | ||
|
|
manuzhang
reviewed
Jun 25, 2026
| cache-read-only: true | ||
| - run: echo -e "$(ip addr show eth0 | grep "inet\b" | awk '{print $2}' | cut -d/ -f1)\t$(hostname -f) $(hostname -s)" | sudo tee -a /etc/hosts | ||
| - run: ./gradlew -DsparkVersions= -DkafkaVersions= -DflinkVersions=${{ matrix.flink }} :iceberg-flink:iceberg-flink-${{ matrix.flink }}:check :iceberg-flink:iceberg-flink-runtime-${{ matrix.flink }}:check -Pquick=true -x javadoc -DtestParallelism=auto | ||
| - env: |
Member
There was a problem hiding this comment.
do we need to change flink ci in this PR
Member
Author
There was a problem hiding this comment.
It is from the dependent PR as mentioned in the description. It will go away after rebase.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
depends on #16945
Summary
This updates Spark CI to build an incremental PR matrix based on the changed Spark version path.
For ordinary PRs, Spark CI uses the JVM policy from the Java CI PR and narrows the Spark matrix when changes are scoped to a known Spark version path:
spark/v3.5/**runs Spark 3.5 onlyspark/v4.0/**runs Spark 4.0 onlyspark/v4.1/**runs Spark 4.1 onlyfull-ci, and push/main/release/tag runs fall back to the full Spark version matrixKnown Spark versions are read from
knownSparkVersionsingradle.properties, so adding/removing Spark versions does not require hard-coding the version list in the planner.This keeps the PR path conservative: stale or unknown planner inputs run more Spark CI, not less.
Testing
actionlint .github/workflows/spark-ci.yml .github/workflows/java-ci.ymlzizmor --min-severity medium --min-confidence medium --no-progress .github/workflows/spark-ci.yml .github/workflows/java-ci.ymlbash -n .github/scripts/ci-plan-common.sh .github/scripts/plan-spark-ci.shgit diff --check codex/ci-java-pr-jdk17..HEADspark/v4.1/**-> Spark 4.1 only on the regular PR JVM matrixfull-ci-> all known Spark versions with the full JVM matrixAI Disclosure