Skip to content

Add caching for the Mill build tool #819

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
90 changes: 90 additions & 0 deletions .github/workflows/e2e-cache.yml
Original file line number Diff line number Diff line change
Expand Up @@ -207,3 +207,93 @@ jobs:
exit 1
fi
ls ~/.cache/coursier
mill-save:
runs-on: ${{ matrix.os }}
defaults:
run:
shell: bash
working-directory: __tests__/cache/mill
strategy:
fail-fast: false
matrix:
os: [macos-13, windows-latest, ubuntu-22.04]
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Run setup-java with the cache for mill
uses: ./
id: setup-java
with:
distribution: 'adopt'
java-version: '11'
cache: mill
- name: Create files to cache
run: ./mill --disable-ticker _.compile
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_.compile is probably wrong (should be __.compile), but we should make this warmup command somethign the user passes in. Some users may want to cache jars, assemblies, or other artifacts beyond just plain compilation

Copy link
Author

@clintval clintval May 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for reviewing @lihaoyi!

_.compile should be OK here since this is just an integration for a tiny test Scala project:

package build
import mill._, scalalib._
object MyProject extends ScalaModule {
def scalaVersion = "2.13.11"
def ivyDeps = Agg(ivy"com.lihaoyi::mainargs:0.6.2")
object test extends ScalaTests {
def ivyDeps = Agg(ivy"com.lihaoyi::utest:0.8.5")
def testFramework = "utest.runner.Framework"
}
}

I could change it to __.compile for safety-sake anyway, though.

And this isn't where the magic happens since these are just tests for this project.

The default caching behavior is set to:

setup-java/src/cache.ts

Lines 64 to 78 in 18d114c

{
id: 'mill',
path: [
join(os.homedir(), '.cache', 'mill')
],
pattern: [
// https://github.com/coursier/cache-action/blob/4e2615869d13561d626ed48655e1a39e5b192b3c/README.md?plain=1#L28-L38
'**/build.sc',
'**/*.sc',
'**/mill',
'**/.mill-version',
// https://github.com/com-lihaoyi/mill/blob/5b88d1e268e6264e44589c5ac82c0fdbd680fd63/mill#L6-L11
'**/.config/mill-version'
]
}

So that anything under ~/.cache/mill is cached.

The GitHub Action cache is rebuilt if anything in these patterns changes:

pattern: [
  // https://github.com/coursier/cache-action/blob/4e2615869d13561d626ed48655e1a39e5b192b3c/README.md?plain=1#L28-L38
  '**/build.sc',
  '**/*.sc',
  '**/mill',
  '**/.mill-version',
  // https://github.com/com-lihaoyi/mill/blob/5b88d1e268e6264e44589c5ac82c0fdbd680fd63/mill#L6-L11
  '**/.config/mill-version'
]


- name: Check files to cache on macos-latest
if: matrix.os == 'macos-13'
run: |
if [ ! -d ~/.cache/mill/download ]; then
echo "::error::The ~/.cache/mill/download directory does not exist unexpectedly"
exit 1
fi
- name: Check files to cache on windows-latest
if: matrix.os == 'windows-latest'
run: |
if [ ! -d %USERPROFILE%/.cache/mill/download ]; then
echo "::error::The %USERPROFILE%/.cache/mill/download directory does not exist unexpectedly"
exit 1
fi
- name: Check files to cache on ubuntu-latest
if: matrix.os == 'ubuntu-latest'
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I copied these patterns from above and notice a potential bug.

The matrix defines ubuntu-22.04 but the conditional here is still set to ubuntu-latest. I imagine this was not intentional. Is there a reason to pin to 22.04? If not, I'd vote reverting back to ubuntu-latest all over.

run: |
if [ ! -d ~/.cache/mill/download ]; then
echo "::error::The ~/.cache/mill/download directory does not exist unexpectedly"
exit 1
fi
mill-restore:
runs-on: ${{ matrix.os }}
defaults:
run:
shell: bash
working-directory: __tests__/cache/mill
strategy:
fail-fast: false
matrix:
os: [macos-13, windows-latest, ubuntu-22.04]
needs: mill-save
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Run setup-java with the cache for mill
uses: ./
id: setup-java
with:
distribution: 'adopt'
java-version: '11'
cache: mill

- name: Confirm that ~/.cache/mill/download directory has been made
if: matrix.os == 'macos-13'
run: |
if [ ! -d ~/.cache/mill/download ]; then
echo "::error::The ~/.cache/mill/download directory does not exist unexpectedly"
exit 1
fi
ls ~/.cache/mill/download
- name: Confirm that %USERPROFILE%/.cache/mill/download directory has been made
if: matrix.os == 'windows-latest'
run: |
if [ ! -d %USERPROFILE%/.cache/mill/download ]; then
echo "::error::The %USERPROFILE%/.cache/mill/download directory does not exist unexpectedly"
exit 1
fi
ls %USERPROFILE%/.cache/mill/download
- name: Confirm that ~/.cache/mill/download directory has been made
if: matrix.os == 'ubuntu-latest'
run: |
if [ ! -d ~/.cache/mill/download ]; then
echo "::error::The ~/.cache/mill/download directory does not exist unexpectedly"
exit 1
fi
ls ~/.cache/mill/download
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# Ignore Scala IDE files
.metals/

# Ignore node_modules, ncc is used to compile nodejs modules into a single file
node_modules/
__tests__/runner/*
Expand Down
29 changes: 23 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ The `setup-java` action provides the following functionality for GitHub Actions
- Caching dependencies managed by Apache Maven.
- Caching dependencies managed by Gradle.
- Caching dependencies managed by sbt.
- Caching dependencies managed by Mill.
- [Maven Toolchains declaration](https://maven.apache.org/guides/mini/guide-using-toolchains.html) for specified JDK versions.

This action allows you to work with Java and Scala projects.
Expand All @@ -39,7 +40,7 @@ This action allows you to work with Java and Scala projects.

- `check-latest`: Setting this option makes the action to check for the latest available version for the version spec.

- `cache`: Quick [setup caching](#caching-packages-dependencies) for the dependencies managed through one of the predefined package managers. It can be one of "maven", "gradle" or "sbt".
- `cache`: Quick [setup caching](#caching-packages-dependencies) for the dependencies managed through one of the predefined package managers. It can be one of "maven", "gradle", "sbt", or "mill".

- `cache-dependency-path`: The path to a dependency file: pom.xml, build.gradle, build.sbt, etc. This option can be used with the `cache` option. If this option is omitted, the action searches for the dependency file in the entire repository. This option supports wildcards and a list of file names for caching multiple dependencies.

Expand Down Expand Up @@ -121,19 +122,20 @@ Currently, the following distributions are supported:
**NOTE:** To comply with the GraalVM Free Terms and Conditions (GFTC) license, it is recommended to use GraalVM JDK 17 version 17.0.12, as this is the only version of GraalVM JDK 17 available under the GFTC license. Additionally, it is encouraged to consider upgrading to GraalVM JDK 21, which offers the latest features and improvements.

### Caching packages dependencies
The action has a built-in functionality for caching and restoring dependencies. It uses [toolkit/cache](https://github.com/actions/toolkit/tree/main/packages/cache) under hood for caching dependencies but requires less configuration settings. Supported package managers are gradle, maven and sbt. The format of the used cache key is `setup-java-${{ platform }}-${{ packageManager }}-${{ fileHash }}`, where the hash is based on the following files:
The action has a built-in functionality for caching and restoring dependencies. It uses [toolkit/cache](https://github.com/actions/toolkit/tree/main/packages/cache) under hood for caching dependencies but requires less configuration settings. Supported package managers are Gradle, Maven, sbt, and Mill. The format of the used cache key is `setup-java-${{ platform }}-${{ packageManager }}-${{ fileHash }}`, where the hash is based on the following files:

- gradle: `**/*.gradle*`, `**/gradle-wrapper.properties`, `buildSrc/**/Versions.kt`, `buildSrc/**/Dependencies.kt`, `gradle/*.versions.toml`, and `**/versions.properties`
- maven: `**/pom.xml`
- Gradle: `**/*.gradle*`, `**/gradle-wrapper.properties`, `buildSrc/**/Versions.kt`, `buildSrc/**/Dependencies.kt`, `gradle/*.versions.toml`, and `**/versions.properties`
- Maven: `**/pom.xml`
- sbt: all sbt build definition files `**/*.sbt`, `**/project/build.properties`, `**/project/**.scala`, `**/project/**.sbt`
- Mill: `**/build.sc`, `**/*.sc`, `**/mill`, `**/.mill-version`, and `**/.config/mill-version`

When the option `cache-dependency-path` is specified, the hash is based on the matching file. This option supports wildcards and a list of file names, and is especially useful for monorepos.

The workflow output `cache-hit` is set to indicate if an exact match was found for the key [as actions/cache does](https://github.com/actions/cache/tree/main#outputs).

The cache input is optional, and caching is turned off by default.

#### Caching gradle dependencies
#### Caching Gradle dependencies
```yaml
steps:
- uses: actions/checkout@v4
Expand All @@ -148,7 +150,7 @@ steps:
- run: ./gradlew build --no-daemon
```

#### Caching maven dependencies
#### Caching Maven dependencies
```yaml
steps:
- uses: actions/checkout@v4
Expand Down Expand Up @@ -178,6 +180,21 @@ steps:
run: sbt package
```

#### Caching Mill dependencies
```yaml
steps:
- uses: actions/checkout@v4
- uses: actions/setup-java@v4
with:
distribution: 'temurin'
java-version: '21'
cache: 'mill'
cache-dependency-path: | # optional
sub-project/build.sc
- name: Build with Mill
run: ./mill _.compile
```

#### Cache segment restore timeout
Usually, cache gets downloaded in multiple segments of fixed sizes. Sometimes, a segment download gets stuck, which causes the workflow job to be stuck. The cache segment download timeout [was introduced](https://github.com/actions/toolkit/tree/main/packages/cache#cache-segment-restore-timeout) to solve this issue as it allows the segment download to get aborted and hence allows the job to proceed with a cache miss. The default value of the cache segment download timeout is set to 10 minutes and can be customized by specifying an environment variable named `SEGMENT_DOWNLOAD_TIMEOUT_MINS` with a timeout value in minutes.

Expand Down
42 changes: 42 additions & 0 deletions __tests__/cache.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -201,6 +201,48 @@ describe('dependency cache', () => {
expect(firstCall).not.toBe(thirdCall);
});
});
describe('for mill', () => {
it('throws error if no build.sc found', async () => {
await expect(restore('mill', '')).rejects.toThrow(
`No file in ${projectRoot(
workspace
)} matched to [**/build.sc,**/*.sc,**/mill,**/.mill-version,**/.config/mill-version], make sure you have checked out the target repository`
);
});
it('downloads cache', async () => {
createFile(join(workspace, 'build.sc'));

await restore('mill', '');
expect(spyCacheRestore).toHaveBeenCalled();
expect(spyGlobHashFiles).toHaveBeenCalledWith(
'**/build.sc\n**/*.sc\n**/mill\n**/.mill-version\n**/.config/mill-version'
);
expect(spyWarning).not.toHaveBeenCalled();
expect(spyInfo).toHaveBeenCalledWith('mill cache is not found');
});
it('detects scala and mill changes under **/mill-build/ folder', async () => {
createFile(join(workspace, 'build.sc'));
createDirectory(join(workspace, 'project'));
createFile(join(workspace, '.config/mill-version'));

await restore('mill', '');
const firstCall = spySaveState.mock.calls.toString();

spySaveState.mockClear();
await restore('mill', '');
const secondCall = spySaveState.mock.calls.toString();

// Make sure multiple restores produce the same cache
expect(firstCall).toBe(secondCall);

spySaveState.mockClear();
createFile(join(workspace, '.mill-version'));
await restore('mill', '');
const thirdCall = spySaveState.mock.calls.toString();

expect(firstCall).not.toBe(thirdCall);
});
});
it('downloads cache based on versions.properties', async () => {
createFile(join(workspace, 'versions.properties'));

Expand Down
1 change: 1 addition & 0 deletions __tests__/cache/mill/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
out/
1 change: 1 addition & 0 deletions __tests__/cache/mill/.mill-version
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0.12.3
12 changes: 12 additions & 0 deletions __tests__/cache/mill/build.sc
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
package build
import mill._, scalalib._

object MyProject extends ScalaModule {
def scalaVersion = "2.13.11"
def ivyDeps = Agg(ivy"com.lihaoyi::mainargs:0.6.2")

object test extends ScalaTests {
def ivyDeps = Agg(ivy"com.lihaoyi::utest:0.8.5")
def testFramework = "utest.runner.Framework"
}
}
26 changes: 26 additions & 0 deletions __tests__/cache/mill/mill
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
#!/usr/bin/env sh
# This is a wrapper script that automatically downloads Mill from GitHub.
set -e

if [ -z "$MILL_VERSION" ] ; then
MILL_VERSION="$(head -n 1 .mill-version 2> /dev/null)"
fi

MILL_DOWNLOAD_PATH="$HOME/.cache/mill/download"
MILL_EXEC_PATH="${MILL_DOWNLOAD_PATH}/$MILL_VERSION"

if [ ! -x "$MILL_EXEC_PATH" ] ; then
mkdir -p "${MILL_DOWNLOAD_PATH}"
DOWNLOAD_FILE=$MILL_EXEC_PATH-tmp-download
MILL_DOWNLOAD_URL="https://github.com/lihaoyi/mill/releases/download/${MILL_VERSION%%-*}/$MILL_VERSION-assembly"
curl --fail -L -o "$DOWNLOAD_FILE" "$MILL_DOWNLOAD_URL"
chmod +x "$DOWNLOAD_FILE"
mv "$DOWNLOAD_FILE" "$MILL_EXEC_PATH"
unset DOWNLOAD_FILE
unset MILL_DOWNLOAD_URL
fi

unset MILL_DOWNLOAD_PATH
unset MILL_VERSION

exec "${MILL_EXEC_PATH}" "$@"
17 changes: 16 additions & 1 deletion src/cache.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ const CACHE_MATCHED_KEY = 'cache-matched-key';
const CACHE_KEY_PREFIX = 'setup-java';

interface PackageManager {
id: 'maven' | 'gradle' | 'sbt';
id: 'maven' | 'gradle' | 'sbt' | 'mill';
/**
* Paths of the file that specify the files to cache.
*/
Expand Down Expand Up @@ -60,6 +60,21 @@ const supportedPackageManager: PackageManager[] = [
'**/project/**.scala',
'**/project/**.sbt'
]
},
{
id: 'mill',
path: [
join(os.homedir(), '.cache', 'mill')
],
pattern: [
// https://github.com/coursier/cache-action/blob/4e2615869d13561d626ed48655e1a39e5b192b3c/README.md?plain=1#L28-L38
'**/build.sc',
'**/*.sc',
'**/mill',
'**/.mill-version',
// https://github.com/com-lihaoyi/mill/blob/5b88d1e268e6264e44589c5ac82c0fdbd680fd63/mill#L6-L11
'**/.config/mill-version'
]
}
];

Expand Down