fix: remove pre-installed Rust stable toolchain to stabilize CI cache keys#4511
Merged
fix: remove pre-installed Rust stable toolchain to stabilize CI cache keys#4511
Conversation
… keys The macOS runner fleet has a mix of runner image versions with different pre-installed Rust stable versions (e.g. 1.93.1 vs 1.94.0). Since Swatinem/rust-cache hashes all installed toolchains into the cache environment key, the hash becomes non-deterministic and causes constant cache misses on macOS. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The `icx-asset ls` command used `slog_async::Async` with a default channel capacity of 128. When listing >128 assets, the process could exit before the background logging thread drained all messages, causing flaky test failures where only 128 of 151 assets appeared in output. Use `build_with_guard()` so the AsyncGuard joins the background thread on drop, ensuring all log lines are flushed before the process exits. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The previous flush-on-exit fix (AsyncGuard) was necessary but not sufficient. slog_async defaults to OverflowStrategy::DropAndReport, which silently drops messages when the 128-capacity channel is full. The `icx-asset ls` command fires 151 info!() calls in a tight loop, so messages 129+ were dropped before entering the channel — no amount of flushing can recover them. Switch to OverflowStrategy::Block so the sender waits when the channel is full, guaranteeing all log lines are delivered. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
viviveevee
approved these changes
Mar 27, 2026
lwshang
added a commit
to dfinity/cdk-rs
that referenced
this pull request
Mar 27, 2026
GitHub runners ship with varying stable Rust versions that pollute the rust-cache environment hash, causing non-deterministic cache keys and constant cache misses. Remove the pre-installed stable toolchain before setup-rust-toolchain in all jobs that use caching. See also: dfinity/sdk#4511 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
lwshang
added a commit
to dfinity/cdk-rs
that referenced
this pull request
Mar 30, 2026
…-types (#704) * fix(ci): remove pre-installed stable toolchain to fix rust-cache misses GitHub runners ship with varying stable Rust versions that pollute the rust-cache environment hash, causing non-deterministic cache keys and constant cache misses. Remove the pre-installed stable toolchain before setup-rust-toolchain in all jobs that use caching. See also: dfinity/sdk#4511 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore(ci): bump actions/checkout from v4 to v6 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: remove ic-management-canister-types from workspace The crate has been migrated to the dfinity/ic repo. Use it as an external crates.io dependency instead of a local workspace member. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * ci: add publish workflow for trusted publishing to crates.io Uses OIDC-based authentication via rust-lang/crates-io-auth-action. Each crate has a toggle input, ordered by dependency graph. ic-cdk and ic-cdk-macros share a single toggle and are published together. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: fmt --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
stabletoolchain beforesetup-rust-toolchainruns in all cached CI workflows (unit, lint, e2e, release)icx-asset lsdropping output beyond 128 assets by blocking on a full async log channel instead of silently dropping messagesWhy
CI cache instability
The
Swatinem/rust-cacheaction (used bysetup-rust-toolchain) hashes all installed Rust toolchains into the cache environment key. GitHub's runner fleet — especially macOS — has a mix of runner image versions where different VMs ship with different pre-installed Rust stable versions (e.g.1.93.1vs1.94.0). Which VM a job gets is random, so the environment hash becomes non-deterministic:5b4abfefa57e084d35ab6fff35ab6fffUbuntu happened to be consistent across VMs, but this is fragile and could break on the next runner image rollout.
By removing the
stabletoolchain before the cache key is computed, only the project toolchain (1.88.0fromrust-toolchain.toml) remains, making the environment hash deterministic regardless of runner image version.icx-asset ls truncated output
The
icx-asset lscommand logs each asset viaslog_async::Async, which has a default channel capacity of 128. Two issues combined to truncate output:OverflowStrategy::DropAndReportsilently drops messages when the channel is full. With 151 assets logged in a tight loop, messages 129+ were dropped before ever entering the channel.Fixed by switching to
OverflowStrategy::Block(sender waits when channel is full) and usingbuild_with_guard()(joins the background thread on drop to flush remaining messages).Test plan
🤖 Generated with Claude Code