From cdffbfbc7b5f9485f68b1a9fffa2b99f7fe7e5f2 Mon Sep 17 00:00:00 2001 From: Sovereign Map Test Suite Date: Sat, 1 Aug 2026 06:28:06 -0700 Subject: [PATCH 1/2] Phase 8: cargo-hack, cargo-minimal-versions, and a real no_std check Closes gaps identified from an external CI/tooling review: feature-powerset testing per crate, minimal-dependency-version testing, and doctest coverage were genuinely missing (the review's other suggestions either duplicated existing decisions - cargo-mutants, cargo-semver-checks - or had low value given the workspace's unsafe_code = "forbid" policy - Miri, sanitizers). - New `hack` CI job: cargo-hack feature-powerset testing per crate, since the `test` job's one fixed feature combination can miss interaction bugs. - New `minimal-versions` CI job: tests against the lowest dependency versions each Cargo.toml constraint allows, catching version bounds that are looser than what the code actually needs. - `core-tests`'s `no_std_support` module's "usable under `#![no_std]`" claim is now compiler-enforced via tests/no_std_check.rs, a genuinely `#![no_std]`-marked integration test, instead of true only by convention/review. - `coverage` job now passes `--doctests` to cargo-llvm-cov so `///` examples count toward the report; moved that job to nightly since doctest coverage needs rustdoc's unstable --persist-doctests. Co-Authored-By: Claude Sonnet 5 --- .github/workflows/ci.yml | 63 +++++++++++++++++++++++-- CHANGELOG.md | 40 ++++++++++++++++ README.md | 25 ++++++---- ci/README.md | 38 ++++++++++++--- crates/core-tests/Cargo.toml | 6 +++ crates/core-tests/src/lib.rs | 25 +++------- crates/core-tests/src/no_std_support.rs | 17 +++++++ crates/core-tests/tests/no_std_check.rs | 24 ++++++++++ justfile | 9 ++++ scripts/coverage.sh | 16 ++++--- 10 files changed, 219 insertions(+), 44 deletions(-) create mode 100644 crates/core-tests/src/no_std_support.rs create mode 100644 crates/core-tests/tests/no_std_check.rs diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 59a8949..7fec0cc 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -146,17 +146,25 @@ jobs: # produce coverage-chasing busywork more than better tests; this is # meant to make gaps visible; instead see the "Coverage, Reporting, and # Debugging" section of README.md for the same thing locally. + # + # Nightly, not stable: `--doctests` (below) instruments doc-tests too — + # without it, every `///` example in the crates (there are several, e.g. + # `core-tests`'s `UserFixture`) is invisible to the coverage report, an + # easy-to-miss gap since nothing about a clean report tells you doctests + # were excluded. cargo-llvm-cov implements doc-test coverage via + # rustdoc's unstable `--persist-doctests`/`-Z unstable-options`, which + # only the nightly compiler accepts. runs-on: ubuntu-latest steps: - uses: actions/checkout@v7 - - name: Install Rust (stable) - uses: dtolnay/rust-toolchain@stable + - name: Install Rust (nightly) + uses: dtolnay/rust-toolchain@nightly with: components: llvm-tools-preview - uses: taiki-e/install-action@cargo-llvm-cov - uses: Swatinem/rust-cache@v2 - name: generate coverage report - run: cargo llvm-cov --workspace --all-features --html + run: cargo llvm-cov --workspace --all-features --doctests --html - name: upload coverage report uses: actions/upload-artifact@v4 with: @@ -171,7 +179,7 @@ jobs: if: vars.COVERAGE_GIST_ID != '' id: coverage_summary run: | - PCT=$(cargo llvm-cov --workspace --all-features --summary-only | tail -1 | awk '{print $NF}' | tr -d '%') + PCT=$(cargo llvm-cov --workspace --all-features --doctests --summary-only | tail -1 | awk '{print $NF}' | tr -d '%') echo "percent=$PCT" >> "$GITHUB_OUTPUT" - name: update coverage badge gist if: vars.COVERAGE_GIST_ID != '' && github.event_name != 'pull_request' && github.ref == 'refs/heads/main' @@ -238,6 +246,53 @@ jobs: - name: test (default features, excluding performance-tests) run: cargo test --workspace --exclude performance-tests + hack: + # Feature-interaction coverage the `test` job's single fixed feature + # list can't give you: with 6+ optional features spread across 7 + # crates, a combination the `test` job never happens to select (e.g. + # `no_std` + `async` together, or `perf` alone without `fuzz`) can + # still be broken. cargo-hack builds every crate's own feature powerset + # independently rather than one workspace-wide `--all-features` blob, + # so a broken pairing is attributed to the crate that actually has it. + # `compile-fail` is excluded for the same reason it's excluded from the + # `test` job — trybuild's UI tests assert on rustc's exact diagnostic + # text, which isn't a feature-interaction concern and belongs to the + # dedicated stable-only `trybuild` job instead. + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v7 + - name: Install Rust (stable) + uses: dtolnay/rust-toolchain@stable + - uses: taiki-e/install-action@cargo-hack + - uses: Swatinem/rust-cache@v2 + - name: cargo hack test --feature-powerset + run: cargo hack test --workspace --feature-powerset --exclude-features compile-fail + + minimal-versions: + # `cargo test --workspace` resolves every dependency to its newest + # semver-compatible release, which can mask a `Cargo.toml` requirement + # that's looser than what the code actually needs (e.g. depending on + # `foo = "1"` while calling an API `foo` only added in 1.4) — a + # downstream adopter with an older lockfile hits a compile failure this + # repo's own CI never sees. `cargo minimal-versions` re-resolves every + # dependency down to the *lowest* version each `Cargo.toml` constraint + # allows (needs nightly for the `-Z minimal-versions` resolution step) + # and tests against that. Scoped to default features and excludes + # `performance-tests`, matching the `msrv` job below — Criterion's own + # transitive graph (`clap`/`plotters`/...) floors well above what this + # template's MSRV-sensitive pins are about, so it would fail here for + # reasons unrelated to this template's own minimum-version claims. + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v7 + - name: Install Rust (nightly) + uses: dtolnay/rust-toolchain@nightly + - uses: taiki-e/install-action@cargo-hack + - uses: taiki-e/install-action@cargo-minimal-versions + - uses: Swatinem/rust-cache@v2 + - name: cargo minimal-versions test + run: cargo minimal-versions test --workspace --exclude performance-tests + deny: runs-on: ubuntu-latest steps: diff --git a/CHANGELOG.md b/CHANGELOG.md index 527fb74..c064988 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,46 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ## [Unreleased] +**Phase 8 — verification-focused CI additions:** + +- `.github/workflows/ci.yml`: new stable-only `hack` job (`cargo hack test + --workspace --feature-powerset --exclude-features compile-fail`) — + the `test` job's fixed feature list only ever builds one combination; + cargo-hack checks each crate's own feature powerset independently so a + broken pairing (e.g. `no_std` + `async` together) is caught and + attributed to the right crate. `compile-fail` excluded for the same + reason as in the `test` job. +- `.github/workflows/ci.yml`: new nightly `minimal-versions` job (`cargo + minimal-versions test --workspace --exclude performance-tests`) — + re-resolves every dependency down to the lowest version each + `Cargo.toml` constraint allows instead of the newest resolvable one, so + a `Cargo.toml` requirement looser than what the code actually needs gets + caught here instead of by a downstream adopter with an older lockfile. + `performance-tests` excluded for the same reason as in the `msrv` job + (Criterion's transitive graph floors above what this template's version + claims are about). CI is now a 12-job pipeline (README/`ci/README.md` + updated to match). +- `crates/core-tests`: the `no_std_support` module's "usable under + `#![no_std]`" claim is now enforced by the compiler, not just true by + convention. Moved its source to its own file + (`src/no_std_support.rs`) and added `tests/no_std_check.rs`, an + integration test genuinely marked `#![no_std]` that re-includes that + file via `#[path]` and calls it — gated behind `required-features = + ["no_std"]`, matching the existing `[[bench]] required-features = + ["perf"]` pattern in `performance-tests`. Since `no_std` is already in + the `test`/`nextest`/`coverage` jobs' feature list, no new CI job was + needed — those jobs now actually exercise the claim. Runs on the normal + host target (where `std` is always available) to check the `#![no_std]` + boundary itself, not cross-compilation to an embedded target — kept out + of scope for the same reason as the WASM-target deferral above. +- `.github/workflows/ci.yml`: the `coverage` job now passes `--doctests` + to `cargo llvm-cov`, so the `///` examples throughout the workspace + count toward the coverage report instead of being silently excluded. + This requires nightly (rustdoc's doc-test coverage support is gated + behind the unstable `--persist-doctests` flag), so the job's toolchain + moved from stable to nightly. `scripts/coverage.sh` updated to match + (`cargo +nightly llvm-cov ... --doctests`). + **Phase 7 — advanced testing capabilities and CI matrix growth:** - `crates/semantic-tests`: new `loom` feature — a dedicated `loom_tests` diff --git a/README.md b/README.md index fed5aa7..dcdac27 100644 --- a/README.md +++ b/README.md @@ -115,7 +115,7 @@ Optional, dependency-pulling tooling is gated behind Cargo features so | Crate | Feature | Adds | What it unlocks | | --- | --- | --- | --- | | `core-tests` | `async` | `tokio` | async fixture helpers (`async_support::default_user_fixture_async`, concurrent fixture loading) | -| `core-tests` | `no_std` | — | `core`-only helper module (`no_std_support`) | +| `core-tests` | `no_std` | — | `core`-only helper module (`no_std_support`), compiled under a genuinely `#![no_std]` crate in `tests/no_std_check.rs` | | `semantic-tests` | `async` | `tokio` | an async ownership test (`cargo test -p semantic-tests --features async`) | | `performance-tests` | `perf` | `criterion` | `cargo bench -p performance-tests --features perf` | | `fuzz-tests` | `fuzz` | `proptest` | property tests (`cargo test -p fuzz-tests --features fuzz`) | @@ -156,15 +156,21 @@ asserting the promise in docs. ## Tooling & Automation - Native `cargo test` is first-class. -- CI is a 10-job pipeline: `fmt`+`clippy`+tests across a stable/beta/nightly +- CI is a 12-job pipeline: `fmt`+`clippy`+tests across a stable/beta/nightly × ubuntu/windows/macos/linux-arm64 matrix, a stable-only `nextest` job, a stable-only `trybuild` job, a stable-only `loom` job (concurrency-permutation testing, see [`docs/adding-tests.md`](docs/adding-tests.md)), a nightly `fuzz-build` job (build + short smoke run per target on every push/PR, plus a longer campaign on the daily schedule — see - [`docs/fuzzing.md`](docs/fuzzing.md)), a stable-only `coverage` job, a - nightly `udeps` job (informational, unused-dependency check), an MSRV - job, a `cargo-deny` supply-chain job, and a stable-only `docs` job. See + [`docs/fuzzing.md`](docs/fuzzing.md)), a nightly `coverage` job + (`--doctests` included, so `///` examples count too — needs nightly for + rustdoc's unstable `--persist-doctests`), a nightly `udeps` job + (informational, unused-dependency check), an MSRV job, a stable-only + `hack` job (per-crate feature-powerset testing via + [cargo-hack](https://github.com/taiki-e/cargo-hack)), a nightly + `minimal-versions` job (tests against the lowest dependency versions each + `Cargo.toml` constraint allows, not just the newest resolvable ones), a + `cargo-deny` supply-chain job, and a stable-only `docs` job. See [`ci/README.md`](ci/README.md) for what each one does and why it's scoped the way it is. - [`justfile`](justfile) — `just check`, `just test-all`, `just bench`, @@ -185,10 +191,11 @@ asserting the promise in docs. Recommended defaults: - Coverage: `scripts/coverage.sh` or `just coverage` (installs guidance - included); or manually: `rustup component add llvm-tools-preview && - cargo install cargo-llvm-cov --locked`, then `cargo llvm-cov --workspace - --all-features --html`. Also runs in CI as a downloadable artifact — see - the `coverage` job. + included); or manually: `rustup toolchain install nightly --component + llvm-tools-preview && cargo install cargo-llvm-cov --locked`, then `cargo + +nightly llvm-cov --workspace --all-features --doctests --html` (nightly + is needed for `--doctests`). Also runs in CI as a downloadable artifact — + see the `coverage` job. - Test output: `cargo test -- --nocapture` - Alternative runner: `cargo nextest run --workspace` (install: `cargo install cargo-nextest --locked`) — faster on larger suites, one diff --git a/ci/README.md b/ci/README.md index 20ebf6d..7ddc7e4 100644 --- a/ci/README.md +++ b/ci/README.md @@ -9,7 +9,11 @@ CI helpers and notes for running fmt, clippy, tests, and optional nightly jobs. `compile-fail`) so the Tokio-backed async test, the proptest property tests, and the Criterion bench code path are all checked, not just the defaults. `compile-fail` is deliberately excluded here — see the - `trybuild` job. + `trybuild` job. The `no_std` feature also enables `core-tests`' + `tests/no_std_check.rs`, a genuinely `#![no_std]`-marked integration test + — so this job (and `nextest`/`coverage`, which enable the same feature) + actually verifies the "usable under `#![no_std]`" claim on + `no_std_support`, not just the code review/convention it used to rely on. - **`nextest` job** — runs on stable only: `cargo nextest run --workspace --features async,no_std,perf,fuzz,edge`. Demonstrates [cargo-nextest](https://nexte.st/) as an opt-in alternative runner; @@ -42,11 +46,15 @@ CI helpers and notes for running fmt, clippy, tests, and optional nightly jobs. coverage tooling. `fuzz/` is a detached workspace (see its own `[workspace]` table), so it's never part of the main `cargo test --workspace` run. -- **`coverage` job** — runs on stable only: `cargo llvm-cov --workspace - --all-features --html`, uploaded as a build artifact. Informational — - doesn't gate merges on a threshold; see `scripts/coverage.sh` to run the - same thing locally. Also has an optional, off-by-default coverage % - badge step — see "Coverage badge setup" below. +- **`coverage` job** — runs on nightly (needed for `--doctests`, below): + `cargo llvm-cov --workspace --all-features --doctests --html`, uploaded + as a build artifact. `--doctests` counts the `///` examples toward + coverage too, not just `#[test]`s — it's implemented via rustdoc's + unstable `--persist-doctests`, which only the nightly compiler accepts. + Informational — doesn't gate merges on a threshold; see + `scripts/coverage.sh` to run the same thing locally. Also has an + optional, off-by-default coverage % badge step — see "Coverage badge + setup" below. - **`udeps` job** — runs on nightly only: `cargo udeps --workspace --all-features`, checking for unused dependencies. Informational — `continue-on-error: true` on the check step, since cargo-udeps analyzes @@ -61,6 +69,24 @@ CI helpers and notes for running fmt, clippy, tests, and optional nightly jobs. format v4, which needs Cargo >= 1.78 to read. `performance-tests` is excluded from the `cargo test` step — see the comment in the workflow file for why. +- **`hack` job** — runs on stable only: `cargo hack test --workspace + --feature-powerset --exclude-features compile-fail`. The `test` job's + fixed `--features async,no_std,perf,fuzz,edge,snapshot` list only ever + builds *one* combination; a pairing it never selects (e.g. `no_std` + + `async` together) can still be broken. cargo-hack checks each crate's own + feature powerset independently, so a failure is attributed to the crate + that actually has the broken combination. `compile-fail` is excluded for + the same reason as in the `test` job — see the `trybuild` job above. +- **`minimal-versions` job** — runs on nightly (needed for the `-Z + minimal-versions` resolution step `cargo minimal-versions` performs + internally): `cargo minimal-versions test --workspace --exclude + performance-tests`. Re-resolves every dependency down to the *lowest* + version each `Cargo.toml` constraint allows, instead of the newest + semver-compatible one `cargo test` would normally pick — catches a + `Cargo.toml` requirement that's looser than what the code actually needs. + `performance-tests` is excluded for the same reason as in the `msrv` job: + Criterion's own transitive graph floors well above what this template's + version claims are about. - **`deny` job** — runs `cargo-deny check` (licenses, security advisories, banned/duplicate dependencies, untrusted sources) against both the main workspace and the detached `fuzz/` workspace. Config lives in diff --git a/crates/core-tests/Cargo.toml b/crates/core-tests/Cargo.toml index 1bc11d4..f4f2665 100644 --- a/crates/core-tests/Cargo.toml +++ b/crates/core-tests/Cargo.toml @@ -16,3 +16,9 @@ tokio = { version = "1", features = ["rt", "macros"], optional = true } default = [] async = ["dep:tokio"] no_std = [] + +# Genuinely `#![no_std]`, gated the same as the module it re-includes — see +# tests/no_std_check.rs. +[[test]] +name = "no_std_check" +required-features = ["no_std"] diff --git a/crates/core-tests/src/lib.rs b/crates/core-tests/src/lib.rs index a164f87..35b39f7 100644 --- a/crates/core-tests/src/lib.rs +++ b/crates/core-tests/src/lib.rs @@ -199,26 +199,13 @@ pub mod async_support { /// adopter compiles this crate under `#![no_std]` (e.g. for embedded /// targets). The rest of `core-tests` keeps using `std` for fixture /// convenience; this module is where `no_std`-safe helpers should live. +/// +/// That claim is enforced by the compiler, not just by convention: `tests/ +/// no_std_check.rs` re-includes this file's source inside a crate genuinely +/// marked `#![no_std]`, so a future edit that sneaks in a `std` reference +/// fails to build instead of just looking fine on review. #[cfg(feature = "no_std")] -pub mod no_std_support { - /// Checks whether an ASCII string reads the same forwards and backwards, - /// without allocating. - pub fn is_ascii_palindrome(input: &str) -> bool { - let bytes = input.as_bytes(); - let mut left = 0; - let mut right = bytes.len(); - - while left < right { - right -= 1; - if !bytes[left].eq_ignore_ascii_case(&bytes[right]) { - return false; - } - left += 1; - } - - true - } -} +pub mod no_std_support; #[cfg(test)] mod tests { diff --git a/crates/core-tests/src/no_std_support.rs b/crates/core-tests/src/no_std_support.rs new file mode 100644 index 0000000..27980b2 --- /dev/null +++ b/crates/core-tests/src/no_std_support.rs @@ -0,0 +1,17 @@ +/// Checks whether an ASCII string reads the same forwards and backwards, +/// without allocating. +pub fn is_ascii_palindrome(input: &str) -> bool { + let bytes = input.as_bytes(); + let mut left = 0; + let mut right = bytes.len(); + + while left < right { + right -= 1; + if !bytes[left].eq_ignore_ascii_case(&bytes[right]) { + return false; + } + left += 1; + } + + true +} diff --git a/crates/core-tests/tests/no_std_check.rs b/crates/core-tests/tests/no_std_check.rs new file mode 100644 index 0000000..2a5f399 --- /dev/null +++ b/crates/core-tests/tests/no_std_check.rs @@ -0,0 +1,24 @@ +//! Compiles `src/no_std_support.rs`'s source inside a crate genuinely marked +//! `#![no_std]`, so the "usable under `#![no_std]`" claim on that module +//! (see `src/lib.rs`) is enforced by the compiler on every `cargo test +//! --features no_std` run, rather than being true only by convention/review. +//! +//! This runs on the normal host target, where `std` is always available — +//! it checks the `#![no_std]` boundary itself (does this code reference +//! `std`?), not cross-compilation to an embedded target. A general +//! cross-compilation matrix is deliberately out of scope for this template; +//! see the WASM-target deferral in `CHANGELOG.md` for the same reasoning +//! applied elsewhere. +#![no_std] + +#[path = "../src/no_std_support.rs"] +mod no_std_support; + +use no_std_support::is_ascii_palindrome; + +#[test] +fn palindrome_logic_compiles_and_works_under_no_std() { + assert!(is_ascii_palindrome("Level")); + assert!(is_ascii_palindrome("")); + assert!(!is_ascii_palindrome("rust")); +} diff --git a/justfile b/justfile index 64147b5..12351c5 100644 --- a/justfile +++ b/justfile @@ -65,6 +65,15 @@ fuzz-build: fuzz-run target: cd fuzz && cargo +nightly fuzz run {{target}} seed_corpus/{{target}} +# Feature-powerset check per crate, matching CI's `hack` job (install: cargo install cargo-hack --locked). +hack: + cargo hack test --workspace --feature-powerset --exclude-features compile-fail + +# Test against the lowest versions every Cargo.toml constraint allows, matching CI's `minimal-versions` job. +# Needs nightly for resolution (install: cargo install cargo-hack cargo-minimal-versions --locked). +minimal-versions: + cargo minimal-versions test --workspace --exclude performance-tests + # Check licenses/advisories/bans/sources for both workspaces (install: cargo install cargo-deny --locked). deny: cargo deny check diff --git a/scripts/coverage.sh b/scripts/coverage.sh index e0227b2..5cf6789 100644 --- a/scripts/coverage.sh +++ b/scripts/coverage.sh @@ -2,8 +2,12 @@ # Generates an HTML coverage report with cargo-llvm-cov. Informational — # there's no enforced threshold; use this to spot untested paths. # Install once with: -# rustup component add llvm-tools-preview +# rustup toolchain install nightly --component llvm-tools-preview # cargo install cargo-llvm-cov --locked +# Runs under nightly: `--doctests` (counting the `///` examples toward +# coverage, not just #[test]s) needs rustdoc's unstable +# --persist-doctests, which only the nightly compiler accepts — matches +# CI's `coverage` job, see .github/workflows/ci.yml. # Usage: scripts/coverage.sh set -euo pipefail @@ -11,15 +15,15 @@ cd "$(dirname "${BASH_SOURCE[0]}")/.." if ! cargo llvm-cov --version >/dev/null 2>&1; then echo "cargo-llvm-cov is not installed. Install it with:" >&2 - echo " rustup component add llvm-tools-preview" >&2 + echo " rustup toolchain install nightly --component llvm-tools-preview" >&2 echo " cargo install cargo-llvm-cov --locked" >&2 exit 1 fi -echo "==> cargo llvm-cov --workspace --all-features --summary-only" -cargo llvm-cov --workspace --all-features --summary-only +echo "==> cargo +nightly llvm-cov --workspace --all-features --doctests --summary-only" +cargo +nightly llvm-cov --workspace --all-features --doctests --summary-only -echo "==> cargo llvm-cov report --html" -cargo llvm-cov report --html +echo "==> cargo +nightly llvm-cov report --html" +cargo +nightly llvm-cov report --html echo "Report written to target/llvm-cov/html/index.html" From 2a8c2591c77e666f3c8fd1c9feabfe4f83278bd3 Mon Sep 17 00:00:00 2001 From: Sovereign Map Test Suite Date: Sat, 1 Aug 2026 06:28:18 -0700 Subject: [PATCH 2/2] Release prep: v0.2.0-alpha - CHANGELOG.md: converted the accumulated Unreleased section (Phases 5-8: docs/lint consolidation, tooling, advanced testing, and verification- focused CI additions) into a dated [0.2.0-alpha] entry, with a fresh empty Unreleased section above for future work. Minor bump rather than a beta/stable relabel - substantial new features landed since v0.1.0-alpha (loom, feature-powerset/minimal-versions CI, a real no_std check), not just fixes, and the project isn't claiming production-readiness yet. - Crate versions (Cargo.toml) intentionally left at 0.1.0, matching the v0.1.0-alpha release: every crate is publish = false, so the CHANGELOG/git-tag version and the Cargo.toml version are independent numbering schemes here - see the [0.1.0] initial-scaffold entry at the bottom of CHANGELOG.md, which predates any tag. Co-Authored-By: Claude Sonnet 5 --- CHANGELOG.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index c064988..f31a792 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ## [Unreleased] +## [0.2.0-alpha] - 2026-08-01 + **Phase 8 — verification-focused CI additions:** - `.github/workflows/ci.yml`: new stable-only `hack` job (`cargo hack test