Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
90 changes: 90 additions & 0 deletions .github/workflows/sovereign-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
# Change once here → applies to all 38 repos instantly.
#
# Self-hosted jobs run inside the sovereign-ci container (built by forjar).
# Each job gets an isolated filesystem — no shared ~/.rustup/, no race conditions.

Check warning on line 7 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

7:81 [line-length] line too long (82 > 80 characters)

Check warning on line 7 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

7:81 [line-length] line too long (82 > 80 characters)
# Image: localhost:5000/sovereign-ci:stable (local registry on mac-server)
# Rebuild: cd infra && make -f machines/intel/Makefile ci-image
#
Expand Down Expand Up @@ -48,25 +48,50 @@
default: false
type: boolean
extra_pkgs:
description: 'Extra apt packages to install in container (e.g. python3-dev libclang-dev)'

Check warning on line 51 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

51:81 [line-length] line too long (97 > 80 characters)

Check warning on line 51 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

51:81 [line-length] line too long (97 > 80 characters)
required: false
default: ''
type: string
enable_sccache:
description: 'sccache compiler cache (build-performance.md §5.3). Fleet-default on 2026-04-18 after Phase 3 pilot: F9 p95=100%, n=24 runs, median=100%. Set false to disable per-repo.'

Check warning on line 56 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

56:81 [line-length] line too long (191 > 80 characters)

Check warning on line 56 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

56:81 [line-length] line too long (191 > 80 characters)
required: false
default: true
type: boolean
use_nextest:
description: 'cargo nextest run instead of cargo test (build-performance.md §4.3 + §7 Phase 2). 30-40% test-job speedup on large suites. Pilot repos only until F11 test-job p95 ≤ 300s verified over 7 days.'

Check warning on line 61 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

61:81 [line-length] line too long (214 > 80 characters)

Check warning on line 61 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

61:81 [line-length] line too long (214 > 80 characters)
required: false
default: false
type: boolean
test_workspace:
description: 'PMAT-159: test all workspace members with `--workspace --lib` (not just root). Opt-in because workspace members may not build in the sovereign-ci container (e.g. aprender-gpu needs cuBLAS). Pair with test_args exclusions as needed.'

Check warning on line 66 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

66:81 [line-length] line too long (254 > 80 characters)

Check warning on line 66 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

66:81 [line-length] line too long (254 > 80 characters)
required: false
default: false
type: boolean
coverage_min:
description: >
OPT-IN line-coverage floor (ratchet, PMAT build-system audit gap #1).
DEFAULT IS EMPTY → behavior is UNCHANGED for every repo that does not
set it (coverage stays purely advisory, exactly as before). When set
to a number (e.g. "96.9"), the coverage job FAILS if measured line
coverage drops below this floor. Additionally, if a committed baseline
file (see coverage_baseline_file) exists, the job FAILS on ANY DROP
below the recorded baseline — so enabling this on an already-green
repo can never break it (the baseline is its current value). This is a
ratchet, not an absolute jump: it blocks regressions without demanding
an immediate coverage increase. Pilot: aprender only.
required: false
default: ''
type: string
coverage_baseline_file:
description: >
Path (repo-relative) to a committed file holding a single float = the
last-known-good line-coverage %. Used as the ratchet floor when
coverage_min is set: the effective floor is max(coverage_min,
baseline). Inert unless coverage_min is non-empty. Default:
.pmat/coverage-baseline.txt
required: false
default: '.pmat/coverage-baseline.txt'
type: string

# HD-02: Least-privilege token — only escalate where needed
permissions:
Expand All @@ -83,7 +108,7 @@
jobs:
test:
name: test
runs-on: [self-hosted, clean-room]

Check failure on line 111 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

label "clean-room" is unknown. available labels are "windows-latest", "windows-latest-8-cores", "windows-2025", "windows-2025-vs2026", "windows-2022", "windows-11-arm", "ubuntu-slim", "ubuntu-latest", "ubuntu-latest-4-cores", "ubuntu-latest-8-cores", "ubuntu-latest-16-cores", "ubuntu-24.04", "ubuntu-24.04-arm", "ubuntu-22.04", "ubuntu-22.04-arm", "macos-latest", "macos-latest-xlarge", "macos-latest-large", "macos-26-intel", "macos-26-xlarge", "macos-26-large", "macos-26", "macos-15-intel", "macos-15-xlarge", "macos-15-large", "macos-15", "macos-14-xlarge", "macos-14-large", "macos-14", "self-hosted", "x64", "arm", "arm64", "linux", "macos", "windows". if it is a custom label for self-hosted runner, set list of labels in actionlint.yaml config file

Check failure on line 111 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

label "clean-room" is unknown. available labels are "windows-latest", "windows-latest-8-cores", "windows-2025", "windows-2025-vs2026", "windows-2022", "windows-11-arm", "ubuntu-slim", "ubuntu-latest", "ubuntu-latest-4-cores", "ubuntu-latest-8-cores", "ubuntu-latest-16-cores", "ubuntu-24.04", "ubuntu-24.04-arm", "ubuntu-22.04", "ubuntu-22.04-arm", "macos-latest", "macos-latest-xlarge", "macos-latest-large", "macos-26-intel", "macos-26-xlarge", "macos-26-large", "macos-26", "macos-15-intel", "macos-15-xlarge", "macos-15-large", "macos-15", "macos-14-xlarge", "macos-14-large", "macos-14", "self-hosted", "x64", "arm", "arm64", "linux", "macos", "windows". if it is a custom label for self-hosted runner, set list of labels in actionlint.yaml config file
container:
image: localhost:5000/sovereign-ci:stable@sha256:10486da5daa3786f3264aa0e19fdde007e7ba1eca1d47ba587947946e42bd871
# Phase 3 §5.3 — sccache rustc cache + /var/log/ci-metrics for F9 stats.
Expand All @@ -98,7 +123,7 @@
# is not concurrent-safe and collides with "No such file or directory"
# errors (15 observed on aprender PR #1019). Mirrors the aprender
# workspace-test pattern (task #134).
- /mnt/nvme-raid0/targets/sovereign-ci-${{ inputs.repo }}/${{ github.event.pull_request.number || github.ref_name }}:/workspace/target

Check warning on line 126 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

126:81 [line-length] line too long (142 > 80 characters)

Check warning on line 126 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

126:81 [line-length] line too long (142 > 80 characters)
# PMAT-159 (2026-04-20): bumped 30→60 min so workspace-mode callers
# (test_workspace: true) have headroom to compile + test large workspaces.
# Default --lib callers are well under 30 min; the ceiling only binds for
Expand All @@ -112,9 +137,9 @@
persist-credentials: false
- name: Install extra packages
if: ${{ inputs.extra_pkgs != '' }}
run: |

Check failure on line 140 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

shellcheck reported issue in this script: SC2015:info:2:25: Note that A && B || C is not if-then-else. C may run when A is true

Check failure on line 140 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

shellcheck reported issue in this script: SC2015:info:2:25: Note that A && B || C is not if-then-else. C may run when A is true
apt-get update -qq && apt-get install -y -qq ${{ inputs.extra_pkgs }} 2>/dev/null || \

Check warning on line 141 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

141:81 [line-length] line too long (96 > 80 characters)

Check warning on line 141 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

141:81 [line-length] line too long (96 > 80 characters)
sudo apt-get update -qq && sudo apt-get install -y -qq ${{ inputs.extra_pkgs }} 2>/dev/null || true

Check warning on line 142 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

142:81 [line-length] line too long (109 > 80 characters)

Check warning on line 142 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

142:81 [line-length] line too long (109 > 80 characters)
- name: Checkout sibling repos (path deps)
run: |
cd "$GITHUB_WORKSPACE/.."
Expand All @@ -125,11 +150,11 @@
# provable-contracts kept (pv codegen still uses it); aprender +
# whisper.apr still active.
for repo in provable-contracts aprender whisper.apr; do
# Skip the current repo — it is already checked out at $GITHUB_WORKSPACE

Check warning on line 153 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

153:81 [line-length] line too long (84 > 80 characters)

Check warning on line 153 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

153:81 [line-length] line too long (84 > 80 characters)
[ "$repo" = "${{ inputs.repo }}" ] && continue
# Remove stale partial clones (file instead of dir)
if [ -e "$repo" ] && [ ! -d "$repo" ]; then rm -f "$repo"; fi
# Validate cached clone. If HEAD missing or fetch fails, nuke + re-clone.

Check warning on line 157 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

157:81 [line-length] line too long (85 > 80 characters)

Check warning on line 157 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

157:81 [line-length] line too long (85 > 80 characters)
# Silent `|| true` on fetch previously masked broken caches (bashrs#197).
if [ -d "$repo" ]; then
if ! git -C "$repo" rev-parse HEAD >/dev/null 2>&1; then
Expand Down Expand Up @@ -160,7 +185,7 @@
# Note: generated contract macros may have unused variables (provable-contracts#64).
# This is handled by adding -A unused-variables to the clippy step.
- name: Generate contract assertions (pv codegen)
run: |

Check failure on line 188 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

shellcheck reported issue in this script: SC2012:info:9:3: Use find instead of ls to better handle non-alphanumeric filenames

Check failure on line 188 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

shellcheck reported issue in this script: SC2012:info:9:3: Use find instead of ls to better handle non-alphanumeric filenames
# pv is baked into sovereign-ci:stable at /usr/local/cargo/bin/pv
PV=""
for candidate in /usr/local/cargo/bin/pv /usr/local/bin/pv; do
Expand Down Expand Up @@ -211,7 +236,7 @@
SCCACHE_DIR: ${{ inputs.enable_sccache && '/sccache' || '' }}
USE_NEXTEST: ${{ inputs.use_nextest }}
TEST_SCOPE: ${{ inputs.test_workspace && '--workspace --lib' || '--lib' }}
run: |

Check failure on line 239 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

shellcheck reported issue in this script: SC2086:info:17:36: Double quote to prevent globbing and word splitting

Check failure on line 239 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

shellcheck reported issue in this script: SC2086:info:16:26: Double quote to prevent globbing and word splitting

Check failure on line 239 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

shellcheck reported issue in this script: SC2086:info:16:14: Double quote to prevent globbing and word splitting

Check failure on line 239 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

shellcheck reported issue in this script: SC2086:info:13:38: Double quote to prevent globbing and word splitting

Check failure on line 239 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

shellcheck reported issue in this script: SC2086:info:12:28: Double quote to prevent globbing and word splitting

Check failure on line 239 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

shellcheck reported issue in this script: SC2086:info:12:16: Double quote to prevent globbing and word splitting

Check failure on line 239 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

shellcheck reported issue in this script: SC2086:info:10:43: Double quote to prevent globbing and word splitting

Check failure on line 239 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

shellcheck reported issue in this script: SC2086:info:17:36: Double quote to prevent globbing and word splitting

Check failure on line 239 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

shellcheck reported issue in this script: SC2086:info:16:26: Double quote to prevent globbing and word splitting

Check failure on line 239 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

shellcheck reported issue in this script: SC2086:info:16:14: Double quote to prevent globbing and word splitting

Check failure on line 239 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

shellcheck reported issue in this script: SC2086:info:13:38: Double quote to prevent globbing and word splitting

Check failure on line 239 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

shellcheck reported issue in this script: SC2086:info:12:28: Double quote to prevent globbing and word splitting

Check failure on line 239 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

shellcheck reported issue in this script: SC2086:info:12:16: Double quote to prevent globbing and word splitting

Check failure on line 239 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

shellcheck reported issue in this script: SC2086:info:10:43: Double quote to prevent globbing and word splitting
# Mark workspace as safe for git operations inside tests (dubious ownership in containers)
git config --global --add safe.directory "$GITHUB_WORKSPACE"
# Phase 2 §4.3 — nextest drops ~35% off test-job wall-clock on large suites.
Expand Down Expand Up @@ -521,6 +546,71 @@
cargo llvm-cov test $TEST_SCOPE --no-cfg-coverage --no-cfg-coverage-nightly --lcov --output-path lcov.info $TEST_ARGS 2>&1 || \
cargo llvm-cov test --lib --no-cfg-coverage --no-cfg-coverage-nightly -p "$REPO_NAME" --lcov --output-path lcov.info 2>&1 || \
{ echo "::error::Coverage failed — check workspace path dependencies"; exit 1; }
- name: Enforce coverage floor (OPT-IN ratchet — PMAT build-system audit gap #1)
# ZERO BEHAVIOR CHANGE GUARANTEE:
# When inputs.coverage_min is empty (the default for EVERY repo that
# does not explicitly set it), this step is SKIPPED entirely via its
# `if:` guard. Coverage stays exactly as advisory as it was before:
# measured, uploaded to codecov (continue-on-error), never gating.
#
# When coverage_min IS set (pilot: aprender), this step:
# 1. Derives line-coverage % from lcov.info (sum of hit DA records ÷
# total DA records). lcov.info was just produced by the step above;
# no extra compile/test cost.
# 2. Computes the effective floor = max(coverage_min, baseline) where
# baseline is read from inputs.coverage_baseline_file IF that file
# is committed in the repo. This is the RATCHET: a repo that is
# green today records its current % as the baseline, so enabling
# the gate can never break the currently-green state — it only
# blocks a DROP. coverage_min is the absolute floor underneath.
# 3. FAILS the job (exit 1) if measured < effective floor.
#
# The gate `coverage` job result is wired into the top-level `gate` job
# (needs: [...coverage...]; fails if coverage.result == failure), so a
# regression now blocks merge instead of merging silently.
if: ${{ inputs.coverage_min != '' }}
env:
COVERAGE_MIN: ${{ inputs.coverage_min }}
COVERAGE_BASELINE_FILE: ${{ inputs.coverage_baseline_file }}
run: |
set -euo pipefail
if [ ! -f lcov.info ]; then
echo "::error::coverage_min is set but lcov.info was not produced — cannot enforce floor"
exit 1
fi
# Derive line coverage from lcov DA records: DA:<line>,<hits>
# covered = count of DA records with hits > 0; total = all DA records.
read -r COVERED TOTAL < <(awk -F'[:,]' '
/^DA:/ { total++; if ($3 > 0) covered++ }
END { printf "%d %d", covered, total }
' lcov.info)
if [ "${TOTAL:-0}" -eq 0 ]; then
echo "::error::coverage_min is set but lcov.info has 0 line records — refusing to pass a gate on empty data"
exit 1
fi
PCT=$(awk -v c="$COVERED" -v t="$TOTAL" 'BEGIN { printf "%.2f", (c / t) * 100 }')
echo "Measured line coverage: ${PCT}% (${COVERED}/${TOTAL} lines)"

# Effective floor = max(coverage_min, committed baseline if present).
FLOOR="$COVERAGE_MIN"
if [ -n "${COVERAGE_BASELINE_FILE}" ] && [ -f "${COVERAGE_BASELINE_FILE}" ]; then
BASELINE=$(tr -dc '0-9.' < "${COVERAGE_BASELINE_FILE}" | head -c 16)
if [ -n "$BASELINE" ]; then
echo "Ratchet baseline (${COVERAGE_BASELINE_FILE}): ${BASELINE}%"
FLOOR=$(awk -v a="$COVERAGE_MIN" -v b="$BASELINE" 'BEGIN { print (a > b) ? a : b }')
else
echo "::warning::${COVERAGE_BASELINE_FILE} present but unparseable — using coverage_min=${COVERAGE_MIN} as floor"
fi
else
echo "No committed baseline file — using coverage_min=${COVERAGE_MIN} as floor"
fi
echo "Effective coverage floor: ${FLOOR}%"

if awk -v p="$PCT" -v f="$FLOOR" 'BEGIN { exit !(p < f) }'; then
echo "::error::Coverage regression: ${PCT}% < floor ${FLOOR}% — this would have merged SILENTLY before the ratchet (PMAT gap #1). Raise coverage or, if intentional, update ${COVERAGE_BASELINE_FILE}."
exit 1
fi
echo "Coverage ${PCT}% ≥ floor ${FLOOR}% — ratchet satisfied"
- name: Record sccache stats
if: ${{ always() && inputs.enable_sccache }}
run: |
Expand Down
Loading