Make a green CI check mean something again

[mihow]

Summary

While getting three long-failing tests to pass (#164), we found three independent reasons a green check on this repository does not currently tell you the code is healthy. None of them are hypothetical — all three are load-bearing right now, and together they explain how main stayed red for four months without anyone noticing.

Filing these together because they share a cause: a check that reports success without having verified anything is worse than one that fails, since it actively discourages looking.

1. The pre-commit check runs no checks

.github/workflows/lint.yaml has its only real step commented out:

jobs:
  pre-commit:
    steps:
      - uses: actions/checkout@v6
      - uses: actions/setup-python@v6
      # - uses: pre-commit/action@v2.0.0

The job checks out the repository, installs Python, and exits green. Every pull request's passing pre-commit status has been meaningless for as long as this has been in place.

Re-enabling it will likely surface accumulated formatting and lint debt across the tree, so it is worth doing deliberately — probably as its own pull request that fixes the fallout in one pass, rather than as a surprise on someone else's branch.

Related: the pinned autoflake==1.4 and flake8==4.0.0 hooks crash under Python 3.12, because distutils and EntryPoints.get were both removed. Running pre-commit locally under 3.12 produces a crash rather than a lint result; 3.10 works. These pins need bumping as part of the same work.

2. main's test suite has not run since 2026-04-14

gh run list --workflow=test.yml --branch main

The most recent real run is four months old. Everything green on main since then belongs to Dependabot, which is a different workflow — those rows are not the test suite, though they look reassuring in the branch's check history.

main was in fact red for much of that period. Checking out unmodified main and running the suite reproduces the failure fixed in #164.

3. A pull request whose base is not main is never tested

Both test.yml and test-ml.yml trigger only on main:

on:
  push:
    branches: ["main"]
  pull_request:
    branches: ["main"]

So a stacked pull request — one opened against another feature branch rather than main — gets no test run at all. Its checks show only CodeRabbit and the no-op pre-commit above, which reads as a fully green PR. #162 is in exactly this position today.

There is also no workflow_dispatch, so a run cannot be triggered by hand, and a run whose workflow file has since changed cannot be re-run (gh run rerun reports the workflow file may be broken). The only way to get a fresh result is to push a commit.

Directions to discuss

Ordered by effort. Nothing here is verified as the right shape yet.

1. Re-enable pre-commit and bump the hook pins, in a pull request that also fixes whatever it turns up. Smallest change with the most immediate effect.
2. Add workflow_dispatch to both test workflows. One line each, and it removes the "push an empty commit to get CI" workaround.
3. Let the test workflows run on pull requests against any base, or at least on branches matching a stacked-PR convention, so a stacked pull request is not green by default.
4. Consider a scheduled run on main. The four-month gap happened because nothing pushes directly to main between merges, so nothing triggered the suite. A nightly or weekly run would have caught it.

What we still need to verify

• Whether re-enabling pre-commit produces a manageable diff or a very large one. Nobody has run the full hook set against the tree yet.
• Whether running tests on every base branch is affordable in Actions minutes, given the ML tests download model weights at test time.