Skip to content

feat: unified label dispatcher, PR gate and .github/ci.toml declarative configuration - #148

Closed
ocots wants to merge 7 commits into
mainfrom
feat/144-unified-label-dispatcher
Closed

ocots wants to merge 7 commits into
mainfrom
feat/144-unified-label-dispatcher

Conversation

@ocots

@ocots ocots commented Nov 10, 2026

Copy link
Copy Markdown
Member

Refs #144

Introduce a centralized, unified label dispatcher and PR gate (dispatch.yml) in CTActions, powered by declarative .github/ci.toml repository configuration.

@ocots
ocots force-pushed the feat/144-unified-label-dispatcher branch from a0e4568 to 0baf50a Compare November 10, 2026 16:39
@ocots

ocots commented Nov 10, 2026

Copy link
Copy Markdown
Member Author

Phase A1 — Declarative Dispatch Engine & Unit Tests

Commit: 8d69aa6

What was done:

  • Implemented scripts/dispatch/dispatch.py:
    • Parses repository configuration from .github/ci.toml (with sensible defaults when absent) using Python's standard tomllib.
    • Determines which jobs to run (run_ci_github, run_ci_kkt, run_documentation, run_breakage_packages, run_release_gate) based on the triggering event, PR state (open vs closed/merged), and attached labels (run ci ..., run doc, run breakage, etc.).
    • Executes optional repository-specific test discovery scripts (scope_script, e.g. tools/ci/scope.py) and passes discovered test arguments cleanly as JSON-encoded workflow outputs (ci_test_args).
    • Emits $GITHUB_OUTPUT parameters and rich $GITHUB_STEP_SUMMARY markdown tables.
  • Implemented scripts/dispatch/test/test_dispatch.py:
    • 6 unit tests covering defaults, TOML overrides, closed PRs fast exit, open PR label evaluation, and push events (all passing).

@ocots

ocots commented Nov 10, 2026

Copy link
Copy Markdown
Member Author

Phase A2 — Reusable Dispatch Workflow & Workflow Hardening

Commit: dab2c94

What was done:

  • Added .github/workflows/dispatch.yml:
    • Reusable central entrypoint for PRs.
    • Sub-second early exit when PR is closed or merged (protecting release workflow step 3.6).
    • Sub-second early exit for non-dispatch labels (e.g. bug, enhancement) routed to an isolated concurrency group so active test runs are never cancelled.
    • 5-second debounce window on labeled / unlabeled events with dynamic API fetch of the latest PR label set (collapsing multi-label bursts into a single run).
  • Added .github/workflows/dispatch-tests.yml:
    • Automated self-test workflow running dispatch.py test suite and verifying dispatch outputs.
  • Updated .github/workflows/breakage-suite.yml:
    • Adjusted baseline run lookup to check both $file and Breakage.yml.
  • Updated .github/workflows/release-gate.yml:
    • Added dedicated ctactions-release-gate-* concurrency group and < 2s fast-exit when PR is closed.
  • Checked with actionlint: all workflows passed with 0 errors/warnings.

@ocots

ocots commented Nov 10, 2026

Copy link
Copy Markdown
Member Author

Phase B — Release Tooling Integration

Commit: ad8c859

What was done:

  • Updated scripts/release/changes.jl:
    • Extended run_labels to read .github/ci.toml (with fallback to regex inspection of caller workflow outputs).
  • Updated scripts/release/gate.jl:
    • Verifies documentation requirements via ci.toml or run_labels.
  • Updated scripts/release/ready.jl & audit.jl:
    • gated_jobs and ready now recognize Dispatcher.yml, parse its jobs and outputs, and mandate that release-gate within Dispatcher.yml succeeds.
  • Updated scripts/release/test/runtests.jl:
    • Added dedicated tests for ci.toml, Dispatcher.yml, and the release-gate requirement.
    • Ran full release test suite: 184,338 passes, 0 failures.

@ocots

ocots commented Nov 10, 2026

Copy link
Copy Markdown
Member Author

Phase C — Documentation & Catalog Entry

Commit: 0baf50a

What was done:

  • Added docs/workflows/dispatch.md:
    • Complete guide for the unified dispatcher, explaining the .github/ci.toml schema, debounce mechanics, concurrency isolation, and caller syntax.
  • Updated docs/README.md:
    • Documented dispatch.yml and dispatch-tests.yml in the reusable workflows and self-tests catalog tables.

@ocots
ocots marked this pull request as ready for review November 10, 2026 17:46
@ocots

ocots commented Nov 10, 2026

Copy link
Copy Markdown
Member Author

Critical review of #148

Conventions applied: ct-conventions v0.11.0, ct-agents v0.4.0. Reviewed at a0c0f78.

The direction is right: one PR entry point, declarative .github/ci.toml, and history left untouched for push/tag workflows. But the PR is not ready to merge. Two issues would break release.jl ready and burn runner time in practice, and the label/config logic is duplicated in a way that will drift. Details below, ordered by severity.


🔴 Blockers

B1. An unrelated label silently invalidates release.jl ready.
ready.jl takes the last non-skipped run of each workflow on the head SHA (findlast(r -> r.conclusion != "skipped", rs)). In Dispatcher.yml, an unrelated label (bug, enhancement, …) still starts a run. Its dispatch job succeeds (the fast path writes outputs), and every downstream job is skipped. So the run concludes success, not skipped. That run then becomes the reference run:

  • release-gate is now required in Dispatcher.yml, and here it is skipped, so we get "did not run (skipped)" → not ready;
  • the earlier run, where the CI/doc/breakage jobs actually passed, is ignored.

Adding any cosmetic label after CI turns the PR "not ready", and the only way out is to re-run everything. Possible fixes:

  • (preferred) filter unrelated labels with a job-level if: in the caller on the dispatch job, so every job is skipped and the run concludes skipped. This is also the only way to make that path really cheap (see M4);
  • or have ready.jl pick the last Dispatcher.yml run whose dispatch job actually evaluated (for example, a marker job or output).

runtests.jl needs a test for this scenario: a successful run, followed by a later run where only dispatch succeeded.

B2. Adding one label re-runs every labelled suite, GPU included.
On a relevant labeled event, the new run cancels the one in progress (cancel-in-progress: true) and re-dispatches every label present on the PR. Example: a PR with run ci kkt-runner that has been green for 40 min. Adding run documentation cancels nothing useful and re-runs the KKT GPU suite from scratch. The debounce only helps within the 5 s window. This is a cost regression compared to independent workflows, and it is tied to B1: ready.jl needs all required jobs in one run. We need an explicit design decision before rollout:

  • accept "one run = full set" and say so in the docs (plus the cost warning), or
  • on labeled, dispatch only the newly added label, and teach ready.jl to aggregate the latest successful result per job across Dispatcher.yml runs on the head SHA.

B3. edited in the caller types cancels running CI.
The documented caller lists types: [opened, labeled, unlabeled, synchronize, reopened, edited]. edited is not labeled/unlabeled, so it lands in pr-dispatch-<PR> with cancel-in-progress: true. Fixing a typo in the PR title or body cancels the CI/GPU runs in progress and re-runs everything (see B2). Remove edited, or route it to the ignore group. If the release gate needs to react to body edits, give it its own trigger.


🟠 Major

M1. Label and default knowledge is duplicated in five places, and they already disagree.

  • dispatch.py load_config defaults (Python);
  • changes.jl DISPATCH_OUTPUT_LABELS (Julia, hard-coded default labels, plus a run_probe output that dispatch.yml does not emit);
  • gate.jl hard-codes "run documentation";
  • the prefix filter (run , test , documentation push preview, release) in dispatch.yml and twice in the caller concurrency expression;
  • docs/workflows/dispatch.md.

As a result, the configurable labels in ci.toml are only half supported. A custom label that does not start with run (for example ci gpu) is treated as unrelated by the fast path and never dispatched. ready.jl maps outputs back to the default label names, not the configured ones. test and release are whitelisted but nothing consumes them. Suggestion: make dispatch.py the single source of truth, with a --print-labels mode or a JSON schema that changes.jl/ready.jl read. Then either restrict labels to the run prefix (documented and validated), or drop the claim that labels are configurable.

M2. run_labels in changes.jl does not apply defaults and swallows errors.

  • If ci.toml sets only some labels (for example [documentation] label = "run doc" and nothing else), the set is non-empty, so it is returned as is. run ci github-runner and run breakage, which are active by default, are missing.
  • If ci.toml sets no labels at all, it falls back to the YAML regex, which maps outputs to the default labels even when enabled = false.
  • try … catch end silently ignores a malformed ci.toml (EXC rules: an error must be typed and surfaced, never swallowed).

M3. ci.toml is never validated.
Unknown sections or keys ([ci.gihub], fail-fast, version) are silently ignored and the defaults apply, so a typo means a different CI with no warning. Types are not checked either (versions = "1.10" is iterated as characters by json.dumps, which only works by accident). The dispatcher should fail with a clear message listing the allowed keys.

M4. The "< 2 s" claim is misleading.
The closed and unrelated fast paths are cheap steps, but each one still allocates an ubuntu-latest runner (queue time of 5–30 s, billed minutes) and shows a run in the PR checks. That is not "< 2 s" in wall time. Only a job-level if: in the caller (see B1) avoids the runner. Also, closed is not in types, so the "closed PR" path only fires on label events after merge. That is fine for recipes/release.md step 3.6, but the docs should say so precisely.

M5. Multi-line values corrupt $GITHUB_OUTPUT.
dispatch.py writes f"{k}={v}\n". ci_github_test_args comes from the scope_script stdout, and test_args from ci.toml. A newline in either one corrupts the output file, or injects extra outputs. Use the key<<EOF delimiter syntax, or reject newlines.

M6. push/workflow_dispatch branches in dispatch.py contradict the design.
The plan keeps CI.yml/Documentation.yml/Breakage.yml for push/tags, and Dispatcher.yml only runs on PRs. Yet evaluate_dispatch turns on CI, doc, breakage and gpu_main on push. If a repository ever adds push to Dispatcher.yml, everything runs twice. Either remove those branches, or document Dispatcher.yml as able to replace the push workflows (and then ReleaseTag.yml/audit.jl must follow).


🟡 Minor

  • Output boilerplate: about 35 outputs are written three times (closed, unrelated, eval) plus the a || b || c mapping, and every new key means touching 5 places. Downstream jobs test == 'true', so an empty output already means "do not run". The closed/unrelated steps only need open/skip.
  • The gh api failure in the debounce step is swallowed (2>/dev/null || echo ''), and the run then falls back silently to the stale labels from the payload, which is what the debounce is meant to avoid. At minimum, emit a ::warning::.
  • The scope_test_dirs tests run inside the dispatcher and a failure is only a warning. Either make it an error or move it into the CI job, because it lengthens the critical path of every run.
  • fetch-depth: 0 is unconditional on the caller checkout. It is only needed by scope_script, so it should be conditional.
  • ctactions_ref defaults to main. A caller pinned at dispatch.yml@vX still runs dispatch.py@main, so the workflow and the script can drift apart. This is the same pattern as release-gate.yml, but it matters more here because the output contract is shared.
  • Workflow-level concurrency added to release-gate.yml: when it is called from Dispatcher.yml, it stacks on top of the caller's group (github.workflow resolves to the caller). The existing ReleaseGate.yml callers now get a second cancellation layer. Check that this does not cancel a gate that ready is waiting for.
  • breakage-suite.yml: hard-coding Breakage.yml as a fallback couples the reusable workflow to a caller's file name, and 2>/dev/null || true hides API errors.
  • dispatch.py: the action argument is unused, and preview_label is read but never emitted.
  • Untracked __pycache__ files are still in the worktree (now ignored, fine). python3 scripts/dispatch/test/test_dispatch.py fails (ModuleNotFoundError); only python -m unittest works, so document the command or fix the import.

🧪 Tests

  • test_dispatch.py (6 tests) covers neither scope_script, nor custom labels, nor enabled = false combined with a label, nor invalid config, nor multi-line values.
  • dispatch-tests.yml only checks the open output. Nothing exercises the unrelated, closed or debounce paths, and unlabeled is not in its types.
  • runtests.jl: the new Dispatcher.yml tests cover the happy path only. B1 (a later unrelated run) and B2 (a cancelled run followed by a new run) are missing.
  • The 184,338 passing assertions are mostly the existing suite. They show that nothing regressed, not that the new paths are correct.

Verdict

Request changes. Fix B1–B3 and M1–M2 before the pilot on OptimalControlProblems.jl. Otherwise the pilot will reproduce exactly the problems #144 is meant to fix (spurious "not ready", lost runner time). B2 needs a human design decision (full re-run vs. per-label aggregation in ready.jl). I suggest settling it in #144 before changing any code.

@ocots
ocots marked this pull request as draft November 10, 2026 20:55
@ocots

ocots commented Nov 10, 2026

Copy link
Copy Markdown
Member Author

Fixes applied addressing the critical review

Commit: 35d8cb2

1. Blockers resolved:

  • B1 (Unrelated label invalidating ready):
    • Updated scripts/release/ready.jl for Dispatcher.yml to aggregate job outcomes across non-skipped runs on the head commit. A subsequent run where only dispatch ran (or where jobs were skipped) does not erase previous successful jobs on the same commit.
    • Added caller-level if: on the dispatch job in docs/workflows/dispatch.md to prevent starting VM runners on unrelated label events.
    • Added unit test in runtests.jl verifying that an unrelated label run does not invalidate readiness.
  • B2 (Re-running all suites on every label):
    • ready.jl now merges completed job results across multiple Dispatcher.yml runs on the same head SHA. If GPU CI passed in run 1 and documentation in run 2, all required jobs are recognized as passed.
    • Added unit test in runtests.jl verifying split multi-run job aggregation.
  • B3 (edited canceling running CI):
    • Removed edited from types in docs/workflows/dispatch.md. Only [opened, labeled, unlabeled, synchronize, reopened] trigger PR dispatch.

2. Major issues resolved:

  • M1 & M2 (Label knowledge duplication, defaults & errors):
    • scripts/release/changes.jl now evaluates .github/ci.toml with the full defaults matching dispatch.py (enabled/disabled sections, default and custom labels) without swallowing errors.
    • Removed obsolete run_probe from DISPATCH_OUTPUT_LABELS.
    • scripts/release/gate.jl inspects whether documentation is configured rather than hardcoding "run documentation".
    • Added unit tests in runtests.jl for run_labels with ci.toml.
  • M3 (ci.toml validation):
    • Implemented strict schema and type validation in dispatch.py (validate_config). Unknown keys (e.g. typos like gihub) and type mismatches immediately exit with clear ::error:: messages.
    • Added unit tests test_invalid_config_unknown_key and test_invalid_config_wrong_type.
  • M4 (Runner allocation on unrelated labels):
    • Documented caller job-level if: so GitHub Actions skips runner allocation completely (0s run time, 0 runner minutes billed).
  • M5 (Multiline output formatting):
    • write_github_output in dispatch.py uses randomized EOF delimiters (secrets.token_hex) for multiline strings, preventing output file corruption.
    • Added unit test test_multiline_output_delimiter.
  • M6 (Scope script failure handling):
    • run_scope_script now raises on test or script failure rather than warning, preventing erroneous fallback.
    • Fixed direct execution of test_dispatch.py without requiring -m unittest.

Check results:

  • scripts/dispatch/test/test_dispatch.py: 11 unit tests passing (100%).
  • scripts/release/test/runtests.jl: 184,342 tests passing, 0 failures.
  • actionlint: 0 errors/warnings.

@ocots

ocots commented Nov 10, 2026

Copy link
Copy Markdown
Member Author

Closing without merging. The single Dispatcher.yml uses concurrency: pr-dispatch-<PR> with cancel-in-progress: true, so adding a label (e.g. run documentation) 20 minutes after run ci github-runner cancels the CI in progress and restarts everything. That is a regression: today, the per-label if: guards already keep the running CI alive. The 5 s debounce only groups labels added within 5 s.

The actual problem (runs hidden on the PR page and ~30 skipped jobs per label) is tracked in #149, with a different approach (label → workflow_dispatch). The .github/ci.toml and centralised-labels ideas are kept for later in #150.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant