feat(scale-down): opt-in idle confirmation window before terminating not-busy runners

[jensenbox]

Problem

Closes the failure mode reported in #5085, including the harder variant discussed in its comments: GitHub's runner busy flag is not a reliable input for termination decisions. We traced 14 terminated-mid-job runners across one incident window (org-level, non-ephemeral, on-demand, module v7.10.0, scale-down on the default 5-minute schedule) and every one was killed through removeRunner's normal path on a busy: false reading, in two distinct shapes:

Variant A — assignment lag. The flag reads false for 25–60+ seconds after a job is already running on the runner:

job started 23:44:18 ... Busy: false at 23:45:18  → terminated → job killed
job started 23:44:23 ... Busy: false at 23:45:15  → terminated → job killed

Variant B — stale flag mid-job. The flag flips to false on a runner that has been continuously executing one job for many minutes:

03:05:10  Runner 'i-0a014...' - Busy: true   → correctly skipped
03:10:12  Runner 'i-0a014...' - Busy: false  → terminated; job died at 98% complete, all tests passing

Another case read false 12m15s into a running job. Because the DELETE call also succeeds in these cases (GitHub's server-side view is consistent with the wrong flag), no re-check-after-deregister sequencing can prevent Variant B — the module needs evidence across time instead of a single reading. With scale-to-zero config (idle_config = []) every instance past minimum_running_time_in_minutes is evaluated on every tick, so one stale-flag window kills several runners in the same invocation — we measured roughly one multi-runner burst per hour under CI load, each costing a full matrix rerun.

Change

Adds an opt-in confirmation window, default off (scale_down_idle_confirmation_seconds = 0 keeps today's single-reading behaviour bit-for-bit):

• On a not-busy reading, scale-down tags the instance ghr:idle_detected_at=<ISO8601> and defers termination.
• It terminates only when not-busy readings span at least the configured window (set it to one scale-down schedule interval to require two consecutive not-busy evaluations).
• Any busy reading clears the tag and resets the window — this is what saves the Variant B runners, which read true on the tick before their fatal stale reading.
• An unparsable tag value restarts the window rather than blocking or terminating.
• Each invocation logs a census (evaluated N runner(s): busy=… idle-deferred=… terminated=…) so "ran and found nothing" is distinguishable from "never ran".

No new IAM: the scale-down role already carries scoped ec2:CreateTags/DeleteTags for ghr:orphan. Threaded through the root module and multi-runner (runner_config.scale_down_idle_confirmation_seconds). Existing behaviours unchanged: bypass-removal, orphan handling, and the "only terminate EC2 after successful de-registration" guard.

Cost trade-off: a genuinely idle runner lives one extra evaluation interval before reap (~5 minutes on the default schedule).

Validation

• Unit tests: 5 new cases (first-reading deferral, in-window deferral, window-elapsed termination, busy-reading reset, unparsable-tag restart); full control-plane suite green on main (542 tests), eslint + prettier clean.
• Production: we have run this patch on our fleet (both pools, 300s window, 5-minute schedule) since 2026-07-28 05:32 UTC. Prior baseline: ~1 kill burst/hour. Since deploy: zero mid-job terminations in 14 hours of CI load, ~35 terminations all via the confirmed-idle path, zero de-registration failures. We directly observed the mechanism catching a would-have-been kill: a runner deferred on a false reading reported busy: true five minutes later and survived.

Happy to adjust naming/defaults or split the census logging out if you'd prefer a narrower diff. cc @npwolf / @ben-smyth from #5085 — this covers Variant B from your reports, which the deregister-then-recheck proposal cannot.

[edersonbrilhante]

@jensenbox Can you rebase? Heads up, I am working to refactor the module to introduce plugin layout. Ref: #5234

[jensenbox]

Rebased onto current main and ported to the compute-provider plugin framework from #5234. Sorry for the delay — the refactor landed the day after your rebase request, so this needed a port rather than a rebase.

Not a breaking interface change

The feature needs to persist "when was this runner first seen idle" across scale-down invocations. Before #5234 it tagged EC2 directly from the orchestration code, which is no longer the right layer.

ScaleDownComputeProvider now has markIdle / unmarkIdle, and RunnerInfo carries idleDetectedAt. Both methods are optional, deliberately:

• a provider with nowhere to persist per-runner state stays type-valid against the interface — nothing existing has to change
• scale-down checks for the capability and, when it is absent, skips the confirmation window entirely and keeps the previous single-reading behaviour rather than failing or deferring forever
• only a provider implementing them opts into the feature

aws/ec2 implements both with instance tags (ghr:idle_detected_at) — the same mechanism ghr:orphan already uses, so no new state store. The scaffold in templates/provider documents both as optional so plugin authors see them.

If you would rather express this as a separate capability interface than two optional methods on ScaleDownComputeProvider, say the word — that is a small change and I would rather match your intent for the framework than guess.

Why the feature

GitHub's busy flag can be stale. It reads false for runners actively executing a job, both shortly after assignment (25-60s lag) and deep into a running job (12+ minutes) — see #5085. A single busy=false reading is not sufficient evidence a runner is idle, so scale-down can terminate mid-job.

SCALE_DOWN_IDLE_CONFIRMATION_SECONDS (default 0, i.e. current behaviour) requires busy=false readings spanning at least that window before terminating. Any busy=true reading clears the marker and restarts the window.

Verification

• 265 passed across the full scale-runners suite, 7 files
• control-plane build clean, 0 TypeScript errors
• 5 new tests: window start, deferral mid-window, elapse-then-terminate, the disabled (0) path, and a provider implementing neither method
• I checked the new tests actually bite rather than assuming: stubbing idleConfirmed to always confirm fails exactly the window-start and deferral cases

Terraform plumbing (scale_down_idle_confirmation_seconds) is unchanged from the original PR and carried through the rebase cleanly.