feat(container-runner): drain child before SIGTERM on engine pause

[abcxff]

No description provided.

[abcxff]

Stack for rivet-dev/actors

• #5594 feat(container-runner): drain engine sleeps unless the idle …
• #5595 feat(container-runner): jitter idle timeout to avoid teardow …
• #5585 feat(container-runner): self-sleep on repeated actor start
• #5586 feat(container-runner): sleep on startup idle timeout
• #5589 chore(container-runner): tighten comments
• #5587 feat(container-runner): exit process when the last child sto …
• #5588 feat(container-runner): drain child before SIGTERM on engine … 👈
• #5584 feat(container-runner): drain children and engine concurrent …
• #5565 fix(envoy-client): ack terminating stop commands so pegboard …
• main

Get stack: forklift get 5588
Push local edits: forklift submit
Merge when ready: forklift merge 5588

_{change yloxstwo}

[claude]

Review: drain child before SIGTERM on engine pause

Re-verified against the current commit (c7bb680981e480ca5aa874be0aae0ab7b24c1ec4, still the sole commit on this PR). Independently re-traced the logic end to end; both previously-reported issues are still present and I confirmed the same root causes by reading container-runner/src/actor.rs, main.rs, child.rs, and the relevant rivetkit-core/rivetkit lifecycle code. Posting as a fresh pass since nothing has changed to resolve them yet.

1. Registry removal now happens after the drain wait, reopening the exact race stop_child's ordering comment exists to prevent

stop_child (container-runner/src/actor.rs:39-57) documents the invariant: "Remove from the registry FIRST so the watchdog treats the exit as deliberate, then stop." That only holds because, previously, on_sleep called stop_child directly — registry removal happened synchronously before anything gave the child a chance to exit on its own.

drain_then_stop_child (actor.rs:65-82) breaks that ordering: it clones the child, waits on child.wait_exit() / sleep(drain_grace()) / exit_token().cancelled(), and only after that select resolves does it call stop_child, which is where the registry removal actually happens.

Meanwhile run() (actor.rs:229-255) — the actor's watchdog — is a separate, long-lived task that also awaits child.wait_exit() on the same ChildProcess, and it keeps running throughout the drain window (it is not torn down until after on_sleep returns). So for the entire DRAIN_GRACE window (900s by default), two independent tasks are both waiting on the same exit signal:

• If the child exits naturally during the drain (the whole point of this feature), both run() and the drain select wake at the same instant and race children().remove_async(&actor_id).
• If run() wins the race, it treats the exit as unexpected: a clean exit calls ctx.destroy(), a nonzero exit anyhow::bail!s an errored stop — while on_sleep/drain_then_stop_child is concurrently finishing and reporting the sleep as successful. An actor meant to sleep (preserving its launch spec for a later wake) can instead be destroyed or reported crashed, nondeterministically, depending on which task's poll happens to run first.
• Both paths also call release_child_port for the same port (run() at actor.rs:238, stop_child at actor.rs:48). RESERVED_PORTS is a plain scc::HashSet, so the loser's redundant release can free a port a different, concurrently-starting actor on the same instance has since reserved. The pre-bind TcpStream::connect check in ChildProcess::spawn will usually catch the resulting collision, but that's still a spurious start failure caused by this race, not a real conflict.

Suggested fix: move children().remove_async / ACTOR_CTXS.remove_async to the top of drain_then_stop_child, before the tokio::select!, so the watchdog can never win the race regardless of how long the drain takes or how the child exits. stop_child can keep its own removal calls for the direct (on_destroy) path — they become idempotent no-ops when reached via drain.

2. Docs still reference the removed RIVET_STOP_GRACE_SECS flag

--stop-grace-secs / RIVET_STOP_GRACE_SECS is removed outright in this PR (it used to back RunnerConfig::stop_grace), but two docs still describe it as a live option:

• container-runner/examples/unity-demo/Dockerfile:32 — RIVET_STOP_GRACE_SECS SIGTERM->SIGKILL grace for the child. (default 25).
• docs/content/docs/container-runner.mdx:81 — the config table still has a --stop-grace-secs / RIVET_STOP_GRACE_SECS row (whose documented default of 25 never even matched the old code default of 10, separately from this PR).

Because the flag is removed rather than deprecated, a deploy that explicitly passes --stop-grace-secs on the command line will now fail to start outright (clap errors on an unrecognized argument) instead of the value being silently ignored. Worth confirming no external deploy configs pass this flag, and updating both docs to drop it in favor of RIVET_SIGTERM_BUDGET_SECS (default 9s) and the new RIVET_DRAIN_GRACE_SECS (default 900s).

Minor / worth double-checking

• DRAIN_GRACE default (900s) vs. the engine's actual per-runner drain window: the doc comment on DRAIN_GRACE (main.rs:118-129) correctly flags that it "must fit inside the engine's per-runner drain_grace_period or a platform reclaim cuts it short," and the engine-side default for that field is 1800s (engine/packages/api-types/src/namespaces/runner_configs.rs:103), which is comfortably above the new 900s+9s+5s sleep_grace_period. But the checked-in container-runner/examples/e2e-test/configure-serverless.mjs:13 configures drain_grace_period: 30 — far below DRAIN_GRACE's default. Per engine/packages/pegboard-outbound/src/lib.rs:583-621, once that window elapses the outbound /start handling signals the actor workflow Lost, which can trigger reallocation while this runner is still mid-drain and still holding the child alive. Worth an explicit end-to-end check that realistic drain_grace_period values are always configured well above RIVET_DRAIN_GRACE_SECS in every deploy path (not just documented in a comment), since a misconfigured pair fails silently rather than loudly.
• No test coverage was added for the new drain/select logic (container-runner/tests/ only covers boot_id and input parsing today). Given issue 1 above is a genuine timing/concurrency race, even a narrow test exercising ChildProcess + the registry interaction (e.g. simulate a natural child exit mid-drain and assert run() never wins the removal race) would meaningfully guard against regressions here.
• The drain_then_stop_child / DRAIN_GRACE doc comments describe the trigger as "sleep, lost, going-away," but the Actor trait (rivetkit-rust/packages/rivetkit/src/actor.rs) and ShutdownKind (rivetkit-core/src/actor/task.rs) only have Sleep and Destroy — there is no separate "lost"/"going-away" hook, and drain_then_stop_child is only ever called from on_sleep. Presumably this refers to engine-side reasons that route into a Sleep command, but as written it reads like there are additional callers/hooks that do not exist; worth tightening the wording so a future reader does not go looking for them.

Solid idea splitting "let it finish work" from "kill it" — the registry-ordering invariant just needs to survive the split (issue 1) before this is safe to merge.