hpc exec runs its command on the login node (via ssh ... bash -s), outside the scheduler. A long-running exec is not the intended use: it is for short commands (setup, inspection), and even the longest legitimate case — a dependency sync or a small build — finishes within minutes. Nothing healthy needs to run unbounded there. What is worth guarding against is the pathological case: a command that never returns (waits on stdin, infinite loop, deadlock) and sits on a shared login node indefinitely.
hpc's other operations are eager and self-bounding, so exec is the one path that can linger. A default wall-clock cap leaves normal use untouched while keeping a never-returns command from occupying the login node forever. A deliberately long exec stays possible by raising the cap explicitly — and owning that choice.
Proposal
- Apply a default wall-clock timeout of 1800s (30 min) to
hpc exec.
- Enforce it remotely by wrapping the remote invocation in coreutils
timeout (e.g. timeout --kill-after=10s <N>s bash -s) rather than relying on a local subprocess timeout — so the cap holds on the login node even if the local client disconnects.
- Scope to
exec only. submit is unaffected (the scheduler enforces its own limits); job-output --follow (tail -F) is unaffected (it is intentionally long-lived).
- Overrides:
--timeout <sec> per invocation, with 0 to disable; optionally an exec.timeout key in hpc.toml for a per-project default.
- On timeout, exit with a clear message naming the cap and the override flag.
Why 30 min
The cap's purpose is to bound the "ran forever / forgotten" tail, not to throttle load. 30 min sits well above any healthy login-node command: routine setup such as a dependency sync is seconds to a minute, and even the slowest legitimate cases — compiling a native extension from source, or fetching a large dataset — rarely approach it. A too-tight default would just get disabled wholesale, which is the worse outcome.
Open implementation questions
- Process-tree kill depth.
timeout signals only its direct child (bash); grandchildren placed in a new session can survive. Decide whether to setsid and signal the process group for a full-tree kill, or accept direct-child termination for the common single-runaway case.
timeout availability. It is part of GNU coreutils and near-universal on Linux HPC, but confirm a graceful fallback (or a documented requirement) for environments without it.
hpc execruns its command on the login node (viassh ... bash -s), outside the scheduler. A long-runningexecis not the intended use: it is for short commands (setup, inspection), and even the longest legitimate case — a dependency sync or a small build — finishes within minutes. Nothing healthy needs to run unbounded there. What is worth guarding against is the pathological case: a command that never returns (waits on stdin, infinite loop, deadlock) and sits on a shared login node indefinitely.hpc's other operations are eager and self-bounding, so
execis the one path that can linger. A default wall-clock cap leaves normal use untouched while keeping a never-returns command from occupying the login node forever. A deliberately longexecstays possible by raising the cap explicitly — and owning that choice.Proposal
hpc exec.timeout(e.g.timeout --kill-after=10s <N>s bash -s) rather than relying on a localsubprocesstimeout — so the cap holds on the login node even if the local client disconnects.execonly.submitis unaffected (the scheduler enforces its own limits);job-output --follow(tail -F) is unaffected (it is intentionally long-lived).--timeout <sec>per invocation, with0to disable; optionally anexec.timeoutkey inhpc.tomlfor a per-project default.Why 30 min
The cap's purpose is to bound the "ran forever / forgotten" tail, not to throttle load. 30 min sits well above any healthy login-node command: routine setup such as a dependency sync is seconds to a minute, and even the slowest legitimate cases — compiling a native extension from source, or fetching a large dataset — rarely approach it. A too-tight default would just get disabled wholesale, which is the worse outcome.
Open implementation questions
timeoutsignals only its direct child (bash); grandchildren placed in a new session can survive. Decide whether tosetsidand signal the process group for a full-tree kill, or accept direct-child termination for the common single-runaway case.timeoutavailability. It is part of GNU coreutils and near-universal on Linux HPC, but confirm a graceful fallback (or a documented requirement) for environments without it.