You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
hermes-workspace auto-spawns its own gateway on boot, colliding with a systemd-managed hermes-gateway.service (real spawn-collision, not the false-alarm case in #487) #788
On a Linux/systemd deployment (hermes-agent installed via the official installer, hermes-gateway.service managed as its own systemd unit — the pattern the docs recommend), hermes-workspace.service (running pnpm dev → vite dev) races the systemd-managed gateway at every boot:
hermes-workspace.service has After=hermes-gateway.service — ordering only, no health-gate.
On start, vite.config.ts's startClaudeAgent() checks http://127.0.0.1:8642/health. This fails because the real gateway isn't warm yet.
startClaudeAgent() spawns hermes gateway run as a child of the vite process, which grabs ~/.hermes/gateway.pid / gateway.lock and binds the gateway port first.
Every subsequent start attempt by the systemd-managed hermes-gateway.service unit now fails immediately with ❌ Gateway already running (PID <vite-spawned-child>), exit code 1. systemd's Restart=always retries every ~15-25s, forever — a genuine crash-loop (observed: 120+ restarts, still looping 45+ minutes after boot when investigated), each failure firing an OnFailure alert.
Impact: the workspace-spawned gateway instance does work (it answers on the port), so end users don't see an outage — but the systemd unit that's supposed to be the source of truth for the service never actually starts, spams failure alerts indefinitely, and the "real" gateway is an unmanaged child process tied to the workspace's vite server lifecycle instead of a supervised systemd service (no proper restart-on-crash, no clean shutdown ordering, no correct process supervision from systemd's point of view).
What's already in the code
There already IS an escape hatch: HERMES_WORKSPACE_AUTO_START_AGENT=false (checked at vite.config.ts, guarding the startClaudeAgent() call). But:
It's not documented anywhere — absent from both .env and .env.example.
Its only code comment scopes it to launchd (macOS): // Skip when launchd manages the gateway (HERMES_WORKSPACE_AUTO_START_AGENT=false) to avoid SIGTERM cycle on close that nukes the launchd-managed process. Nothing mentions systemd/Linux, so a Linux user has no reason to discover or set it.
Suggested fix
Document HERMES_WORKSPACE_AUTO_START_AGENT=false in .env.example and the systemd-deployment section of the README, generalizing the comment beyond launchd ("set this if any external process manager — launchd, systemd, Docker, etc. — already supervises the gateway").
Consider a safer default for the race itself: instead of spawning immediately on the first failed health check, retry the health check a few times with backoff before falling back to auto-spawn — this alone would close the boot-order race for systemd deployments using After= (ordering-only) without requiring any manual env var.
Repro environment
hermes-agent installed via the official installer (binary at ~/.hermes/hermes-agent/venv/bin/hermes)
hermes-gateway.service and hermes-workspace.service both managed as systemd system units, hermes-workspace.service has After=hermes-gateway.service (ordering only)
Confirmed via journalctl -u hermes-gateway.service: repeated ❌ Gateway already running (PID <n>) / Main process exited, code=exited, status=1/FAILURE cycling every 15-25s from boot until the externally-spawned process was manually stopped
Confirmed via ps: the PID holding the gateway port/lock had a vite dev process (spawned by hermes-workspace.service's pnpm dev) as an ancestor, not systemd (PID 1)
Problem
On a Linux/systemd deployment (hermes-agent installed via the official installer,
hermes-gateway.servicemanaged as its own systemd unit — the pattern the docs recommend),hermes-workspace.service(runningpnpm dev→vite dev) races the systemd-managed gateway at every boot:hermes-workspace.servicehasAfter=hermes-gateway.service— ordering only, no health-gate.vite.config.ts'sstartClaudeAgent()checkshttp://127.0.0.1:8642/health. This fails because the real gateway isn't warm yet.resolveClaudeBinary()(vite.config.ts ~line 55) resolves~/.hermes/hermes-agent/venv/bin/hermes— the actual installer path — so the spawn path IS reached (unlike the false-alarm in feat: env flag to coexist with an externally-managed Hermes gateway (skip auto-spawn) #487, where the reporter's resolver returned null on their setup).startClaudeAgent()spawnshermes gateway runas a child of the vite process, which grabs~/.hermes/gateway.pid/gateway.lockand binds the gateway port first.hermes-gateway.serviceunit now fails immediately with❌ Gateway already running (PID <vite-spawned-child>), exit code 1. systemd'sRestart=alwaysretries every ~15-25s, forever — a genuine crash-loop (observed: 120+ restarts, still looping 45+ minutes after boot when investigated), each failure firing an OnFailure alert.Impact: the workspace-spawned gateway instance does work (it answers on the port), so end users don't see an outage — but the systemd unit that's supposed to be the source of truth for the service never actually starts, spams failure alerts indefinitely, and the "real" gateway is an unmanaged child process tied to the workspace's vite server lifecycle instead of a supervised systemd service (no proper restart-on-crash, no clean shutdown ordering, no correct process supervision from systemd's point of view).
What's already in the code
There already IS an escape hatch:
HERMES_WORKSPACE_AUTO_START_AGENT=false(checked at vite.config.ts, guarding thestartClaudeAgent()call). But:.envand.env.example.// Skip when launchd manages the gateway (HERMES_WORKSPACE_AUTO_START_AGENT=false) to avoid SIGTERM cycle on close that nukes the launchd-managed process.Nothing mentions systemd/Linux, so a Linux user has no reason to discover or set it.Suggested fix
HERMES_WORKSPACE_AUTO_START_AGENT=falsein.env.exampleand the systemd-deployment section of the README, generalizing the comment beyond launchd ("set this if any external process manager — launchd, systemd, Docker, etc. — already supervises the gateway").After=(ordering-only) without requiring any manual env var.Repro environment
~/.hermes/hermes-agent/venv/bin/hermes)hermes-gateway.serviceandhermes-workspace.serviceboth managed as systemd system units,hermes-workspace.servicehasAfter=hermes-gateway.service(ordering only)journalctl -u hermes-gateway.service: repeated❌ Gateway already running (PID <n>)/Main process exited, code=exited, status=1/FAILUREcycling every 15-25s from boot until the externally-spawned process was manually stoppedps: the PID holding the gateway port/lock had avite devprocess (spawned byhermes-workspace.service'spnpm dev) as an ancestor, not systemd (PID 1)