Skip to content

hermes-workspace auto-spawns its own gateway on boot, colliding with a systemd-managed hermes-gateway.service (real spawn-collision, not the false-alarm case in #487) #788

Description

@hbigot1970

Problem

On a Linux/systemd deployment (hermes-agent installed via the official installer, hermes-gateway.service managed as its own systemd unit — the pattern the docs recommend), hermes-workspace.service (running pnpm devvite dev) races the systemd-managed gateway at every boot:

  1. hermes-workspace.service has After=hermes-gateway.service — ordering only, no health-gate.
  2. On start, vite.config.ts's startClaudeAgent() checks http://127.0.0.1:8642/health. This fails because the real gateway isn't warm yet.
  3. resolveClaudeBinary() (vite.config.ts ~line 55) resolves ~/.hermes/hermes-agent/venv/bin/hermes — the actual installer path — so the spawn path IS reached (unlike the false-alarm in feat: env flag to coexist with an externally-managed Hermes gateway (skip auto-spawn) #487, where the reporter's resolver returned null on their setup).
  4. startClaudeAgent() spawns hermes gateway run as a child of the vite process, which grabs ~/.hermes/gateway.pid / gateway.lock and binds the gateway port first.
  5. Every subsequent start attempt by the systemd-managed hermes-gateway.service unit now fails immediately with ❌ Gateway already running (PID <vite-spawned-child>), exit code 1. systemd's Restart=always retries every ~15-25s, forever — a genuine crash-loop (observed: 120+ restarts, still looping 45+ minutes after boot when investigated), each failure firing an OnFailure alert.

Impact: the workspace-spawned gateway instance does work (it answers on the port), so end users don't see an outage — but the systemd unit that's supposed to be the source of truth for the service never actually starts, spams failure alerts indefinitely, and the "real" gateway is an unmanaged child process tied to the workspace's vite server lifecycle instead of a supervised systemd service (no proper restart-on-crash, no clean shutdown ordering, no correct process supervision from systemd's point of view).

What's already in the code

There already IS an escape hatch: HERMES_WORKSPACE_AUTO_START_AGENT=false (checked at vite.config.ts, guarding the startClaudeAgent() call). But:

  • It's not documented anywhere — absent from both .env and .env.example.
  • Its only code comment scopes it to launchd (macOS): // Skip when launchd manages the gateway (HERMES_WORKSPACE_AUTO_START_AGENT=false) to avoid SIGTERM cycle on close that nukes the launchd-managed process. Nothing mentions systemd/Linux, so a Linux user has no reason to discover or set it.

Suggested fix

  1. Document HERMES_WORKSPACE_AUTO_START_AGENT=false in .env.example and the systemd-deployment section of the README, generalizing the comment beyond launchd ("set this if any external process manager — launchd, systemd, Docker, etc. — already supervises the gateway").
  2. Consider a safer default for the race itself: instead of spawning immediately on the first failed health check, retry the health check a few times with backoff before falling back to auto-spawn — this alone would close the boot-order race for systemd deployments using After= (ordering-only) without requiring any manual env var.

Repro environment

  • hermes-agent installed via the official installer (binary at ~/.hermes/hermes-agent/venv/bin/hermes)
  • hermes-gateway.service and hermes-workspace.service both managed as systemd system units, hermes-workspace.service has After=hermes-gateway.service (ordering only)
  • Confirmed via journalctl -u hermes-gateway.service: repeated ❌ Gateway already running (PID <n>) / Main process exited, code=exited, status=1/FAILURE cycling every 15-25s from boot until the externally-spawned process was manually stopped
  • Confirmed via ps: the PID holding the gateway port/lock had a vite dev process (spawned by hermes-workspace.service's pnpm dev) as an ancestor, not systemd (PID 1)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions