Problem: runner leaks TCP connections — newHTTPClient() on every poll cycle exhausts ephemeral ports

[BK0STAR]

Issue

Issue

The remote runner leaks TCP connections to the server until it exhausts all ephemeral source ports, after which it can no longer poll for jobs and all tasks remain stuck in waiting.

Root cause (in services/runners/job_pool.go):

sendProgress() and checkNewJobs() each call newHTTPClient() on every invocation, and both are invoked every second by the requestTimer ticker in JobPool.Run():

func (p *JobPool) sendProgress() (ok bool) {
      client := newHTTPClient()   // new http.Transport on every call
      ...
}

func (p *JobPool) checkNewJobs() {
      ...
      client := newHTTPClient()   // new http.Transport on every call
      ...
}

Each newHTTPClient() creates a fresh http.Transport with its own keep-alive connection pool. The connection is therefore never reused across poll cycles, and the idle connection is never closed — it stays
ESTABLISHED inside an abandoned transport. At ~2 requests/second this accumulates roughly 2 new connections per second.

Observed impact (Pro v2.18.8, Kubernetes/OpenShift, remote runners):

After ~12h of uptime, each runner pod held ~28,200 ESTABLISHED connections to the server service:

$ netstat -tn | awk '{print $6}' | sort | uniq -c
  28232 ESTABLISHED

all pointing to the server:

  28232 172.30.54.155:3000

At that point every new outbound dial fails and the runner logs, twice per second:

level=error msg="Put \"http://semaphore:3000/api/internal/runners\": dial tcp 172.30.54.155:3000: connect: cannot assign requested address" action="send request" context=sending_progress
level=error msg="Get \"http://semaphore:3000/api/internal/runners\": dial tcp 172.30.54.155:3000: connect: cannot assign requested address" action="send request" context="checking new jobs"

Tasks dispatched to the runner stay in waiting forever. The accumulated connections (TCP buffers + transports held by GC) also inflate memory usage over time — we initially hit OOMKills at a 768Mi limit
before identifying this leak.

Aggravating detail: in checkNewJobs(), defer resp.Body.Close() is placed after the early return for resp.StatusCode >= 400, so in error mode response bodies leak as well.

Suggested fix

Create the http.Client once (e.g. a field on JobPool, or a package-level lazily-initialized client) and reuse it in sendProgress(), checkNewJobs(), tryRegisterRunner() and Unregister(). The default
http.Transport keep-alive pool will then reuse a single connection instead of leaking one per request. Also move defer resp.Body.Close() before the status-code check in checkNewJobs().

Workaround

Kubernetes liveness probe that restarts the runner pod before port exhaustion:

livenessProbe:
  exec:
    command:
      - sh
      - -c
      - '[ "$(netstat -tn 2>/dev/null | grep -c ESTABLISHED)" -lt 5000 ]'
  initialDelaySeconds: 120
  periodSeconds: 60
  failureThreshold: 3

Impact

Ansible (task execution)

Installation method

Kubernetes

Database

Postgres

Semaphore Version

Pro v2.18.8 (server and runner) — bug still present in current develop source

Additional information

Two independent runner pods on the same cluster exhibited identical behavior (28,232 and 28,234 connections respectively after ~12h).

Impact

Configuration

Installation method

Docker

Database

No response

Browser

No response

Semaphore Version

Pro v2.18.8

Ansible Version

Logs & errors

No response

Manual installation - system information

No response

Configuration

No response

Additional information

No response

[thenorad]

The same issue. Semaphore runner can't connect to the Semaphore UI after ~12 hours after start. If I restart runner container, it works for the same time.

I've added healthcheck to the runner container and watchdog that restart container if it lost connection to the semaphore ui. Just as workaround.

services:
    semaphore:
        ports:
            - 127.0.0.1:3000:3000
        image: semaphoreui/semaphore:latest
        restart: unless-stopped
        environment:
            SEMAPHORE_DB_DIALECT: sqlite
            SEMAPHORE_ADMIN: admin
            SEMAPHORE_ADMIN_PASSWORD: XXXYYY
            SEMAPHORE_ADMIN_NAME: Admin
            SEMAPHORE_ADMIN_EMAIL: admin@example.com
            SEMAPHORE_USE_REMOTE_RUNNER: "True"
            SEMAPHORE_SCHEDULE_TIMEZONE: "UTC"
            SEMAPHORE_COOKIE_HASH: "XXXYYY"
            SEMAPHORE_COOKIE_ENCRYPTION: "XXXYYY"
            SEMAPHORE_ACCESS_KEY_ENCRYPTION: "XXXYYY"
            SEMAPHORE_TOTP_ENABLED: "True"
            SEMAPHORE_TOTP_ALLOW_RECOVERY: "True"
            SEMAPHORE_TELEGRAM_TOKEN: "XXXYYY"
            SEMAPHORE_TELEGRAM_ALERT: "1"
        volumes:
            - /opt/semaphore/storage/data:/var/lib/semaphore
            - /opt/semaphore/storage/config:/etc/semaphore
            - /opt/semaphore/storage/tmp:/tmp/semaphore

    runner:
        build:
          context: .
          dockerfile: Dockerfile.runner
        #image: semaphoreui/runner:v2.18.10
        restart: unless-stopped
        depends_on:
            - semaphore
        environment:
            SEMAPHORE_WEB_ROOT: "http://semaphore:3000"
            SEMAPHORE_RUNNER_TOKEN: "XXXYYY"
            SEMAPHORE_RUNNER_PRIVATE_KEY_FILE: "/config.runner.key"
        volumes:
            - /opt/semaphore/storage/runner/config.runner.key:/config.runner.key
            - /var/run/docker.sock:/var/run/docker.sock
            - /opt/teleport/system/bin/tsh:/usr/local/bin/tsh:ro
            - /opt/teleport/system/bin/tbot:/usr/local/bin/tbot:ro
            - /opt/tbot/storage:/opt/tbot/storage:ro
        healthcheck:
          test: ["CMD-SHELL", "curl -s -o /dev/null -w '%{http_code}' http://semaphore:3000/api/internal/runners | grep -Eq '^(200|401)$'"]
          interval: 30s
          timeout: 5s
          retries: 3
          start_period: 30s

    autoheal:
      image: willfarrell/autoheal:latest
      restart: unless-stopped
      environment:
        - AUTOHEAL_CONTAINER_LABEL=all
        - AUTOHEAL_INTERVAL=30
        - AUTOHEAL_START_PERIOD=60
        - AUTOHEAL_DEFAULT_STOP_TIMEOUT=10
      volumes:
        - /var/run/docker.sock:/var/run/docker.sock

[z-xavier]

Docker Compose users, I am using a workaround that is closer to the Kubernetes livenessProbe example from the issue.

The Kubernetes workaround works because a failing liveness probe causes kubelet to restart the runner container:

  livenessProbe:
    exec:
      command:
        - sh
        - -c
        - '[ "$(netstat -tn 2>/dev/null | grep -c ESTABLISHED)" -lt 5000 ]'
    initialDelaySeconds: 120
    periodSeconds: 60
    failureThreshold: 3

Docker Compose healthchecks are different: an unhealthy healthcheck only marks the container as unhealthy; Compose does not automatically restart the container because of that. So in Compose I made the healthcheck perform the same liveness-
style check, but when the threshold is exceeded it terminates the runner process. Then the normal restart: unless-stopped policy starts the container again.

Minimal example:

  services:
    semaphore:
      image: semaphoreui/semaphore:latest
      restart: unless-stopped
      environment:
        SEMAPHORE_USE_REMOTE_RUNNER: "True"
        # other Semaphore settings omitted
      healthcheck:
        test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://127.0.0.1:3000"]
        interval: 10s
        timeout: 5s
        retries: 3

    semaphore-runner:
      image: semaphoreui/runner:latest
      restart: unless-stopped
      environment:
        SEMAPHORE_WEB_ROOT: http://semaphore:3000
        SEMAPHORE_RUNNER_TOKEN: "<redacted>"
        SEMAPHORE_RUNNER_PRIVATE_KEY_FILE: /var/lib/semaphore/runner.key
      depends_on:
        semaphore:
          condition: service_healthy
      healthcheck:
        test:
          - CMD-SHELL
          - >
            established="$$(awk 'NR>1 && $$4 == "01" {n++} END {print n+0}' /proc/net/tcp)";
            if [ "$$established" -lt 5000 ]; then
              exit 0;
            fi;
            echo "too many ESTABLISHED tcp sockets: $$established" >&2;
            runner_pid="$$(ps -o pid,comm,args | awk 'NR>1 && $$2 == "semaphore" && $$0 ~ /runner start/ {print $$1; exit}')";
            if [ -z "$$runner_pid" ]; then
              echo "no semaphore runner process found" >&2;
              exit 1;
            fi;
            kill -TERM "$$runner_pid";
            sleep 30;
            kill -0 "$$runner_pid" 2>/dev/null && kill -KILL "$$runner_pid";
            exit 1
        interval: 60s
        timeout: 45s
        retries: 3
        start_period: 2m

This is still only a workaround for the TCP connection leak, not a fix for the underlying runner bug. The threshold can be adjusted per environment. I used /proc/net/tcp instead of netstat so the check does not depend on extra packages being installed in the runner image.