Add SIGUSR1 drain-and-exit handler to worker; enable safer auto-recycle

[mihow]

Background

Silent slow-failure modes have been observed in worker processes during sustained job ingestion. They are tracked in:

• Clarify and document tuning parameters #138 — memory creep / RSS not returning to baseline after batches
• DataLoader subprocess deadlock: MainThread hangs forever on queue.get after shared-memory unlink failure #140 — DataLoader IPC deadlock causing PID-alive but zero-progress hangs
• DataLoader pipe FD leak: ~10k pipes/day accumulate, worker hits EMFILE silently after ~6 days uptime #145 — file-descriptor / pipe leak (~10k pipes/day, EMFILE after ~6 days uptime)

Each of these is being investigated separately, but they share a property: the worker process keeps running while no longer making progress, and the supervisor sees RUNNING. The result is wasted GPU cycles, unacked NATS tasks, and downstream redelivery noise — sometimes for hours before an operator notices.

A defense-in-depth measure is to recycle the worker process periodically, so any unknown leak/hang mode becomes noise rather than an incident. Recycling is not a substitute for fixing the underlying bugs, but it is a useful floor of correctness while those fixes land.

Scope of this issue

This issue covers the worker-side change required to make recycling safe and ergonomic:

• A drain-and-exit signal handler so the worker can be told "finish the current batch, ack the message, then exit cleanly" — without losing in-flight work or leaving NATS messages unacked.

The operator-side pieces (cron schedules, health-driven scripts, container-level health checks) live in private deployment tooling and are out of scope for this issue.

A separate change in the platform repo will add --max-tasks-per-child to the Celery result-callback workers; that is a different fork-pool runtime and is not addressed here.

Proposal: SIGUSR1 = drain-and-exit

Standard Unix signal flow. Operator (or the supervisor's restart command) sends SIGUSR1, the worker sets a flag, finishes the in-flight batch, acks NATS, and exits cleanly with status 0. Whatever process supervisor is in use (supervisord, systemd, docker restart: always) respawns it with a fresh DataLoader / FD pool / RSS.

# trapdata/cli/start_worker.py (or equivalent worker main)
import signal

DRAIN_REQUESTED = False

def request_drain(signum, frame):
    global DRAIN_REQUESTED
    logger.info("SIGUSR1 received — drain requested, will exit after current batch")
    DRAIN_REQUESTED = True

signal.signal(signal.SIGUSR1, request_drain)

# main poll loop:
while not DRAIN_REQUESTED:
    msg = pull_from_nats()
    if msg is None:
        if DRAIN_REQUESTED:
            break
        continue
    process_batch(msg)
    ack(msg)

logger.info("drain complete, exiting cleanly")
sys.exit(0)

Trigger: supervisorctl signal SIGUSR1 <program>, or any other process supervisor that can deliver a signal to the PID. Falls back to existing SIGTERM behavior if not implemented (so this is a strict superset).

Alternative considered: drain flag file

A ~/.drain flag file checked at the top of the poll loop. Simpler, but introduces shared filesystem state and a cleanup race. Rejected in favor of SIGUSR1.

Why this matters now

Without the drain handler, the only way to recycle a worker is SIGTERM (immediate). That kills any in-flight batch mid-processing, leaves the NATS message unacked, and forces redelivery on another worker after ack_wait. Wasted GPU cycles + observable as num_redelivered climbing.

With the drain handler, a routine recycle is invisible from the job's perspective: the batch finishes, the message is acked, and the worker comes back fresh.

This unblocks both health-driven recycle (e.g. "log mtime > 10 min while pending > 0 → recycle this PID") and time-based rolling recycle (e.g. low-traffic daily restart) as cheap operator-side scripts.

Effort + risk

• Effort: ~2–3 hours. Single file, single signal handler, single change to the poll loop.
• Risk: low. Falls back to existing SIGTERM behavior when not used. Worst case during draft: handler missed and worker SIGKILL'd at supervisor's stopwaitsecs — same as today.

Acceptance criteria

• Worker installs a SIGUSR1 handler at startup
• On SIGUSR1, worker finishes the current batch, acks the NATS message, then exits with status 0
• If no batch is in flight, worker exits within one poll-loop iteration
• Logged at INFO when signal is received and again when drain completes
• No change to existing SIGTERM / SIGINT behavior
• Optional: stopwaitsecs recommendation noted in deploy docs (e.g. 600s) so process supervisors give the in-flight batch time to finish

Out of scope

• Cron / schedule for recycle (operator side)
• Health-driven recycle script (operator side)
• Celery worker max_tasks_per_child (separate change in the platform repo)
• Any of the underlying bugs (Clarify and document tuning parameters #138, DataLoader subprocess deadlock: MainThread hangs forever on queue.get after shared-memory unlink failure #140, DataLoader pipe FD leak: ~10k pipes/day accumulate, worker hits EMFILE silently after ~6 days uptime #145) — those remain tracked in their own issues

Related

• Clarify and document tuning parameters #138 (memory creep)
• DataLoader subprocess deadlock: MainThread hangs forever on queue.get after shared-memory unlink failure #140 (DataLoader IPC deadlock)
• DataLoader pipe FD leak: ~10k pipes/day accumulate, worker hits EMFILE silently after ~6 days uptime #145 (FD / pipe leak)

[mihow]

Claude says: Concrete use case from the 2026-05-19 OOM (see #140).

Plain-language: This issue proposes giving the worker a way to finish its current batch, ack its messages, and shut down cleanly when something asks it to recycle (instead of getting SIGKILLed mid-job). The 2026-05-19 incident is exactly the scenario: workers were killed by OOM mid-batch, held NATS messages they never acked, hit max_deliver=2, and ~201/408 images got orphaned. A drain-and-exit recycler triggered at an RSS threshold (e.g. SIGUSR1 at 60% host RAM) would have let the worker finish its batch and ack cleanly before respawn — converting a 49% failure rate into a throughput dip.

Scope: this is a compensating mitigation that runs alongside the root-cause fix in #140, not a substitute for it. Order of operations: ship #140 fix first, watch a week, then decide whether the recycler is still worth the engineering cost.