feat(observability): collect host scheduler-pressure metrics (PSI, procs, reclaim, per-NUMA)

[nv-yna]

Problem

Tachometer is on by default for every run, but its host-side scrape surface omits the
scheduler-pressure signal set that the retired steady_probe.sh sampler carried. The
node_exporter launch enabled only cpu/infiniband/meminfo, and host_sampler.py
read /proc/stat + /proc/meminfo but no PSI. Net result: none of these were
collected anywhere —

• PSI stall time (/proc/pressure/{cpu,memory,io})
• procs_running / procs_blocked / context-switch counters (/proc/stat)
• major-fault and page-reclaim counters (/proc/vmstat: pgmajfault, pgsteal_*)
• per-NUMA free memory (/sys/devices/system/node/*/meminfo)

This is exactly the class of signal that distinguishes "the box is busy" from "real
work is blocked waiting for a resource" — the question a host sampler exists to answer.

Change

Two complementary, minimal changes. Both are cheap procfs/sysfs reads with no measured
decode-latency cost (unlike dense NVML sampling, which already has a cadence guard).

1. node_exporter (all worker nodes) — add the stat, vmstat, pressure, and
   meminfo_numa collectors, and widen --collector.vmstat.fields to include pgsteal
   (node_exporter's default vmstat field regex ships pgmajfault but not reclaim). The
   command is extracted into tachometer_node_exporter_command_template() mirroring the
   existing DCGM template; an explicit recipe node_exporter.command still wins via
   resolve_exporter_command. The vendored NodeExporterFilter passes the new families
   through its default arm, so no scraper/Rust change is needed.

2. host_sampler (orchestrator/head node) — add a /proc/pressure/{cpu,memory,io} PSI
   read to each sample. The exporters are launched on backend nodes only, so the sampler
   is the orchestrator node's only PSI source; this covers the frontend node the exporters
   never reach.

Maps the probe's host columns 1:1: procs_running/blocked → node_procs_running/blocked,
ctxt → node_context_switches_total, pgmajfault/pgsteal → node_vmstat_*,
numa*_free → node_memory_numa_MemFree{node=N}, PSI → node_pressure_* / sampler psi.

Verification — real run on hecate (aarch64)

Ran the pinned image quay.io/prometheus/node-exporter:v1.8.2 with the exact generated
flags via pyxis on a live hecate node (kernel 6.17.0-1029-nvidia-64k), scraped from the
host side:

family	value	status
node_procs_running	3	✅
node_context_switches_total	1.29e10	✅
node_vmstat_pgmajfault	9.07e6	✅
node_vmstat_pgsteal_kswapd	5.00e6	✅ (override works)
node_memory_numa_MemFree{node=0}	6.22e11	✅
node_pressure_cpu_waiting_*	—	⚠️ absent

Kernel-gated PSI, discovered by this real run: hecate's kernel is built without
CONFIG_PSI — /proc/pressure does not exist — so the pressure collector and the
sampler's PSI read yield nothing there. Both paths degrade gracefully (node_exporter omits
the family; _pressure() returns {} and the psi key is empty). On PSI-enabled kernels
(verified on a workstation with the same image + flags) all node_pressure_* families and
the sampler psi block populate. This is documented in-code so nobody expects PSI columns
from a hecate run.

Full evidence: tmp_context/srt-slurm/20260909-tachometer-host-metrics/VERIFICATION.md.

Tests

make-level unit tests (9 new/adjacent green; full test_telemetry.py = 69 green):

• node_exporter template asserts the four collectors, the pgsteal field override, and the
  explicit-command override still winning;
• host_sampler._pressure() parse of PSI totals and the CONFIG_PSI-absent → {} case.

Out of scope (noted for reviewers)

• Per-process top-CPU introspection (the probe's top1_comm/top_pid/top_threads/
  vol+nonvol_ctxt) is already covered by host_sampler._proc_sample() when
  observability.enabled — on the orchestrator node.
• Extending exporter placement to frontend/client nodes on dedicated placement is a separate
  orchestration change.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 95.00000% with 1 line in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@e60c64a). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
src/srtctl/analysis/host_sampler.py	93.33%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #415   +/-   ##
=======================================
  Coverage        ?   73.88%           
=======================================
  Files           ?      101           
  Lines           ?    14128           
  Branches        ?        0           
=======================================
  Hits            ?    10438           
  Misses          ?     3690           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.