xprobe engineering rules

Sources of truth

Code, tests, and checked-in JSON schemas define implemented behavior.
README.md and docs/ describe released behavior. Rewrite stale sections in place when behavior changes; do not append historical caveats or parallel inventories.
Treat user reports and external reviews as evidence to investigate, not as implemented fact. Verify claims about bounds, clocks, filtering, lifecycle, and errors against code and tests before documenting or changing them.

The public CLI is exactly doctor, discover, validate, and measure.
Keep public contracts in xprobe/protocol, deterministic Linux logic in xprobe/core, collection in xprobe/collector, matching in xprobe/correlator, export in xprobe/exporter, and orchestration in the CLI.
Do not add a daemon or persistent session service without a concrete workflow that cannot be expressed by the four bounded commands.
Keep eBPF and CUPTI callback hot paths bounded and free of blocking I/O.
Resolve architecture-specific syscall names in core. In BPF, filter process identity and syscall number before reading scalar registers or reserving a record. Arm multi-link collectors only after every link is attached.

Return malformed data, unexpected I/O, command failures, and cleanup failures as explicit errors. Do not replace them with defaults or partial success.
Identify a process by PID plus procfs start time and verify it around attach.
Every attach and injection path must restore target state on failure and have deterministic cleanup tests.
validate is read-only. measure may inject CUPTI only after validation reports injection_required; log the mutation and include a JSON warning.
Stop CUPTI logically after collection. Do not dlclose the injected agent.
NVTX callback routing must be initialized before the target's first NVTX API call. Once initialized, keep the CUPTI subscriber mapped and registered; logical stop disables callback domains instead of unsubscribing it.
Correlate NVTX IDs as ordered lifecycles. Thread range keys reuse push levels, and a bounded capture may end with one range open; retain earlier closed pairs and report only the trailing start as unmatched.
Never collect pointer-referenced payloads, environments, or GPU buffer data by default. Named tracepoints retain identity and timestamps unless a versioned scalar payload is explicitly designed. Never describe temporal correlation as exact causality.

Follow skills/xprobe-measure-latency/SKILL.md for measurement tasks.
Choose the shortest Skill route supported by the evidence. Existing artifacts do not require live readiness checks; known selectors may proceed directly to validation and bounded measurement. Use broad-to-narrow inventory only when selectors are unknown. Classify live work as CPU-only or GPU/mixed before choosing collectors; discover is only for CUDA context holders. Scope breadth and capture duration are independent. Do not guess selectors.
Read evidence pairs, artifact analysis, stream identity, collection quality, and profiler overhead before interpreting latency. Summed concurrent GPU duration is not wall time.
Keep reusable analysis deterministic and bundled with the Skill. Keep model orchestration in the caller rather than adding CLI commands or a service.

Start post-release work on a dedicated feature or fix branch. Rebase pull requests into master; do not create merge commits.
Build Linux release artifacts on Ubuntu 22.04 and enforce a GLIBC_2.34 ceiling for the CLI and every shipped Agent ELF.
A release is complete only after downloading the public archive, checking its digest, testing installation, and inspecting all shipped ELF compatibility.
Retry CI only when logs identify a transient infrastructure failure. Preserve real compiler, test, packaging, and compatibility failures for diagnosis.

Run just fmt-check, just lint, and just test for Rust changes.
Run just test-bpf-live for BPF attachment changes.
Run just test-cupti-live for CUPTI ABI or callback changes.
Run just test-nvtx-live for NVTX callback, filtering, or lifecycle changes.
Run just test-injection-live for injection or agent lifecycle changes.
Run just test-multisource-live for host/GPU orchestration changes.
Run just test-agent-contract for CLI, schema, docs, or Skill changes.
Test bundled Skill scripts with deterministic fixtures and include them in just test-agent-contract.
Run just benchmark-gpu for callback hot-path changes.
Run just benchmark-aggregate for aggregate inventory hot-path or capacity changes.
Run just benchmark-multiprocess for concurrent worker orchestration changes.
Run just benchmark-pytorch for PyTorch workflow or framework-overhead changes.
Use emoji conventional commits, for example 🐛 fix: restore target registers.