paiml · noahgift · May 23, 2026 · May 23, 2026 · May 23, 2026 · May 23, 2026
diff --git a/book/src/cli/bench.md b/book/src/cli/bench.md
@@ -18,9 +18,51 @@ apr bench [OPTIONS]
 apr bench qwen2.5-coder-1.5b-instruct-q4_k_m.gguf --iterations 10
 ```
 
-## Full help
+## What this does
 
-Run `apr bench --help` for the complete option list.
+`apr bench` measures decode throughput (tok/s) with warmup. By default it
+generates 32 tokens per iteration after 3 warmup iterations, averages, and
+reports p50/p95/p99 latencies. Use `--fast` to route through `realizar` (the
+production inference engine, 5-20x faster than the `aprender` debug path).
+The hard floor in spec H12 is 10 tok/s — anything below that is a release
+blocker.
+
+## Key flags
+
+| Flag | What it does | Example |
+|------|-------------|---------|
+| `--iterations N` | Measurement iterations (default 5) | `--iterations 20` |
+| `--warmup N` | Warmup iterations (default 3) | `--warmup 5` |
+| `--max-tokens N` | Tokens per iteration (default 32) | `--max-tokens 128` |
+| `--prompt TEXT` | Custom benchmark prompt | `--prompt "def fizzbuzz"` |
+| `--fast` | Route through realizar (production path) | `--fast` |
+| `--percentiles LIST` | Latency percentiles (default `50,95,99`) | `--percentiles 50,90,99` |
+
+## Common workflows
+
+**Production benchmark vs Ollama reference.**
+
+```bash
+apr bench qwen2.5-coder-1.5b.apr --fast --iterations 20 --max-tokens 128 --json | \
+    jq '{tok_s, p50_ms, p99_ms}'
+```
+
+**Compare CPU vs GPU on the same model.**
+
+```bash
+apr bench qwen2.5-coder-1.5b.apr --fast --iterations 10              # CUDA (auto)
+apr bench qwen2.5-coder-1.5b.apr --fast --iterations 10 --no-gpu     # Trueno SIMD
+```
+
+## Troubleshooting
+
+- **Throughput tanks across iterations** — likely thermal throttling on a laptop.
+  Watch `nvidia-smi -l 1` or `htop` for sustained clock drops.
+- **"failed to load model"** — confirm the file extension matches the format;
+  `.apr`, `.gguf`, `.safetensors` are auto-detected.
+- **Numbers don't match the spec table** — the spec table uses `--fast`
+  (realizar). Without it you're benching the slower debug path. Always pass
+  `--fast` for headline numbers.
 
 ## See also
 

diff --git a/book/src/cli/canary.md b/book/src/cli/canary.md
@@ -18,9 +18,49 @@ apr canary [OPTIONS]
 apr canary qwen2.5-coder-1.5b-instruct-q4_k_m.gguf
 ```
 
-## Full help
+## What this does
 
-Run `apr canary --help` for the complete option list.
+`apr canary` records and checks behavioral fixtures — a known input (audio,
+text, JSON) paired with the model's reference output. Use it to guard against
+silent regressions: if Qwen2.5-Coder used to translate `"def add(a, b):"` into
+exactly `"\n    return a + b\n"`, a canary captures that, and any later change
+that flips the answer fails CI.
+
+## Key flags
+
+| Subcommand | What it does | Example |
+|-----------|-------------|---------|
+| `canary create` | Record a new canary from a model + input | `--input prompt.txt --output ref.json` |
+| `canary check` | Compare model against a saved canary | `--canary ref.json` |
+| `--json` | Machine-readable output | `--json` |
+
+## Common workflows
+
+**Capture a regression fixture during golden-model bring-up.**
+
+```bash
+apr canary create qwen2.5-coder-0.5b.apr \
+    --input prompts/fizzbuzz.txt \
+    --output canaries/qwen-0.5b-fizzbuzz.json
+```
+
+**Gate every PR on the saved canary set.**
+
+```bash
+for c in canaries/*.json; do
+    apr canary check qwen2.5-coder-0.5b.apr --canary "$c" --json || exit 1
+done
+```
+
+## Troubleshooting
+
+- **Canary fails after harmless refactor** — sampler stochasticity. Re-run with
+  `--temperature 0.0 --seed 299792458` (the project's standard deterministic
+  seed); regenerate the canary if the change is intentional.
+- **`canary create` segfaults on audio input** — confirm the input file format
+  matches the model's expected modality (WAV 16kHz mono for Whisper).
+- **All canaries fail after a model swap** — that's the point. Canaries are
+  per-model fixtures; recreate them when you intentionally change weights.
 
 ## See also
 

diff --git a/book/src/cli/chat.md b/book/src/cli/chat.md
@@ -18,9 +18,51 @@ apr chat [OPTIONS]
 apr chat qwen2.5-coder-1.5b
 ```
 
-## Full help
+## What this does
 
-Run `apr chat --help` for the complete option list.
+`apr chat` opens a multi-turn REPL against a local model. Each turn is wrapped in
+the model's ChatML template, so Instruct-tuned checkpoints (Qwen, LLaMA, Mistral)
+respond conversationally instead of completing your text. Conversation history is
+kept in-process — exit the REPL and history is gone. For persisted sessions use
+`apr code --resume`.
+
+## Key flags
+
+| Flag | What it does | Example |
+|------|-------------|---------|
+| `--system MSG` | System prompt that frames every turn | `--system "You are a Rust expert."` |
+| `--temperature T` | Sampling temperature (0.7 is the default sweet spot) | `--temperature 0.2` |
+| `--top-p P` | Nucleus sampling cutoff | `--top-p 0.95` |
+| `--max-tokens N` | Cap per response (default 512) | `--max-tokens 1024` |
+| `--inspect` | Show top-k probs + tok/s per turn | `--inspect` |
+| `--backend B` | Force backend (`cuda`/`cpu`/`wgpu`) | `--backend cpu` |
+
+## Common workflows
+
+**Sandbox a system prompt before promoting it to production.**
+
+```bash
+apr chat qwen2.5-coder-7b.apr \
+    --system "Reply only with valid Python. No prose." \
+    --temperature 0.1
+```
+
+**Profile per-turn latency while you converse.**
+
+```bash
+apr chat qwen2.5-coder-1.5b.apr --inspect --backend cuda
+# After each turn you'll see: 87 tok @ 412 tok/s, ttft 38ms
+```
+
+## Troubleshooting
+
+- **Model echoes the prompt or returns gibberish** — the chat template wasn't
+  applied. `apr chat` auto-detects Instruct models but a base model will need
+  `apr run --chat` instead. Confirm with `apr inspect <model> | grep template`.
+- **CUDA OOM on a 7B model** — drop `--max-tokens` (KV cache scales linearly with
+  context), or pass `--backend cpu` to fall back to Trueno SIMD.
+- **Responses get cut mid-sentence** — bump `--max-tokens`; the default 512 is
+  conservative. For long-form generation, 1024-2048 is standard.
 
 ## See also
 

diff --git a/book/src/cli/check.md b/book/src/cli/check.md
@@ -18,9 +18,49 @@ apr check [OPTIONS]
 apr check qwen2.5-coder-1.5b-instruct-q4_k_m.gguf
 ```
 
-## Full help
+## What this does
 
-Run `apr check --help` for the complete option list.
+`apr check` is the 10-stage end-to-end pipeline self-test from APR-TRACE-001:
+tokenize, embed, RMSNorm, QKV, attention, FFN gate+up, FFN down, residual, LM
+head, sample. Each stage must produce numerically sane output (no NaN/Inf,
+within distribution band). A passing `apr check` means the model runs; it
+doesn't mean it's accurate (use `apr eval` / `apr qa` for that).
+
+## Key flags
+
+| Flag | What it does | Example |
+|------|-------------|---------|
+| `--no-gpu` | Run on CPU (Trueno SIMD) | `--no-gpu` |
+| `--json` | One JSON envelope per stage | `--json` |
+| `-v, --verbose` | Per-stage tensor stats | `--verbose` |
+
+## Common workflows
+
+**Quick post-convert sanity check.**
+
+```bash
+apr check qwen2.5-coder-0.5b.apr --json | \
+    jq '.stages[] | select(.status != "PASS")'
+```
+
+**Compare CPU and GPU pipelines stage-by-stage.**
+
+```bash
+apr check qwen2.5-coder-1.5b.apr --no-gpu --json > cpu.json
+apr check qwen2.5-coder-1.5b.apr         --json > gpu.json
+jq -s '.[0].stages - .[1].stages' cpu.json gpu.json
+```
+
+## Troubleshooting
+
+- **NaN at attention stage** — almost always a softmax numerical issue caused
+  by a missing causal mask. Cross-check with `apr trace --layer "blk.0.attn"`.
+- **"unsupported architecture"** — `apr check` needs a known dispatch table.
+  Confirm `apr inspect --json | jq .arch`; if `unknown`, add the arch to
+  the model dispatch contract.
+- **All stages pass but `apr run` is gibberish** — the 10 stages cover
+  numerical sanity, not tokenizer correctness. Run `apr tokenize <model>
+  <prompt>` to verify the tokenizer round-trips.
 
 ## See also
 

diff --git a/book/src/cli/code.md b/book/src/cli/code.md
@@ -18,9 +18,54 @@ apr code [OPTIONS]
 apr code -p "review this Python function" --max-turns 1
 ```
 
-## Full help
+## What this does
 
-Run `apr code --help` for the complete option list.
+`apr code` is the local-first coding agent — a Claude Code / Cursor analogue with
+all inference performed by `realizar` against a local model (`qwen2.5-coder-*` is
+the default sweet spot). It reads `APR.md` / `CLAUDE.md` for project context,
+supports tool use, multi-turn sessions, and emits Claude-Code-compatible traces
+for the `ccpa measure` parity harness. Used non-interactively (`-p`) it behaves
+like `claude -p`; interactively it opens a TUI loop.
+
+## Key flags
+
+| Flag | What it does | Example |
+|------|-------------|---------|
+| `-p, --print PROMPT` | Non-interactive: print response and exit | `-p "explain this diff"` |
+| `--model M` | Override default model path | `--model qwen2.5-coder-7b.apr` |
+| `--project DIR` | Project root (loads APR.md / CLAUDE.md) | `--project ../mylib` |
+| `--max-turns N` | Stop after N agent turns | `--max-turns 5` |
+| `--resume [ID]` | Resume a previous session | `--resume sess-abc123` |
+| `--output-format FMT` | `text` or `json` (Claude Code-style envelope) | `--output-format json` |
+| `--emit-trace PATH` | Write a CCPA trace JSONL | `--emit-trace run.jsonl` |
+
+## Common workflows
+
+**One-shot code review from a CI job.**
+
+```bash
+git diff HEAD~1 | apr code -p "Review this diff. Flag bugs only." \
+    --model qwen2.5-coder-7b.apr --max-turns 1 --output-format json
+```
+
+**Iterative refactor with session persistence.**
+
+```bash
+apr code --project . --model qwen2.5-coder-7b.apr
+# > rename `Foo` to `Bar` across the crate
+# (exit, then continue later)
+apr code --resume                       # picks the most recent session
+```
+
+## Troubleshooting
+
+- **`No APR.md or CLAUDE.md found`** — agent context is empty. Either add a
+  `CLAUDE.md` at the project root or pass `--project /path/with/context`.
+- **Slow first turn (30s+ to TTFT)** — that's model load + KV warm-up, not a hang.
+  Subsequent turns reuse the loaded weights. Use a smaller model
+  (`qwen2.5-coder-0.5b`) for snappier iteration.
+- **JSON output is missing `result` field** — confirm `--output-format json` is
+  passed AFTER `-p`. The non-interactive envelope is only emitted in print mode.
 
 ## See also
 

diff --git a/book/src/cli/compare-hf.md b/book/src/cli/compare-hf.md
@@ -18,9 +18,49 @@ apr compare-hf [OPTIONS]
 apr compare-hf model.apr --hf-repo Qwen/Qwen2.5-Coder-1.5B-Instruct
 ```
 
-## Full help
+## What this does
 
-Run `apr compare-hf --help` for the complete option list.
+`apr compare-hf` downloads the original SafeTensors from a HuggingFace repo and
+runs an element-wise diff against your local APR conversion. It's the gold
+standard for "did my converter actually preserve the weights" — if the maximum
+absolute difference exceeds `--threshold` (default 1e-5), the converter has a
+bug. Use this as the discharge gate for any new arch in the import pipeline.
+
+## Key flags
+
+| Flag | What it does | Example |
+|------|-------------|---------|
+| `--hf REPO` | HuggingFace repo ID | `--hf Qwen/Qwen2.5-Coder-1.5B-Instruct` |
+| `--tensor PAT` | Filter to one tensor pattern | `--tensor "lm_head"` |
+| `--threshold N` | Max allowed abs diff (default 1e-5) | `--threshold 1e-4` |
+| `--json` | JSON output with per-tensor diff stats | `--json` |
+| `-v, --verbose` | Print every tensor (not just failures) | `--verbose` |
+
+## Common workflows
+
+**Verify a fresh import.**
+
+```bash
+apr import hf://Qwen/Qwen2.5-Coder-1.5B-Instruct -o qwen.apr
+apr compare-hf qwen.apr --hf Qwen/Qwen2.5-Coder-1.5B-Instruct --json | \
+    jq '.tensors[] | select(.max_abs_diff > 1e-5)'
+```
+
+**Drill into the LM head specifically (where transpose bugs surface).**
+
+```bash
+apr compare-hf qwen.apr --hf Qwen/Qwen2.5-Coder-1.5B-Instruct \
+    --tensor "lm_head" --threshold 1e-6 --verbose
+```
+
+## Troubleshooting
+
+- **Large diff on `lm_head` only** — classic LAYOUT-001 bug. The converter
+  forgot to transpose. Fix in `crates/aprender-core/src/format/converter/`.
+- **All tensors diverge equally** — quantization mismatch. APR Q4K vs HF F16
+  will always diverge; compare F32 APR against HF SafeTensors instead.
+- **`--hf` download is slow** — set `HF_HUB_ENABLE_HF_TRANSFER=1` to use the
+  Rust-native downloader. Auth via `HF_TOKEN` for gated models.
 
 ## See also
 

diff --git a/book/src/cli/compile.md b/book/src/cli/compile.md
@@ -18,9 +18,56 @@ apr compile [OPTIONS]
 apr compile model.apr --target cuda -o compiled.apr
 ```
 
-## Full help
+## What this does
 
-Run `apr compile --help` for the complete option list.
+`apr compile` produces a single, self-contained executable with the model
+weights embedded — pass it around, double-click it, no `apr` binary required.
+Useful for demos, sovereign air-gapped deployments, and reproducible bug
+reports. Internally it statically links `realizar` and embeds the model bytes
+as a `.rodata` section. Optionally LTO + strip for the smallest possible
+binary.
+
+## Key flags
+
+| Flag | What it does | Example |
+|------|-------------|---------|
+| `-o PATH` | Output binary path | `-o ./qwen-cli` |
+| `--target TRIPLE` | Cross-compile target | `--target x86_64-unknown-linux-musl` |
+| `--quantize FMT` | Quantize before embedding | `--quantize int4` |
+| `--release` | Optimized build | `--release` |
+| `--strip` | Strip debug symbols | `--strip` |
+| `--lto` | Enable Link-Time Optimization | `--lto` |
+| `--list-targets` | Show supported targets | `--list-targets` |
+
+## Common workflows
+
+**Build a static, distributable demo binary.**
+
+```bash
+apr compile qwen2.5-coder-0.5b.apr \
+    --target x86_64-unknown-linux-musl --quantize int4 --release --strip --lto \
+    -o demo/qwen-coder
+./demo/qwen-coder "fn fizzbuzz(n: u32) {"
+```
+
+**Ship a CUDA-specific build for a known GPU farm.**
+
+```bash
+apr compile qwen2.5-coder-7b.apr --target x86_64-unknown-linux-gnu --release \
+    -o farm/qwen7b-cuda
+```
+
+## Troubleshooting
+
+- **"linker not found for target"** — cross-compilers aren't installed.
+  Install `musl-tools` (Linux) or use `rustup target add`.
+- **Output binary is huge** — `--release --strip --lto` reduces by 30-50%.
+  Beyond that, the model weights dominate; quantize harder (`--quantize int4`)
+  or use a smaller model.
+- **Static CUDA binary fails on a different driver** — CUDA isn't truly
+  static. The libcuda.so on the host must match the embedded CUDA toolkit.
+  Ship the Trueno SIMD path instead with `--no-default-features` if you
+  need real portability.
 
 ## See also