From b0568c4f61431ef218fe20290f77c50f8da256f9 Mon Sep 17 00:00:00 2001 From: Noah Gift Date: Sat, 23 May 2026 18:13:29 +0200 Subject: [PATCH 1/4] docs(book): expand walkthroughs for apr run, chat, serve, code, inspect, validate, lint, tensors, hex, tree, flow, explain, debug, trace (Refs #1902) Adds "What this does", "Key flags", "Common workflows", and "Troubleshooting" sections to 14 inference and inspection CLI chapters. Each chapter now 60-150 lines (was ~25). Keeps existing PCU header, bash example, and See-also block intact. Co-Authored-By: Claude Opus 4.7 --- book/src/cli/chat.md | 46 ++++++++++++++++++++++++++++++++++-- book/src/cli/code.md | 49 +++++++++++++++++++++++++++++++++++++-- book/src/cli/debug.md | 43 ++++++++++++++++++++++++++++++++-- book/src/cli/explain.md | 44 +++++++++++++++++++++++++++++++++-- book/src/cli/flow.md | 42 +++++++++++++++++++++++++++++++-- book/src/cli/hex.md | 46 ++++++++++++++++++++++++++++++++++-- book/src/cli/inspect.md | 47 +++++++++++++++++++++++++++++++++++-- book/src/cli/lint.md | 43 ++++++++++++++++++++++++++++++++-- book/src/cli/run.md | 50 ++++++++++++++++++++++++++++++++++++++-- book/src/cli/serve.md | 45 ++++++++++++++++++++++++++++++++++-- book/src/cli/tensors.md | 43 ++++++++++++++++++++++++++++++++-- book/src/cli/trace.md | 47 +++++++++++++++++++++++++++++++++++-- book/src/cli/tree.md | 41 ++++++++++++++++++++++++++++++-- book/src/cli/validate.md | 45 ++++++++++++++++++++++++++++++++++-- 14 files changed, 603 insertions(+), 28 deletions(-) diff --git a/book/src/cli/chat.md b/book/src/cli/chat.md index b5bf1d7e90..00d1b58122 100644 --- a/book/src/cli/chat.md +++ b/book/src/cli/chat.md @@ -18,9 +18,51 @@ apr chat [OPTIONS] apr chat qwen2.5-coder-1.5b ``` -## Full help +## What this does -Run `apr chat --help` for the complete option list. +`apr chat` opens a multi-turn REPL against a local model. Each turn is wrapped in +the model's ChatML template, so Instruct-tuned checkpoints (Qwen, LLaMA, Mistral) +respond conversationally instead of completing your text. Conversation history is +kept in-process — exit the REPL and history is gone. For persisted sessions use +`apr code --resume`. + +## Key flags + +| Flag | What it does | Example | +|------|-------------|---------| +| `--system MSG` | System prompt that frames every turn | `--system "You are a Rust expert."` | +| `--temperature T` | Sampling temperature (0.7 is the default sweet spot) | `--temperature 0.2` | +| `--top-p P` | Nucleus sampling cutoff | `--top-p 0.95` | +| `--max-tokens N` | Cap per response (default 512) | `--max-tokens 1024` | +| `--inspect` | Show top-k probs + tok/s per turn | `--inspect` | +| `--backend B` | Force backend (`cuda`/`cpu`/`wgpu`) | `--backend cpu` | + +## Common workflows + +**Sandbox a system prompt before promoting it to production.** + +```bash +apr chat qwen2.5-coder-7b.apr \ + --system "Reply only with valid Python. No prose." \ + --temperature 0.1 +``` + +**Profile per-turn latency while you converse.** + +```bash +apr chat qwen2.5-coder-1.5b.apr --inspect --backend cuda +# After each turn you'll see: 87 tok @ 412 tok/s, ttft 38ms +``` + +## Troubleshooting + +- **Model echoes the prompt or returns gibberish** — the chat template wasn't + applied. `apr chat` auto-detects Instruct models but a base model will need + `apr run --chat` instead. Confirm with `apr inspect | grep template`. +- **CUDA OOM on a 7B model** — drop `--max-tokens` (KV cache scales linearly with + context), or pass `--backend cpu` to fall back to Trueno SIMD. +- **Responses get cut mid-sentence** — bump `--max-tokens`; the default 512 is + conservative. For long-form generation, 1024-2048 is standard. ## See also diff --git a/book/src/cli/code.md b/book/src/cli/code.md index 592c20bf82..9a229c7135 100644 --- a/book/src/cli/code.md +++ b/book/src/cli/code.md @@ -18,9 +18,54 @@ apr code [OPTIONS] apr code -p "review this Python function" --max-turns 1 ``` -## Full help +## What this does -Run `apr code --help` for the complete option list. +`apr code` is the local-first coding agent — a Claude Code / Cursor analogue with +all inference performed by `realizar` against a local model (`qwen2.5-coder-*` is +the default sweet spot). It reads `APR.md` / `CLAUDE.md` for project context, +supports tool use, multi-turn sessions, and emits Claude-Code-compatible traces +for the `ccpa measure` parity harness. Used non-interactively (`-p`) it behaves +like `claude -p`; interactively it opens a TUI loop. + +## Key flags + +| Flag | What it does | Example | +|------|-------------|---------| +| `-p, --print PROMPT` | Non-interactive: print response and exit | `-p "explain this diff"` | +| `--model M` | Override default model path | `--model qwen2.5-coder-7b.apr` | +| `--project DIR` | Project root (loads APR.md / CLAUDE.md) | `--project ../mylib` | +| `--max-turns N` | Stop after N agent turns | `--max-turns 5` | +| `--resume [ID]` | Resume a previous session | `--resume sess-abc123` | +| `--output-format FMT` | `text` or `json` (Claude Code-style envelope) | `--output-format json` | +| `--emit-trace PATH` | Write a CCPA trace JSONL | `--emit-trace run.jsonl` | + +## Common workflows + +**One-shot code review from a CI job.** + +```bash +git diff HEAD~1 | apr code -p "Review this diff. Flag bugs only." \ + --model qwen2.5-coder-7b.apr --max-turns 1 --output-format json +``` + +**Iterative refactor with session persistence.** + +```bash +apr code --project . --model qwen2.5-coder-7b.apr +# > rename `Foo` to `Bar` across the crate +# (exit, then continue later) +apr code --resume # picks the most recent session +``` + +## Troubleshooting + +- **`No APR.md or CLAUDE.md found`** — agent context is empty. Either add a + `CLAUDE.md` at the project root or pass `--project /path/with/context`. +- **Slow first turn (30s+ to TTFT)** — that's model load + KV warm-up, not a hang. + Subsequent turns reuse the loaded weights. Use a smaller model + (`qwen2.5-coder-0.5b`) for snappier iteration. +- **JSON output is missing `result` field** — confirm `--output-format json` is + passed AFTER `-p`. The non-interactive envelope is only emitted in print mode. ## See also diff --git a/book/src/cli/debug.md b/book/src/cli/debug.md index e0f58e9fb6..cf2585d4d1 100644 --- a/book/src/cli/debug.md +++ b/book/src/cli/debug.md @@ -18,9 +18,48 @@ apr debug [OPTIONS] apr debug qwen2.5-coder-1.5b-instruct-q4_k_m.gguf ``` -## Full help +## What this does -Run `apr debug --help` for the complete option list. +`apr debug` is the catch-all "tell me what's in this file" command — magic bytes, +metadata, ASCII string extraction, optional hex dump. It is intentionally less +strict than `apr validate` (won't fail on warnings) and less specialized than +`apr tensors` / `apr inspect`. The `--drama` flag is the verbose, narrated +variant used for runbooks and screencasts. + +## Key flags + +| Flag | What it does | Example | +|------|-------------|---------| +| `--drama` | Theatrical narrated output | `--drama` | +| `--hex` | Include hex dump section | `--hex` | +| `--strings` | Extract ASCII strings | `--strings` | +| `--limit N` | Cap output lines (default 256) | `--limit 100` | + +## Common workflows + +**Quick triage when `apr run` fails.** + +```bash +apr debug suspect.gguf --strings | head -50 +# Look for unexpected vocab tokens, broken metadata strings, etc. +``` + +**Capture a model snapshot for a bug report.** + +```bash +apr debug qwen2.5-coder-1.5b.apr --drama --hex --limit 512 > bug-1234-debug.txt +gh issue create --title "..." --body-file bug-1234-debug.txt +``` + +## Troubleshooting + +- **Output empty for a known-good model** — most likely a permission issue; + `apr debug` will exit silently if it can't read the file. Check with + `ls -l `. +- **`--strings` returns gibberish** — the file is encrypted or compressed. + `apr debug` doesn't decrypt; use `apr decrypt` first if applicable. +- **Drama mode mangles JSON pipelines** — drama mode prints to stderr. Use + `--json` instead for any scripting use. ## See also diff --git a/book/src/cli/explain.md b/book/src/cli/explain.md index b44453fbe2..8a2ec92e56 100644 --- a/book/src/cli/explain.md +++ b/book/src/cli/explain.md @@ -18,9 +18,49 @@ apr explain [OPTIONS] apr explain qwen2.5-coder-1.5b-instruct-q4_k_m.gguf ``` -## Full help +## What this does -Run `apr explain --help` for the complete option list. +`apr explain` is the human-readable companion to the rest of the inspection +suite. Pass an error code (like `LAYOUT-001`), a model file, an architecture +family name, or a tensor — and it returns a paragraph explaining what it is, +why it matters, and where to look next. With `--kernel` it walks the kernel +dispatch pipeline (which fused kernel handles `attn_q.weight` on this backend? +why?). + +## Key flags + +| Flag | What it does | Example | +|------|-------------|---------| +| `--tensor NAME` | Explain a specific tensor | `--tensor lm_head.weight` | +| `--kernel` | Walk the kernel dispatch tree | `--kernel` | +| `--proof-status` | Per-kernel proof status from contracts | `--proof-status` | +| `-v, --verbose` | Include contract details + obligations | `--verbose` | + +## Common workflows + +**Look up an error code from a stack trace.** + +```bash +apr explain LAYOUT-001 +# "GGUF column-major data was loaded without transpose..." +``` + +**Audit which kernel will run for a given tensor on this backend.** + +```bash +apr explain qwen2.5-coder-1.5b.apr --tensor "blk.0.ffn_gate.weight" --kernel --verbose +# Shows: fused_q4k_parallel_matvec via Trueno SIMD (row-major), proof status: VERIFIED +``` + +## Troubleshooting + +- **"no such error code"** — the code may be from a downstream crate (trueno, + realizar). Cross-check with `pmat query ""`. +- **`--proof-status` reports `UNVERIFIED`** — the kernel is in the dispatch + table but its falsification suite has gaps. File a contract bump rather than + shipping. +- **Architecture name not recognized** — pass a model file directly so explain + can read `general.architecture` from metadata. ## See also diff --git a/book/src/cli/flow.md b/book/src/cli/flow.md index 796c5ac737..d327ef5408 100644 --- a/book/src/cli/flow.md +++ b/book/src/cli/flow.md @@ -18,9 +18,47 @@ apr flow [OPTIONS] apr flow qwen2.5-coder-1.5b-instruct-q4_k_m.gguf ``` -## Full help +## What this does -Run `apr flow --help` for the complete option list. +`apr flow` renders the model's data-flow graph — how a token embedding becomes +logits — as a directed graph of operators (matmul, attn, RMSNorm, SwiGLU, etc.) +rather than a tensor-storage tree (which is `apr tree`'s job). Use this to +understand WHERE in the pipeline a fused kernel will apply, or to author a +roofline plan for a new architecture. + +## Key flags + +| Flag | What it does | Example | +|------|-------------|---------| +| `--component C` | `full`, `encoder`, `decoder` | `--component decoder` | +| `--layer PAT` | Filter to specific layer pattern | `--layer "blk.0"` | +| `--json` | JSON output for graph tools | `--json` | +| `-v, --verbose` | Include per-edge tensor stats | `--verbose` | + +## Common workflows + +**Print the decoder data-flow for SwiGLU fusion analysis.** + +```bash +apr flow qwen2.5-coder-1.5b.apr --component decoder --layer "blk.0" --verbose +# Shows: rmsnorm -> attn_qkv (fused) -> attn_o -> rmsnorm -> ffn_gate+up (fused) -> ffn_down +``` + +**Export the graph for an external visualizer.** + +```bash +apr flow qwen2.5-coder-7b.apr --json | jq '.nodes | length' +``` + +## Troubleshooting + +- **`unknown architecture`** — flow uses an arch-specific graph. If `apr inspect` + reports `arch: unknown`, the model lacks the `general.architecture` metadata + field; stamp it with `apr stamp` or re-convert. +- **Output is huge for 30B models** — scope with `--layer "blk.0"` first and + expand from there. +- **Verbose stats show zeros** — `--verbose` requires the model to be in a + format with embedded tensor stats; otherwise it falls back to shape-only. ## See also diff --git a/book/src/cli/hex.md b/book/src/cli/hex.md index 5d2588e13c..c8b9ba3404 100644 --- a/book/src/cli/hex.md +++ b/book/src/cli/hex.md @@ -18,9 +18,51 @@ apr hex [OPTIONS] apr hex qwen2.5-coder-1.5b-instruct-q4_k_m.gguf | head -20 ``` -## Full help +## What this does -Run `apr hex --help` for the complete option list. +`apr hex` is xxd that understands model formats. It overlays APR/GGUF/SafeTensors +structure on top of the byte view: header fields, Q4K/Q6K/Q8_0 super-block layout, +per-region entropy, value distribution, and the layout contract from +`contracts/tensor-layout-v1.yaml`. Use it to debug new format support or to prove +a transpose was applied correctly. + +## Key flags + +| Flag | What it does | Example | +|------|-------------|---------| +| `--header` | Annotated header (magic, version, tensor count) | `--header` | +| `--blocks` | Q4K/Q6K/Q8_0 super-block structure | `--blocks --tensor lm_head` | +| `--distribution` | Value histogram + entropy + kurtosis | `--distribution` | +| `--contract` | Overlay tensor-layout contract per tensor | `--contract` | +| `--tensor PAT` | Filter to one tensor | `--tensor "blk.0.attn"` | +| `--raw` | xxd-style raw view with ASCII column | `--raw --offset 0x100` | + +## Common workflows + +**Verify GGUF -> APR import transposed correctly.** + +```bash +apr hex qwen2.5-coder-1.5b.gguf --blocks --tensor "blk.0.attn_q.weight" > gguf.txt +apr hex qwen2.5-coder-1.5b.apr --blocks --tensor "blk.0.attn_q.weight" > apr.txt +# Row-major APR view should show the col-major GGUF rows as columns +``` + +**Audit the file header without loading the whole model.** + +```bash +apr hex unknown-model.gguf --header +# Confirms magic bytes, version, tensor count before you try `apr run` +``` + +## Troubleshooting + +- **`magic bytes mismatch` on a known-good file** — corrupted download or wrong + format extension. Cross-check with `file ` (POSIX) and re-pull if + needed. +- **Block view looks random** — you're probably looking at an F32 tensor, not + Q4K. Confirm with `apr tensors --filter ` to see the dtype. +- **Slow on 30B models** — `--raw` reads sequentially; scope with `--offset` and + `--limit` to inspect just the region of interest. ## See also diff --git a/book/src/cli/inspect.md b/book/src/cli/inspect.md index 2c23b49b1f..9b0e557a61 100644 --- a/book/src/cli/inspect.md +++ b/book/src/cli/inspect.md @@ -18,9 +18,52 @@ apr inspect [OPTIONS] apr inspect qwen2.5-coder-1.5b-instruct-q4_k_m.gguf ``` -## Full help +## What this does -Run `apr inspect --help` for the complete option list. +`apr inspect` reads the model header (APR, GGUF, or SafeTensors) and prints +architecture, vocab size, tensor count, license, and HF identity metadata. +With `--quality` it emits a 0-100 score covering physics (no NaN/Inf), structural +completeness, provenance, and tokenizer presence — the SHIP-TWO-001 §84 +ship-gate rubric. This is the first command to run on any model you didn't build +yourself. + +## Key flags + +| Flag | What it does | Example | +|------|-------------|---------| +| `--quality` | Print 0-100 SHIP-TWO score block | `--quality` | +| `--weights` | Tensor statistics (mean, std, min, max) | `--weights` | +| `--vocab` | Tokenizer / vocab details | `--vocab` | +| `--filters` | Filter/security checks | `--filters` | +| `--json` | Machine-readable output | `--json | jq .arch` | + +## Common workflows + +**Pre-ship gate: confirm score >= 90 before publishing to HF.** + +```bash +apr inspect qwen2.5-coder-1.5b.apr --quality --json | jq '.score' +# Must be >= 90 per AC-SHIP2-007 +``` + +**Diff two checkpoints' metadata after a finetune.** + +```bash +apr inspect qwen2.5-coder-0.5b.apr --json > before.json +apr inspect qwen2.5-coder-0.5b-ft.apr --json > after.json +diff <(jq -S . before.json) <(jq -S . after.json) +``` + +## Troubleshooting + +- **Quality score < 90** — drill into the breakdown. Missing `data_source` or + `data_license` is the most common cause (provenance is 20 points). Fix with + `apr stamp --field data_source=...`. +- **`hf_architecture` field missing** — common on models converted before PMAT-690. + Re-convert from the original SafeTensors with current `apr convert` to get the + stamp. +- **NaN/Inf detected in weights** — almost always a training divergence or a + bad quantization step. Trace back with `apr tensors --stats --filter `. ## See also diff --git a/book/src/cli/lint.md b/book/src/cli/lint.md index f94b76118a..5035af7caf 100644 --- a/book/src/cli/lint.md +++ b/book/src/cli/lint.md @@ -18,9 +18,48 @@ apr lint [OPTIONS] apr lint qwen2.5-coder-1.5b-instruct-q4_k_m.gguf ``` -## Full help +## What this does -Run `apr lint --help` for the complete option list. +`apr lint` is to model files what clippy is to Rust — it flags conventions, style +issues, and footguns that aren't strictly invalid but should be fixed before +shipping. Examples: missing metadata fields, non-canonical tensor names, untagged +quantization formats, or vocabulary anomalies. Use it as a pre-publish gate that +complements `apr validate` (which checks integrity, not style). + +## Key flags + +| Flag | What it does | Example | +|------|-------------|---------| +| `--json` | Machine-readable lint output | `--json` | +| `-v, --verbose` | Show rationale for each lint | `--verbose` | +| `--skip-contract` | Skip the tensor-layout contract | `--skip-contract` | +| `--offline` | Suppress all network access | `--offline` | + +## Common workflows + +**Pre-publish lint sweep.** + +```bash +apr lint qwen2.5-coder-1.5b.apr --verbose +# Review warnings, fix metadata, re-run until clean. +``` + +**Lint every cached APR file.** + +```bash +find ~/.cache/aprender/models -name "*.apr" -print0 | \ + xargs -0 -I{} apr lint {} --json +``` + +## Troubleshooting + +- **"non-canonical tensor name detected"** — usually a stale converter. Re-run + `apr convert` on the original source; the current writer normalizes + `model.layers.N.*` to the canonical form. +- **"missing data_source / data_license"** — provenance fields are not strictly + required but lint flags them. Use `apr stamp` to add them. +- **Lint hangs on a huge model** — the model is being mmap-walked; this is + expected for 30B+ checkpoints. Add `--quiet` if you only care about exit code. ## See also diff --git a/book/src/cli/run.md b/book/src/cli/run.md index 057b110132..8e6e8c497e 100644 --- a/book/src/cli/run.md +++ b/book/src/cli/run.md @@ -18,9 +18,55 @@ apr run [OPTIONS] apr run qwen2.5-coder-1.5b "What is 2+2?" --max-tokens 16 ``` -## Full help +## What this does -Run `apr run --help` for the complete option list. +`apr run` is the one-shot inference command — give it a model and a prompt and it +auto-downloads (from the Hugging Face mirror), caches under `~/.cache/aprender/`, +loads, and generates. It's the "hello world" entry point and the default smoke +test for any model. Backend selection is automatic: CUDA when available, Trueno +SIMD on CPU otherwise. + +## Key flags + +| Flag | What it does | Example | +|------|-------------|---------| +| `-n, --max-tokens N` | Generation budget in tokens | `--max-tokens 256` | +| `--chat` | Wrap prompt in ChatML for Instruct models | `--chat` | +| `--temperature T` | Sampling temperature (0 = greedy) | `--temperature 0.7` | +| `--top-p P` | Nucleus sampling cutoff | `--top-p 0.9` | +| `--stream` | Stream tokens as they're generated | `--stream` | +| `--trace` | Emit per-layer inference trace | `--trace --trace-level layer` | +| `--backend B` | Force a backend (`cuda`, `cpu`, `wgpu`) | `--backend cuda` | +| `--benchmark` | Print tok/s + latency at end | `--benchmark` | + +## Common workflows + +**Smoke test a freshly converted model.** + +```bash +apr convert qwen2.5-coder-1.5b.safetensors -o qwen2.5-coder-1.5b.apr --quantize q4k +apr run qwen2.5-coder-1.5b.apr "fn main() {" --max-tokens 32 --benchmark +``` + +**Reproduce a CPU/GPU parity bug with deterministic seed.** + +```bash +apr run qwen2.5-coder-1.5b.apr "def fizzbuzz(n):" --seed 42 --backend cpu --max-tokens 64 > cpu.txt +apr run qwen2.5-coder-1.5b.apr "def fizzbuzz(n):" --seed 42 --backend cuda --max-tokens 64 > gpu.txt +diff cpu.txt gpu.txt +``` + +## Troubleshooting + +- **Gibberish or repeated tokens** — run `apr qa --verbose` first; it covers + the eight gates that catch tokenizer / KV-cache / quantization mismatches before + you start reading code. See [`apr qa first, hypotheses second`](https://github.com/paiml/aprender/issues/202). +- **"unsupported file format"** — the path must be `.apr`, `.gguf`, or `.safetensors`. + For Hugging Face shortnames (`qwen2.5-coder-1.5b`) the resolver expects the model + to exist under `~/.cache/aprender/models/`; pull it explicitly with `apr pull`. +- **Slow on GPU (20 tok/s instead of 400)** — the `cuda` feature wasn't compiled in. + Verify with `apr run ... -v` (look for `backend: cuda`); rebuild with + `cargo install aprender --features cuda` if needed. ## See also diff --git a/book/src/cli/serve.md b/book/src/cli/serve.md index 5fe476b5ae..427068c33b 100644 --- a/book/src/cli/serve.md +++ b/book/src/cli/serve.md @@ -18,9 +18,50 @@ apr serve [OPTIONS] apr serve run qwen2.5-coder-1.5b --port 8080 ``` -## Full help +## What this does -Run `apr serve --help` for the complete option list. +`apr serve` exposes a model as an OpenAI-compatible HTTP API with `/v1/chat/completions`, +`/v1/completions`, Prometheus metrics, and Server-Sent Events streaming. Use it +when an IDE or agent client expects an Ollama / OpenAI endpoint. The `plan` +subcommand answers "does this model fit in my VRAM, and what's the roofline +ceiling?" without actually loading weights. + +## Key flags + +| Flag | What it does | Example | +|------|-------------|---------| +| `--port P` | Listen port (default 8080) | `--port 11434` | +| `--host H` | Bind host (default 127.0.0.1) | `--host 0.0.0.0` | +| `--batch` | Enable batched GPU prefill (2x+ throughput) | `--batch` | +| `--gpu` / `--no-gpu` | Force backend selection | `--gpu` | +| `--no-metrics` | Disable `/metrics` endpoint | `--no-metrics` | +| `--otlp-endpoint URL` | Export distributed traces | `--otlp-endpoint http://jaeger:4317` | + +## Common workflows + +**Drop-in Ollama replacement on the standard port.** + +```bash +apr serve run qwen2.5-coder-7b.apr --port 11434 --host 0.0.0.0 --batch +curl http://localhost:11434/v1/chat/completions \ + -d '{"model":"qwen2.5-coder-7b","messages":[{"role":"user","content":"hi"}]}' +``` + +**Capacity-check before deploying a larger model.** + +```bash +apr serve plan qwen3-coder-30b-a3b.apr --batch-size 4 +# Reports: VRAM budget, KV cache size, roofline tok/s ceiling, contract status +``` + +## Troubleshooting + +- **`address already in use` on 8080** — another process owns the port. List with + `lsof -i :8080` or pick a new one via `--port 8081`. +- **Throughput stuck ~20 tok/s on a CUDA host** — `--batch` is off by default for + determinism. Add it for production; expect ~8x prefill speedup on Qwen2.5. +- **Prometheus scrape returns 404** — `/metrics` is gated behind `--no-metrics` + being absent; double-check the flag didn't sneak in via your systemd unit. ## See also diff --git a/book/src/cli/tensors.md b/book/src/cli/tensors.md index 78fe31a50f..3f8158c2a6 100644 --- a/book/src/cli/tensors.md +++ b/book/src/cli/tensors.md @@ -18,9 +18,48 @@ apr tensors [OPTIONS] apr tensors qwen2.5-coder-1.5b-instruct-q4_k_m.gguf --json | jq length ``` -## Full help +## What this does -Run `apr tensors --help` for the complete option list. +`apr tensors` lists every tensor in a model with name, shape, dtype, and offset. +With `--stats` it adds mean/std/min/max — the fastest way to spot a divergent +finetune or a quantization step that introduced NaN. Filter by name pattern with +`--filter` to scope to one layer or projection. + +## Key flags + +| Flag | What it does | Example | +|------|-------------|---------| +| `--stats` | Compute mean/std/min/max per tensor | `--stats` | +| `--filter PAT` | Substring filter on tensor name | `--filter "layers.0"` | +| `--limit N` | Cap the number of rows | `--limit 20` | +| `--json` | JSON output for scripting | `--json` | + +## Common workflows + +**Confirm a finetune actually changed the weights.** + +```bash +apr tensors qwen2.5-coder-0.5b.apr --stats --json > before.json +apr tensors qwen2.5-coder-0.5b-ft.apr --stats --json > after.json +jq -s '.[0] - .[1]' before.json after.json # diffs cleanly when stats are identical +``` + +**Find the LM head and check its shape.** + +```bash +apr tensors qwen2.5-coder-1.5b.apr --filter "lm_head" +# Expect [vocab_size, hidden_size] for tied embeddings +``` + +## Troubleshooting + +- **Tensor count is half of the source** — `apr tensors` doesn't deduplicate + shared (tied) tensors, but a converter might. Check `apr inspect --json | jq + .tensor_count` against the source's reported count. +- **Stats show `nan` / `inf`** — bad weight. Trace with `apr trace --layer + ` to find where corruption entered. +- **Long output for 30B models** — pipe through `--limit 50` or `--filter` to + scope. The full list is 600+ rows for Qwen3 30B. ## See also diff --git a/book/src/cli/trace.md b/book/src/cli/trace.md index f65927efb9..06d9385408 100644 --- a/book/src/cli/trace.md +++ b/book/src/cli/trace.md @@ -18,9 +18,52 @@ apr trace [OPTIONS] apr trace qwen2.5-coder-1.5b-instruct-q4_k_m.gguf --prompt "What is 2+2?" --max-tokens 4 ``` -## Full help +## What this does -Run `apr trace --help` for the complete option list. +`apr trace` runs a prompt through the model and emits per-layer tensor stats +(or full F32 payloads) at every stage: embedding, qkv, attn output, FFN gate/up, +FFN down, RMSNorm, LM head. It's the surgical tool for CPU/GPU divergence +hunting — diff a trace from `--backend cpu` against `--backend cuda` and the +first row to disagree tells you which kernel diverged. + +## Key flags + +| Flag | What it does | Example | +|------|-------------|---------| +| `--layer PAT` | Filter to one layer pattern | `--layer "blk.0"` | +| `--payload` | Trace full F32 tensor payloads | `--payload` | +| `--diff` | Diff mode (against `--reference`) | `--diff` | +| `--save-tensor STAGES` | Save per-stage F32 tensors to disk | `--save-tensor embedding,qkv_matmul` | +| `--save-tensor-layers RANGE` | Layer range (default `0..1`) | `--save-tensor-layers 0..2` | +| `--reference PATH` | Reference model for `--diff` | `--reference golden.apr` | + +## Common workflows + +**Find the layer where GPU diverges from CPU on a quantized model.** + +```bash +apr trace qwen2.5-coder-1.5b.apr --backend cpu --save-tensor all --save-tensor-dir /tmp/cpu +apr trace qwen2.5-coder-1.5b.apr --backend cuda --save-tensor all --save-tensor-dir /tmp/gpu +diff <(ls /tmp/cpu) <(ls /tmp/gpu) +# Then numpy-diff each stage to find the first divergent kernel +``` + +**Diff two quantization levels at the same prompt.** + +```bash +apr trace qwen2.5-coder-1.5b-q4k.apr --diff --reference qwen2.5-coder-1.5b-q6k.apr \ + --prompt "fn main() {" --max-tokens 8 +``` + +## Troubleshooting + +- **`--save-tensor` writes nothing** — confirm the stage names match + `contracts/apr-cli-trace-save-tensor-v1.yaml` (e.g. `embedding`, not + `embed`). Use `all` to capture every stage. +- **Trace is slow for 7B+ models** — that's the F32 payload write. Drop + `--payload` if you only need stats, or restrict to one layer with `--layer`. +- **Diff shows divergence at layer 0** — likely a tokenizer or embedding bug, + not a kernel bug. Confirm with `apr tokenize ` against the reference. ## See also diff --git a/book/src/cli/tree.md b/book/src/cli/tree.md index 752bb1eb28..bf8b29ac70 100644 --- a/book/src/cli/tree.md +++ b/book/src/cli/tree.md @@ -18,9 +18,46 @@ apr tree [OPTIONS] apr tree qwen2.5-coder-1.5b-instruct-q4_k_m.gguf ``` -## Full help +## What this does -Run `apr tree --help` for the complete option list. +`apr tree` renders the architecture as a `tree(1)`-style hierarchy: model -> +blocks -> sub-modules -> tensors. With `--sizes` it sums tensor bytes at each +node, so you can see exactly where memory goes (typically embeddings + LM head +dominate 1B-3B models). Output formats include ASCII, Graphviz `dot`, Mermaid, +and JSON. + +## Key flags + +| Flag | What it does | Example | +|------|-------------|---------| +| `--format FMT` | `ascii` (default), `dot`, `mermaid`, `json` | `--format mermaid` | +| `--sizes` | Show tensor sizes at each node | `--sizes` | +| `--filter PAT` | Filter by component pattern | `--filter "attn"` | +| `--depth N` | Cap tree depth | `--depth 3` | + +## Common workflows + +**Generate a Mermaid diagram for an architecture spec doc.** + +```bash +apr tree qwen2.5-coder-1.5b.apr --format mermaid --depth 4 > qwen-arch.mmd +``` + +**Find the largest tensors at a glance.** + +```bash +apr tree qwen2.5-coder-7b.apr --sizes --depth 2 | sort -k 2 -h | tail -10 +``` + +## Troubleshooting + +- **Tree is flat (one level)** — naming convention doesn't match a known + architecture. Run `apr inspect --json | jq .arch` and confirm the arch is + detected; fall back to `apr tensors` for a raw view. +- **`--sizes` shows 0** — model uses sentinel-only tensors (rare). Check + `apr tensors --stats` to see if shapes are real. +- **Mermaid output too large to render** — clip with `--depth 3` and + `--filter` to one block, then expand layer-by-layer. ## See also diff --git a/book/src/cli/validate.md b/book/src/cli/validate.md index 31699ec95b..debd3e47ac 100644 --- a/book/src/cli/validate.md +++ b/book/src/cli/validate.md @@ -18,9 +18,50 @@ apr validate [OPTIONS] apr validate qwen2.5-coder-1.5b-instruct-q4_k_m.gguf --quality ``` -## Full help +## What this does -Run `apr validate --help` for the complete option list. +`apr validate` verifies a model file's structural integrity (magic bytes, header, +tensor offsets, shape contract) and optionally emits a 100-point quality score. +Use it as a CI gate — it returns non-zero on any error and, with `--strict`, on +any warning. Where `apr inspect --quality` summarizes the score, `apr validate +--quality` blocks the pipeline when the score is below threshold. + +## Key flags + +| Flag | What it does | Example | +|------|-------------|---------| +| `--quality` | Compute 100-point quality assessment | `--quality` | +| `--strict` | Treat warnings as errors | `--strict` | +| `--min-score N` | Fail if quality score < N | `--min-score 90` | +| `--json` | JSON output for CI parsing | `--json` | + +## Common workflows + +**CI gate before pushing to crates.io / HF.** + +```bash +apr validate qwen2.5-coder-1.5b.apr --quality --min-score 90 --strict +# exit 0 = ship; non-zero = block release +``` + +**Audit every model in your cache.** + +```bash +for m in ~/.cache/aprender/models/*.apr; do + echo "=== $m ===" + apr validate "$m" --quality --json | jq '{score, errors: .errors|length}' +done +``` + +## Troubleshooting + +- **`magic bytes mismatch`** — file is corrupted or not an APR/GGUF file. Re-pull + with `apr pull` or verify the SHA256 against the source. +- **Score lower than `apr inspect --quality` reported** — `validate` runs full + tensor-layout contracts, not just metadata. The two diverge when LAYOUT-001/002 + catches a row-major / column-major mismatch. +- **`tensor offset out of bounds`** — partial download. Delete from cache + (`apr rm `) and re-pull. ## See also From d752480eea8e979dd5dc28bdd8177e25f275cd91 Mon Sep 17 00:00:00 2001 From: Noah Gift Date: Sat, 23 May 2026 18:16:38 +0200 Subject: [PATCH 2/4] docs(book): expand walkthroughs for apr qa, qualify, check, bench, eval, canary, parity, profile, compare-hf (Refs #1902) Adds "What this does", "Key flags", "Common workflows", and "Troubleshooting" sections to 9 quality / evaluation CLI chapters. Co-Authored-By: Claude Opus 4.7 --- book/src/cli/bench.md | 46 +++++++++++++++++++++++++++++++++++-- book/src/cli/canary.md | 44 +++++++++++++++++++++++++++++++++-- book/src/cli/check.md | 44 +++++++++++++++++++++++++++++++++-- book/src/cli/compare-hf.md | 44 +++++++++++++++++++++++++++++++++-- book/src/cli/eval.md | 46 +++++++++++++++++++++++++++++++++++-- book/src/cli/parity.md | 43 ++++++++++++++++++++++++++++++++-- book/src/cli/profile.md | 46 +++++++++++++++++++++++++++++++++++-- book/src/cli/qa.md | 47 ++++++++++++++++++++++++++++++++++++-- book/src/cli/qualify.md | 44 +++++++++++++++++++++++++++++++++-- 9 files changed, 386 insertions(+), 18 deletions(-) diff --git a/book/src/cli/bench.md b/book/src/cli/bench.md index f5a8ffa1d3..d3564ae00b 100644 --- a/book/src/cli/bench.md +++ b/book/src/cli/bench.md @@ -18,9 +18,51 @@ apr bench [OPTIONS] apr bench qwen2.5-coder-1.5b-instruct-q4_k_m.gguf --iterations 10 ``` -## Full help +## What this does -Run `apr bench --help` for the complete option list. +`apr bench` measures decode throughput (tok/s) with warmup. By default it +generates 32 tokens per iteration after 3 warmup iterations, averages, and +reports p50/p95/p99 latencies. Use `--fast` to route through `realizar` (the +production inference engine, 5-20x faster than the `aprender` debug path). +The hard floor in spec H12 is 10 tok/s — anything below that is a release +blocker. + +## Key flags + +| Flag | What it does | Example | +|------|-------------|---------| +| `--iterations N` | Measurement iterations (default 5) | `--iterations 20` | +| `--warmup N` | Warmup iterations (default 3) | `--warmup 5` | +| `--max-tokens N` | Tokens per iteration (default 32) | `--max-tokens 128` | +| `--prompt TEXT` | Custom benchmark prompt | `--prompt "def fizzbuzz"` | +| `--fast` | Route through realizar (production path) | `--fast` | +| `--percentiles LIST` | Latency percentiles (default `50,95,99`) | `--percentiles 50,90,99` | + +## Common workflows + +**Production benchmark vs Ollama reference.** + +```bash +apr bench qwen2.5-coder-1.5b.apr --fast --iterations 20 --max-tokens 128 --json | \ + jq '{tok_s, p50_ms, p99_ms}' +``` + +**Compare CPU vs GPU on the same model.** + +```bash +apr bench qwen2.5-coder-1.5b.apr --fast --iterations 10 # CUDA (auto) +apr bench qwen2.5-coder-1.5b.apr --fast --iterations 10 --no-gpu # Trueno SIMD +``` + +## Troubleshooting + +- **Throughput tanks across iterations** — likely thermal throttling on a laptop. + Watch `nvidia-smi -l 1` or `htop` for sustained clock drops. +- **"failed to load model"** — confirm the file extension matches the format; + `.apr`, `.gguf`, `.safetensors` are auto-detected. +- **Numbers don't match the spec table** — the spec table uses `--fast` + (realizar). Without it you're benching the slower debug path. Always pass + `--fast` for headline numbers. ## See also diff --git a/book/src/cli/canary.md b/book/src/cli/canary.md index a3ac4d0de3..81382c8496 100644 --- a/book/src/cli/canary.md +++ b/book/src/cli/canary.md @@ -18,9 +18,49 @@ apr canary [OPTIONS] apr canary qwen2.5-coder-1.5b-instruct-q4_k_m.gguf ``` -## Full help +## What this does -Run `apr canary --help` for the complete option list. +`apr canary` records and checks behavioral fixtures — a known input (audio, +text, JSON) paired with the model's reference output. Use it to guard against +silent regressions: if Qwen2.5-Coder used to translate `"def add(a, b):"` into +exactly `"\n return a + b\n"`, a canary captures that, and any later change +that flips the answer fails CI. + +## Key flags + +| Subcommand | What it does | Example | +|-----------|-------------|---------| +| `canary create` | Record a new canary from a model + input | `--input prompt.txt --output ref.json` | +| `canary check` | Compare model against a saved canary | `--canary ref.json` | +| `--json` | Machine-readable output | `--json` | + +## Common workflows + +**Capture a regression fixture during golden-model bring-up.** + +```bash +apr canary create qwen2.5-coder-0.5b.apr \ + --input prompts/fizzbuzz.txt \ + --output canaries/qwen-0.5b-fizzbuzz.json +``` + +**Gate every PR on the saved canary set.** + +```bash +for c in canaries/*.json; do + apr canary check qwen2.5-coder-0.5b.apr --canary "$c" --json || exit 1 +done +``` + +## Troubleshooting + +- **Canary fails after harmless refactor** — sampler stochasticity. Re-run with + `--temperature 0.0 --seed 299792458` (the project's standard deterministic + seed); regenerate the canary if the change is intentional. +- **`canary create` segfaults on audio input** — confirm the input file format + matches the model's expected modality (WAV 16kHz mono for Whisper). +- **All canaries fail after a model swap** — that's the point. Canaries are + per-model fixtures; recreate them when you intentionally change weights. ## See also diff --git a/book/src/cli/check.md b/book/src/cli/check.md index 7869f9ea40..f05eb06731 100644 --- a/book/src/cli/check.md +++ b/book/src/cli/check.md @@ -18,9 +18,49 @@ apr check [OPTIONS] apr check qwen2.5-coder-1.5b-instruct-q4_k_m.gguf ``` -## Full help +## What this does -Run `apr check --help` for the complete option list. +`apr check` is the 10-stage end-to-end pipeline self-test from APR-TRACE-001: +tokenize, embed, RMSNorm, QKV, attention, FFN gate+up, FFN down, residual, LM +head, sample. Each stage must produce numerically sane output (no NaN/Inf, +within distribution band). A passing `apr check` means the model runs; it +doesn't mean it's accurate (use `apr eval` / `apr qa` for that). + +## Key flags + +| Flag | What it does | Example | +|------|-------------|---------| +| `--no-gpu` | Run on CPU (Trueno SIMD) | `--no-gpu` | +| `--json` | One JSON envelope per stage | `--json` | +| `-v, --verbose` | Per-stage tensor stats | `--verbose` | + +## Common workflows + +**Quick post-convert sanity check.** + +```bash +apr check qwen2.5-coder-0.5b.apr --json | \ + jq '.stages[] | select(.status != "PASS")' +``` + +**Compare CPU and GPU pipelines stage-by-stage.** + +```bash +apr check qwen2.5-coder-1.5b.apr --no-gpu --json > cpu.json +apr check qwen2.5-coder-1.5b.apr --json > gpu.json +jq -s '.[0].stages - .[1].stages' cpu.json gpu.json +``` + +## Troubleshooting + +- **NaN at attention stage** — almost always a softmax numerical issue caused + by a missing causal mask. Cross-check with `apr trace --layer "blk.0.attn"`. +- **"unsupported architecture"** — `apr check` needs a known dispatch table. + Confirm `apr inspect --json | jq .arch`; if `unknown`, add the arch to + the model dispatch contract. +- **All stages pass but `apr run` is gibberish** — the 10 stages cover + numerical sanity, not tokenizer correctness. Run `apr tokenize + ` to verify the tokenizer round-trips. ## See also diff --git a/book/src/cli/compare-hf.md b/book/src/cli/compare-hf.md index 32e4e3b0a0..05e919a031 100644 --- a/book/src/cli/compare-hf.md +++ b/book/src/cli/compare-hf.md @@ -18,9 +18,49 @@ apr compare-hf [OPTIONS] apr compare-hf model.apr --hf-repo Qwen/Qwen2.5-Coder-1.5B-Instruct ``` -## Full help +## What this does -Run `apr compare-hf --help` for the complete option list. +`apr compare-hf` downloads the original SafeTensors from a HuggingFace repo and +runs an element-wise diff against your local APR conversion. It's the gold +standard for "did my converter actually preserve the weights" — if the maximum +absolute difference exceeds `--threshold` (default 1e-5), the converter has a +bug. Use this as the discharge gate for any new arch in the import pipeline. + +## Key flags + +| Flag | What it does | Example | +|------|-------------|---------| +| `--hf REPO` | HuggingFace repo ID | `--hf Qwen/Qwen2.5-Coder-1.5B-Instruct` | +| `--tensor PAT` | Filter to one tensor pattern | `--tensor "lm_head"` | +| `--threshold N` | Max allowed abs diff (default 1e-5) | `--threshold 1e-4` | +| `--json` | JSON output with per-tensor diff stats | `--json` | +| `-v, --verbose` | Print every tensor (not just failures) | `--verbose` | + +## Common workflows + +**Verify a fresh import.** + +```bash +apr import hf://Qwen/Qwen2.5-Coder-1.5B-Instruct -o qwen.apr +apr compare-hf qwen.apr --hf Qwen/Qwen2.5-Coder-1.5B-Instruct --json | \ + jq '.tensors[] | select(.max_abs_diff > 1e-5)' +``` + +**Drill into the LM head specifically (where transpose bugs surface).** + +```bash +apr compare-hf qwen.apr --hf Qwen/Qwen2.5-Coder-1.5B-Instruct \ + --tensor "lm_head" --threshold 1e-6 --verbose +``` + +## Troubleshooting + +- **Large diff on `lm_head` only** — classic LAYOUT-001 bug. The converter + forgot to transpose. Fix in `crates/aprender-core/src/format/converter/`. +- **All tensors diverge equally** — quantization mismatch. APR Q4K vs HF F16 + will always diverge; compare F32 APR against HF SafeTensors instead. +- **`--hf` download is slow** — set `HF_HUB_ENABLE_HF_TRANSFER=1` to use the + Rust-native downloader. Auth via `HF_TOKEN` for gated models. ## See also diff --git a/book/src/cli/eval.md b/book/src/cli/eval.md index 64b3fb65b0..fc6f4d8336 100644 --- a/book/src/cli/eval.md +++ b/book/src/cli/eval.md @@ -18,9 +18,51 @@ apr eval [OPTIONS] apr eval qwen2.5-coder-1.5b-instruct-q4_k_m.gguf --suite humaneval --limit 5 ``` -## Full help +## What this does -Run `apr eval --help` for the complete option list. +`apr eval` runs perplexity (PPL) on wikitext-2 or lambada by default, with a +default H13 threshold of PPL <= 20. With `--task classify` it runs F1 / accuracy +on a JSONL test set for fine-tuned classifiers. pass@k for code benchmarks +(humaneval, mbpp) is gated through `--samples N --temperature 0.8`. + +## Key flags + +| Flag | What it does | Example | +|------|-------------|---------| +| `--dataset NAME` | `wikitext-2`, `lambada`, or `custom` | `--dataset lambada` | +| `--threshold N` | PPL fail threshold (default 20) | `--threshold 15` | +| `--task TASK` | Omit for PPL, `classify` for F1/accuracy | `--task classify` | +| `--samples N` | pass@k samples per problem | `--samples 5` | +| `--temperature T` | Sampling temperature for pass@k | `--temperature 0.8` | +| `--device DEV` | `cpu` or `cuda` | `--device cuda` | +| `--generate-card` | Write HF model card on success | `--generate-card` | + +## Common workflows + +**Pre-publish PPL gate.** + +```bash +apr eval qwen2.5-coder-1.5b.apr --dataset wikitext-2 --threshold 15 --json +# exit 0 = ship; non-zero = block +``` + +**pass@1 on HumanEval for a code model.** + +```bash +apr eval qwen2.5-coder-7b.apr --dataset humaneval --samples 1 --temperature 0.0 \ + --device cuda --json +``` + +## Troubleshooting + +- **PPL > 20 on a known-good base model** — confirm the chat template wasn't + accidentally applied. PPL is for base models; chat-templated text inflates + PPL by 2-5x. +- **HumanEval pass@1 is 0%** — the prompt-extraction heuristic may be missing + (see [`§70 RC3 fix`](https://github.com/paiml/aprender/pull/1641)). + Enable `APR_EVAL_DEBUG=1` to see the actual prompt sent to the model. +- **OOM on CUDA for 7B** — drop `--samples 1`, or use `--device cpu` (slower + but bounded by RAM not VRAM). ## See also diff --git a/book/src/cli/parity.md b/book/src/cli/parity.md index 5d279b150c..f26d206ac6 100644 --- a/book/src/cli/parity.md +++ b/book/src/cli/parity.md @@ -18,9 +18,48 @@ apr parity [OPTIONS] apr parity model.gguf --backends cpu,gpu ``` -## Full help +## What this does -Run `apr parity --help` for the complete option list. +`apr parity` runs the same prompt through CPU and GPU and compares outputs +token-by-token. When CPU and GPU disagree, the divergence is almost always a +quantization kernel that lost precision differently on each backend, or a +layout-001/002 transpose error. With `--assert` it exits non-zero on divergence, +making it a CI gate. + +## Key flags + +| Flag | What it does | Example | +|------|-------------|---------| +| `-p, --prompt TEXT` | Prompt to compare (default "What is 2+2?") | `--prompt "fn main()"` | +| `--assert` | Exit non-zero on divergence | `--assert` | +| `--json` | JSON output with per-token diff | `--json` | +| `-v, --verbose` | Per-layer divergence details | `--verbose` | + +## Common workflows + +**Block PR if a kernel change introduces GPU divergence.** + +```bash +apr parity qwen2.5-coder-1.5b.gguf --prompt "def hello():" --assert --json +``` + +**Drill into the layer that diverges (genchi genbutsu).** + +```bash +apr parity qwen2.5-coder-1.5b.gguf --verbose 2>&1 | grep DIVERGE +# Then surgically trace that layer: +apr trace qwen2.5-coder-1.5b.gguf --layer "blk.7" --save-tensor all +``` + +## Troubleshooting + +- **Parity fails on first token** — likely tokenizer or embedding bug, not a + kernel bug. Confirm with `apr tokenize ` on both backends. +- **Token 50 diverges but tokens 1-49 match** — slow precision drift from + accumulator differences. Often acceptable for quantized models; investigate + if it crosses a sampling boundary. +- **"no GPU detected"** — `apr parity` requires CUDA. Build with `--features + cuda` or use `apr trace` for offline comparison via saved tensors. ## See also diff --git a/book/src/cli/profile.md b/book/src/cli/profile.md index bfb553a2d9..bce234e7b9 100644 --- a/book/src/cli/profile.md +++ b/book/src/cli/profile.md @@ -18,9 +18,51 @@ apr profile [OPTIONS] apr profile qwen2.5-coder-1.5b-instruct-q4_k_m.gguf ``` -## Full help +## What this does -Run `apr profile --help` for the complete option list. +`apr profile` is the deep performance microscope. It runs inference under +roofline analysis, classifies every operator as memory-bound or compute-bound, +detects naive (non-fused, non-SIMD) implementations, and optionally writes a +flamegraph. Use it when `apr bench` says you're slow and you need to know WHY. +With `--ci` and `--assert-throughput` it doubles as a regression gate. + +## Key flags + +| Flag | What it does | Example | +|------|-------------|---------| +| `--format FMT` | `human`, `json`, `flamegraph` | `--format flamegraph` | +| `-o FILE` | Output path for flamegraph SVG | `-o profile.svg` | +| `--detect-naive` | Flag non-fused kernels | `--detect-naive` | +| `--perf-grade` | Compute grade vs Ollama baseline | `--perf-grade` | +| `--ci` | CI mode (exit non-zero on regression) | `--ci --assert-throughput 100` | +| `--energy` | Energy measurement via RAPL | `--energy` | +| `--measure N` | Measurement passes (default 10) | `--measure 20` | + +## Common workflows + +**Generate a flamegraph SVG for an inference run.** + +```bash +apr profile qwen2.5-coder-1.5b.apr --format flamegraph -o qwen-profile.svg \ + --measure 20 --tokens 64 +xdg-open qwen-profile.svg +``` + +**CI gate: fail if throughput drops below 100 tok/s.** + +```bash +apr profile qwen2.5-coder-1.5b.apr --ci --assert-throughput 100 --assert-p99 50 +``` + +## Troubleshooting + +- **"naive implementation detected"** — a kernel is missing its SIMD/fused path. + Cross-check with `apr explain --kernel --tensor `; file a contract for + the missing fused kernel. +- **Flamegraph SVG is empty** — measurement was too short. Bump `--measure 20 + --tokens 64`. +- **`--energy` shows zeros** — RAPL is unavailable (containers, non-Intel CPUs). + Drop the flag. ## See also diff --git a/book/src/cli/qa.md b/book/src/cli/qa.md index 8ecb79747f..95396e7f05 100644 --- a/book/src/cli/qa.md +++ b/book/src/cli/qa.md @@ -18,9 +18,52 @@ apr qa [OPTIONS] apr qa qwen2.5-coder-1.5b-instruct-q4_k_m.gguf ``` -## Full help +## What this does -Run `apr qa --help` for the complete option list. +`apr qa` runs the eight falsifiable ship-gates on a model: golden output, throughput, +Ollama parity, GPU-vs-CPU speedup, tensor contract, cross-format parity, PTX parity, +and metadata plausibility. Every gate has a measurable pass/fail condition — there +are no judgment calls. This is the FIRST tool to run on any model regression, and +the CI gate that blocks a HF or crates.io publish. + +## Key flags + +| Flag | What it does | Example | +|------|-------------|---------| +| `--assert-tps N` | Fail if throughput < N tok/s | `--assert-tps 100` | +| `--assert-speedup R` | Fail if speedup vs Ollama < R | `--assert-speedup 1.0` | +| `--iterations N` | Bench iterations (default 10) | `--iterations 20` | +| `--max-tokens N` | Generate budget per iteration | `--max-tokens 64` | +| `--json` | Emit one JSON envelope for CI | `--json` | +| `--previous-report FILE` | Regression check against prior run | `--previous-report last.json` | +| `--regression-threshold R` | Allowed degradation ratio | `--regression-threshold 0.05` | + +## Common workflows + +**Block a release until every gate is GREEN.** + +```bash +apr qa qwen2.5-coder-1.5b.apr \ + --assert-tps 100 --assert-speedup 1.0 --json > qa-report.json +jq '.gates[] | select(.status != "PASS")' qa-report.json +``` + +**Regression-detect against the previous nightly build.** + +```bash +apr qa qwen2.5-coder-1.5b.apr --json --previous-report nightly-prev.json \ + --regression-threshold 0.10 +``` + +## Troubleshooting + +- **Golden output fails on a known-good model** — re-pull the model; the golden + fixture is content-hashed against the canonical model in the contract. See + [`apr qa first, hypotheses second`](https://github.com/paiml/aprender/issues/202). +- **Throughput gate fails on first run** — increase `--warmup` to 5+; cold-cache + numbers are not representative. +- **Cross-format parity skipped** — pass `--safetensors-path` to enable + GGUF/SafeTensors comparison (F-QUAL-032 contract). ## See also diff --git a/book/src/cli/qualify.md b/book/src/cli/qualify.md index bd325e90c2..76ef03624c 100644 --- a/book/src/cli/qualify.md +++ b/book/src/cli/qualify.md @@ -18,9 +18,49 @@ apr qualify [OPTIONS] apr qualify qwen2.5-coder-1.5b-instruct-q4_k_m.gguf ``` -## Full help +## What this does -Run `apr qualify --help` for the complete option list. +`apr qualify` runs the entire `apr` subcommand suite — inspect, validate, lint, +tensors, hex, tree, flow, run, bench, eval — against a single model and reports +which tools succeeded. It exists because models often work with one tool and +break another (a converter might emit valid metadata but a malformed tensor +table). Qualify catches "format not supported" surprises before users do. + +## Key flags + +| Flag | What it does | Example | +|------|-------------|---------| +| `--tier T` | `smoke` (Phase 1), `standard` (+contracts), `full` (+playbook) | `--tier standard` | +| `--timeout SEC` | Per-gate timeout (default 120s) | `--timeout 60` | +| `--skip LIST` | Comma-separated gates to skip | `--skip bench,eval` | +| `--json` | One JSON envelope per gate | `--json` | +| `-v, --verbose` | Show each subcommand's stdout | `--verbose` | + +## Common workflows + +**Smoke a model fresh from the converter.** + +```bash +apr qualify newly-converted.apr --tier smoke --json | \ + jq '.gates[] | select(.status == "FAIL")' +``` + +**Pre-publish full sweep including 10-stage check + canary + eval.** + +```bash +apr qualify qwen2.5-coder-1.5b.apr --tier full --timeout 300 +``` + +## Troubleshooting + +- **"timeout exceeded"** — bump `--timeout`; 7B models on CPU can need 300s+ for + bench/eval gates. +- **One gate hangs the whole run** — `apr qualify` enforces per-gate timeouts, + but a deadlocked subcommand may still need a manual SIGTERM. Use + `--skip ` to bypass and file a bug. +- **Different tiers contradict each other** — `smoke` only checks command exit + codes; `standard` adds contract validation; `full` adds the playbook. If + smoke passes but standard fails, the bug is in tensor-layout contracts. ## See also From 9a36da0c87da0e8c6f58b5a94802c4477effd678 Mon Sep 17 00:00:00 2001 From: Noah Gift Date: Sat, 23 May 2026 18:19:00 +0200 Subject: [PATCH 3/4] docs(book): expand walkthroughs for apr convert, export, import, quantize, merge, prune, compile (Refs #1902) Adds "What this does", "Key flags", "Common workflows", and "Troubleshooting" sections to 7 model-transform CLI chapters. Co-Authored-By: Claude Opus 4.7 --- book/src/cli/compile.md | 51 ++++++++++++++++++++++++++++++++++++++-- book/src/cli/convert.md | 44 ++++++++++++++++++++++++++++++++-- book/src/cli/export.md | 46 ++++++++++++++++++++++++++++++++++-- book/src/cli/import.md | 49 ++++++++++++++++++++++++++++++++++++-- book/src/cli/merge.md | 49 ++++++++++++++++++++++++++++++++++++-- book/src/cli/prune.md | 48 +++++++++++++++++++++++++++++++++++-- book/src/cli/quantize.md | 47 ++++++++++++++++++++++++++++++++++-- 7 files changed, 320 insertions(+), 14 deletions(-) diff --git a/book/src/cli/compile.md b/book/src/cli/compile.md index 344cda720f..001208d69e 100644 --- a/book/src/cli/compile.md +++ b/book/src/cli/compile.md @@ -18,9 +18,56 @@ apr compile [OPTIONS] apr compile model.apr --target cuda -o compiled.apr ``` -## Full help +## What this does -Run `apr compile --help` for the complete option list. +`apr compile` produces a single, self-contained executable with the model +weights embedded — pass it around, double-click it, no `apr` binary required. +Useful for demos, sovereign air-gapped deployments, and reproducible bug +reports. Internally it statically links `realizar` and embeds the model bytes +as a `.rodata` section. Optionally LTO + strip for the smallest possible +binary. + +## Key flags + +| Flag | What it does | Example | +|------|-------------|---------| +| `-o PATH` | Output binary path | `-o ./qwen-cli` | +| `--target TRIPLE` | Cross-compile target | `--target x86_64-unknown-linux-musl` | +| `--quantize FMT` | Quantize before embedding | `--quantize int4` | +| `--release` | Optimized build | `--release` | +| `--strip` | Strip debug symbols | `--strip` | +| `--lto` | Enable Link-Time Optimization | `--lto` | +| `--list-targets` | Show supported targets | `--list-targets` | + +## Common workflows + +**Build a static, distributable demo binary.** + +```bash +apr compile qwen2.5-coder-0.5b.apr \ + --target x86_64-unknown-linux-musl --quantize int4 --release --strip --lto \ + -o demo/qwen-coder +./demo/qwen-coder "fn fizzbuzz(n: u32) {" +``` + +**Ship a CUDA-specific build for a known GPU farm.** + +```bash +apr compile qwen2.5-coder-7b.apr --target x86_64-unknown-linux-gnu --release \ + -o farm/qwen7b-cuda +``` + +## Troubleshooting + +- **"linker not found for target"** — cross-compilers aren't installed. + Install `musl-tools` (Linux) or use `rustup target add`. +- **Output binary is huge** — `--release --strip --lto` reduces by 30-50%. + Beyond that, the model weights dominate; quantize harder (`--quantize int4`) + or use a smaller model. +- **Static CUDA binary fails on a different driver** — CUDA isn't truly + static. The libcuda.so on the host must match the embedded CUDA toolkit. + Ship the Trueno SIMD path instead with `--no-default-features` if you + need real portability. ## See also diff --git a/book/src/cli/convert.md b/book/src/cli/convert.md index 6f30a5782f..1c7cd4835a 100644 --- a/book/src/cli/convert.md +++ b/book/src/cli/convert.md @@ -18,9 +18,49 @@ apr convert [OPTIONS] apr convert model.safetensors --quantize q4_k -o model-q4k.apr ``` -## Full help +## What this does -Run `apr convert --help` for the complete option list. +`apr convert` takes a SafeTensors or GGUF model and produces a `.apr` file — +the project's native, row-major-only, quantization-aware format. It performs +the LAYOUT-001/002 transpose on GGUF input, optionally quantizes (int8 / int4 / +fp16 / Q4K), and optionally compresses the result with zstd. The output is +designed for `realizar` to mmap-load and feed straight into fused kernels. + +## Key flags + +| Flag | What it does | Example | +|------|-------------|---------| +| `-o, --output PATH` | Output `.apr` path (required) | `-o qwen.apr` | +| `--quantize FMT` | `int8`, `int4`, `fp16`, `q4k` | `--quantize q4k` | +| `--compress C` | `none`, `zstd`, `zstd-max`, `lz4` | `--compress zstd` | +| `-f, --force` | Overwrite an existing output | `--force` | + +## Common workflows + +**Convert HF SafeTensors to APR Q4K (the most common case).** + +```bash +apr import hf://Qwen/Qwen2.5-Coder-1.5B-Instruct -o qwen.safetensors +apr convert qwen.safetensors --quantize q4k -o qwen2.5-coder-1.5b.apr +apr inspect qwen2.5-coder-1.5b.apr --quality +``` + +**Re-convert with compression to shrink disk footprint.** + +```bash +apr convert qwen2.5-coder-1.5b.apr --compress zstd-max -o qwen-compressed.apr +ls -lh qwen2.5-coder-1.5b.apr qwen-compressed.apr +``` + +## Troubleshooting + +- **"layout contract violation"** — the input GGUF has an unexpected tensor + shape. Re-pull the GGUF and confirm with `apr hex --header`. +- **Q4K output looks nothing like the GGUF Q4K** — that's expected. APR Q4K + is row-major; GGUF Q4K is column-major. Compare via `apr compare-hf`, not + byte-level diff. +- **Slow conversion (minutes for 1.5B)** — quantization is single-threaded for + determinism. For batch conversions, parallelize over multiple files instead. ## See also diff --git a/book/src/cli/export.md b/book/src/cli/export.md index f0437a8d14..f53955af8f 100644 --- a/book/src/cli/export.md +++ b/book/src/cli/export.md @@ -18,9 +18,51 @@ apr export [OPTIONS] apr export model.apr --format gguf -o model.gguf ``` -## Full help +## What this does -Run `apr export --help` for the complete option list. +`apr export` is the outbound bridge — convert a `.apr` model into a format that +other ecosystems understand: SafeTensors (HuggingFace native), GGUF (llama.cpp / +Ollama), MLX (Apple Silicon), ONNX, OpenVINO, CoreML. The row-major-to-target +transpose runs automatically. Use `--batch` to fan out to several targets in +one pass; use `--plan` to preview without writing. + +## Key flags + +| Flag | What it does | Example | +|------|-------------|---------| +| `--format FMT` | `safetensors`, `gguf`, `mlx`, `onnx`, `openvino`, `coreml` | `--format gguf` | +| `-o, --output PATH` | Output file/directory | `-o qwen.gguf` | +| `--quantize FMT` | Apply quantization during export | `--quantize int4` | +| `--batch LIST` | Multi-format fan-out | `--batch gguf,mlx,safetensors` | +| `--plan` | Validate inputs + print plan, don't write | `--plan` | +| `--list-formats` | Print supported targets | `--list-formats` | + +## Common workflows + +**Publish your trained model to three ecosystems at once.** + +```bash +apr export qwen2.5-coder-1.5b.apr --batch gguf,mlx,safetensors -o dist/ +ls dist/ +# dist/qwen2.5-coder-1.5b.gguf dist/qwen2.5-coder-1.5b.mlx dist/qwen2.5-coder-1.5b.safetensors +``` + +**Plan a GGUF export before committing disk space.** + +```bash +apr export qwen2.5-coder-7b.apr --format gguf --quantize int4 --plan +# Reports: estimated output size, transpose plan, contract status +``` + +## Troubleshooting + +- **GGUF export fails on "no general.architecture"** — `apr inspect` shows + metadata is missing. Stamp it with `apr stamp` before export. +- **CoreML / OpenVINO export silently emits SafeTensors** — those targets are + graph-level converters that need a complete dispatch table. Cross-check with + `--list-formats`. +- **Exported GGUF triples in size** — `--quantize` wasn't passed. Without it, + export materializes F32. Always pass `--quantize int4` for ship artifacts. ## See also diff --git a/book/src/cli/import.md b/book/src/cli/import.md index 75c752a4ed..01e9755b1d 100644 --- a/book/src/cli/import.md +++ b/book/src/cli/import.md @@ -18,9 +18,54 @@ apr import [OPTIONS] apr import hf://openai/whisper-tiny -o whisper.apr --arch whisper ``` -## Full help +## What this does -Run `apr import --help` for the complete option list. +`apr import` is the all-in-one inbound pipeline: download from a `hf://` URL +(authenticated via `HF_TOKEN` for gated repos), parse SafeTensors / GGUF / URL +inputs, apply the LAYOUT-001/002 transpose, infer or accept the architecture, +optionally preserve Q4K, and write a `.apr`. Unlike `apr convert` (which works +on a local file), `apr import` understands `hf://` and provenance. + +## Key flags + +| Flag | What it does | Example | +|------|-------------|---------| +| `--arch ARCH` | `llama`, `qwen2`, `qwen3`, `whisper`, `auto`, ... | `--arch qwen2` | +| `-o, --output PATH` | Output `.apr` path | `-o qwen.apr` | +| `--quantize FMT` | Quantize during import | `--quantize int4` | +| `--preserve-q4k` | Keep GGUF Q4K quantization | `--preserve-q4k` | +| `--strict` | Reject unverified archs / fail on warnings | `--strict` | +| `--enforce-provenance` | Reject pre-baked GGUF (only SafeTensors) | `--enforce-provenance` | +| `--allow-no-config` | Allow import without `config.json` (risky) | `--allow-no-config` | + +## Common workflows + +**Standard HF -> APR import.** + +```bash +HF_TOKEN=hf_xxx apr import hf://Qwen/Qwen2.5-Coder-1.5B-Instruct \ + -o qwen2.5-coder-1.5b.apr --arch qwen2 +apr inspect qwen2.5-coder-1.5b.apr --quality +``` + +**Single-provenance ship pipeline (rejects pre-baked GGUF).** + +```bash +apr import hf://Qwen/Qwen2.5-Coder-7B-Instruct -o qwen-7b.apr \ + --arch qwen2 --enforce-provenance --strict +``` + +## Troubleshooting + +- **"refusing to import without config.json"** — without `config.json`, + hyperparameters like `rope_theta` are inferred from tensor shapes and may + be wrong (GH-223). Either upload a `config.json` or, only if you've + verified the inference, pass `--allow-no-config`. +- **HF download 401 / 403** — set `HF_TOKEN` (it's already in our environment + per the memory note). Gated repos require accepting the license on the HF + website first. +- **"unknown architecture"** — pass `--arch` explicitly; `auto` only works on + well-known archs (llama, qwen2/3, whisper, gpt2, ...). ## See also diff --git a/book/src/cli/merge.md b/book/src/cli/merge.md index c14778bba6..2ab3bc946a 100644 --- a/book/src/cli/merge.md +++ b/book/src/cli/merge.md @@ -18,9 +18,54 @@ apr merge [OPTIONS] apr merge model1.apr model2.apr --strategy weighted --weights 0.7,0.3 -o merged.apr ``` -## Full help +## What this does -Run `apr merge --help` for the complete option list. +`apr merge` combines two or more compatible models into one — same architecture, +same vocab, same shape. Supports the standard model-merging strategies: +`average`, `weighted`, `slerp`, `ties` (trim by task-vector magnitude), `dare` +(drop and rescale). Used to fold finetune deltas back into the base model, or +to assemble a "best of N" checkpoint across runs. + +## Key flags + +| Flag | What it does | Example | +|------|-------------|---------| +| `--strategy S` | `average`, `weighted`, `slerp`, `ties`, `dare` | `--strategy slerp` | +| `--weights LIST` | Weights for `weighted` | `--weights 0.7,0.3` | +| `--base-model M` | Required for TIES / DARE | `--base-model qwen2.5-coder-0.5b.apr` | +| `--drop-rate R` | DARE drop probability (default 0.9) | `--drop-rate 0.7` | +| `--density D` | TIES trim density (default 0.2) | `--density 0.3` | +| `--seed N` | RNG seed for DARE | `--seed 42` | +| `--plan` | Validate inputs + show plan | `--plan` | + +## Common workflows + +**Linearly blend two finetunes weighted 70/30.** + +```bash +apr merge qwen-ft-rust.apr qwen-ft-python.apr \ + --strategy weighted --weights 0.7,0.3 \ + -o qwen-ft-blend.apr +``` + +**TIES-merge two task finetunes against the base.** + +```bash +apr merge qwen-ft-code.apr qwen-ft-chat.apr \ + --strategy ties --base-model qwen2.5-coder-1.5b.apr --density 0.2 \ + -o qwen-ft-ties.apr +``` + +## Troubleshooting + +- **"shape mismatch"** — all input models must share the exact architecture + and quantization scheme. Cross-check with `apr inspect` on each. F32 + Q4K + cannot merge. +- **TIES / DARE require `--base-model`** — the strategy computes task vectors + as deltas from a base. Pass it explicitly. +- **Merged model fails `apr qa`** — merge can degrade quality if weight + spaces aren't aligned. Try `slerp` for two models or fall back to weighted + averaging. ## See also diff --git a/book/src/cli/prune.md b/book/src/cli/prune.md index ca28767ac6..c27711a98d 100644 --- a/book/src/cli/prune.md +++ b/book/src/cli/prune.md @@ -18,9 +18,53 @@ apr prune [OPTIONS] apr prune model.apr --ratio 0.5 -o pruned.apr ``` -## Full help +## What this does -Run `apr prune --help` for the complete option list. +`apr prune` removes weights that contribute least to model output. `magnitude` +zeros the smallest-absolute-value weights; `structured` removes whole channels; +`depth` removes whole transformer blocks; `wanda` / `sparsegpt` use calibration +data for a better quality/sparsity trade. Output is a smaller, faster model +(after a sparse-aware kernel pass) or the same size with zeros (for sparsity +experiments). + +## Key flags + +| Flag | What it does | Example | +|------|-------------|---------| +| `-m, --method M` | `magnitude`, `structured`, `depth`, `width`, `wanda`, `sparsegpt` | `--method wanda` | +| `--target-ratio R` | Fraction to prune (default 0.5) | `--target-ratio 0.3` | +| `--remove-layers RANGE` | Layer range for depth pruning | `--remove-layers 20-24` | +| `--calibration FILE` | Calibration JSONL for Wanda / SparseGPT | `--calibration calib.jsonl` | +| `--analyze` | Analyze pruning opportunities (no write) | `--analyze` | +| `--plan` | Estimate only | `--plan` | + +## Common workflows + +**Magnitude-prune a 1.5B model to 50% sparsity.** + +```bash +apr prune qwen2.5-coder-1.5b.apr --method magnitude --target-ratio 0.5 \ + -o qwen-pruned.apr +apr eval qwen-pruned.apr --dataset wikitext-2 --threshold 25 +``` + +**Wanda-prune with calibration data for better quality.** + +```bash +apr prune qwen2.5-coder-1.5b.apr --method wanda --target-ratio 0.5 \ + --calibration calib.jsonl -o qwen-wanda.apr +apr eval qwen-wanda.apr --dataset wikitext-2 # Wanda typically beats magnitude +``` + +## Troubleshooting + +- **Pruned model is slower than the original** — sparse kernels only speed + things up at 70%+ structured sparsity. At 50% magnitude (unstructured), + expect the same speed but smaller disk after compression. +- **PPL doubles after pruning** — too aggressive a ratio or no calibration. + Drop to 0.3 ratio or switch to `wanda` with calibration data. +- **`--remove-layers` produces a broken model** — depth pruning is risky; + the deleted layers' residuals propagate. Always `apr check` afterward. ## See also diff --git a/book/src/cli/quantize.md b/book/src/cli/quantize.md index b6622c5bf1..c920e3d06d 100644 --- a/book/src/cli/quantize.md +++ b/book/src/cli/quantize.md @@ -18,9 +18,52 @@ apr quantize [OPTIONS] apr quantize model.apr --to q4_k -o model-q4k.apr ``` -## Full help +## What this does -Run `apr quantize --help` for the complete option list. +`apr quantize` reduces model weight precision — int8 / int4 / fp16 / Q4K +(GGML-style super-block) — for smaller footprint and faster decode. Unlike +`apr convert --quantize`, this command works on an already-APR file and is the +right tool for re-quantizing (e.g. F16 -> Q4K) post-finetune. With `--plan` it +estimates output size without writing; with `--batch` it produces several +quantization levels in one pass. + +## Key flags + +| Flag | What it does | Example | +|------|-------------|---------| +| `-s, --scheme FMT` | `int8`, `int4`, `fp16`, `q4k` (default `int4`) | `--scheme q4k` | +| `-o, --output PATH` | Output file | `-o qwen-q4k.apr` | +| `--format FMT` | Override output container (`apr`, `gguf`, `safetensors`) | `--format gguf` | +| `--batch LIST` | Fan out to multiple schemes | `--batch q4k,int8,fp16` | +| `--plan` | Estimate size + plan, don't write | `--plan` | +| `-f, --force` | Overwrite existing output | `--force` | + +## Common workflows + +**Re-quantize a finetuned F32 checkpoint to Q4K for shipping.** + +```bash +apr quantize qwen2.5-coder-1.5b-ft.apr --scheme q4k -o ship/qwen-1.5b-ft-q4k.apr +apr qa ship/qwen-1.5b-ft-q4k.apr --assert-tps 100 +``` + +**Generate three quantization variants for benchmarking.** + +```bash +apr quantize qwen2.5-coder-1.5b.apr --batch q4k,int8,fp16 -o ship/ +ls ship/ +for m in ship/*.apr; do apr bench "$m" --fast --iterations 5; done +``` + +## Troubleshooting + +- **PPL increases > 5% after Q4K** — calibration data may help (currently the + scheme is data-free). Use `int8` for a smaller quality hit at the cost of + ~2x more disk. +- **`--scheme fp16` produces a larger file than expected** — that's correct + if the source was already Q4K; fp16 is bigger than int4. Verify the source + dtype with `apr tensors --stats`. +- **"output already exists"** — pass `--force` or pick a new `-o` path. ## See also From 6a501059d41d89d9bdf4c033c3138d36833ee34b Mon Sep 17 00:00:00 2001 From: Noah Gift Date: Sat, 23 May 2026 18:23:03 +0200 Subject: [PATCH 4/4] docs(book): expand walkthroughs for apr pull, list, rm, gpu, registry, finetune, distill, train, pretrain, tokenize, tune, data (Refs #1902) Adds "What this does", "Key flags", "Common workflows", and "Troubleshooting" sections to the 5 registry chapters and the 7 training chapters. Completes the 42-chapter walkthrough batch. Co-Authored-By: Claude Opus 4.7 --- book/src/cli/data.md | 45 ++++++++++++++++++++++++++++++-- book/src/cli/distill.md | 53 +++++++++++++++++++++++++++++++++++-- book/src/cli/finetune.md | 53 +++++++++++++++++++++++++++++++++++-- book/src/cli/gpu.md | 42 ++++++++++++++++++++++++++++-- book/src/cli/list.md | 40 ++++++++++++++++++++++++++-- book/src/cli/pretrain.md | 56 ++++++++++++++++++++++++++++++++++++++-- book/src/cli/pull.md | 51 ++++++++++++++++++++++++++++++++++-- book/src/cli/registry.md | 43 ++++++++++++++++++++++++++++-- book/src/cli/rm.md | 42 ++++++++++++++++++++++++++++-- book/src/cli/tokenize.md | 50 +++++++++++++++++++++++++++++++++-- book/src/cli/train.md | 48 ++++++++++++++++++++++++++++++++-- book/src/cli/tune.md | 54 ++++++++++++++++++++++++++++++++++++-- 12 files changed, 553 insertions(+), 24 deletions(-) diff --git a/book/src/cli/data.md b/book/src/cli/data.md index 6f1472e2eb..67a11a0239 100644 --- a/book/src/cli/data.md +++ b/book/src/cli/data.md @@ -18,9 +18,50 @@ apr data [OPTIONS] apr data inspect train.jsonl ``` -## Full help +## What this does -Run `apr data --help` for the complete option list. +`apr data` is the training-data hygiene suite — audit for quality issues, +stratified split, contamination check against benchmarks, and class balancing. +A finetune run is only as good as its dataset; this command catches the +mundane issues (label imbalance, near-duplicates, benchmark leakage) before +they ruin a 12-hour GPU run. + +## Key subcommands + +| Subcommand | What it does | Example | +|-----------|-------------|---------| +| `data audit` | Quality scan (duplicates, label balance, length stats) | `apr data audit train.jsonl` | +| `data split` | Stratified train/val/test split | `apr data split train.jsonl --ratios 0.8,0.1,0.1` | +| `data decontaminate` | N-gram overlap against benchmark sets | `apr data decontaminate train.jsonl --against humaneval` | +| `data balance` | Resample to fix class imbalance | `apr data balance train.jsonl --strategy oversample` | + +## Common workflows + +**Pre-finetune hygiene pass.** + +```bash +apr data audit ./data/train.jsonl --json | jq '.warnings' +apr data decontaminate ./data/train.jsonl --against humaneval --against mbpp +apr data split ./data/train.jsonl --ratios 0.9,0.05,0.05 -o ./data/splits/ +``` + +**Fix imbalanced classification dataset.** + +```bash +apr data audit ./data/labels.jsonl --json | jq '.class_distribution' +apr data balance ./data/labels.jsonl --strategy oversample -o ./data/balanced.jsonl +``` + +## Troubleshooting + +- **`decontaminate` flags too aggressively** — default n-gram threshold is + strict. Tune via `--ngram-size 13 --min-overlap 0.5` for less-strict + matching. +- **`split` produces a tiny val set** — small datasets + stratification can + yield val sets too small to be useful. Use a held-out file instead. +- **`balance --strategy oversample` blows up dataset size** — major class + imbalance. Try `undersample` for the majority class, or use + `apr finetune --oversample` (in-loop oversampling) instead. ## See also diff --git a/book/src/cli/distill.md b/book/src/cli/distill.md index 3b03dee878..7dcf00c9d2 100644 --- a/book/src/cli/distill.md +++ b/book/src/cli/distill.md @@ -18,9 +18,58 @@ apr distill [OPTIONS] apr distill --teacher 7b.apr --student 0.5b.apr --data train.jsonl ``` -## Full help +## What this does -Run `apr distill --help` for the complete option list. +`apr distill` trains a small "student" model to imitate a large "teacher" by +matching the teacher's softmax distribution over a corpus. The result is a model +with ~5-10x fewer parameters that retains most of the teacher's quality. Two-stage +mode (`--config two_stage.yaml`) precomputes teacher logits to disk, then trains +the student offline — much faster than running both forward passes per step. + +## Key flags + +| Flag | What it does | Example | +|------|-------------|---------| +| `` | Teacher model (positional) | `qwen2.5-coder-7b.apr` | +| `--student FILE` | Student model | `--student qwen2.5-coder-0.5b.apr` | +| `-d, --data FILE` | Training JSONL | `--data train.jsonl` | +| `--strategy S` | `standard`, `progressive`, `ensemble` | `--strategy progressive` | +| `--temperature T` | Softmax scaling (default 3.0) | `--temperature 4.0` | +| `--alpha A` | KL vs task loss weight (default 0.7) | `--alpha 0.5` | +| `--stage S` | `precompute`, `train`, `generate` | `--stage precompute` | +| `--config FILE` | Two-stage YAML config | `--config distill.yaml` | +| `--backend B` | `fixture` (default), `gpu` | `--backend gpu` | + +## Common workflows + +**Standard one-stage distill (small datasets, GPU memory permitting).** + +```bash +apr distill qwen2.5-coder-7b.apr \ + --student qwen2.5-coder-0.5b.apr \ + --data ./data/distill.jsonl \ + --strategy standard --epochs 3 -o ./student-out/ +``` + +**Two-stage: precompute teacher logits, then train the student offline.** + +```bash +apr distill qwen2.5-coder-7b.apr --stage precompute \ + --data ./data/distill.jsonl -o ./teacher-logits/ +apr distill --stage train --config two_stage.yaml -o ./student-final/ +``` + +## Troubleshooting + +- **Student plateaus at the teacher's loss** — that's the ceiling; distillation + cannot exceed the teacher. Try `--strategy progressive` (curriculum) or use a + stronger teacher. +- **Teacher OOM in single-stage mode** — switch to `--stage precompute` first, + then run `--stage train` with only the student on the GPU. See + [PMAT-701](https://github.com/paiml/aprender/issues/701) for the + TEACHER==STUDENT==0.5B smoke-defaults-leak lesson. +- **NaN losses early in training** — drop `--temperature` to 2.0; T=3.0 can + cause flat distributions that produce vanishing gradients. ## See also diff --git a/book/src/cli/finetune.md b/book/src/cli/finetune.md index 438390c95f..c3fab7b890 100644 --- a/book/src/cli/finetune.md +++ b/book/src/cli/finetune.md @@ -18,9 +18,58 @@ apr finetune [OPTIONS] apr finetune qwen2.5-coder-0.5b --data train.jsonl --epochs 3 ``` -## Full help +## What this does -Run `apr finetune --help` for the complete option list. +`apr finetune` runs LoRA / QLoRA / full-finetune on a base model with a JSONL +training set. Method is `auto` by default — it inspects available VRAM and +model size and picks the right configuration. With `--quantize-nf4` it +activates QLoRA (frozen-weight 4-bit + LoRA adapters, ~8x less VRAM). Multi-GPU +data-parallel works via `--gpus 0,1`. Output is a PEFT-compatible adapter +directory (or merged model when `--merge` is set). + +## Key flags + +| Flag | What it does | Example | +|------|-------------|---------| +| `-m, --method M` | `auto`, `full`, `lora`, `qlora` | `--method qlora` | +| `-d, --data FILE` | Training JSONL | `--data train.jsonl` | +| `-o, --output PATH` | Adapter dir or merged model | `-o ./adapter/` | +| `-r, --rank N` | LoRA rank (auto-selected by default) | `--rank 16` | +| `--epochs N` | Epoch count (default 3) | `--epochs 5` | +| `--learning-rate LR` | Learning rate (default 2e-4) | `--learning-rate 5e-5` | +| `--quantize-nf4` | Activate QLoRA (4-bit frozen base) | `--quantize-nf4` | +| `--gpus LIST` | Data-parallel device list | `--gpus 0,1` | +| `--merge` | Merge adapter into base when done | `--merge` | + +## Common workflows + +**LoRA finetune Qwen2.5-Coder-0.5B on a custom JSONL.** + +```bash +apr finetune qwen2.5-coder-0.5b.apr \ + --data ./data/train.jsonl --method lora --rank 8 \ + --epochs 3 -o ./qwen-ft-adapter/ +apr eval ./qwen-ft-adapter/ --dataset wikitext-2 +``` + +**QLoRA on a 7B model with a single 16GB GPU.** + +```bash +apr finetune qwen2.5-coder-7b.apr \ + --data ./data/train.jsonl --method qlora --quantize-nf4 \ + --vram 16 --rank 32 -o ./qwen7b-qlora/ +``` + +## Troubleshooting + +- **CUDA OOM mid-epoch** — bump `--quantize-nf4` (QLoRA), or drop + `--max-seq-len`, or reduce `--rank`. The auto-planner uses `--vram`; if you + pass too high a value it'll try too aggressive a config. +- **Loss diverges to NaN** — drop `--learning-rate` by 5x. Default 2e-4 is + for LoRA; QLoRA usually wants 1e-4. +- **Adapter doesn't load post-merge** — check `--checkpoint-format` + (default `apr,safetensors`); the adapter dir needs both files for PEFT + compatibility. ## See also diff --git a/book/src/cli/gpu.md b/book/src/cli/gpu.md index 6b1222b4f9..ec803f25c0 100644 --- a/book/src/cli/gpu.md +++ b/book/src/cli/gpu.md @@ -18,9 +18,47 @@ apr gpu [OPTIONS] apr gpu --json ``` -## Full help +## What this does -Run `apr gpu --help` for the complete option list. +`apr gpu` is `nvidia-smi` with an `apr`-aware overlay: it shows each visible +GPU, its compute capability, total / free VRAM, current power draw, and any +active `apr serve` reservations from the GPU-SHARE-001 advisory lock. On +multi-GPU hosts use it to confirm which device CUDA will pick before launching +an inference server. + +## Key flags + +| Flag | What it does | Example | +|------|-------------|---------| +| `--json` | JSON output with one record per GPU | `--json` | +| `-v, --verbose` | Include reservation details + holders | `--verbose` | +| `-q, --quiet` | Numeric VRAM summary only | `--quiet` | + +## Common workflows + +**Confirm GPU is healthy before benchmarking.** + +```bash +apr gpu --json | jq '.devices[] | {idx, name, vram_free_mb, sm}' +apr bench qwen2.5-coder-1.5b.apr --fast --iterations 20 +``` + +**Watch VRAM during an inference run.** + +```bash +apr serve run qwen2.5-coder-7b.apr & +while sleep 2; do apr gpu --quiet; done +``` + +## Troubleshooting + +- **"no GPU detected" on a known-good CUDA host** — `apr` was built without + `--features cuda`. Verify with `apr --version -v` and rebuild. +- **Reported VRAM differs from `nvidia-smi`** — `nvidia-smi` shows OS-level + usage; `apr gpu` filters to processes started by `apr`. Use both together + to find non-`apr` GPU users. +- **Multi-GPU host always picks device 0** — set `CUDA_VISIBLE_DEVICES=N` + before launching; `apr` honors the env var. ## See also diff --git a/book/src/cli/list.md b/book/src/cli/list.md index f10dccd836..ebd0a2969a 100644 --- a/book/src/cli/list.md +++ b/book/src/cli/list.md @@ -18,9 +18,45 @@ apr list [OPTIONS] apr list --json | jq length ``` -## Full help +## What this does -Run `apr list --help` for the complete option list. +`apr list` enumerates models in the local cache (`~/.cache/aprender/`). Output +includes name, size on disk, last-used time, and format. Combined with `--json` +and `jq` it's the inventory query for "what models do I have, sorted by size" +or "which model haven't I used in 30 days?" Use `apr rm` to delete unused +checkpoints when disk gets tight. + +## Key flags + +| Flag | What it does | Example | +|------|-------------|---------| +| `--json` | Machine-readable output | `--json` | +| `-v, --verbose` | Include extra metadata (arch, dtype) | `--verbose` | +| `-q, --quiet` | Names only | `--quiet` | + +## Common workflows + +**Find your biggest cached models.** + +```bash +apr list --json | jq -r '.[] | "\(.size_mb) \(.name)"' | sort -rn | head -10 +``` + +**Garbage-collect anything older than 30 days.** + +```bash +apr list --json | jq -r '.[] | select(.last_used_days_ago > 30) | .name' | \ + xargs -I{} apr rm {} +``` + +## Troubleshooting + +- **Empty list, yet `apr run` works** — your `APR_CACHE_DIR` may differ from + the default `~/.cache/aprender/`. Confirm with `apr list --verbose`. +- **`last_used_days_ago` missing for some entries** — older cached models + pre-date the access-time tracking. They'll start populating once used. +- **Sizes don't match `du -sh ~/.cache/aprender`** — `apr list` shows logical + model size; `du` includes shard metadata + temporary download artifacts. ## See also diff --git a/book/src/cli/pretrain.md b/book/src/cli/pretrain.md index 5fd7f92c3e..ea9fc54f38 100644 --- a/book/src/cli/pretrain.md +++ b/book/src/cli/pretrain.md @@ -18,9 +18,61 @@ apr pretrain [OPTIONS] apr pretrain config.yaml ``` -## Full help +## What this does -Run `apr pretrain --help` for the complete option list. +`apr pretrain` drives a from-scratch pretraining loop per +`contracts/training-loop-pretrain-v1.yaml` — the SHIP-TWO-001 MODEL-2 workflow. +Unlike `apr finetune` (small LoRA adapter on a frozen base), pretrain updates +every parameter from a random init or a checkpoint. Defaults atomically flip +between `finetune` mode (post-divergence remedy, lr=5e-5, warmup=100) and +`from-scratch` mode (lr=3e-4, warmup=1000) so you can't accidentally apply +finetune LRs to a cold start. + +## Key flags + +| Flag | What it does | Example | +|------|-------------|---------| +| `--dataset PATH` | Tokenized shard index or raw corpus | `--dataset ./data/shards.json` | +| `--tokenizer DIR` | `vocab.json` + `merges.txt` | `--tokenizer ./tok/` | +| `--run-dir DIR` | Output dir for ckpts + metadata | `--run-dir ./runs/v1/` | +| `--mode M` | `finetune` or `from-scratch` (default `finetune`) | `--mode from-scratch` | +| `--lr LR` | Peak learning rate (overrides mode default) | `--lr 3e-4` | +| `--num-steps N` | Warmup + cosine decay total | `--num-steps 100000` | +| `--warmup-steps N` | Warmup steps (overrides mode default) | `--warmup-steps 1000` | +| `--batch-size N` | Micro-batch size (default 16) | `--batch-size 32` | +| `--seq-length N` | Tokens per example | `--seq-length 2048` | + +## Common workflows + +**From-scratch 370M cold start (MODEL-2 lane).** + +```bash +apr pretrain --mode from-scratch \ + --dataset ./data/csn-python-shards.json \ + --tokenizer ./tokenizers/csn-bpe-50257/ \ + --run-dir ./runs/csn-370m-cold/ \ + --num-steps 50000 --batch-size 32 --seq-length 2048 +``` + +**Continual pretraining (warm start from a checkpoint).** + +```bash +apr pretrain --mode finetune \ + --dataset ./data/domain-shards.json \ + --tokenizer ./tokenizers/qwen-bpe/ \ + --run-dir ./runs/qwen-continual/ \ + --lr 5e-5 --warmup-steps 100 --num-steps 5000 +``` + +## Troubleshooting + +- **val_loss plateaus at the Chinchilla floor** — you're undertrained for the + model size. Add more tokens (10-20 tok/param is the Chinchilla optimum) or + reduce the model. See [a-priori theoretical falsification](https://github.com/paiml/aprender/issues/700). +- **NaN loss at step 1** — usually a learning-rate-too-high crash. Drop `--lr` + by 5x, or rely on the mode default. +- **Throughput collapses after 500 steps** — memory fragmentation. Check + `apr gpu --verbose` for VRAM growth; restart with `apr train watch`. ## See also diff --git a/book/src/cli/pull.md b/book/src/cli/pull.md index 86eed6e180..077e5970e5 100644 --- a/book/src/cli/pull.md +++ b/book/src/cli/pull.md @@ -18,9 +18,56 @@ apr pull [OPTIONS] apr pull hf://Qwen/Qwen2.5-Coder-0.5B-Instruct ``` -## Full help +## What this does -Run `apr pull --help` for the complete option list. +`apr pull` is `ollama pull` for `apr` — give it a short alias (`qwen2.5-coder-1.5b`), +an `hf://` URI, or an `org/repo` ID, and it downloads + caches under +`~/.cache/aprender/`. It also pulls datasets when invoked as `apr pull dataset +`. Auth via `HF_TOKEN` (already in the environment per the project memory +note). Use `--revision` to pin to a tag / SHA for reproducibility. + +## Key flags + +| Flag | What it does | Example | +|------|-------------|---------| +| `--force` | Re-download even if cached | `--force` | +| `--dry-run` | Resolve to canonical URL, don't download | `--dry-run` | +| `--revision REV` | Pin to branch / tag / SHA | `--revision v1.0` | +| `--offline` | Forbid network I/O | `--offline` | +| `--include GLOB` | (dataset) shard glob filter | `--include "train-00*"` | +| `-o, --output DIR` | (dataset) output directory | `-o ./mydata/` | + +## Common workflows + +**Pull a model and verify integrity.** + +```bash +apr pull qwen2.5-coder-1.5b +apr validate ~/.cache/aprender/models/qwen2.5-coder-1.5b.apr --quality +``` + +**Reproducibility-pin a model to a specific commit.** + +```bash +apr pull hf://Qwen/Qwen2.5-Coder-1.5B-Instruct --revision a1b2c3d4 +# Record the SHA in CI; future pulls with the same --revision are byte-identical +``` + +**Pull a dataset subset.** + +```bash +apr pull dataset HuggingFaceH4/no_robots --include "train-*.parquet" -o ./data/ +``` + +## Troubleshooting + +- **HTTP 401 / 403** — set `HF_TOKEN`; for gated models accept the license on + the HF website first. +- **"alias not in registry"** — short names live in `configs/aliases.yaml`. + Use the full `hf://org/repo` URI, or `apr registry aliases` to see what's + registered. +- **Partial / corrupted download** — `apr pull --force` re-fetches. If the + underlying disk is full, `apr rm` old models first. ## See also diff --git a/book/src/cli/registry.md b/book/src/cli/registry.md index 62763edf58..51d5db34ba 100644 --- a/book/src/cli/registry.md +++ b/book/src/cli/registry.md @@ -18,9 +18,48 @@ apr registry [OPTIONS] apr registry status ``` -## Full help +## What this does -Run `apr registry --help` for the complete option list. +`apr registry` exposes the alias map at `configs/aliases.yaml` — the table that +lets you write `apr run qwen2.5-coder-1.5b` instead of `apr run +hf://Qwen/Qwen2.5-Coder-1.5B-Instruct`. Aliases are version-controlled, so two +developers running the same short name will pull the same canonical revision. +Use `registry aliases` to dump the table; future subcommands will add health +and provenance checks. + +## Key flags + +| Subcommand | What it does | Example | +|-----------|-------------|---------| +| `registry aliases` | List short-name -> canonical-URL pairs | `apr registry aliases --json` | +| `--json` | Machine-readable output | `--json` | +| `-v, --verbose` | Include revision pins + descriptions | `--verbose` | + +## Common workflows + +**Audit which short names point to which HF repos.** + +```bash +apr registry aliases --json | jq '.[] | {alias: .name, hf: .canonical_url}' +``` + +**Verify a `apr run` resolves to the expected canonical URL.** + +```bash +apr pull qwen2.5-coder-0.5b --dry-run +# Prints resolved hf://... URL without downloading +``` + +## Troubleshooting + +- **"unknown alias"** — the short name isn't in `configs/aliases.yaml`. Use + the full `hf://org/repo` URI, or open a PR adding the alias. +- **Two short names map to the same URL** — that's a registry duplicate. + File an issue; the lint will flag this once + [the registry-lint contract](https://github.com/paiml/aprender/blob/main/contracts/apr-registry-aliases-v1.yaml) + ships. +- **Alias resolves but `apr pull` fails** — revision pin in the alias map + may point to a deleted SHA. Update `configs/aliases.yaml`. ## See also diff --git a/book/src/cli/rm.md b/book/src/cli/rm.md index a0af48439f..56d652551d 100644 --- a/book/src/cli/rm.md +++ b/book/src/cli/rm.md @@ -18,9 +18,47 @@ apr rm [OPTIONS] apr rm # from \`apr list\` output ``` -## Full help +## What this does -Run `apr rm --help` for the complete option list. +`apr rm` deletes a cached model from `~/.cache/aprender/`. It's the cleanup +counterpart to `apr pull` — useful when disk is tight, when a corrupted +download needs to be wiped, or when you want a fresh fetch to verify +reproducibility. The model ID is exactly what `apr list` shows in the first +column. + +## Key flags + +| Flag | What it does | Example | +|------|-------------|---------| +| `--json` | Machine-readable confirmation | `--json` | +| `-v, --verbose` | Show byte count + path freed | `--verbose` | +| `-q, --quiet` | Silent success (errors only) | `--quiet` | + +## Common workflows + +**Wipe a corrupted download and re-pull.** + +```bash +apr rm qwen2.5-coder-1.5b +apr pull qwen2.5-coder-1.5b +apr validate ~/.cache/aprender/models/qwen2.5-coder-1.5b.apr +``` + +**Bulk-clean stale models.** + +```bash +apr list --json | jq -r '.[] | select(.last_used_days_ago > 60) | .name' | \ + xargs -I{} apr rm {} --verbose +``` + +## Troubleshooting + +- **"model not found"** — the ID must match `apr list` exactly. Names are + case-sensitive and include the format suffix (e.g. `qwen2.5-coder-1.5b.apr`). +- **`apr rm` doesn't free disk** — Linux page cache. Run `sync; echo 3 > + /proc/sys/vm/drop_caches` (root) or just wait; the disk really is free. +- **Removed model still appears in `apr list`** — stale index. Run any + `apr pull` or `apr inspect`; the index is rebuilt on next use. ## See also diff --git a/book/src/cli/tokenize.md b/book/src/cli/tokenize.md index 231b5536a6..b10a0b5fb2 100644 --- a/book/src/cli/tokenize.md +++ b/book/src/cli/tokenize.md @@ -18,9 +18,55 @@ apr tokenize [OPTIONS] apr tokenize "Hello world" --tokenizer qwen2.5-coder-1.5b ``` -## Full help +## What this does -Run `apr tokenize --help` for the complete option list. +`apr tokenize` trains BPE tokenizers from a JSONL corpus per +`contracts/tokenizer-bpe-v1.yaml`, imports HuggingFace `tokenizer.json` files +into the project's two-file `vocab.json` + `merges.txt` layout, and pretokenizes +corpora into `.bin` shards ready for training. It also serves as a one-shot +"tokenize this text" inspector — handy for verifying that a prompt round-trips +through the model's tokenizer. + +## Key subcommands + +| Subcommand | What it does | Example | +|-----------|-------------|---------| +| `tokenize plan` | Validate + estimate training time | `apr tokenize plan corpus.jsonl` | +| `tokenize apply` | Train BPE on the corpus | `apr tokenize apply corpus.jsonl` | +| `tokenize train` | Direct BPE train per MODEL-2 contract | `apr tokenize train corpus.jsonl` | +| `tokenize import-hf` | Import a HF `tokenizer.json` | `apr tokenize import-hf tokenizer.json` | +| `tokenize encode-corpus` | Pretokenize JSONL into `.bin` shards | `apr tokenize encode-corpus train.jsonl` | +| `tokenize repair-manifest` | Reconstruct manifest from shard files | `apr tokenize repair-manifest dir/` | + +## Common workflows + +**Train a 50k-vocab BPE on a code corpus.** + +```bash +apr tokenize plan ./corpus/python-csn.jsonl --vocab-size 50257 +apr tokenize apply ./corpus/python-csn.jsonl --vocab-size 50257 -o ./tok/python-50k/ +``` + +**Pretokenize for pretrain consumption.** + +```bash +apr tokenize encode-corpus ./corpus/train.jsonl \ + --tokenizer ./tok/python-50k/ \ + -o ./shards/python-train/ +ls ./shards/python-train/ # shard-0000.bin, shard-0001.bin, manifest.json +``` + +## Troubleshooting + +- **Vocab smaller than requested** — corpus too small; BPE merges run out before + reaching the budget. Use a larger corpus or accept the resulting smaller + vocab. +- **`import-hf` rejects `tokenizer.json`** — the layout must be + HuggingFace-standard. Some old tokenizers use a different schema; convert + via `transformers.AutoTokenizer.save_pretrained` first. +- **`.bin` shard size doesn't match `manifest.json`** — interrupted encode. + Re-run `encode-corpus` or use `repair-manifest` to rebuild the index from + existing shards. ## See also diff --git a/book/src/cli/train.md b/book/src/cli/train.md index 5df21ac86f..5978383c3f 100644 --- a/book/src/cli/train.md +++ b/book/src/cli/train.md @@ -18,9 +18,53 @@ apr train [OPTIONS] apr train config.yaml ``` -## Full help +## What this does -Run `apr train --help` for the complete option list. +`apr train` is the production training pipeline driver. Unlike `apr finetune` +(which takes flags), `apr train` reads a declarative YAML plan and supports +plan/apply/watch/sweep/halving/archive subcommands — the same workflow shape +as Terraform. `plan` validates inputs and shows what would happen; `apply` +allocates GPUs and runs; `watch` restarts on crash with hang detection; +`halving` does successive-halving HPO; `archive` packages a checkpoint for +release. + +## Key subcommands + +| Subcommand | What it does | Example | +|-----------|-------------|---------| +| `train plan CONFIG` | Validate + estimate without touching the GPU | `apr train plan run.yaml` | +| `train apply CONFIG` | Execute the plan | `apr train apply run.yaml` | +| `train watch RUN_DIR` | Auto-restart on crash / hang | `apr train watch ./runs/v1/` | +| `train sweep BASE` | Generate HPO configs from a base YAML | `apr train sweep base.yaml` | +| `train halving CONFIGS` | Successive-halving HPO (C-HPO-001) | `apr train halving sweep/` | +| `train archive RUN_DIR` | Package a checkpoint for release | `apr train archive ./run/` | + +## Common workflows + +**Plan-then-apply finetune.** + +```bash +apr train plan train.yaml +# Shows: GPU budget, expected wall-time, contract status. No GPU allocation yet. +apr train apply train.yaml +``` + +**Resilient training with auto-restart.** + +```bash +apr train apply train.yaml & +apr train watch ./runs/qwen-ft/ # restarts apply if it crashes +``` + +## Troubleshooting + +- **`plan` succeeds but `apply` fails on "VRAM exceeded"** — `plan` uses static + estimates. Add headroom in the YAML's `vram_gb` field, or pass + `--wait-gpu 3600` to queue until VRAM is free. +- **HPO `halving` always picks the same config** — sweep generated identical + configs. Verify with `apr train sweep base.yaml --json | jq length`. +- **`watch` restart-loops on the same crash** — hang detector wins over crash + detector. Inspect `./runs/.../stderr.log`; bug is repeatable, fix the cause. ## See also diff --git a/book/src/cli/tune.md b/book/src/cli/tune.md index ff6a70d5e6..c1df9024ed 100644 --- a/book/src/cli/tune.md +++ b/book/src/cli/tune.md @@ -18,9 +18,59 @@ apr tune [OPTIONS] apr tune qwen2.5-coder-0.5b --data train.jsonl ``` -## Full help +## What this does -Run `apr tune --help` for the complete option list. +`apr tune` is `apr finetune`'s smarter sibling: it can plan a LoRA / QLoRA +configuration given VRAM constraints, OR run a full HPO sweep (TPE / grid / +random with ASHA / median scheduling) and pick the best hyperparameters +automatically. `--scout` does 1-epoch-per-trial exploration to find promising +regions cheaply, and `--from-scout` warm-starts a full sweep from the scout +results. + +## Key flags + +| Flag | What it does | Example | +|------|-------------|---------| +| `-m, --method M` | `auto`, `full`, `lora`, `qlora` | `--method qlora` | +| `--plan` | Plan config without training | `--plan` | +| `--vram N` | Available VRAM in GB | `--vram 24` | +| `--model SIZE` | Size hint for planning (`7B`, `1.5B`) | `--model 1.5B` | +| `--task TASK` | `classify` (SPEC-TUNE-2026-001) | `--task classify` | +| `--strategy S` | HPO search: `tpe`, `grid`, `random` | `--strategy tpe` | +| `--scheduler S` | HPO scheduler: `asha`, `median`, `none` | `--scheduler asha` | +| `--budget N` | HPO trial count (default 10) | `--budget 50` | +| `--scout` | 1-epoch-per-trial fast exploration | `--scout` | +| `--from-scout DIR` | Warm-start from scout phase | `--from-scout ./scout/` | +| `--time-limit T` | Wall-clock cap | `--time-limit 8h` | + +## Common workflows + +**Plan-only: figure out the best config for your VRAM budget.** + +```bash +apr tune qwen2.5-coder-1.5b.apr --plan --vram 16 --model 1.5B --method auto +# Outputs: recommended rank, batch size, sequence length, est. tokens/sec +``` + +**Two-stage HPO: scout first, then full sweep on the promising region.** + +```bash +apr tune qwen2.5-coder-0.5b.apr --data train.jsonl --task classify \ + --scout --budget 20 -o ./scout/ +apr tune qwen2.5-coder-0.5b.apr --data train.jsonl --task classify \ + --from-scout ./scout/ --budget 50 --time-limit 4h +``` + +## Troubleshooting + +- **HPO picks the smallest rank every trial** — likely `--vram` is too low for + larger configs to fit; bump it or pass `--method qlora` to enable 4-bit + weights. +- **ASHA aggressively kills good trials** — switch to `--scheduler median` + or `none` for short sweeps where promotion thresholds matter less. +- **`--scout` results don't match full-sweep ordering** — scout uses 1 epoch + which is noisy. Treat scout as a feasibility filter, not a ranking; the + full sweep does the ranking. ## See also