Add Python inference benchmark CLI

python/README.md

@@ -123,6 +123,87 @@ if result["cloud_handoff"]:
 }
 ```
 
+## Benchmarking
+
+Use `cactus benchmark` when comparing inference changes across devices,
+quantization settings, or long-context profiles. It runs warmups, resets the KV
+cache between runs by default, records every trial as JSONL, and writes a grouped
+summary with mean, p50, p95, min, and max values for TTFT, total latency,
+prefill throughput, decode throughput, RAM, and token counts.
+
+```bash
+cactus benchmark LiquidAI/LFM2-VL-450M \
+  --profile short_chat \
+  --profile long_prefill \
+  --sweep-token-counts 512,2048,4096 \
+  --iterations 5 \
+  --warmup 2 \
+  --max-tokens 128 \
+  --output runs/lfm2_vl.jsonl \
+  --summary-json runs/lfm2_vl_summary.json \
+  --markdown-report runs/lfm2_vl_report.md
+```
+
+Custom profiles can be checked into a PR so reviewers can rerun the exact same
+prompt shapes:
+
+```json
+{
+  "profiles": [
+    {
+      "name": "long_context_decode",
+      "messages": [
+        {"role": "user", "content": "Repeat your benchmark context here..."}
+      ],
+      "options": {"max_tokens": 128, "temperature": 0.0}
+    }
+  ]
+}
+```
+
+```bash
+cactus benchmark ./weights/lfm2-vl --profiles-file profiles.json --output bench.jsonl
+```
+
+Compare two summary files to catch regressions before sending a runtime change
+for review:
+
+```bash
+cactus benchmark \
+  --compare runs/main_summary.json runs/pr_summary.json \
+  --compare-metric time_to_first_token_ms \
+  --compare-metric decode_tps \
+  --regression-threshold 5 \
+  --fail-on-regression \
+  --markdown-report runs/regression_report.md
+```
+
+Use a budget file when CI or device-lab runs need different tolerances per
+metric or profile. For example, a long-context sweep can use p95 decode
+throughput while RAM gets a tighter global budget:
+
+```json
+{
+  "threshold_pct": 5,
+  "metrics": {
+    "ram_usage_mb": {"threshold_pct": 2}
+  },
+  "profiles": {
+    "context_sweep_4096": {
+      "decode_tps": {"stat": "p95", "threshold_pct": 3},
+      "time_to_first_token_ms": {"threshold_pct": 8}
+    }
+  }
+}
+```
+
+```bash
+cactus benchmark \
+  --compare runs/main_summary.json runs/pr_summary.json \
+  --budget-json runs/mobile_budgets.json \
+  --fail-on-regression
+```
+
 ### Prefill
 
 Pre-processes input text and populates the KV cache without generating output tokens. This reduces latency for subsequent calls to `cactus_complete`.