Packed-panel CQ4 weight format: NEON + SME2 kernels

[ncylich]

The packed-panel weight format and the kernels over it — NEON and SME2. Stacked on #717 (the fused GEMV dispatch); retarget to main after it merges. Besides the format and SME2 work this PR carries two small pipeline fixes its bundles depend on (see "Pipeline fixes"); all numbers below are measured against #717's engine on the same bundles/devices, so they isolate exactly what this PR adds.

Packed-panel format

A kernel-ready layout for CQ4 weights, emitted into the .cact weight files by the transpiler and consumed zero-copy from the mmap. No runtime cache, no second resident weight copy — the file is the kernel's operand stream.

• Panels (data region): [SB64][num_groups][gs/4][128B]; each 128B kg-block is 256 nibbles (low-first) = the int8 SDOT panel [4 vec][16 ch][4 K]. Channels past N zero-padded.
• Scales region: folded fp32 panel norms alongside the legacy fp16 norms (embedding row decode), plus the transposed rotation for orthogonal lm_heads (virtual 128-wide groups written directly — an exact regrouping).
• Header: FLAG_PACKED_PANELS (1<<5) + unique .cq4p.weights suffix; new bundles ship only that file, so pre-panel engines fail loudly.
• The kg-block is simultaneously the NEON SDOT micro-tile and the SME2 LUTI4/ZT0 streaming layout — one format serves both ISAs, which is what lets the runtime cache die. The compatibility cost is two vzips in the NEON unpack (9 vs 7 ops/32 weights): free on wide-issue cores, visible only compute-bound on Cortex-X4-class (see phone numbers).

Kernels

NEON: panel SDOT GEMV; fused panel GEMM (M>1) consuming file bytes directly — the legacy GEMM re-expands packed nibbles into a 4×-larger int8 buffer per call, panels never do; orthogonal lm_head GEMV via the file's transposed rotation; batched orthogonal embedding row decode.

SME2 (compile probe + runtime cpu_has_sme2() with streaming-SVL==64 gate): LUTI4/ZT0 GEMV leaf, fused SMOPA GEMM for all M>1 panel matmuls, SME prefill attention. SME workers replace NEON workers inside the same thread budget (flat k=min(2,nt−1), the measured speed/power frontier). All SME leaves emit raw int32 partials; the fp32 rescale stays in NEON — SME numerics are identical to NEON (locked by an agreement test). Non-SME2 silicon runs the NEON panel path bit-for-bit. SME TUs build at -march=armv8.2-a+…+sme2 with the vg1x4 GEMV TU at -O1 (documented clang miscompile); knowledge base in cactus-kernels/docs/sme/.

Tests

Panel GEMV/GEMM/orth/embed vs FP32 oracle with [sme2] variants of every test (backend control), M/N tails included; NEON↔SME agreement (identical int32 partials); SME attention differentials vs hybrid NEON + double-precision oracle; panel file loaded through the real graph matches direct kernels at M=1/M>1; transpiler panels byte-identical to the C reference encoder (ctypes); panel qdq round-trips legacy dequant at 0.0 diff.

Measured — M4 Pro

Kernel (Gemma shapes, GF; baseline = #717 fused-dispatch legacy):

shape	legacy (fused)	panel NEON	panel + SME2
kv_proj 1×1536×512	158	125	281
down 1×6144×1536	268	229	337
GEMM M16 1024²	175	285	856
GEMM M256 1024²	1188	1764	2557
prefill attention	—	—	4.6× hybrid NEON

E2E (~1k prompt + 32 decode, alternating cycles, coherent; baseline = same engine, legacy bundle):

model	config	prefill tok/s	decode tok/s	TTFT ms
gemma-4-e2b-it	legacy (#717)	302	49.2	3263
	panel + SME2	350 (+16%)	50.4	2811 (−14%)
qwen3-1.7b	legacy (#717)	279	52.3	3545
	panel + SME2	400 (+43%)	53.0	2471 (−30%)
lfm2-350m	legacy (#717)	183	173	5685
	panel + SME2	195 (+7%)	185 (+7%)	5339 (−6%)

Beyond throughput: decode energy 358 → 294 mJ/token (−18%, 2 SME workers replace ~10 cores); chunk-aligned prompts reach 977 tok/s prefill / 1.10 s TTFT (1024-token prompt — arbitrary-length prompts pay a per-token tail that caps blended prefill, independent of this PR); cold start carries zero cache build vs the prior runtime-cache SME design (3.3 s vs 5.9–7.4 s first-message TTFT, ~1.2 GB less RAM).

Measured — on-device (#698 harness, 512+32 spec; NEON panel path, SME gates self-disable — portability validated)

device	model	prefill tok/s	decode tok/s
Samsung SM-S942U1	gemma-4-e2b-it	304 → 402 (+32%)	26.2 → 24.0 (−8%)
	qwen3-1.7b	99 → 110 (+11%)	23.1 → 18.4 (−20%)
	lfm2-350m	105 → 96 (−9%)	102 → 88 (−14%)
Pixel 10a (Tensor G4)	gemma-4-e2b-it	84 → 104 (+24%)	10.5 → 8.0 (−24%)
	qwen3-1.7b	34 → 40 (+18%)	10.9 → 9.1 (−17%)
	lfm2-350m	44 → 45 (par)	41 → 47 (+15%)

(Legacy baselines are #717's fused dispatch — a deliberately hard bar: #717 already captured the dispatch wins on the old format.) The split is clean: panels win prefill on phones (+11–32%) and everything on M-class; the legacy format with #717's dispatch keeps the phone decode crown (the 2-op NEON unpack tax is visible on compute-bound Cortex cores, and phone decode at the DRAM roofline gains nothing from the format). Both formats remain fully supported by one engine — bundle choice per deployment, with sliding-window-safe behavior unchanged.

Pipeline fixes carried by this PR

• --torch-dtype for convert (bf16-safe model loading). Convert hard-coded fp16 HF loading; Gemma weights exceed fp16 range and were silently corrupted (the historical source of garbled Gemma bundles). Panel bundles for Gemma are built with --torch-dtype bfloat16; quantization math unchanged.
• Dense-MLP TQ fusion disabled by default. The fused node costs ~15 ms/decode-step on Tensor G4 (+2% on M4) and its gate keys on ".cq" in the weight path — which this PR's .cq4p.weights suffix would satisfy, silently enabling the regression on panel bundles. Re-enable programmatically via FusionConfig(enable_dense_mlp_tq_fused=True).

Loader / transpiler

io.cpp points packed_panels / norm_panels / rotation_t straight into the mmap; cache/registry/madvise machinery removed by construction. write_cq_tensor emits panels for CQ4 matmuls and orthogonal lm_heads/embeddings (chunked for large vocabs), byte-identical to the engine reference encoder; qdq dequantizes panels for export round-trips.

[ncylich]

E2E validation on Gemma 4 E2B IT (M4 Pro)

Converted google/gemma-4-E2B-it end-to-end with this branch (cactus.convert --bits 4 --torch-dtype bfloat16 → all CQ4 matmul weights + the tied orthogonal lm_head emitted as .cq4p.weights panels → component transpile), and A/B'd it against the existing legacy interleaved-format bundle on the same engine build.

Protocol: ~1.2k-token prompt, 32 decode tokens, 6 alternating cycles, cycle 1 discarded. Both bundles produce coherent on-task completions.

	prefill tok/s	decode tok/s	TTFT ms	RAM MB
legacy interleaved bundle	261.2	42.02	3772	2341
packed-panel bundle	300.4	49.54	3279	2311
delta	+15.0%	+17.9%	−13.1%	−30

Per-cycle decode spread is <0.5 tok/s on both sides. The legacy baseline matches its previously documented numbers, and the panel bundle carries no load-time transformation — the gains come purely from the file-borne layout.

Note: converting Gemma requires the new --torch-dtype bfloat16 (2d68be8) — fp16 loading corrupts its out-of-range weights.

[ncylich]

Generalization: the panel format (NEON path) on two more architectures

Same protocol as the Gemma table (~1k-token prompt, 32 decode tokens, 3 alternating cycles, cycle 1 dropped, coherent output checked); legacy = the existing production interleaved bundles, panel = fresh converts on this branch; both on this PR's engine.

model	config	prefill tok/s	decode tok/s	TTFT ms
qwen3-1.7b (global GQA, chunked prefill)	legacy	239.4	36.96	4123
	panel	260.0 (+8.6%)	51.76 (+40.0%)	3796 (−7.9%)
lfm2-350m (hybrid conv+attention)	legacy	143.6	139.7	7249
	panel	184.9 (+28.8%)	176.7 (+26.5%)	5630 (−22.3%)
gemma-4-e2b-it (sliding+global, multimodal)	legacy	261.2	42.02	3772
	panel	300.4 (+15.0%)	49.54 (+17.9%)	3279 (−13.1%)

Decode gains of +18–40% across three very different architectures; panel conv projections on the LFM2 hybrid work out of the box.

One transpile-workflow note surfaced by this: the chunked-prefill component's chunk length is captured from the shape-capture prompt/ids, so short capture prompts produce small chunks (a 45-token chunk bundle prefilled ~40% slower than a 128-token one purely from per-chunk dispatch overhead). Worth pinning --input-ids length in bundle recipes; not a format issue (both layouts affected equally).

[ncylich]

On-device A/B (Android, PR #698 harness, 512+32 spec)

Same six bundles as the host tables, run on real phones via tests/android-e2e/run_spec.sh (1 warmup + 3 rounds, devices powered, Thermal Status 0). The engine binary is the full stacked build — neither device has SME2, so the runtime gates self-disable and this exercises exactly this PR's NEON panel path on third-party silicon (that fallback working untouched is itself the portability validation).

Samsung SM-S942U1 (big-core SoC) — round means:

model	legacy pf/dec tok/s	panel pf/dec tok/s	delta
lfm2-350m	70.4 / 69.8	96.1 / 87.6	+36% / +25%
qwen3-1.7b	69.3 / 18.2	109.8 / 18.4	+58% / +1%
gemma-4-e2b-it	204.1 / 17.4	300.0 / 17.9	+47% / +3%

Pixel 10a (Tensor G4):

model	legacy pf/dec	panel pf/dec	delta
lfm2-350m	45.0 / 35.2	45.0 / 47.4	par / +35%
qwen3-1.7b	33.9 / 10.4	40.5 / 9.1	+19% / ≈par
gemma-4-e2b-it	77.8 / 9.9	104.4 / 6.4	+33% / −35% ⚠

Decode parity on the 1.7–2B models on Samsung is expected: phone decode sits at the DRAM roofline and both formats stream the same 4-bit bytes — the format's decode wins show where compute/dispatch headroom exists (small models, and all models on M-class).

⚠ The one regression (Pixel gemma decode) is not the weight format: on-device CACTUS_PROFILE shows the entire delta is the gemma-only DENSE_MLP_TQ_FUSED graph node (100 of 162 ms/step; the qwen/lfm panel bundles contain no such node and show parity/wins on the same device, and the same gemma bundle is fine on Samsung and M4). Reproduces order-reversed, with thread-budget overrides, and big-cores-only — filed as a Tensor-G4-specific follow-up on the fused dense-MLP execution path.

[ncylich]

Pixel/Gemma decode regression: root-caused and fixed (54f2749)

Follow-up to the Android tables — the one regressing cell (Pixel 10a gemma decode, −35%) decomposed into three independent causes, none of them the weight format's bytes:

1. An accidentally-enabled graph fusion (fixed here). fuse_dense_mlp_tq gates on ".cq" appearing in the bound weight path, so convert-produced bundles (plain .weights names) never triggered it — the .cq4p.weights suffix switched it on for every CQ4 tensor as a side effect. The fused node runs identical kernels yet costs ~15 ms/decode-step on Tensor G4 (−15% decode), confirmed by re-transpiling the same panel weights with fusion off (6.5 → 7.7 tok/s) and by per-op profile diff (fused node 100.5 ms vs 124.7 ms of equivalent unfused matmuls + 4 ms gelu/multiply). It gains only ~2% on M4 and nothing on Samsung. Fix: fusion is now opt-in (--fuse-dense-mlp-tq).
2. The panel GEMV inner loop is Cortex-X4-unfriendly. Single-core-pinned (taskset 80) on the same prime core: panels 6.28 vs legacy 8.86 decode (−29%). The panel unpack costs 9 ops/32 weights vs the interleaved kernel's 7 (the two vzips) — free on M-class/8-Elite issue width, ~−29% per-byte on X4-class cores when not DRAM-bound. Gemma is the only suite model compute-bound enough on G4 to expose it (qwen/lfm sit at the bandwidth roofline → parity/wins). Filed as kernel follow-up: a Cortex-tuned panel unpack variant.
3. Thread budget: the panel budget (SB_PER_THREAD=8, the M4 power-frontier point) halves thread counts on small-N shapes vs the legacy formula; on G4's weaker cores CACTUS_GEMV_SB_PER_THREAD=4 recovers ~+7% decode.

Also ruled out with on-device experiments: thermal (order-reversed reruns), the orth lm_head driver and panel lm_head file (byte-swapped into the legacy bundle under its bound name: 10.0–10.5 tok/s, no cost), scheduling (big-cores-only pin), and the safe-product-scale path (trace shows scale=0).

Post-fix numbers (unfused graphs):

• Pixel gemma: decode 6.4 → 7.7–8.0 (8.3 with SB_PER_THREAD=4) vs legacy 9.9; prefill stays +33–44% (104–112 vs 78). Remaining gap is item 2.
• Samsung gemma, cool alternated pair: legacy 284 pf / 24.8 dec vs panels 402 pf / 24.0 dec — decode parity at the DRAM roofline, prefill +41% (the earlier +3%-decode table row was thermally tangled; both bundles now measure at the README's reference decode range).
• M4: defusing costs ~1 tok/s (53.5 → 52.3); recipes that want it back can pass the flag.