Qwen3.5 MoE support

[howard0su]

Summary

Implements full MoE inference with expert weight swapping for the Qwen3.6-35B-A3B model (256 experts/layer, 8 active) on a single 2080 Ti. The model's expert weights (~18.6 GB) far exceed VRAM, so we use a two-graph-per-layer execution pattern with a dynamic LRU
cache and optional layer pinning.

Key Changes

1. Expert Cache (expert_cache.h/.cpp) — LRU cache with configurable slots, batch async H2D transfers, per-layer miss tracking, priority-layer eviction protection (first 6 + last 3)
2. Two-Graph Execution — Graph A (attention + router + top-K) → CPU reads router results, loads misses → Graph B (expert FFN + shared expert + residual)
3. PinnedExperts — Bulk-load all 256 experts for selected layers into VRAM (zero cache misses). Strategy: always pin first + last layer, fill remaining budget from layer 1
4. DDTree for MoE — Tree-structured speculative decoding wired through MoE path (enabled but not recommended — wider batches amplify cache misses)

Performance (64 tokens, budget=6, 2080 Ti 22GB)

Configuration	tok/s	Hit Rate
No pinning, 5000 dynamic slots	12.9	79%
pin=16 (layers 0-14,39), 2000 slots	17.2	86%
pin=20 (layers 0-18,39), 2000 slots	18.2	87%
pin=38, slots=100	28	71%
DDTree budget=22, no pin	3.1	—

Usage

 cd dflash && cmake --build build --target test_dflash

 # Recommended (20 pinned layers):
 ./build/test_dflash \
   models/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf \
   models/draft/draft-Qwen3.6-35B-A3B.gguf \
   test/prompt_test.bin 64 /dev/null \
   --cache-slots=2000 --budget=6 --pin-layers=20

 # Tune for your VRAM:
 #   --pin-layers=N   (each layer ≈ 470 MB; first+last always pinned)
 #   --cache-slots=M  (each slot ≈ 1.8 MB)

[cubic-dev-ai]

5 issues found across 14 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="dflash/test/smoke_moe_forward.cpp">

<violation number="1" location="dflash/test/smoke_moe_forward.cpp:188">
P1: NaN/Inf only breaks the loop; the smoke test still exits 0, hiding forward-pass corruption.</violation>
</file>

<file name="dflash/src/qwen35moe_target_graph.cpp">

<violation number="1" location="dflash/src/qwen35moe_target_graph.cpp:30">
P2: Mutable global debug/timing state is updated from the forward path without synchronization, so concurrent inference can race and trigger UB.</violation>

<violation number="2" location="dflash/src/qwen35moe_target_graph.cpp:309">
P2: Graph A CUDA-debug error path returns without ggml_free(ctx_a), causing a context leak</violation>
</file>

<file name="dflash/src/gguf_target_loader.cpp">

<violation number="1" location="dflash/src/gguf_target_loader.cpp:681">
P2: MoE expert-source setup should validate tensor shapes/strides against the metadata before trusting `nb[2]` and expert counts.</violation>
</file>

<file name="dflash/test/test_dflash.cpp">

<violation number="1" location="dflash/test/test_dflash.cpp:3121">
P1: MoE path ignores the draft backend and computes the draft graph on the target backend, breaking split target/draft GPU setups.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}