Cannot run Qwen3.6-27B on RTX 4090 24 GB: server.py loads both models eagerly, bench protocol works

[n00b001]

Summary

The README claims Qwen3.6-27B Q4_K_M at 128K on a single RTX 3090 (24 GB). I cannot reproduce this through server.py. The benchmark (pflash/tests/bench_niah_cpp.py) works because it uses a park/unpark dance and sets --max-ctx 16384 sized for the compressed prompt — not the full source.

server.py loads both target + draft at startup, keeps them resident, and allocates KV cache eagerly based on --max-ctx. This leaves no headroom on a 24 GB card regardless of flag combinations.

Hardware

• GPU: NVIDIA GeForce RTX 4090, 24564 MiB VRAM, SM 8.9
• CUDA: 13.2, nvcc 13.2.78
• Driver: 595.71.05
• Target: Qwen3.6-27B-Q4_K_M.gguf (17 GB on disk)
• Draft: z-lab/Qwen3.6-27B-DFlash model.safetensors
• Drafter: Qwen3-0.6B-BF16.gguf

The gap: bench vs server

The benchmark (pflash/tests/bench_niah_cpp.py) does this per-request dance:

# 1. Compress with drafter (daemon idles at ~3 GB after parking target+draft)
compressed_ids = dflash.compress(ids, args.keep_ratio, args.drafter_gguf)

# 2. Free drafter, restore target+draft for generation
dflash.free_drafter()       # release drafter weights + KV + BSA scratch
dflash.unpark_target()      # ~16 GB
dflash.unpark_draft()       # +draft weights for spec decode

# 3. Generate on compressed prompt (fits in small max_ctx)
out_ids = dflash.generate(target_ids, args.n_gen)

# 4. Re-park draft for next iteration
dflash.park_draft()

Key: --max-ctx 16384 is sized for the compressed prompt, not the source. After compressing 128K → 2.5K tokens (keep_ratio=0.02), generation fits with minimal KV cache.

server.py does none of this:

• Loads target + draft at startup, keeps them resident forever
• Allocates max_ctx KV cache eagerly on daemon spawn
• Has no park/unpark between prefill and generate
• --budget 0 does not skip draft loading (clamped to 64 in C++ config)

Configurations tested via server.py

All failed with OOM. Each reached [daemon] ready then crashed on first request (~20-23K token prompt).

Boot log (common)

[target] target loaded: 851 tensors on GPU 14.99 GiB, tok\_embd 682 MiB CPU-only (q4\_K)
[draft]  loaded
[daemon] ready

Target + draft = ~20 GB VRAM. Only ~4 GB headroom.

1. PFlash on, max-ctx 200K

python scripts/server.py --target models/Qwen3.6-27B-Q4_K_M.gguf \
  --draft models/draft --bin build/test_dflash \
  --max-ctx 200000 --budget 16 --fa-window 0 \
  --prefill-compression auto --prefill-drafter models/Qwen3-0.6B-BF16.gguf

Result: OOM during target prefill intermediates (before PFlash even fires)

[prompt] 16904 tokens
[prefill] token-seg ubatch=384
ggml_backend_cuda_buffer_type_alloc_buffer: allocating 1215.29 MiB on device 0: cudaMalloc failed: out of memory
ggml_gallocr_reserve_n_impl: failed to allocate CUDA0 buffer of size 1274320896
prefill build @src/qwen3_0p6b_loader.cpp

2. PFlash on, max-ctx 131K, larger FA window

--max-ctx 131072 --budget 22 --fa-window 32000 --prefill-compression auto ...

Result: Prefill succeeded but rollback cache OOM

[prefill] 20042 tokens in 15.23 s, last_tok=0
ggml_backend_cuda_buffer_type_alloc_buffer: allocating 1854.38 MiB on device 0: cudaMalloc failed: out of memory
cache migration: ggml_backend_alloc_ctx_tensors failed for rollback cache

3. PFlash off, budget=0 (no spec-decode), max-ctx 65K

--max-ctx 65536 --budget 0 --fa-window 16384 --prefill-compression off

Result: Prefill succeeded but rollback cache OOM (~5 GB)

[cfg] ... budget=64 temp=1.00 ... fa_window=16384
[prefill] 21051 tokens in 16.30 s, last_tok=0
ggml_backend_cuda_buffer_type_alloc_buffer: allocating 4957.12 MiB on device 0: cudaMalloc failed: out of memory
alloc_tensor_range: failed to allocate CUDA0 buffer of size 5197922304
cache migration: ggml_backend_alloc_ctx_tensors failed for rollback cache

4. PFlash off, budget=0, max-ctx 24K (smallest tested)

--max-ctx 24000 --budget 0 --fa-window 16384 --prefill-compression off

Result: Same ~5 GB rollback OOM — identical allocation size regardless of max-ctx

[cfg] ... budget=64 temp=1.00 ... fa_window=16384
[prefill] 23071 tokens in 17.96 s, last_tok=0
ggml_backend_cuda_buffer_type_alloc_buffer: allocating 4957.12 MiB on device 0: cudaMalloc failed: out of memory
alloc_tensor_range: failed to allocate CUDA0 buffer of size 5197922304
cache migration: ggml_backend_alloc_ctx_tensors failed for rollback cache

Bugs observed

1. --budget 0 does not disable draft loading. C++ config clamps to budget=64. Draft model (~5 GB) still loaded → dead VRAM when spec-decode is effectively off.

2. Rollback cache allocated unconditionally. ~5 GB regardless of max-ctx, budget, or prompt length. Happens even with effective budget=0 (no draft tokens).

3. server.py does not implement the park/unpark dance that makes 24 GB cards work. The benchmark proves it is possible — but requires the two-phase compress → generate protocol from bench_niah_cpp.py.

What works

The bench harness at pflash/tests/bench_niah_cpp.py completes successfully:

• Spawns daemon with --max-ctx 16384 (sized for compressed prompt)
• Parks target+draft → compresses with drafter → frees drafter
• Unparks target+draft → generates on compressed prompt → parks draft
• 128K source → 2.5K compressed → fits in small KV cache

Request

Either:

1. Make server.py implement the park/unpark dance for PFlash requests (matching bench protocol)
2. Add a --no-draft flag to skip draft loading when budget=0
3. Skip rollback cache allocation when budget ≤ 0
4. Document that server.py does not support 24 GB cards and provide an alternative entry point

The benchmark proves the algorithm works on 24 GB — but only with careful VRAM sequencing that server.py does not implement.

[davide221]

Thanks for reporting it, server does implement unpark/park logic. server.py:199-233. Am I missing something? We will look into the OOM errors

[n00b001]

You're right, lines 199-233 (_maybe_compress) do implement the park/unpark dance via compress_text_via_daemon. But that code path only runs when all three conditions are met:

1. --prefill-compression is set to auto or always (default is off)
2. The prompt token count exceeds --prefill-threshold (default 32000)
3. --prefill-drafter path is provided

When any of these conditions isn't met, _maybe_compress returns early at line 206 and the park/unpark dance never executes. Both target (~15 GB) + draft (~5 GB) stay resident for the entire generate call.

The OOM still happens in two scenarios:

A. PFlash disabled (default): Target + draft = ~20 GB VRAM, leaving only ~4 GB headroom. The daemon then tries to allocate rollback cache (~5 GB unconditionally, even with budget=0) → OOM. This is what Configs 3 and 4 in the issue reproduce.

B. PFlash enabled but prefill crashes first: Even with --prefill-compression auto, the daemon boot log shows target + draft both loaded at startup (~20 GB). When a long prompt arrives, prefill intermediates need extra scratch space on top of that 20 GB → OOM before compression even fires. This is Config 1.

The root cause: The C++ daemon loads both target and draft eagerly on boot (the [target] loaded + [draft] loaded + [daemon] ready sequence). The park/unpark in _maybe_compress only runs after the daemon is already at 20 GB. The bench works because it starts with a small --max-ctx (16384, sized for compressed prompt) and immediately parks target+draft before loading drafter.

Two things needed to make server.py work on 24 GB:

1. The daemon should park target+draft before prefilling a long prompt, not just during compression
2. Rollback cache allocation (~5 GB) is unconditional regardless of budget or max-ctx — it seems to allocate based on something other than the actual context size

Can you confirm whether the README's "128K on RTX 3090" claim assumes PFlash compression is always active? And is there a recommended server.py flag combination for 24 GB cards?

[adybag14-cyber]

correct fails on rtx 4090

[n00b001]

Quick clarification on a point that may be confusing: the "prefill" that succeeds in Configs 3 and 4 is normal prompt prefill (the target model processes the user's tokens through its attention layers). PFlash compression was completely OFF (--prefill-compression off) and not involved at all.

The sequence is:

1. Daemon boot: load target (~15 GB) + draft (~5 GB) = ~20 GB VRAM
2. Request arrives → normal prefill of 21K tokens succeeds (KV cache squeezes into remaining headroom)
3. After prefill, DDTree speculative decoding tries to allocate rollback cache (~5 GB unconditionally)
4. cudaMalloc fails: only ~4 GB free

So the failure has nothing to do with PFlash. It's that even a completely plain request (no compression, no spec-prefill, budget=0) still OOMs because:

• The draft model stays loaded for DDTree verification
• --budget 0 is clamped to 64, so rollback cache is allocated regardless
• ~25 GB total > 24 GB GPU

The park/unpark in _maybe_compress only helps PFlash compression requests. It does nothing for the base case.

[Quitetall]

Same issue on RTX 4090 (24 GB), filed separately as #88 before finding this.

In my case it manifests as GGML_ASSERT(ggml_nelements(a) == ggml_nelements(b)) failed in qwen35_target_graph.cpp which I now believe is a cascade from the rollback cache OOM, not a standalone bug. The partial allocation corrupts tensor shapes, then ggml_cpy hits the mismatch.

Tested at commit ca87136 (pre-refactor) and HEAD (69ebfb9), both Qwen3.5 and 3.6 targets, budget 1/4/6, GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 on and off. Unified memory lets the rollback cache "succeed" (migration takes ~12s) but the assertion still fires so it may not just be an allocation failure but also a shape computation issue in the rollback path.
+1 on the request to skip rollback cache when budget=0 and honor the flag without clamping to 64.

[noonghunna]

Hitting the same eager-load behavior on dual 3090 (24 GB each) and wanted to add data + a related finding from running verify-stress against server.py with PFlash enabled.

Setup

• 2× RTX 3090, CUDA 12.x, sm_86
• Target: Qwen3.5-27B-Q4_K_M (/mnt/models/gguf/qwen3.5-27b/..., 14.91 GiB on GPU per [daemon] [target] target loaded log)
• DFlash draft: z-lab/Qwen3.5-27B-DFlash (~2 GB safetensors)
• PFlash drafter: unsloth/Qwen3-0.6B-GGUF BF16 (1.2 GB)
• server.py --max-ctx 65536 --fa-window 2048 --daemon --prefill-compression auto --prefill-threshold 8000 --prefill-keep-ratio 0.05

Single-GPU result (default config — both models on GPU 0)

[drafter] forward+score in 0.58s S=9806
[drafter] score_and_compress total 0.58s S=9806 kept=462 (15/307 chunks)
  [daemon] [drafter] freed
  [daemon] [unpark] target restored
  [daemon] [unpark] draft restored
  [daemon] [snap] inline slot=4 cur_pos=473
ggml_backend_cuda_buffer_type_alloc_buffer: allocating 75.63 MiB on device 0: cudaMalloc failed: out of memory
ggml_gallocr_reserve_n_impl: failed to allocate CUDA0 buffer of size 79299072
ddtree verify build failed

OOMs at 75 MiB on the 2nd request. Your description matches: target+draft+pflash drafter+KV pool all on a single 24 GB card has no headroom for the gallocr workspace on the second cycle, even after [drafter] freed / [unpark] cycles run.

Workaround on dual-GPU (and a missing piece)

PR #80 added --target-gpu / --draft-gpu / --draft-feature-mirror to the daemon binary. But server.py doesn't forward those flags to the daemon subprocess — so the OpenAI-compatible serving path can't use the split.

A 6-line edit in build_app (read env vars LUCEBOX_TARGET_GPU / LUCEBOX_DRAFT_GPU / LUCEBOX_DRAFT_FEATURE_MIRROR, append matching CLI flags to cmd) is enough to wire it through. With LUCEBOX_TARGET_GPU=0 LUCEBOX_DRAFT_GPU=1 LUCEBOX_DRAFT_FEATURE_MIRROR=1 the daemon cfg flips:

[daemon] [cfg] ... draft_feature_mirror=1 target_gpu=0 draft_gpu=1

VRAM split lands at GPU 0 = 17.5 GiB, GPU 1 = 3.5 GiB, and the 30K compress cycle survives. (Filed as a separate comment on #42.)

But — a different bug surfaces on the 3rd cycle (filed separately)

After splitting target/draft, server.py makes it through the 10K + 30K probes but then dies on the 25K tool-prefill probe with CUDA error: an illegal memory access was encountered in ggml_backend_cuda_buffer_context::~ggml_backend_cuda_buffer_context → cudaFree(dev_ptr). Different signature from the closed #46/#49 (those were 2nd-prompt gallocr leaks → OOM; ours is 3rd-cycle illegal-mem-access on free, not OOM). Filed as new issue (link forthcoming).

Hope the data helps. Tracking PFlash on club-3090 stack here — happy to provide additional logs if useful.

[noonghunna]

Cross-link: filed the 3rd-cycle CUDA illegal-mem-access as #95.