Feature request: support DeepSeek-V4 KV cache pooling in vLLM and SGLang

[shipiyouniao]

Feature request

Would the maintainers be open to adding official kvcached support for DeepSeek-V4 KV cache pooling in both vLLM and SGLang?

DeepSeek-V4 uses model/runtime-specific KV cache paths rather than only the common MHA/GQA/MLA pool classes. For SGLang, the relevant path appears to use dedicated DeepSeek-V4 pool classes such as DeepSeekV4TokenToKVPool / DeepSeekV4SingleKVPool, so support may need a dedicated adapter instead of relying only on the existing MHATokenToKVPool / MLATokenToKVPool replacement path.

The vLLM side may also need a separate compatibility check because DeepSeek-V4 support has been moving quickly across vLLM versions and may not share the exact same KV cache interfaces as older DeepSeek-V2/V3 MLA models.

Before sending a PR, I wanted to ask whether this model family is something the project would like to support upstream, and whether there is a preferred design direction for model-specific KV pool families like DeepSeek-V4.

One specific design question: should kvcached aim to fully replace the DeepSeek-V4 KV/cache pool stack, including compressed/cache side pools, or should the first upstream step only manage the full-attention KV portion and leave DeepSeek-V4-specific compressed/indexer/state pools owned by the serving runtime?

Thanks.

[jiarong0907]

@shipiyouniao Thanks! We are definitely interested in supporting this model family.

For the first step, let's leave the V4-specific pools (compressed/HCA KV, indexer, compressor state) statically reserved and owned by the runtime since the V4 paths in both vLLM and SGLang are moving fast and are very implementation-specific.

Binding kvcached's allocator to them now would mean re-validating on every patch release, with real corruption risk on the indexer (decode hot path, aliases compressed entries). Not worth it until those interfaces settle.

So for the PR: a dedicated V4 adapter, and have kvcached just account for the runtime's static reservation so co-located models don't OOM against memory it can't see. Elastic management of the compressed store is future work once V4 stabilizes.

Happy to discuss more. Thanks again!

[shipiyouniao]

That sounds reasonable.

For context, we have a local prototype on the SGLang side that only routes the full-attention KV portion of the DeepSeek-V4 path through kvcached while leaving the V4-specific compressed/indexer/state pools owned by SGLang. That prototype made us lean toward the same staged design: do not bind kvcached to the compressed/indexer/state pools yet, and start with a narrow adapter around the parts that are stable enough to reason about.

I would avoid treating our prototype as an upstream-ready implementation for now because it was tested together with other local serving changes. But the direction seems consistent with your suggestion: first make kvcached aware of V4 runtime-owned reservations, then decide later whether any part of the V4 cache stack is stable enough for elastic ownership.

One heads-up for reproduction: in our local SGLang DeepSeek-V4 tests, the stable launch path needed a few runtime-side workarounds:

• --dsa-topk-backend torch
• SGLANG_DSA_FUSE_TOPK=false
• with kvcached enabled, SGLANG_OPT_DEEPGEMM_HC_PRENORM=0

In our MPS/router-style multi-worker setup, we also needed:

• SGLANG_OPT_USE_TILELANG_MHC_PRE=0

The last one may not be needed for a single-worker reproduction, but I mention it in case you test with MPS or co-located workers. I think these are mostly SGLang DeepSeek-V4 kernel/runtime issues rather than kvcached design issues, but they may help avoid false negatives while testing the adapter.

For the first kvcached PR, would you prefer the adapter to report one total runtime-owned reservation per rank/GPU, or separate reservations for the V4 side pools such as SWA, C4, C128, indexer, and compressor state?

Also, would you be open to an external feature PR for this? It will take us some time to prepare because our local prototype depends on a few earlier kvcached compatibility fixes, some of which are still under review, and I would like to keep the feature PR small enough to review cleanly.

[jiarong0907]

@shipiyouniao Great, glad we're aligned.

On granularity: report per-pool not a single total could be better. kvcached just consumes the sum for budgeting in v1, but the breakdown is nearly free now (SGLang allocates them separately) and saves re-plumbing when we later check whether any pool — likely the compressed store — is stable enough for elastic ownership.

An external PR is very welcome, and small is rightt.

Thanks for the repro flags and agreed those are SGLang V4 runtime issues, not kvcached design ones. Put them in the PR notes and pin the SGLang version you validated against. Looking forward to it.

[shipiyouniao]

Thanks again for the guidance.

I opened #376 as a small first step following the per-pool reservation direction: it only accounts for SGLang DeepSeek-V4 runtime-owned pools and keeps those pools owned by SGLang. It does not take over full-attention, compressed, indexer, or state-pool lifecycle in this PR.

A note on dependencies: #376 is stacked on a few earlier compatibility fixes (#336, #340, #341, #365, #373), because without those fixes the multi-process/TP validation path can fail or hang before the DeepSeek-V4 reservation logic is exercised. I added that dependency note to the PR body.

If this reservation-only direction looks acceptable, would the maintainers prefer the next PR to add SGLang DeepSeek-V4 full-attention KV pool takeover, while still leaving compressed/indexer/state pools runtime-owned? Or would it be better to keep the first upstream step strictly reservation-only until the dependency fixes land?

No performance claims from my side yet; I am keeping this PR focused on accounting/budgeting correctness.