MLX_KV_BITS=4 breaks tool arguments partway through a session (persists via reused prompt cache)

[KaoCSC]

MLX_KV_BITS=4 degrades agentic tool calling far more than "occasional garbled tool calls" — and because the prompt cache is reused, the damage persists for the life of the server process.

I hit this while tuning for memory headroom on a 32 GB machine, where 4-bit KV looks like an easy ~5 GB win. It isn't. Sharing the numbers because the failure is quiet and easy to attribute to "local models are just flaky."

Caveat up front: I tested divinetribe/Hermes-4-14B-abliterated-4bit-mlx, not the Qwen 2.5 Coder 14B that Claude Agentico.command ships with. I have not verified Qwen shows the same behaviour. Flagging it because Agentico defaults to MLX_KV_BITS=4 with MLX_KV_QUANT_START=0 — quantizing from token 0, i.e. more aggressive than the config I measured.

Numbers

scripts/test_mlx_server.py, same machine, same model, same build — only MLX_KV_BITS changed. Fresh server per sweep, multiple runs because the sampler makes single runs noisy:

MLX_KV_BITS	Passed / 14, across runs
0	13, 13, 13, 12
4	10, 5, 7, 3

4-bit isn't just lower, it's unstable — 3 to 10 out of 14.

The failure mode matters more than the pass rate

Dominant error is Tool 'Bash' has empty arguments — 12, 20, and 14 occurrences in the three 4-bit runs. The model emits a syntactically valid tool call with the right name and an empty argument object.

Critically, once these start they largely don't stop. Run 1, in execution order:

  ✗ Simple Bash command — Expected tool 'Bash' but got: ['Glob']
  ✓ Create directory with mkdir
  ✗ Read a file — Expected tool 'Read' but got: ['Bash']
  ✓ Complex Bash with pipes
  ✓ Edit a file
  ✓ Multi-tool: find then read
  ✓ Rapid-fire #1: Run: echo 'test 1'
  ✗ Rapid-fire #2: Run: echo 'test 2' — Tool 'Bash' has empty arguments
  ✗ Rapid-fire #3: Run: pwd — Tool 'Bash' has empty arguments
  ✗ Rapid-fire #4: Run: date — Tool 'Bash' has empty arguments
  ✗ Rapid-fire #5: Run: whoami — Tool 'Bash' has empty arguments
  ✗ Calendar Step 1: Create month folders — Tool 'Bash' has empty arguments
  ✗ Calendar Step 2: Delete all months except September — Tool 'Bash' has empty arguments
  ✗ Calendar Step 3: Verify remaining folders — Tool 'Bash' has empty arguments

Seven healthy requests, then every subsequent one fails the same way. Run 3 degrades from the second request and never recovers (3/14). The same suite at MLX_KV_BITS=0 on the same server:

  ✗ Simple Bash command — Expected tool 'Bash' but got: ['Glob']   <- assertion artifact, Glob is a fine answer here
  ✓ Create directory with mkdir
  ✓ Read a file
  ✓ Complex Bash with pipes
  ✓ Edit a file
  ✓ Multi-tool: find then read
  ✓ Rapid-fire #1..#5
  ✓ Calendar Step 1: Create month folders
  ✓ Calendar Step 2: Delete all months except September

Why it persists

server.py keeps a single module-level _prompt_cache and reuses KV state across requests — that's the point of the caching design, and the logs confirm it (Cache hit: 727 reused, 390 new tokens to prefill). With quantization on, error introduced into that cache is carried into every subsequent request rather than discarded. So this is not per-request numeric noise you can retry your way out of; it's state that goes bad and stays bad until the process restarts.

Worth noting recover_garbled_tool_json / MAX_TOOL_RETRIES doesn't help here: the JSON isn't garbled. It parses cleanly into {}. The retry path keys off tool-intent phrases in unparsed text, so a well-formed call with empty arguments sails straight through.

Evidence this is cache-state rather than sampling: 16 identical single-shot requests against a warm 4-bit server (same 6-tool schema, same prompt shape) returned Bash with correct arguments 16/16. The corruption only shows up once a session has accumulated enough reused cache.

It also doesn't buy what you'd expect

• Not faster. 44.4s vs 38.5s total across the suite (mean 3.2s vs 2.8s per turn). Quantize/dequantize overhead exceeds the bandwidth saving at these context lengths.
• No memory saved at realistic prompt sizes. 8222 MB vs 8229 MB resident (footprint -p). With MLX_KV_QUANT_START=256 and ~700–1100 token prompts the cache is small either way. For this model (40 layers, 8 KV heads, head_dim 128) the KV cache is only 0.67 GB at 4K context; the ~5 GB saving needs something near the full 40K.

So on a 32 GB machine the trade is strictly bad: no speed, no meaningful memory saved, materially worse tool calling.

On Claude Agentico.command

The launcher's stated rationale is sound — Claude Code's ~5860-token system prompt does make the KV cache the dominant consumer, and kIOGPUCommandBufferCallbackErrorOutOfMemory is a real failure on 16 GB. I'm not arguing the default is wrong for 16 GB, where the alternative may be not running at all.

But the header currently sets expectations at "occasional garbled tool calls," and at least on Hermes the actual behaviour is closer to "tool arguments stop being populated partway through a session and don't come back." If Qwen 2.5 Coder behaves similarly, that's worth saying more loudly in the comment, since the symptom in a real session looks like the model getting confused rather than a tuning knob misfiring.

Possible directions, in rough order of effort:

1. Document the trade-off honestly in the Agentico header, and note that MLX_KV_BITS=0 is strongly preferred wherever RAM allows.
2. Drop MLX_KV_BITS to 0 by default on machines with headroom — the launcher could pick from hw.memsize the way setup.sh already picks a model.
3. Invalidate or rebuild the prompt cache periodically when KV_BITS is on, trading some prefill cost for bounded error accumulation.
4. Add an empty-arguments check to the retry path: a tool_use block whose input is {} for a tool with required properties is almost certainly a miss and could trigger the existing retry.

Happy to test any of these on Hermes, and to re-run the sweep against Qwen 2.5 Coder 14B to confirm whether Agentico's shipped default is affected — just say which would be useful.

Environment

• MacBook Air, Apple M5 (base), 32 GB unified memory, fanless, macOS 25.5.0
• Python 3.12.13, mlx 0.32.0, mlx-lm 0.31.3
• Model: divinetribe/Hermes-4-14B-abliterated-4bit-mlx (Qwen3 arch, 40 layers, 40K context)
• MLX_KV_QUANT_START=256 for the 4-bit runs
• Server run from proxy/server.py at a88bc96, plus the two fixes in fix: stop generation on the tokenizer's real EOS, and tolerate empty env ints #41 (without the EOS fix in that PR, this model can't complete the suite at all, so the comparison isn't runnable on a clean checkout)

[KaoCSC]

Correction — I tested this against Qwen 2.5 Coder and Gemma 4, and the scope of my original report was too broad. Retracting the part about Claude Agentico.command.

Same machine, same harness, 3 runs per config.

Qwen 2.5 Coder 14B — Agentico's shipped default is fine

mlx-community/Qwen2.5-Coder-14B-Instruct-4bit, tested at Agentico's exact config (MLX_KV_BITS=4, MLX_KV_QUANT_START=0 — more aggressive than the QUANT_START=256 I used on Hermes):

MLX_KV_BITS	QUANT_START	Passed / 14
0	256	14, 14, 14
4	0	14, 14, 14

Zero empty-argument failures in either arm. No degradation at all. My suggestion that Agentico's default might be hurting tool calls is not supported — please disregard that section of the original report. The default looks correct for the model it ships with.

Gemma — the trade-off doesn't exist, confirming by code and by run

server.py:141 force-disables KV quantization for any path containing gemma, so a 4-vs-0 comparison is 0-vs-0. Confirmed empirically — requesting MLX_KV_BITS=4 on divinetribe/gemma-4-12B-it-abliterated-4bit-mlx-text logs:

[21:13:31] Gemma detected: disabling KV cache quantization (RotatingKVCache NYI)
[21:13:31] KV cache: full precision

Gemma users are unaffected by construction. Worth stating plainly somewhere, since MLX_KV_BITS reads like a working knob for them.

"Empty arguments" is not a reliable KV-quantization signature

I leaned on that error string too heavily. Gemma 4 12B produces 4 empty-argument failures per run at full precision (Tool 'Write' has empty arguments, Tool 'Glob' has empty arguments), scattered rather than cascading. So it's a general small-model failure mode, not proof of cache corruption.

What still stands

The Hermes-specific comparison, which was a controlled A/B — same model, same build, same harness, only MLX_KV_BITS changed:

MLX_KV_BITS	Passed / 14	empty-arg failures per run
0	13, 13, 13, 12	~0
4	10, 5, 7, 3	12, 20, 14

and the cascade shape within a run (7 clean requests, then near-total failure that never recovers) still points at accumulated error in the reused _prompt_cache.

Revised scope: this is specific to divinetribe/Hermes-4-14B-abliterated-4bit-mlx, and possibly to Qwen3-architecture models generally — I've only tested the one. It is not a property of MLX_KV_BITS=4 as such. Qwen 2.5 Coder (Qwen2 arch) is completely unaffected; Gemma can't reach the code path.

Given that, the actionable items from my original list shrink to: document that MLX_KV_BITS is a no-op on Gemma, and note that KV quantization is worth A/B-ing per model rather than assuming it's free. The Agentico suggestions can be dropped.

Apologies for the overreach in the first post — I should have tested the shipped model before drawing the launcher into it.

---

Separately: Qwen 2.5 Coder scores 0/12 on the stock server

Worth its own issue, and I'll open one with a fix, but flagging here since it directly affects Agentico and Chat.

Before I added bare-JSON parsing, Qwen 2.5 Coder scored 0 passed / 12 in every run at both KV settings. Not a tool-calling weakness — it emits textbook-correct calls with no wrapper tags:

{
  "name": "Bash",
  "arguments": {
    "command": "ls /tmp",
    "description": "List files in /tmp directory"
  }
}

parse_tool_calls accepts <|tool_call>…<tool_call|>, <tool_call>{…}</tool_call>, and <function=…><parameter=…>, but not unwrapped top-level JSON, so every call is missed. The retry path re-prompts twice explicitly demanding <tool_call> tags and Qwen declines both times; recover_garbled_tool_json can't help because the JSON was never garbled.

Adding a ~10-line bare-JSON branch took it from 0/12 to 14/14 — the best score I've measured on this machine, better than Hermes' 13/14. That also means my KV numbers above are only meaningful with that fix: at 0/12 both arms sit on the floor, so the comparison was undecidable until the parser could see the calls.

Environment

MacBook Air, Apple M5 (base), 32 GB, macOS 25.5.0 · Python 3.12.13 · mlx 0.32.0 · mlx-lm 0.31.3 · server at a88bc96 + #41.

[nicedreamzapp]

re-measured this on main now that #47 (the prompt-cache trim bug from #46) is fixed. short version: this is real and it is its own bug, but it is a lot milder than the original numbers, and those numbers were contaminated.

same model as your original report, divinetribe/Hermes-4-14B-abliterated-4bit-mlx, scripts/test_mlx_server.py, MLX_KV_QUANT_START=256, 3 runs each. i added a third arm: a build with KV reuse disabled entirely, as a ground truth (equivalent to your fresh-server-per-session control, without 14 model reloads).

build	KV_BITS=0	KV_BITS=4
no cache reuse (ground truth)	14, 14, 14	13, 13, 13 (1 empty-arg each)
main (cache fixed)	13, 14, 14	13, 10, 11 (1-2 empty-arg each)
pre-#47 (cache broken)	14, 14, 14	12, 13, 13

three things fall out of it:

1. it is not the cache bug wearing a disguise. with KV reuse switched off completely, KV_BITS=4 still drops a test and still produces Tool 'Bash' has empty arguments — your exact signature. so quantized KV degrades tool-argument emission on its own.

2. the magnitude is much smaller than reported. you measured 3-10 of 14. ground truth here is a stable 13 of 14. the difference is that your runs had the trim bug stacked underneath, which is what you suspected in #46.

3. quantization and cache reuse compound. ground-truth KV_BITS=4 is a stable 13/13/13. with reuse on it becomes 13/10/11 and gets noisier. trimming and re-extending an already-quantized cache appears to accumulate error in a way neither does alone. that is probably the more interesting half of this issue now.

worth adding: on a different model (Qwen3-Coder-30B) the old cache bug inflated the same suite from a true 12/14 to a fake 14/14, by priming the model with the previous request's tool call. so pre-#47 numbers are unreliable in both directions, not just downward. anything benchmarked against a long-lived server before #47 needs re-running, including the pre-#47 row above.

your correction stands — Qwen 2.5 Coder at KV_BITS=4, QUANT_START=0 was 14/14 for you, and that is what Claude Agentico.command and Claude Chat.command actually ship. those two set KV_BITS=4 deliberately to keep a 16 GB Mac under the memory ceiling, so leaving them alone for now.

keeping this open for the compounding effect in point 3.