[Swift runtime] Sequential VLM engine rejects any decoder with more than two states

[baozzz1]

Model name

qwen3_vl and any other VLM whose text decoder mixes linear attention with full attention
(gated-delta / Mamba-style hybrids). The text-only engine already accepts these; the VLM engine
does not.

swift/Sources/CoreAILanguageModels/InferenceEngines/CoreAISequentialVLMEngine.swift

Command run

No command — this is a construction-time guard, and it is unconditional. Loading any
embeddings-input VLM bundle whose LLM function declares more than two states throws from
CoreAISequentialVLMEngine.init, before any inference runs.

All line numbers below are on 475c585.

macOS / iOS target

Reported against the source on 475c585. I could not run the Swift package here — it declares
platforms: [.macOS("27.0"), .iOS("27.0")] and this machine is on macOS 26.7 — so this report is
a code-level one: the guard and the machinery it blocks are quoted with line numbers rather than
with a runtime log.

Xcode version

Xcode 27.0 (27A266a)

Python / uv version

Not applicable — Swift runtime only.

Full error output

CoreAISequentialVLMEngine.swift, lines 235–239:

guard llmDesc.stateNames.count == 2 else {
    throw InferenceRuntimeError.invalidOutputType(
        "VLM LLM function expected 2 states (KV cache), "
            + "got \(llmDesc.stateNames.count): \(llmDesc.stateNames)")
}

A hybrid decoder declares four states — a key/value pair plus a convolution state and a recurrent
state — so it is refused at init with

VLM LLM function expected 2 states (KV cache), got 4: [keyCache, valueCache, convState, recurrentState]

Anything else?

The rest of the engine already handles this. The guard is the only thing in the way; every
piece of four-state support is present and, on the VLM path, currently dead:

• Lines 277–284 build the handlers through the shared factory and keep both extra results:

  let stateHandlers = try StateHandlerFactory.createSyncHandlers(
      descriptor: llmDesc,
      maxContextLength: config.maxContextLength,
      options: options
  )
  self.kvCache = stateHandlers.kvCache
  self.additionalStates = stateHandlers.additionalStates
  self.hasNonTruncatableStates = stateHandlers.hasNonTruncatableStates

• StateHandlerFactory.createSyncHandlers
  (swift/Sources/CoreAILanguageModels/Handlers/StateHandlerFactory.swift, lines 100–174) already
  splits the growing KV pair from everything else, allocates the remainder as a
  FixedNDArrayState, and sets hasNonTruncatableStates for .fixed states (lines 131–134).

• Line 711 binds additionalStates on every call as runWithStates(secondary:).

• Line 912 resets them alongside the KV cache in clearGenerationState().

• Lines 875–879 already refuse a partial rewind when a non-truncatable state is present:

  if tokenIndex != 0 && hasNonTruncatableStates {
      throw InferenceRuntimeError.invalidState(
          "Partial reset is not supported for hybrid models with recurrent state. "
              + "Use reset(to: 0) and replay the prefix.")
  }

The engine's own property comment says as much — line 113:

/// Additional non-KV states (nil for the VLM's two-state KV contract; carried for symmetry).

It is nil only because the guard above makes it impossible for it to be anything else.

The text-only engine takes the same contract. CoreAISequentialEngine.swift, lines 117–121:

guard descriptor.stateNames.count >= 2 && descriptor.stateNames.count <= 4 else {
    throw InferenceRuntimeError.invalidOutputType(
        "Expected 2–4 states (KV cache + optional persistent states), got \(descriptor.stateNames.count): "
            + "states=\(descriptor.stateNames), outputs=\(descriptor.outputNames)")
}

That has been the case since "Add state handlers for hybrid model support (>2 states)" (#132). The
VLM engine was consolidated onto the same shared handlers afterwards (#249, #251, #262) but kept
the == 2 check. The result is that a hybrid decoder can be exported, can run under the text-only
engine, and is refused by the VLM engine for a reason that no longer holds.

Suggested fix. Relax the guard to >= 2, matching CoreAISequentialEngine. Reference patch:
baozzz1@91265fd
(swift/Sources/CoreAILanguageModels/InferenceEngines/CoreAISequentialVLMEngine.swift).

A related documentation gap, while you are in this file. StateHandlerFactory.inferKind
(StateHandlerFactory.swift, lines 84–97) classifies states by shape and then by name:

let hasDynamicDim = desc.shape.contains(where: { $0 < 0 })
if hasDynamicDim {
    return .kvCache
}
let lower = name.lowercased()
if lower.contains("cache") || lower.contains("kv") {
    return .slidingCache
}
return .fixed

So a statically shaped state whose name happens to contain cache or kv is treated as a
truncatable sliding-window cache. A conv or recurrent state is neither — it advances with every
token and cannot be rewound to a midpoint. If such a state is named, say, conv_cache, it is
classified as .slidingCache, hasNonTruncatableStates stays false, and the partial-reset guard
at line 875 never fires: reset(to:) silently accepts a rewind that corrupts the state, with no
error and only wrong output to show for it.

That naming rule is a real part of the export contract and I could not find it written down
anywhere. Either state it in the export docs (states that must not be truncated must not have
cache or kv in their name), or lean on the explicit "states" field in metadata.json that
classifyStates already prefers — the heuristic's own log line points at it, but nothing in the
model-authoring docs tells an author to populate it.

[carinapeng]

Thanks for filing this issue. Was able to reproduce this, we will fix on our end!