feat(desktop): flag huddle Smart Turn gate after VAD silence

[YoraiLevi]

feat(desktop): flag huddle Smart Turn gate after VAD silence

Verdict: after the existing ~300 ms VAD silence, an optional local classifier can HOLD (keep listening) or SHIFT (today's Parakeet flush). Default is off. PTT, the wire, and SILENCE_FLUSH_FRAMES are unchanged.

Implements the flagged prototype in #6346. Closest existing issue: that RFC. No duplicate implementation PR.

Enable: BUZZ_HUDDLE_SMART_TURN=1 (strict == "1"; 0 / true / yes stay off).

Why

Hands-free huddles treat any ~300 ms pause as end of turn, so the agent wakes mid-sentence. Retuning the silence window already failed (450 ms sluggish, 150 ms split sentences). This PR asks a small local model "did they finish?" at the same silence point.

What changed

16 kHz speech_buf
  → (flag on + session ready) pad → normalize → 80×800 log-mel
  → Smart Turn v3.1 CPU ONNX (existing sherpa-linked ORT)
  → probability > 0.5 ? SHIFT : HOLD
  → on_silence_threshold
      HOLD  → keep speech_buf / in_speech / voiced_frames; silence_frames = 0
      SHIFT → today's Parakeet flush + reset

Flag off, missing weights, load error, or inference error → decision=None → today's flush.

Component map

Grey is unchanged. Only the boxed gate is new.

flowchart TB
  subgraph unchanged [Unchanged]
    mic[AudioWorklet 48 kHz]
    resample[rubato 48k to 16k]
    vad[earshot VAD]
    stt[Parakeet sherpa-onnx]
    tx["text_tx Receiver String"]
    post[pipeline spawn_transcription_task]
    relay[kind:9 to relay]
    acp[buzz-acp]
  end
  gate{{Smart Turn v3.1 HOLD or SHIFT}}
  mic --> resample --> vad
  vad -->|19 silent frames ~300 ms| gate
  gate -->|HOLD flag on| vad
  gate -->|SHIFT or flag off| stt --> tx --> post --> relay --> acp

Loading

flowchart LR
  subgraph desktop [desktop/src-tauri huddle]
    sttFile[stt.rs worker]
    modelsFile[models.rs + models_smart_turn.rs cache]
    pipeFile[pipeline.rs kind:9]
  end
  subgraph voice [crates/buzz-voice]
    smartTurn[smart_turn.rs pad/norm/log-mel + ort]
  end
  subgraph acp [crates/buzz-acp]
    harness[event handling unchanged]
  end
  modelsFile --> sttFile
  smartTurn --> sttFile
  sttFile --> pipeFile --> harness

Loading

Red-proof

These tests were inverted on purpose, watched fail, then restored.

Gate	Sabotage	Observed failure
B2.1	HOLD mapped to Flush	left Flush / right Keep, exit 101
G3.1	Keep path still speech_buf.clear()	left 256 / right 768, exit 101
G4.1	clip-relative floor max − 8 changed to max − 7	span assertion failed, exit 101

Production Keep only zeros silence_frames (stt.rs on_silence_threshold). Production classify uses probability > 0.5 with no extra sigmoid (smart_turn.rs). 0.5 is HOLD; 0.500_001 is SHIFT.

G4.1 vs Fizz's pinned-Python golden (transformers==4.44.0, not the Rust port): max abs err 1.192e-7 at row-major index 11100 (mel 13, frame 700). Floor on this clip is -0.110, not silence's -1.5. G4.1 is a feature-extraction gate, not an accuracy gate.

Latency (kill-arm 2: GO)

Measured on MacBook Air, Apple M5, 32 GB. Implementation SHA b4a8fada. Pinned CPU graph SHA-256 fb68d55c…c6594. Exact T4g PCM fixture. 3 untimed warm-ups, then 20 classify calls (pad/norm/log-mel + ONNX).

	ms
Session load (once)	45.943
min	70.762
p50	72.565
p95	76.049
max	76.055

19-frame VAD window is 304 ms. p50 / 304 ≈ 0.24. Added SHIFT delay is tens of ms, not hundreds.

Caveat: one fast ARM class. The flag stays default off, so a slower laptop is not the shipping path.

Do not read Pipecat's 10–65 ms as a Buzz number.

Packaging

*.f32 binary in .gitattributes (under * text=auto eol=lf). The 256 KB golden has no NUL in the first 8 KB, so text=auto classified it as text. Today it round-trips (zero CRLF). A later fixture with 0x0D 0x0A would strip a byte and break G4.1.

Smart Turn download/cache lives in models_smart_turn.rs so models.rs stays under the 1000-line ratchet (994 → 999). Flag, SHA pin, and fail-open are unchanged.

Out of scope (intentional)

• Default-on, settings UI, non-English eval
• Retuning SILENCE_FLUSH_FRAMES
• PTT p-tags (separate track)
• ACP steer races ([Bug] Agents never respond in voice huddles: transcripts arrive but every utterance hits ExpectedRunIdMissing and cancels the in-flight run #3071 / buzz-acp: thread-reply @mention to an idle agent is swallowed by the steer path (ExpectedRunIdMissing, no turn in flight) #3282)
• New Nostr / ACP wire types
• 8 s model window vs 30 s Buzz buffer — truncation can hide HOLD vs SHIFT on a long turn; unmeasured product question, not this latency arm

How to test

1. Default (flag unset): huddle STT behavior is today's VAD flush. No Smart Turn session is constructed.
2. BUZZ_HUDDLE_SMART_TURN=1 with weights cached under ~/.buzz/models/smart-turn-v3.1/: a mid-sentence pause should HOLD; a finished turn should SHIFT into Parakeet.
3. Flag on, weights missing: log + today's flush (fail open).
4. Held PTT: silence still does not flush (existing vad_flush_allowed).

Commits

• bcac097c9 spike(voice): prove Smart Turn ONNX runtime
• 6ffd155f4 feat(desktop): add pure huddle turn gate
• a6a1e40e8 feat(desktop): wire huddle turn gate state
• b4a8fada3 feat(huddle): classify semantic turn boundaries
• 122233401 chore: mark float fixtures as binary
• 2897d060f chore(desktop): split Smart Turn cache out of models.rs

HEAD: 2897d060fb31abd8b4be74fbbd4ab46d11368f4b on spike/huddle-smart-turn (base 934f3325c).

just ci green on this tree: file-size-check (models.rs 994 → 999), workspace clippy/fmt, desktop pnpm check/test/build, cargo test --workspace in desktop/src-tauri, web build, flutter test 1544 passed.