[BUG] Live preview falls progressively behind on long dictations — every tick re-transcribes the entire buffer from sample 0

[aa-hh]

Describe the bug

On long dictations the live streaming preview falls progressively further behind speech, and on a busy machine it can appear to stop updating entirely for many seconds at a time. The final transcript is unaffected — Transcription History always has the complete text — so this is a preview-latency problem, not a transcription problem.

The cause is that the preview is not incremental: every tick re-transcribes the entire recording from sample 0.

ASRService.processStreamingChunk():

// Sources/Fluid/Services/ASRService.swift:4394
let chunk = self.audioBuffer.getPrefix(currentSampleCount)

currentSampleCount is the whole buffer, so per-tick decode cost grows linearly with how long you have been speaking, while the tick interval stays fixed (streamingPreviewIntervalSeconds, 0.6 s for parakeet-tdt-v2). Three behaviors compound it once decode time crosses the interval:

• guard !self.isProcessingChunk (:4358) drops ticks that arrive while a decode is in flight.
• The slow-path at :4464 sets skipNextChunk = true, deliberately skipping the next tick as well, so the visible gap becomes roughly decode + 2 × interval.
• The catch at :4482 also sets skipNextChunk = true and then retries with an even larger buffer. If a preview decode starts failing mid-recording, every subsequent attempt is more expensive than the one that just failed, so the preview can stay frozen for the remainder of the recording with no user-visible error.

Worth noting: the overlay only ever displays the tail of the text (transcriptionPreviewCharLimit, default 150 characters; NotchContentState additionally stores only the last 800). So the app is paying to re-decode the full recording every 0.6 s in order to display roughly the last sentence.

Reproduction steps

1. Select a non-streaming engine (parakeet-tdt-v2 here) and enable the streaming preview.
2. Start a dictation and keep speaking continuously for several minutes.
3. Watch the preview text cadence against your speech; it drifts steadily further behind.
4. Measure it directly in ~/Library/Logs/Fluid/Fluid.log — the existing benchmark lines make this visible without extra tooling:
   ASR_BENCH ... chunk_done index=N elapsedMs=<decode> ... samples=<total>
elapsedMs climbs as samples grows, because each decode re-processes everything.

The lag is deterministic and grows with recording length; whether it reads as "laggy" or "frozen" depends on machine load. The worst cases here coincided with the Neural Engine being busy (e.g. the AI-enhancement model loading on the first dictation after launch).

Expected behavior

Preview cadence stays roughly constant regardless of how long the dictation has been running — the preview only needs recent audio to show the last ~150 characters.

Actual behavior

Per-tick decode time grows linearly with recording length, and the skip-on-slow logic multiplies the resulting gap. Measured on an idle M-series Mac, single dictation session, parakeet-tdt-v2 (elapsedMs vs. buffered audio, from the ASR_BENCH lines):

  1.3 s of audio  ->  136 ms decode
 11.6 s of audio  ->  134 ms decode
 22.5 s of audio  ->  213 ms decode
 33.9 s of audio  ->  332 ms decode
 46.1 s of audio  ->  263 ms decode
 58.6 s of audio  ->  306 ms decode
 71.8 s of audio  ->  363 ms decode
 85.9 s of audio  ->  397 ms decode

Roughly +3.5 ms of decode per additional second of speech at RTF ~0.005. Extrapolating on an otherwise idle machine: ~1.2 s per update at 5 minutes, ~2.3 s at 10 minutes — before the skipNextChunk penalty, and before any contention for the Neural Engine, which multiplies it.

App Version

1.6.8 (build 19), built from main @ 600506a. The same code path is present in released 1.6.7 — b0a0417 ("fix cancelled preview handoff") touches the microphone preview, not this.

macOS Version

macOS 27.0 beta (26A5388g)

Architecture

Apple Silicon

Logs or crash report

Same session as the table above, verbatim (note elapsedMs rising with samples, while rawChars has stopped growing — the tail being displayed is unchanged, but the decode keeps getting more expensive):

[INFO] [ASRBenchmark] ASR_BENCH t=296159.118213 session=5 chunk_done index=3 elapsedMs=136 ageMs=1516 samples=20651 rawChars=16 cleanedChars=16 rtf=0.105
[INFO] [ASRBenchmark] ASR_BENCH t=296180.385269 session=5 chunk_done index=31 elapsedMs=213 ageMs=22784 samples=359766 rawChars=177 cleanedChars=177 rtf=0.009
[INFO] [ASRBenchmark] ASR_BENCH t=296216.627557 session=5 chunk_done index=73 elapsedMs=306 ageMs=59027 samples=938155 rawChars=228 cleanedChars=228 rtf=0.005
[INFO] [ASRBenchmark] ASR_BENCH t=296244.978717 session=5 chunk_done index=102 elapsedMs=393 ageMs=87379 samples=1390422 rawChars=228 cleanedChars=228 rtf=0.005

The compounding skip, from other sessions on this machine:

[DEBUG] [ASRService] ⚠️ Transcription slow (0.83s > 0.6s), skipping next chunk
[DEBUG] [ASRService] ⚠️ Transcription slow (1.20s > 0.6s), skipping next chunk

Possible direction

Bound the preview decode instead of re-running the whole buffer — e.g. feed only the last N seconds of audio to transcribeStreaming and append to the displayed tail, or reuse the windowed chunking the final path already performs via ChunkProcessor. Since the overlay only renders the last ~150 characters, a fixed-size trailing window would keep preview cost constant regardless of dictation length. Happy to help test a fix.

[altic-dev]

This is a very sensitive issue to be honest. I've tried fixing it but it misses the last few words sometimes

And merging it by doing the last few seconds also has to be tested very thoroughly because it might miss a couple of words or put duplicates in between

That's why I made the parakeet implementation like really really faster so you don't really see much difference overall while I try to find a fix for this

You can try nematron streaming or some other model and that could possibly help you save time if you care about those 100 milliseconds.

Another thing is the accuracy goes down for the last bit if you try to chunk it manually and then do it. Feeding it all at once gives high accuracy output

[altic-dev]

Since I processed the preview again and again, you can make corrections to words, and it will understand things better. I don't know if you noticed that it's not just about saving a few hundred milliseconds, but also about making sure it works better overall in terms of accuracy. Feel free to fork it and play with it if you'd like. This change is very specific, and I won't be able to do it as soon as possible, but it's at the top of my mind.