"Whisper Has an Internal Word Aligner" paper

[garfieldnate]

Just wanted to call your attention to a new paper on forced-alignment using Whisper that claims very good accuracy: [https://arxiv.org/abs/2509.09987](Whisper Has an Internal Word Aligner). The repo is here.

It doesn't work nearly as well as whisper-stable-ts out-of-the-box because of the lack of VAD or other pre-processing steps, but in principle it should be possible to port the algorithm to this repo and see if it improves timestamp accuracy.

[jianfch]

Implemented in 55669d7.
Python

result = model.transcribe(... aligner='new')

CLI

stable-ts audio.mp3 -to aligner=new -o output.srt

[garfieldnate]

Wow, that was quick! Thank you!

Did you get any impressions on the performance from trying it out? You called the previous algorithm "legacy", which implies it will no longer be used in the future, but you also kept the "legacy" algorithm as the default, so I'm not sure what to think :D

I have passed this info on to my co-worker to try it on our data.

[jianfch]

Although the new algorithm appears to be better from limited testing I've done, the performance varies depending the value you set for its parameter e.g. topk. So it might still need some tuning or changes/fixes for some potential unexpected behaviors before it becomes the default algorithm.

[terryops]

hi @jianfch, can we use that on refiner too?

[jianfch]

@terryops No because the refinement process does not use any alignment heads.