v2-large vs v3-large test

[cagenuts]

I needed to create subtitles for a 90 min documentary so thought it would be a decent test to see if the new v3-large model had any significant improvements over the v2-large.

This post is not about an issue though, just my findings.

I upgraded to Whisper-Faster_r160.3_windows and then ran these commands against the same video file.

whisper-faster "%vidname%" -l=en -m=large-v2 -ct=int8_float32 -o=source --verbose=true -bs=1
using the v2-large model

and
whisper-faster "%vidname%" -l=en -m=large-v3 -ct=int8_float32 -o=source --verbose=true -bs=1
using the v3-large model

I compared the two outputs in Subtitle Edit and there were plenty of differences, too many to list here.

Is this a fair comparison or should I have used the old binary (r160) with the v2-large model and then the new binary with the v3-large model?

Just remembered I can change the verbose command to just -v

[Purfview]

Read there -> Some problems with large-v3

I compared the two outputs in Subtitle Edit and there were plenty of differences, too many to list here.

That's normal.

Is this a fair comparison or should I have used the old binary (r160) with the v2-large model and then the new binary with the v3-large model?

Yes, test with same r160.3 version.

[cagenuts]

Ok thanks. I did test with the same binary. I will run some more parallel subtitle creations and see if I can pick up a trend in the v3-large model.

To be clear, I'm not complaining about this.

[Purfview]

Looks like a general consensus is that large-v3 is much worse than large-v2 on English.

Whisper-v3 Hallucinations on Real World Data

Whisper's original large-v3 tread: openai/whisper#1762