fix(cli): keep phrase-level CJK and Thai transcripts as separate cues - #3436
Open
rajanpanth wants to merge 1 commit into
Open
fix(cli): keep phrase-level CJK and Thai transcripts as separate cues#3436rajanpanth wants to merge 1 commit into
rajanpanth wants to merge 1 commit into
Conversation
wordsToCues inferred whether entries were already grouped into phrases by testing for internal whitespace. Chinese, Japanese, Thai and the other scripts written without inter-word spaces never satisfy that test, so their phrase-level transcripts were treated as word-level and re-grouped into a single cue covering the whole clip. A three-phrase Chinese transcript produced one cue; the same transcript in English produced three. The failure was silent: the export succeeded, and the user found out by watching the captions. For entries with no whitespace at all, fall back to entry length when they are in a spaceless script. Whisper emits word-level tokens for those scripts one or two characters at a time, while a phrase-level cue runs to several times that. The median is used so one long token cannot declare word-level input pre-grouped, and a couple of short cues cannot declare a real transcript word-level. --preserve-cues still forces the same thing, and behaviour for space-separated scripts is unchanged. Fixes heygen-com#3353
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3353
Problem
wordsToCuesdecided whether its input was already grouped into phrases by testing for internal whitespace:Chinese, Japanese and Thai do not put spaces between words, so for those scripts the test is always false. Phrase-level entries were treated as individual words and re-grouped into one cue covering the whole transcript. Three Chinese phrases produced one cue; the same three phrases in English produced three.
The failure was silent, which is what made it expensive: as the issue records, two projects hit it and each built their own pipeline rather than finding
--preserve-cues.Approach
The issue lists three options. I took the second, keeping the whitespace test and adding a codepoint check alongside it, but with the length signal from the first, because a codepoint check on its own is not enough to decide the question.
Whether entries are pre-grouped is really "does an entry hold more than one token". Whitespace answers that for space-separated scripts. For spaceless scripts nothing in the codepoints answers it: word-level whisper output and phrase-level cues are both unbroken runs of Han characters. Treating every CJK transcript as pre-grouped would emit one cue per token and break the normal
transcribepath, which the existing"joins CJK word-level tokens without inserting spaces"test covers.So for spaceless scripts the fallback is entry length. Whisper emits word-level tokens for those scripts one or two characters at a time, while a phrase-level cue runs to several times that, and four sits comfortably between.
The median is used rather than
someorevery:somewould let one long token declare word-level input pre-grouped, andeverywould let a couple of short cues (a bare yes or no) declare a real transcript word-level.I did not take option 1 wholesale. Replacing the whitespace test with a duration heuristic for all scripts would put every existing English transcript through a new classifier, and this change deliberately leaves space-separated input on exactly the path it is on today.
Detection also covers Thai, Lao, Myanmar and Khmer, which have the same problem. I left
joinTokensand itsCJK_CHARalone: widening the separator rule is a real change to output for those scripts and belongs in its own PR.Testing
Added to
normalize.test.ts:The existing
"joins CJK word-level tokens without inserting spaces"test passes unchanged: its tokens have a median length of 1, well under the threshold.Reverting the
inferPreGroupedcall while keeping the tests fails the CJK and Thai cases, so they cover the change rather than restating current behaviour.bunx vitest run packages/cli/src/whisper/passes at 106 tests with no pre-existing failures, andbunx oxlint/bunx oxfmt --checkare clean on both files.