Skip to content

fix(cli): keep phrase-level CJK and Thai transcripts as separate cues - #3436

Open
rajanpanth wants to merge 1 commit into
heygen-com:mainfrom
rajanpanth:fix/cjk-phrase-level-cues
Open

fix(cli): keep phrase-level CJK and Thai transcripts as separate cues#3436
rajanpanth wants to merge 1 commit into
heygen-com:mainfrom
rajanpanth:fix/cjk-phrase-level-cues

Conversation

@rajanpanth

Copy link
Copy Markdown

Fixes #3353

Problem

wordsToCues decided whether its input was already grouped into phrases by testing for internal whitespace:

const preGrouped = opts.preGrouped ?? words.some((w) => /\s/.test(w.text.trim()));

Chinese, Japanese and Thai do not put spaces between words, so for those scripts the test is always false. Phrase-level entries were treated as individual words and re-grouped into one cue covering the whole transcript. Three Chinese phrases produced one cue; the same three phrases in English produced three.

The failure was silent, which is what made it expensive: as the issue records, two projects hit it and each built their own pipeline rather than finding --preserve-cues.

Approach

The issue lists three options. I took the second, keeping the whitespace test and adding a codepoint check alongside it, but with the length signal from the first, because a codepoint check on its own is not enough to decide the question.

Whether entries are pre-grouped is really "does an entry hold more than one token". Whitespace answers that for space-separated scripts. For spaceless scripts nothing in the codepoints answers it: word-level whisper output and phrase-level cues are both unbroken runs of Han characters. Treating every CJK transcript as pre-grouped would emit one cue per token and break the normal transcribe path, which the existing "joins CJK word-level tokens without inserting spaces" test covers.

So for spaceless scripts the fallback is entry length. Whisper emits word-level tokens for those scripts one or two characters at a time, while a phrase-level cue runs to several times that, and four sits comfortably between.

The median is used rather than some or every: some would let one long token declare word-level input pre-grouped, and every would let a couple of short cues (a bare yes or no) declare a real transcript word-level.

I did not take option 1 wholesale. Replacing the whitespace test with a duration heuristic for all scripts would put every existing English transcript through a new classifier, and this change deliberately leaves space-separated input on exactly the path it is on today.

Detection also covers Thai, Lao, Myanmar and Khmer, which have the same problem. I left joinTokens and its CJK_CHAR alone: widening the separator rule is a real change to output for those scripts and belongs in its own PR.

Testing

Added to normalize.test.ts:

  • three phrase-level Chinese entries produce three cues with their original timings, the issue's repro
  • two phrase-level Thai entries produce two cues
  • four short word-level CJK tokens still group into one cue, the regression guard for the case above

The existing "joins CJK word-level tokens without inserting spaces" test passes unchanged: its tokens have a median length of 1, well under the threshold.

Reverting the inferPreGrouped call while keeping the tests fails the CJK and Thai cases, so they cover the change rather than restating current behaviour.

bunx vitest run packages/cli/src/whisper/ passes at 106 tests with no pre-existing failures, and bunx oxlint / bunx oxfmt --check are clean on both files.

wordsToCues inferred whether entries were already grouped into phrases
by testing for internal whitespace. Chinese, Japanese, Thai and the
other scripts written without inter-word spaces never satisfy that
test, so their phrase-level transcripts were treated as word-level and
re-grouped into a single cue covering the whole clip.

A three-phrase Chinese transcript produced one cue; the same transcript
in English produced three. The failure was silent: the export
succeeded, and the user found out by watching the captions.

For entries with no whitespace at all, fall back to entry length when
they are in a spaceless script. Whisper emits word-level tokens for
those scripts one or two characters at a time, while a phrase-level cue
runs to several times that. The median is used so one long token cannot
declare word-level input pre-grouped, and a couple of short cues cannot
declare a real transcript word-level.

--preserve-cues still forces the same thing, and behaviour for
space-separated scripts is unchanged.

Fixes heygen-com#3353
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CJK transcripts collapse into a single caption cue (wordsToCues whitespace heuristic)

1 participant