A way to resume a run that went bad?

[jerkstorecaller]

Consider the following scenario:

• You're transcribing a long audio file
• After some hours, you realize Whisper has gone off the rails and is looping the same word over and over

This happened to me with whisper.cpp each time I attempted to use it on a long audio file, usually after 2-3 hours. When this happens, the first thing I need to do is CTRL-C to stop wasting CPU and time. But then I will lose the valid transcriptions I did so far, and I have no way to resume from the last known good segment.

Could stable-ts provide a way to resume transcription at a specific time of the audio, without starting over and redoing the valid segments?

[jianfch]

Yes, this would be a very useful feature. Since the long transcription is processed in segments, implementing this should be straightforward.

[jianfch]

Feature added in 765a081.
For CLI:

stable-ts/stable_whisper/whisper_word_level/cli.py

Lines 132 to 139 in 765a081

	parser.add_argument("--save_unfinished", "-su", action='store_true',
	help="whether to save unfinished outputs caused by KeyboardInterrupt; "
	"outputs are saved as JSON with suffix '-UNFINISHED.json'")
	parser.add_argument("--resume_input", "-ri", nargs="+", type=str,
	help="JSON of unfinished output filepaths(s) to continue transcription from end of last word; "
	"use '+' as suffix to redo the last segment (e.g 'output-UNFINISHED.json+')")
	parser.add_argument("--delete_resume", "-dr", action='store_true',
	help="whether to delete file(s) from '--resume_input'/'-ri' when transcription finishes")

stable-ts audio.mp3 -o output.srt -su
stable-ts audio.mp3 -o output.srt -ri output-UNFINISHED.json -dr

For Python:

stable-ts/stable_whisper/whisper_word_level/original_whisper.py

Lines 205 to 207 in 765a081

	resume : stable_whisper.result.WhisperResult or str or dict or list
	Path/data of an unfinished transcription output to continue transciption from.
	Use "+" as suffix of the path to resume from the end of second last segment (e.g "output-UNFINISHED.json+").

result = model.transcribe('audio.mp3', resume='output-UNFINISHED.json')

[jerkstorecaller]

@jianfch after further testing, your feature works correctly. I wrote comments above that were actually caused by --word_timestamps=False.

TLDR: If --word_timestamps is set to False, this feature doesn't work: saving an unfinished output works the 1st time, then subsequent resumes that are interrupted will fail to update UNFINISHED.json, and also if a resumed run finishes to 100%, it will then fail to save the srt.

You show a warning "UserWarning: Result is missing word timestamps. Word-level timing cannot be exported. Use ``word_level=False`` to avoid this warning" .
I would recommend turning this into an error when resume is active, since you will get an exception anyway later after wasting a lot of CPU. (I ignored this warning because I thought "Oh I just want regular subtitles and I don't care about word timestamps". )

Commands that reproduce the issue (in previous deleted comments I didn't include all the settings for brevity, sorry, I removed those comments). As you can see, output.srt (and UNFINISHED.json) remains the same after the 2nd CTRL+C, losing all processing done between 30% and 58%:

#(PART 1: RUN THEN CTRL+C)
$ stable-ts ./input.wav --model tiny -o output.srt --language es --word_timestamps False --temperature 0.1 --save_unfinished
Loaded Whisper tiny model
/home/user/.local/share/pipx/venvs/stable-ts/lib/python3.12/site-packages/stable_whisper/whisper_word_level/original_whisper.py:253: UserWarning: FP16 is not supported on CPU; using FP32 instead
  warnings.warn("FP16 is not supported on CPU; using FP32 instead")
Transcribe:  30%|██████████████████████▋                                                     | 28.32/94.77 [00:03<00:08,  7.94sec/s]Interrupted at 00:28.320
Transcribe:  30%|██████████████████████▋                                                     | 28.32/94.77 [00:04<00:10,  6.53sec/s]
/home/user/.local/share/pipx/venvs/stable-ts/lib/python3.12/site-packages/stable_whisper/text_output.py:183: UserWarning: Result is missing word timestamps. Word-level timing cannot be exported. Use ``word_level=False`` to avoid this warning
  warnings.warn('Result is missing word timestamps. Word-level timing cannot be exported. '
Saved: /home/user/dev/ai/my-subtitle-pipeline/output.srt
Saved: /home/user/dev/ai/my-subtitle-pipeline/output-UNFINISHED.json
[aost#0:0/pcm_s16le @ 0x55de0f92e680] Error submitting a packet to the muxer: Broken pipe
[out#0/s16le @ 0x55de0f901100] Error muxing a packet
[out#0/s16le @ 0x55de0f901100] Error writing trailer: Broken pipe
[out#0/s16le @ 0x55de0f901100] Error closing file: Broken pipe

$ tail output.srt
00:00:23,480 --> 00:00:25,200
¿taza, estaba en el gimnasio?

12
00:00:25,380 --> 00:00:26,560
Pues, ten...

13
00:00:26,680 --> 00:00:28,320
Y ti me y el chata, van en gimnasio.



#(PART 2: RESUME THEN CTRL+C)

$ stable-ts ./input.wav --model tiny -o output.srt --language es --word_timestamps False --temperature 0.1 --resume_input "./output-UNFINISHED.json"
"output.srt" already exist, overwrite (y/n)? y
Loaded Whisper tiny model
/home/user/.local/share/pipx/venvs/stable-ts/lib/python3.12/site-packages/stable_whisper/whisper_word_level/original_whisper.py:253: UserWarning: FP16 is not supported on CPU; using FP32 instead
  warnings.warn("FP16 is not supported on CPU; using FP32 instead")
Resuming from 00:28.320
Transcribe:  58%|████████████████████████████████████████████▊                                | 55.2/94.77 [00:12<00:09,  4.32sec/s]Interrupted at 00:55.200
Transcribe:  58%|████████████████████████████████████████████▊                                | 55.2/94.77 [00:13<00:09,  3.97sec/s]
Traceback (most recent call last):
  File "/home/user/.local/bin/stable-ts", line 7, in <module>
    sys.exit(cli())
             ^^^^^
  File "/home/user/.local/share/pipx/venvs/stable-ts/lib/python3.12/site-packages/stable_whisper/whisper_word_level/cli.py", line 783, in cli
    _cli(cmd=cmd, _cache=cache)
  File "/home/user/.local/share/pipx/venvs/stable-ts/lib/python3.12/site-packages/stable_whisper/whisper_word_level/cli.py", line 743, in _cli
    result: WhisperResult = call_method_with_options(transcribe_method, transcribe_options)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/share/pipx/venvs/stable-ts/lib/python3.12/site-packages/stable_whisper/whisper_word_level/cli.py", line 564, in call_method_with_options
    return method(**options)
           ^^^^^^^^^^^^^^^^^
  File "/home/user/.local/share/pipx/venvs/stable-ts/lib/python3.12/site-packages/stable_whisper/whisper_word_level/original_whisper.py", line 748, in transcribe_stable
    resume.fill_in_gaps(final_result, verbose=False)
  File "/home/user/.local/share/pipx/venvs/stable-ts/lib/python3.12/site-packages/stable_whisper/result.py", line 2479, in fill_in_gaps
    first_word = None if seg0 is None else seg0.words[-1]
                                           ~~~~~~~~~~^^^^
TypeError: 'NoneType' object is not subscriptable
[aost#0:0/pcm_s16le @ 0x61f670ec1540] Error submitting a packet to the muxer: Broken pipe
[out#0/s16le @ 0x61f670e84080] Error muxing a packet
[out#0/s16le @ 0x61f670e84080] Error writing trailer: Broken pipe
[out#0/s16le @ 0x61f670e84080] Error closing file: Broken pipe

$ tail output.srt
00:00:23,480 --> 00:00:25,200
¿taza, estaba en el gimnasio?

12
00:00:25,380 --> 00:00:26,560
Pues, ten...

13
00:00:26,680 --> 00:00:28,320
Y ti me y el chata, van en gimnasio.

[jerkstorecaller]

Last time I bother you, promise.

When trying to use the Python library rather than CLI, what's the equivalent of the CLI's --save_unfinished? I see the new 'resume' in the doc but nothing about saving partial JSON.

Can you edit this sample to show me?

model = stable_whisper.load_model('tiny')
result = model.transcribe(audio="input.wav", verbose=True)
result.to_srt_vtt(filepath='output.srt', word_level=False)

[jianfch]

The issue should be fixed in c63366e.

I'd advise against using word_timestamps=False for general cases because many features rely on word timestamps. Even accurate segment timestamps rely on word timestamps.

Can you edit this sample to show me?

model = stable_whisper.load_model('tiny')
result = model.transcribe(audio="input.wav", verbose=True)
# result should still return with partial transcription after KeyboardInterrupt
result.save_as_json('output-UNFINISHED.json')
result.to_srt_vtt(filepath='output.srt', word_level=False)

# in the next run
result = model.transcribe(audio="input.wav", verbose=True, resume='output-UNFINISHED.json')