subsai: whisper-timestamped

[dgoryeo]

Thanks again for the notebook example. I just got to run the notebook for model whisper-timestamped. Couple of observations / questions on the results:

(1) The srt output is in word level not in sentence level. Is that because of subsai's default config for whsiper-timestamp?

(2) There is a dependency on matplotlib for default config of whisper-timestamp. May be that should be added to the dependencies?

Kind regards

[absadiki]

Thanks @dgoryeo for taking the time to test the notebook.

(1) Yes you are right, maybe we should add a config for this, so users can choose between word-level and sentence-level ?

(2) Could please let me know what dependency is it ? And what is its effect ? So I can add it to the dependencies.

[dgoryeo]

Thanks @abdeladim-s .
About(1): I think that is a great idea. I'm guessing that users of subsai would be more keen to get "finalised" results, so probbaly sentence level would be a better default option.

About (2): I didn't copy the output text but what I recall: during my first run the ouput in my notebook for each chunk was something like: " ... requires matplotlib ....". This was not an error and the code continued to process through all chunks and finished without any error. However the generated SRT file was empty.

I then installed matplotlib, and import it to the notebook. Then the code generated a plot of timestamps for each processed chunk. In addition it gave a warning that it misses fonts for Japanese (my audio language). The SRT was generated correctly this time.

Hope this helps.
cheers

[absadiki]

(1) The original whsiper is giving sentence-level by default, but yeah I agree, I will try to add a config for whisper-timestamped` to choose between word and sentence levels.

(2) So maybe I will try to add matplotlib in the dependencies section of the notebook.

Thanks @dgoryeo for the suggestions.
Let me know if you find any other issue.

[absadiki]

Hi @dgoryeo,

Regarding the issues,

(1) I've added support for sentence level by default for whisper_timestamped, one can switch between levels by using

 {'segment_type': 'sentence'}  # or {'segment_type': 'word'}

in the model_configs

(2) The matplotlib issue is fixed as well, I found a bug that sets plot_word_alignment to True. That's why it was asking about matplotlib, it should be False by default. I tested it with a Japanese video (from an Anime :) ), it seems good now.

[dgoryeo]

Thanks @abdeladim-s . I'll test it and will report back.

Side note: are you planning to do more work on the translation notebook? I gave it a try the other day, during the last step (cell) it went to an infinite loop. I didn't dig into it more, I thought you probably are still working on it. Let me know and I can try to test that as well.

Kind regards

[absadiki]

You are welcome @dgoryeo,

The translation notebook was working (on the few tests I did at least).

Could you please test it again on your end as well.
Thank you!

[dgoryeo]

Hi @abdeladim-s , has something got broken in the code? I get an error at the sub.save() line:

AttributeError                            Traceback (most recent call last)
[<ipython-input-8-8d3cead3ffac>](https://localhost:8080/#) in <module>
     46         # save subs file
     47         print(f"Saving subtitles file {subs_file} ...")
---> 48         subs.save(subs_file)
     49         # save chunk_object
     50         print(f"Saving chunks object ...")

AttributeError: 'list' object has no attribute 'save'

[absadiki]

Hi @dgoryeo,

Thanks for reporting the issue.
I've found that I did a small mistake. I have fixed it. Please give it another try and let me know how it goes.

Sorry for that.

[dgoryeo]

Worked well. Thanks @abdeladim-s .

[absadiki]

You are welcome @dgoryeo :)