Train on other language

[yiwei0730]

Hello, this is amazing.
I want to ask is it can be trained in other languages, or even if can be trained in multiple languages ​​at the same time.

[zhangshaolei1998]

Hi, thanks for your attention.
StreamSpeech architecture can support multilingual speech-to-speech translation, which we have also explored above. Since multilingual is not the core highlight of this work, we did not cover it in our paper.

If you want to train a multilingual StreamSpeech on CVSS-C, you only need to modify the data processing part. The training part is the same.

• Change lang in preprocess_scripts/preprocess.sh into 'all'
• Appropriately adjust vocab-size in preprocess_scripts/7.prep_cvss_c_multitask_asr_data.sh to support multilingual vocabulary.

Hope these can help you.

[tiannanzhang]

Hi, what changes should be made for speech translation to a language other than English, what parts need to be modified apart from data processing?

Thanks!

[zhangshaolei1998]

@arararz
Hi,
If you want to train StreamSpeech that translate speech to other languages ​​(other than English), in addition to data preparation, there are two points to note:

1. To extract the units of the target speech, you need to use the Vocoder of the corresponding language, which can be found here.
2. Appropriately adjust --ctc-upsample-rate. You can refer to Appendix D of our paper and adjust it to 2-3 times the unit/word sequence length ratio.

Hope these can help you~

[thetushargoyal]

@zhangshaolei1998 hey very interesting work. I was wondering about the training time and what system configuration did you use? thanks

[zhangshaolei1998]

@thetushargoyal
Hi, the training takes less than 1 day on 8 NVIDIA 3090 GPUs.

[nasirudeenraheem]

Hi, thanks for your attention. StreamSpeech architecture can support multilingual speech-to-speech translation, which we have also explored above. Since multilingual is not the core highlight of this work, we did not cover it in our paper.

If you want to train a multilingual StreamSpeech on CVSS-C, you only need to modify the data processing part. The training part is the same.

• Change lang in preprocess_scripts/preprocess.sh into 'all'
• Appropriately adjust vocab-size in preprocess_scripts/7.prep_cvss_c_multitask_asr_data.sh to support multilingual vocabulary.

Hope these can help you.

Thanks for this information. Ihave the following questions?

1. How can we train with our data?
2. Is it possible to use some Hugging face pre-trained model on something like En-ar, if Yes? How can I do that.
   Thank you for answering.

[Gql-1]

Hi, what changes should be made for speech translation to a language other than English, what parts need to be modified apart from data processing?

Thanks!

@tiannanzhang Have you successfully trained a new model on other languages?

[EmreOzkose]

@Gql-1 , I trained successfully on English-Turkish with ~90 hours of data. However, translation is not robust enough when I try to make a demo and talk randomly. I think it is because robustness in translation requires a big enough LLM. As far as I observe, if you have a small in-domain case, this model can achieve really good results, but if your case has an open-ended case, then it will not be enough for you.