Issue with Jsonl and code_type="SNAC"

[thangpnb]

System Info

Torch version 2.3.0, CUDA version 12.4, Ubuntu 22.04, GTX 1080TI

Information

• The official example scripts
• My own modified scripts

🐛 Describe the bug

Originally posted by @Lalaramarya and support by @cwx-worst-one in #216

if snac_tokens_str is a 'list' of 'int', I have error in here:
def get_snac_answer_token(snac_tokens_str):
snac_tokens = snac_tokens_str.split()
audio_length = len(snac_tokens) // 8 + 8

Error logs

snac_tokens = snac_tokens_str.split()
^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'list' object has no attribute 'split'

Expected behavior

Can run training with Jsonl and code_type="SNAC"

[cwx-worst-one]

As mentioned in the README, when training with the Mini-Omni approach, you’ll need to use their original dataset provided here: https://huggingface.co/datasets/gpt-omni/VoiceAssistant-400K.

If you're using the jsonl format, it's recommended to follow their data structure conventions. For example, the value in the answer_snac field should be a string like:

# 1942 2031 2825 2111 2743 271 3545 # 868 3867 50 2736 3070 806 1462

[thangpnb]

Thank you for your prompt response.
However, in the example of the JSONL file format here:
(https://github.com/X-LANCE/SLAM-LLM/blob/main/examples/s2s/demo/demo_data/jsonl_demo-en.jsonl)
It seems that, if I am using jsonl format, the value of answer_snac (target_token) is actually a list of integers (int).
[745, 1479, 3132, 3132, 160, 160, 1531, 218, ....

[cwx-worst-one]

Yes, that's because the demo we provided is primarily designed for training SLAM-Omni. The Mini-Omni version is included as a bonus. In the jsonl file, you’ll need to update the target_token field to match the original Mini-Omni string format I mentioned earlier.

We may consider integrating the Mini-Omni data format into the SLAM-Omni framework in future updates. If you'd prefer to use the list format, feel free to make the necessary adjustments (rewrite the get_snac_answer_token function)—it shouldn’t be too complicated.

Apologies for the inconsistency in data formatting, and thanks for your understanding!

[thangpnb]

Thank you for your clear explanation.

[thangpnb]

@cwx-worst-one can I train SLAM-Omni with 1 Gtx 3090 24GB

[cwx-worst-one]

The official training script for SLAM-Omni is designed for 80GB GPUs.

However, I believe it's still possible to train using the finetune_s2s_group.sh script, as long as you reduce the batch size accordingly (the speed maybe much slower though).

[thangpnb]

Could you please tell me the main difference between finetune_s2s.sh and finetune_s2s_group.sh script?

[cwx-worst-one]

The finetune_s2s_group.sh script uses a semantic group modeling strategy to reduce the audio sequence length and accelerate model training. For more details, please refer to our paper: https://arxiv.org/abs/2412.15649.

[thangpnb]

thank you for your info

[thangpnb]

If I want to train with a custom dataset in Korean or Vietnamese, are there any considerations I should keep in mind when using Whisper and CosyVoice, since they don't officially support Vietnamese and Korean yet?

[cwx-worst-one]

As far as I know, Whisper supports both Korean and Vietnamese, while CosyVoice currently supports Korean. You can verify Whisper's support by testing its ASR performance, and check CosyVoice support by evaluating the TTS output.

For languages not supported by CosyVoice (e.g., Vietnamese), they are likely not suitable for model training either—since our experiments are based on CosyVoice tokens. That said, if reconstruction using CosyVoice tokens still works well for unsupported languages, it might still be feasible.

[thangpnb]

I have trained whisper small model on vietnamese and korean. I think it will be better if train cosyvoice on vietnamese and korean to get tokens.

[cwx-worst-one]

Yes, it might be necessary to further fine-tune the "Supervised Semantic Speech Tokenizer" mentioned in the CosyVoice paper, though this could require more effort..

[thangpnb]

@cwx-worst-one: Have you tested the reproduction Mini-Omni version with SNAC yet? How does its speed and response quality compare to SLAM-Omni version?