Text-to-token LLM from CosyVoice-300M-SFT

[Lalaramarya]

How can I generate a target semantic token for my data using the CosyVoice CosyVoice-300M-SFT model? I am unable to locate any code in the CosyVoice model.

Have you generated the semantic token using the target text or target speech only?
Will there be any difference if we generate the Target discrete token using Hubert from the target speech? and fine-tune the model

Currently, I am getting the result like this

Generating: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 256/256 [00:22<00:00, 11.37it/s]
[2025-05-15 15:23:14][root][INFO] - Generated Text: hoteleot lot lot lot lot Lot lot lotlot lot lotLot lot lot lot lotlota lote le lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot litotal lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lotit lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot lot
[2025-05-15 15:23:14][root][WARNING] - Audio token is too long, skipping.
Generating: 0%| | 0/256 [00:00<?, ?it/s]torch.Size([3000, 128])
300
torch.Size([3, 83])
83
Generating: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 256/256 [00:23<00:00, 10.70it/s]
[2025-05-15 15:23:38][root][INFO] - Generated Text: hote hote hotte hot te tete te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te teet ete te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te Te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te te
[2025-05-15 15:23:38][root][WARNING] - Audio token is too long, skipping.
torch.Size([3000, 128])

[cwx-worst-one]

Hi, you can refer to #217 (comment)

[Lalaramarya]

Hi @cwx-worst-one, thank you very much for your prompt reply
Can you tell, for fine-tuning on new languages, min how much data is required? Currently, I have been training it for 3 days on H100, still, I was getting the result (same word repeating) previously that I have shared.

[cwx-worst-one]

I think your training time should be sufficient — in our case, training on 4 A100s took less than a day.
It seems that the beginning of your output is abnormal (otherwise this could likely be addressed by adjusting decoding strategies like repetition penalty).
I suggest checking the following:

1. Whether the base models (Qwen LLM and CosyVoice) support the new language you're working with
2. Whether the training curve shows proper convergence

[Lalaramarya]

@cwx-worst-one Thank you very much for your prompt reply
Yes, both model supports my new languages, on which I am fine-tuning. Can you tell me the minimum amount of data needed for fine-tuning?

[cwx-worst-one]

In our experiments, we used around 400K dialogue samples, but I believe that roughly 100K SFT samples should already be sufficient.

[Lalaramarya]

@cwx-worst-one Thank you very much
I will try out the semantic speech tokenizer you suggested with a 100k sample.

[Lalaramarya]

@cwx-worst-one Hi
I am replicating the model with VoiceAssistant-400K-SLAM-Omni with the following settings

train_config: {'model_name': 's2s', 'enable_ddp': False, 'enable_deepspeed': False, 'enable_fsdp': False, 'low_cpu_fsdp': False, 'run_validation': True, 'batch_size_training': 1, 'batching_strategy': 'custom', 'context_length': 4096, 'gradient_accumulation_steps': 8, 'num_epochs': 10, 'num_workers_dataloader': 0, 'warmup_steps': 1000, 'total_steps': 100000, 'validation_interval': 3000, 'lr': 0.0001, 'weight_decay': 0.0, 'gamma': 0.85, 'seed': 42, 'use_fp16': True, 'mixed_precision': True, 'val_batch_size': 1, 'use_peft': False, 'peft_config': {'peft_method': 'lora', 'r': 8, 'lora_alpha': 32, 'target_modules': ['q_proj', 'v_proj'], 'bias': 'none', 'task_type': 'CAUSAL_LM', 'lora_dropout': 0.05, 'inference_mode': False}, 'output_dir': '/DATA/Lalaram/SLAM_omni_Jsn/SLAM-LLM/s2s_train_v4-Qwen2-0.5B-gpu1-btz1-lr1e-4-fp16-epochs10-whisper_small-latency0-group1', 'freeze_layers': False, 'num_freeze_layers': 1, 'quantization': False, 'one_gpu': False, 'save_model': True, 'dist_checkpoint_root_folder': 'PATH/to/save/FSDP/model', 'dist_checkpoint_folder': 'fine-tuned', 'save_optimizer': False, 'use_fast_kernels': False, 'run_test_during_validation': False, 'run_test_during_validation_file': 'test.wav', 'run_test_during_validation_prompt': '<|S2S|>', 'freeze_llm': False, 'freeze_encoder': True, 'train_embed_only': False, 'train_audio_embed_only': False, 'train_embed': False, 'task_type': 's2s', 'freeze_encoder_projector': False, 'freeze_group_decode_adapter': False}
[2025-05-16 19:25:46][root][INFO] - fsdp_config: {'mixed_precision': True, 'use_fp16': True, 'sharding_strategy': 'NO_SHARD', 'checkpoint_type': 'SHARDED_STATE_DICT', 'fsdp_activation_checkpointing': True, 'fsdp_cpu_offload': False, 'pure_bf16': False, 'optimizer': 'AdamW'}
[2025-05-16 19:25:46][root][INFO] - model_config: {'file': 'examples/s2s/model/slam_model_s2s.py:model_factory', 'llm_name': 'Qwen2-0.5B', 'llm_path': '/DATA/Lalaram/SLAM_omni/SLAM-LLM/models/Qwen2-0.5B', 'llm_type': 'decoder_only', 'llm_dim': 896, 'encoder_name': 'whisper', 'encoder_ds_rate': 2, 'encoder_path': '/DATA/Lalaram/SLAM_omni_Jsn/SLAM-LLM/model/encoder/small.pt', 'encoder_dim': 768, 'encoder_projector': 'linear', 'encoder_projector_ds_rate': 5, 'modal': 'audio', 'normalize': False, 'encoder_type': 'finetune', 'vocab_config': {'text_vocabsize': 151936, 'text_specialtokens': 64, 'audio_vocabsize': 4096, 'audio_specialtokens': 64, 'code_layer': 1, 'padded_text_vocabsize': 152000, 'padded_audio_vocabsize': 4160, 'total_audio_vocabsize': 4160, 'total_vocabsize': 156160, 'eot': 151936, 'pad_t': 151937, 'input_t': 151938, 'answer_t': 151939, 'asr': 151940, 'eoa': 4096, 'pad_a': 4097, 'input_a': 4098, 'answer_a': 4099, 'split': 4100}, 'codec_decode': False, 'codec_decoder_type': 'SNAC', 'codec_decoder_path': None, 'tts_adapter': False, 'tts_adapter_config': {'add_qkv_bias': True, 'bias': False, 'gelu_approximate': None, 'head_size': 64, 'intermediate_size': 4864, 'lm_head_bias': False, 'mlp_class_name': 'GptNeoxMLP', 'n_layer': 6, 'n_head': 14, 'n_embd': 896, 'n_query_groups': 2, 'norm_class_name': 'RMSNorm', 'norm_eps': 1e-07, 'parallel_residual': False, 'rotary_percentage': 1.0, 'shared_attention_norm': False}, 'encoder_path_hf': None, 'code_type': 'CosyVoice', 'group_decode': False, 'group_decode_adapter_type': 'linear', 'whisper_decode': False}
[2025-05-16 19:25:46][root][INFO] - log_config: {'use_wandb': True, 'wandb_dir': '/DATA/Lalaram/SLAM_omni_Jsn/SLAM-LLM/s2s_train_v4-Qwen2-0.5B-gpu1-btz1-lr1e-4-fp16-epochs10-whisper_small-latency0-group1', 'wandb_entity_name': 'lucky_arya_004', 'wandb_project_name': 'test', 'wandb_exp_name': 's2s_train_v4-Qwen2-0.5B-gpu1-btz1-lr1e-4-fp16-epochs10-whisper_small-latency0-group1', 'log_file': '/DATA/Lalaram/SLAM_omni_Jsn/SLAM-LLM/s2s_train_v4-Qwen2-0.5B-gpu1-btz1-lr1e-4-fp16-epochs10-whisper_small-latency0-group1/exp.log', 'log_interval': 100, 'online_output_dir': None}
wandb: Using wandb-core as the SDK backend. Please refer to https://wandb.me/wandb-core for more information.

[2025-05-16 19:26:03][root][INFO] - dataset_config: {'dataset': 'speech_dataset_s2s', 'file': 'examples/s2s/speech_dataset_s2s.py:get_speech_dataset', 'train_data_path': '/DATA/Lalaram/SLAM_omni/SLAM-LLM/Dataset/VoiceAssistant-400K-SLAM-Omni/data3', 'val_data_path': '/DATA/Lalaram/SLAM_omni/SLAM-LLM/Dataset/VoiceAssistant-400K-SLAM-Omni/data3', 'train_split': 'train', 'test_split': 'validation', 'prompt': 'Conduct a spoken conversation with the user.', 'data_path': None, 'max_words': None, 'max_mel': None, 'fix_length_audio': -1, 'inference_mode': False, 'input_type': 'mel', 'mel_size': 80, 'normalize': False, 'seed': 42, 'manifest_format': 'parquet', 'split_size': 0.01, 'vocab_config': {'text_vocabsize': 151936, 'text_specialtokens': 64, 'audio_vocabsize': 4096, 'audio_specialtokens': 64, 'code_layer': 1, 'padded_text_vocabsize': 152000, 'padded_audio_vocabsize': 4160, 'total_audio_vocabsize': 29120, 'total_vocabsize': 156160, 'eot': 151936, 'pad_t': 151937, 'input_t': 151938, 'answer_t': 151939, 'asr': 151940, 'eoa': 4096, 'pad_a': 4097, 'input_a': 4098, 'answer_a': 4099, 'split': 4100}, 'load_from_cache_file': True, 'task_type': 's2s', 'upsample_text_tokens': False, 'upsampling_factor': 1, 'upsample_method': 'repeat', 'code_type': 'CosyVoice', 'num_latency_tokens': 0, 'do_layershift': False}

I have fine-tuned the Qwen2-0.5B with whisper small for s2s_epoch_1_step_108000/model.pt, still in the inference,e I am getting the result

Generating: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3000/3000 [04:38<00:00, 10.78it/s]
[2025-05-17 20:28:17][root][INFO] - LLM Inference Time: 279.19s
[2025-05-17 20:28:17][root][INFO] - Question: In Ancient Mesopotamian mythology, who is considered the god of wisdom and freshwater, often depicted with streams of water flowing from his shoulders?
[2025-05-17 20:28:17][root][INFO] - Generated Text: The ofantian is,zā..ienāTAL?ING..ANT.EDALY. WAN. TING..ING.ING.&.....!.............. (.
[2025-05-17 20:28:17][root][WARNING] - Audio token is too long, skip. You can try to increase the max_new_tokens in the decode_config.

**can you tell what can be the issue currently my eval_ppl=tensor(8.1625, device='cuda:0') eval_epoch_loss=tensor(2.0996, device='cuda:0') eval_epoch_acc=tensor for step 108000

Can you tell what your loss was after 100000 steps?

[cwx-worst-one]

In our experiments, both the training and validation losses converged to around 1.5–2.0, so your loss seems fairly normal.

Could it be that some settings between training and inference are inconsistent, leading to unexpected decoding behavior?
For example, I noticed that you didn’t use the group strategy during fine-tuning — did you also set group_decode=true (fix: group_decode=false) and code_layer=1 during inference to match?

[Lalaramarya]

@cwx-worst-one Thank you for your reply

Currently, by changing code_layer=1, the inference generates the result, but I am still facing the issues Audio token is too long, skip. You can try to increase the max_new_tokens in the decode_config.

Generating: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3000/3000 [34:55<00:00, 1.43it/s]
[2025-05-19 11:38:18][root][INFO] - LLM Inference Time: 2096.32s
[2025-05-19 11:38:18][root][INFO] - Question: In Ancient Mesopotamian mythology, who is considered the god of wisdom and freshwater, often depicted with streams of water flowing from his shoulders?
[2025-05-19 11:38:18][root][INFO] - Generated Text: The God of wisdom and freshwater is considered the patron saint. He is often depicted with strikes of water from his shoulders, symbolizing his role in preserving knowledge and order.
[2025-05-19 11:38:18][root][WARNING] - Audio token is too long, skip. You can try to increase the max_new_tokens in the decode_config.
[2025-05-19 11:38:18][root][INFO] - Audio output[0] shape: 3000
[2025-05-19 11:38:18][root][INFO] - Audio output[0] shape: 3000
[2025-05-19 11:38:18][root][INFO] - Batch keys: None
Generating: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3000/3000 [26:31<00:00, 1.88it/s]
[2025-05-19 12:04:49][root][INFO] - LLM Inference Time: 1591.60s
[2025-05-19 12:04:49][root][INFO] - Question: How can I evade traffic on the road every day?
[2025-05-19 12:04:49][root][INFO] - Generated Text: To avoid traffic on the road every day, you can use a few strategies. First, make your route clear and concise for everyone to follow. Second, try to find alternative routes or take advantage of holidays like Easter or March. Third, keep an eye out for any road conditions that might be in good weather. Fourth, consider using GPS tracking apps to check road conditions before you go. Fifth, stay alert with local authorities if you're unsure about directions. Lastly, drive safely and within the speed limits.
[2025-05-19 12:04:49][root][WARNING] - Audio token is too long, skip . You can try to increase the max_new_tokens in the decode_config.
[2025-05-19 12:04:49][root][INFO] - Audio output[0] shape: 3000
[2025-05-19 12:04:49][root][INFO] - Audio output[0] shape: 3000
[2025-05-19 12:04:49][root][INFO] - Batch keys: None

current max token I am using max_new_tokens=3000 # 500 for SNAC, 3000 for CosyVoice-single

By changing to group 3, fine-tuning it will solve the issue (Audio token is too long, skip.), or can you suggest a solution?

Also, for inference, you are using the VoiceAssistant-400K-v1/test dataset. What is this dataset, from where I can download it?
If you have prepared it, can you tell how you have prepared it?

[cwx-worst-one]

Oh, sorry — I think I gave the wrong config suggestion earlier. When code_layer=1, you should set group_decode=false, since the grouping strategy isn’t actually used in that case. You can try changing this setting for inference — it should work as expected.

Using a group-3 strategy significantly improves both training and inference efficiency, while also enhancing knowledge retention. So I would recommend using the grouped strategy.

As for the test set, it's something I manually split from the VoiceAssistant dataset — just for reference.
You're free to run inference on any custom dataset. We support both jsonl and parquet formats.

[Lalaramarya]

@cwx-worst-one Thank you very much, let me try with group 3.
Is there any optimal number for deciding the number of groups? Why 3?
how this will affect? do_layershift=false the inference and fine-tuning?

[cwx-worst-one]

This is an empirical finding from our experiments — you can refer to our paper for more details.
Please make sure that the group value remains consistent between training and inference.

Also, note that the do_layershift parameter is unrelated to grouping:

• Set do_layershift=false when training models using CosyVoice, which use single-layer tokens.
• Set do_layershift=true when training models using SNAC, which use multi-layer tokens.

[Lalaramarya]

@cwx-worst-one Thank you for your reply

Could you please tell, do_sample=false, what it will do if true?

As you are using Scheduler type: LambdaLR for Learning rate, is it necessary, and what will happen if I change it to cosine?

[cwx-worst-one]

Setting do_sample=true means that sampling-based decoding is used.
The scheduler type is not mandatory — you can experiment with different schedulers to potentially achieve better results.

[Lalaramarya]

@cwx-worst-one Thank You
Can you tell me how I can generate the semantic tokens from Cosysvoice?
Which code or part of the code can generate the tokens from the target text?
How much time will it take to generate the semantic tokens from 50K sentences?