Fine-Tuning F5-TTS for Arabic

[ibrahimabdelaal]

Hi everyone,
I've started fine-tuning F5-TTS for Arabic and uploaded an initial checkpoint to Hugging Face: https://huggingface.co/IbrahimSalah/F5-TTS-Arabic . The results are not the best yet, but the process is ongoing. I expect to upload a better checkpoint today or tomorrow.

[80boys]

good luck

[ahorazahedi]

Thank you for sharing your work on Arabic F5-TTS! I have a question regarding the handling of Arabic diacritics (tashkeel/تشكيل) in your model:

In Arabic, the same consonant skeleton can have different pronunciations based on diacritical marks. For example:
كَتَبَ (kataba - he wrote) vs كُتِبَ (kutiba - it was written)

I'd like to understand:

1. How does your model handle diacritics during training and inference?
2. Does the training dataset contain fully diacritized text?
3. Can the model handle both diacritized and non-diacritized input?
4. For non-diacritized input, how does the model determine the correct pronunciation?

This is particularly important since modern Arabic text is often written without diacritics, but the correct pronunciation depends on these marks and context.

[ibrahimabdelaal]

Thank you for your question! In my approach to handling Arabic diacritics (tashkeel) in the F5-TTS model, I followed a straightforward method by including diacritics in the vocabulary, treating them as individual tokens just like any other character. This allows the model to learn diacritics as part of the natural tokenization process.

How the Model Handles Diacritics:
Training with Diacritized Text:

The best way for the model to learn correct pronunciation is by training on fully diacritized text. However, large-scale diacritized Arabic datasets are scarce.
My training dataset consists of a mix of both diacritized and non-diacritized text.
To reinforce learning, I duplicated the diacritized data while removing diacritics in the second copy. This allows the model to encounter both versions and learn pronunciation from context.
for the Inference with Non-Diacritized Input:
There are two possible ways the model can handle non-diacritized text:
If the model learns diacritics well from training, it can infer them based on context, though this requires a massive dataset to be highly accurate.
Alternatively, for more reliable results, I can pre-process non-diacritized input by applying a diacritization model before feeding it to F5-TTS.
In case the model struggles with mixed input, another approach is to remove all diacritics during inference to ensure consistency.
so in points :
The model was trained with both diacritized and non-diacritized text.
It learns to predict diacritics from context, but the accuracy depends on the dataset size and quality.
For non-diacritized input, two strategies can be used:
Using a pre-trained diacritization model before inference.
Relying on the model's learned ability to infer diacritics (which depends on data coverage).
Diacritized text provides better supervision during training, making pronunciation prediction easier.

[SNAKEIX]

i used the last checkpoint in custom option and put the path where i store the checkpoint model_380000.pt and the vocab.txt files , the voice speak gabriel XD, i tried many things to work it out but it speak non arabic , do i missing something here ?, when i use the default english checkpoint everything work correctly but the moment i go to custom tap , and put the path , it sees the model but it speak in alien language , so did you try it by your self and worked ? or it speak non arabic , what did i miss

[ibrahimabdelaal]

Yes @SNAKEIX , I’ve tried it. If you plan to do inference with it, just make sure the prompt is in Arabic both the audio and the text. Also, set use_ema=False.One thing to note: I know the model isn't great as it is, but the main idea was to give people a starting point to help them achieve faster convergence. It does speak Arabic,not well,but it does.
I always used the CLI code, not the interface. If you could use that to double check all the paths are correct, that would be great.
Finally, if your plan is to continue fine-tuning from this checkpoint, you can get excellent results. I continued training it with additional data and achieved great improvements (I'll be releasing a much better checkpoint soon).So yes,if you're going to continue from it, just wait a little -won’t be long- you’ll get much faster convergence.
final thought , although it is not usable as it is in production per say ,but will provide faster convergence if finetuned further

[m15kh]

@ibrahimabdelaal
i, thanks for your checkpoint on Hugging Face.
I cloned your vocab.txt and the best checkpoint you mentioned:
model_380000.pt.

For inference, I used the Pinokio app, but the results were very bad.
Did I make a mistake?

[ibrahimabdelaal]

sorry for the late response.Could you please try this notebook : https://colab.research.google.com/drive/1kX7HB05CouHa5A-4Wy0UPqMuW4APqDBr?usp=sharing ,it will work as expected.
also please take a look at update 1 on HF repo

[sridi-Rania]

Hello I'am trying to fine tune F5_TTS into arabic too for my final year project , first when i'am finetuning and using in vocav.txt my own arabic chars it raises an error saying that there is a size mismitch but i'am not using pretrained model ( i didn't put finetune or pretrainargs ) second I can't see how to test the model after finetune thank you for your help

[ibrahimabdelaal]

The size mismatch between your vocabulary and the model is expected, since you extended the vocabulary. You’ll also need to extend the model’s embedding layer accordingly.
To do this, you can use the Gradio finetuning script,it automatically adjusts the model's embeddings to match your updated vocabulary. Alternatively, you can write a simple script to manually extend the embedding layer to accommodate the new number of characters.
To test the model, you should use the inference script provided in the repository, or you can use the notebook I mentioned in my previous reply to listen to the generated output.

[asmaaabdelkader123-maker]

How did you prepare your Arabic dataset? Specifically, how did you collect the data and handle text segmentation or record splitting?