Replies: 97 comments 840 replies
-
|
hello @SWivid do you think is possible fine tuning pretrained model on new language planing to add another language italian + english (to avoid catastrophic forgetting) |
Beta Was this translation helpful? Give feedback.
-
def is_japanese(c):
return (
"\u3040" <= c <= "\u309f" # Hiragana
or "\u30a0" <= c <= "\u30ff" # Katakana
or "\uff66" <= c <= "\uff9f" # Half-width Katakana
)
else: # if mixed chinese characters, alphabets and symbols
for c in seg:
if ord(c) < 256:
char_list.extend(c)
elif is_japanese(c):
char_list.append(c)
else:
if c not in "。，、；：？！《》【】—…":
if not char_list or not is_japanese(char_list[-1]):
char_list.append(" ")
char_list.extend(lazy_pinyin(c, style=Style.TONE3, tone_sandhi=True))
else: # if is zh punc
char_list.append(c)Thanks for your repo, i see..., first i need to use pykakasi to preprocessed my dataset，and add the convert code into, then i can use the original vocab to finetuning, is it corrected? if i missing something, tell me thanks! |
Beta Was this translation helpful? Give feedback.
-
Thanks, this help me a lot! |
Beta Was this translation helpful? Give feedback.
-
|
About right, however, a small clarification is that you do not need that conversion code for training, you only need it for inference. The steps would be:
Then after training, you can implement the is_japanese function when you use convert to pinyin in inference |
Beta Was this translation helpful? Give feedback.
-
|
Yeah! got it |
Beta Was this translation helpful? Give feedback.
-
Hi @JarodMica,
I'd be incredibly grateful for any insights or advice from @JarodMica or anyone else in the community who might have experience with these aspects of f5-tts for Japanese. |
Beta Was this translation helpful? Give feedback.
-
audio files of maybe 3 - 12s duration, i'm not sure what's good, and their transcripts /your_dataset
|-- metadata.csv
|-- wavs/
| |-- audio_0001.wav
| |-- audio_0002.wav
| `-- ...metadata.csv contents:
python scripts/prepare_csv_wavs.py <path_to_your_dataset> <F5-TTS_repo_data_path>/<dataset_name>_pinyinexample: python scripts/prepare_csv_wavs.py /my_pc/your_dataset /my_pc/F5-TTS/data/your_dataset_pinyin
set dataset name to name of your dataset in f5-tts data folder dataset_name = "your_dataset"play around with these parameters and see what give the best results: set max samples to 2, or whatever you seem fit max_samples = 2also play around with learning rate, don't know which one is best learning_rate = 5e-06change epochs and warmup to whatver you seem fit for your dataset epochs = 10 # use linear decay, thus epochs control the slope
num_warmup_updates = 20 # warmup stepsadjust this to your dataset size, eg for 100 audio files and 2 max samples, maybe 500 last_per_steps = 500 # save last checkpoint per steps
hopefully we find good hyperparams for good finetuning results could put it doesn't handle other tokenizers, always assumes english dataset and import sys, os
sys.path.append(os.getcwd())
from pathlib import Path
import json
import shutil
import argparse
from tqdm import tqdm
from datasets.arrow_writer import ArrowWriter
from model.utils import (
convert_char_to_pinyin,
)
PRETRAINED_VOCAB_PATH = Path(__file__).parent.parent / "data/Emilia_ZH_EN_pinyin/vocab.txt"
def is_csv_wavs_format(input_dataset_dir):
fpath = Path(input_dataset_dir)
metadata = fpath / "metadata.csv"
wavs = fpath / 'wavs'
return metadata.exists() and metadata.is_file() and wavs.exists() and wavs.is_dir()
def prepare_csv_wavs_dir(input_dir):
assert is_csv_wavs_format(input_dir), f"not csv_wavs format: {input_dir}"
input_dir = Path(input_dir)
metadata_path = input_dir / "metadata.csv"
audio_path_text_pairs = read_audio_text_pairs(metadata_path.as_posix())
sub_result, durations = [], []
vocab_set = set()
polyphone = True
for audio_path, text in audio_path_text_pairs:
if not Path(audio_path).exists():
print(f"audio {audio_path} not found, skipping")
continue
audio_duration = get_audio_duration(audio_path)
# assume tokenizer = "pinyin" ("pinyin" | "char")
text = convert_char_to_pinyin([text], polyphone=polyphone)[0]
sub_result.append({"audio_path": audio_path, "text": text, "duration": audio_duration})
durations.append(audio_duration)
vocab_set.update(list(text))
return sub_result, durations, vocab_set
def get_audio_duration(audio_path):
import torchaudio
audio, sample_rate = torchaudio.load(audio_path)
num_channels = audio.shape[0]
return audio.shape[1] / (sample_rate * num_channels)
def read_audio_text_pairs(csv_file_path):
import csv
audio_text_pairs = []
parent = Path(csv_file_path).parent
with open(csv_file_path, mode='r', newline='', encoding='utf-8') as csvfile:
reader = csv.reader(csvfile, delimiter='|')
next(reader) # Skip the header row
for row in reader:
if len(row) >= 2:
audio_file = row[0].strip() # First column: audio file path
text = row[1].strip() # Second column: text
audio_file_path = parent / audio_file
audio_text_pairs.append((audio_file_path.as_posix(), text))
return audio_text_pairs
def save_prepped_dataset(out_dir, result, duration_list, text_vocab_set, is_finetune):
out_dir = Path(out_dir)
# save preprocessed dataset to disk
out_dir.mkdir(exist_ok=True, parents=True)
print(f"\nSaving to {out_dir} ...")
# dataset = Dataset.from_dict({"audio_path": audio_path_list, "text": text_list, "duration": duration_list}) # oom
# dataset.save_to_disk(f"data/{dataset_name}/raw", max_shard_size="2GB")
raw_arrow_path = out_dir / "raw.arrow"
with ArrowWriter(path=raw_arrow_path.as_posix(), writer_batch_size=1) as writer:
for line in tqdm(result, desc=f"Writing to raw.arrow ..."):
writer.write(line)
# dup a json separately saving duration in case for DynamicBatchSampler ease
dur_json_path = out_dir / "duration.json"
with open(dur_json_path.as_posix(), 'w', encoding='utf-8') as f:
json.dump({"duration": duration_list}, f, ensure_ascii=False)
# vocab map, i.e. tokenizer
# add alphabets and symbols (optional, if plan to ft on de/fr etc.)
# if tokenizer == "pinyin":
# text_vocab_set.update([chr(i) for i in range(32, 127)] + [chr(i) for i in range(192, 256)])
voca_out_path = out_dir / "vocab.txt"
with open(voca_out_path.as_posix(), "w") as f:
for vocab in sorted(text_vocab_set):
f.write(vocab + "\n")
if is_finetune:
file_vocab_finetune = PRETRAINED_VOCAB_PATH.as_posix()
shutil.copy2(file_vocab_finetune, voca_out_path)
else:
with open(voca_out_path, "w") as f:
for vocab in sorted(text_vocab_set):
f.write(vocab + "\n")
dataset_name = out_dir.stem
print(f"\nFor {dataset_name}, sample count: {len(result)}")
print(f"For {dataset_name}, vocab size is: {len(text_vocab_set)}")
print(f"For {dataset_name}, total {sum(duration_list)/3600:.2f} hours")
def prepare_and_save_set(inp_dir, out_dir, is_finetune: bool = True):
if is_finetune:
assert PRETRAINED_VOCAB_PATH.exists(), f"pretrained vocab.txt not found: {PRETRAINED_VOCAB_PATH}"
sub_result, durations, vocab_set = prepare_csv_wavs_dir(inp_dir)
save_prepped_dataset(out_dir, sub_result, durations, vocab_set, is_finetune)
def cli():
# finetune: python script.py /path/to/input_dir /path/to/output_dir
# pretrain: python script.py /path/to/input_dir /path/to/output_dir --pretrain
parser = argparse.ArgumentParser(description="Prepare and save dataset.")
parser.add_argument('inp_dir', type=str, help="Input directory containing the data.")
parser.add_argument('out_dir', type=str, help="Output directory to save the prepared data.")
parser.add_argument('--pretrain', action='store_true', help="Enable for new pretrain, otherwise is a fine-tune")
args = parser.parse_args()
prepare_and_save_set(args.inp_dir, args.out_dir, is_finetune=not args.pretrain)
if __name__ == "__main__":
cli() |
Beta Was this translation helpful? Give feedback.
-
While using the |
Beta Was this translation helpful? Give feedback.
-
|
the csv_wavs script is mostly copy paste from prepare_emilia.py this is how they deal with different tokenizers: i assume at least this line would need to change, it always assumes pinyin tokenizer: # assume tokenizer = "pinyin" ("pinyin" | "char")
text = convert_char_to_pinyin([text], polyphone=polyphone)[0]i'm not sure about the implications and maybe there's more that needs to change. if you know what's going on there feel free to adjust accordingly and make a merge reuquest @kostum123 |
Beta Was this translation helpful? Give feedback.
-
|
Hi @kunibald413, I'm currently finetuning F5-TTS with gradio interface, it's really amazing. But while I'm trying to finetune it from the previous checkpoint, I'm facing the below issue:
Can you please help me with this. thank you. |
Beta Was this translation helpful? Give feedback.
-
Try changing --file_checkpoint_train into --pretrain in finetune_cli.py (Line 454). There is confusion about the argument between finetune_cli.py and finetune_gradio.py |
Beta Was this translation helpful? Give feedback.
-
I have the same error. |
Beta Was this translation helpful? Give feedback.
-
|
@kunibald413 Thank you for the script. I've already created something similar here. #62 (comment) can you update this part i think be nice to have like this before after |
Beta Was this translation helpful? Give feedback.
-
|
it's merged into main repo, feel free to adjust to your liking |
Beta Was this translation helpful? Give feedback.
-
|
The code @kunibald413 has provided works. However, when training it seems to initialize from a model with random weights. Can we initialize from the trained model weights instead? |
Beta Was this translation helpful? Give feedback.
-
|
By default, the training script will look for an existing model in If you place a model at that path, it will be loaded in as the base model |
Beta Was this translation helpful? Give feedback.
-
|
Just started my spanish finetune from the facebook libraspeech dataset. Single 4090 so it will take a while. |
Beta Was this translation helpful? Give feedback.
-
|
can you report back the time it takes? |
Beta Was this translation helpful? Give feedback.
-
|
Unfortunately I did it wrong and have to start over. But I ran it over 1 Epoch overnight, which took |
Beta Was this translation helpful? Give feedback.
-
|
Small update on the spanish finetune. original.wav (in training data): finetuned.wav (same input voice): Still much to go |
Beta Was this translation helpful? Give feedback.
-
|
Hello @jpgallegoar , thank you for the model in Spanish. I tried cloning a voice with a Cuban accent, and it’s not bad at all, even though it’s an accent you didn’t use during fine-tuning. I wonder if it’s possible to do fine-tuning starting from the model you trained... I only have 12 hours of Spanish with a Cuban accent. |
Beta Was this translation helpful? Give feedback.
-
|
I would like to help you adding Costa Rican accent. How can I provide training data? |
Beta Was this translation helpful? Give feedback.
-
|
I am using a Chinese dataset (about 33hr) to fine-tune my model. The loss is continuously decreasing, and the generated voice tone is getting closer to the target. However, as the training steps increase, the pronunciation of words is becoming increasingly unclear.
parm
|
Beta Was this translation helpful? Give feedback.
-
|
@jpgallegoar There are different accents in Chinese, and fine-tuning can also make the model's voice closer to the dataset. |
Beta Was this translation helpful? Give feedback.
-
|
I'm sorry, but my training dataset is private and cannot be provided. |
Beta Was this translation helpful? Give feedback.
-
Ah yes, thank you for the explanation |
Beta Was this translation helpful? Give feedback.
-
Hello friend, I also want to finetune a chinese model, because I think the voice generated by the current model is not very similar to the target voice sometimes. I want to train a generic model that with 5-10s reference voice by any person, the model can mimic that person's voice. May I ask if your finetuned model get better result in mimicing the reference voice? And how many person's voices did you use and how long for each person? Thanks a lot ! |
Beta Was this translation helpful? Give feedback.
-
Hello, may I inquire whether you have identified a solution to the issue? I have also encountered the same challenge and would greatly appreciate |
Beta Was this translation helpful? Give feedback.
-
|
hi i just create gradio interface for easy user-friendly and accessible for beginners you can see here Features |
Beta Was this translation helpful? Give feedback.
-
|
can confirm also training work i training 3 language indonesia-italian-english eng: https://vocaroo.com/1mGEFlRNgouY italian: https://voca.ro/1l6SYplhnSxz (Quattro imperdibili appuntamenti con l’Orchestra da Camera di Caserta e solisti internazionali.) indonesia: it even can do code switching (eng-indonesia): using same config as train 
|
Beta Was this translation helpful? Give feedback.
-
|
Just to confirm: you have used a dataset composed of english, Italian and Indonesian audios do fine-tune this model, right? Can you share how many hours of each you used for each language, and how long are the clips in seconds? |
Beta Was this translation helpful? Give feedback.
-
|
Very good , when was integrated italian language in main f5-tts? |
Beta Was this translation helpful? Give feedback.
-
@acul3 |
Beta Was this translation helpful? Give feedback.
-
|
@acul3 can you tell me how many epochs you used, especially for indo voice? |
Beta Was this translation helpful? Give feedback.
-
|
Boleh minta model fine-tunenya gan? @acul3 |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I was just wondering why you dont try to train on small data first instead of starting with a large dataset. For me, I trained for only 40 hours greek and with 20 hours (LibriTTS-R) focused on English, and it’s working fine speak very well. in half a day about with the 4090, and after about 100k to 150k steps, the model can speak greek and english in same time, very well and have great zero shot , try see if ths working for you i hope this help |
Beta Was this translation helpful? Give feedback.
-
|
Can you please elaborate? Why did you train in English again? |
Beta Was this translation helpful? Give feedback.
-
|
because i want speak both english and greek for example i give english and speak greek , and like i see this working gine with this method , if i train only greek not speak well english |
Beta Was this translation helpful? Give feedback.
-
|
Can you share the greek model? |
Beta Was this translation helpful? Give feedback.
-
|
here the setting i use |
Beta Was this translation helpful? Give feedback.
-
|
here sample greek and english you see my method working great ;) BTW: i working in finetune and plan to suport easy finetune for other language i have some ideas to add and tips , you can see here #143 already merge in main repo |
Beta Was this translation helpful? Give feedback.
-
|
@lpscr how did you load the model for the inference since you have changed the language from chinese to greek the vocab.txt must have chanced and so with the default config it would error out, right? |
Beta Was this translation helpful? Give feedback.
-
|
i have create my api script class to work easy load model and more stuff but if you want now something simple just change in interface-clu.py ckpt_path with the file .pt your model you have finetune other take preetrain ! |
Beta Was this translation helpful? Give feedback.
-
|
Hi all, this is very important and might be confusing for some. You need to copy the original model If you start training without copying this model, it will train from scratch! I’ve created a script called finetune-cli.py that can automate this process. However, before running the script, you need to update all the settings accordingly. Please make sure to do this before you start. or you can run simple run simple change only the dataname my_speak in 3090 with about 60-80 hours dataset working well for 4090 like say @JarodMica working very well and also with very big dataset about the vocab i dont replace anything because suport all symbols in language i train make sure if suport all symbols in your language you want to train if miss symbols not working correct or another idea it's in case miss symbols , you can simple covert all symbols in english language , here how check the vocab in finetune_gradio.py make sure in data/project_name/ you have inside metadata.csv for all text
thats why i make gradio_finetune.py to dont confuse for begin users i hope this help |
Beta Was this translation helpful? Give feedback.
-
|
@jpgallegoar Ah! My bad, yes, I'm pretty much topped out on RAM if you had 32GB, but chrome takes up 2 gb 💀 and vscode another 2 gb in this screenshot However, I just stopped training just to check and I'm idling at 20 GB used... Idk what's taking it up exactly but possible some data didn't cleared. Anywho, you can try to reduce the num_workers that are being used, the default it 16. If I looked at Ipscr's code, it's not setting it here so you can pass it, This should help to lower RAM usage I think as it should reduce how much data is being prepared before hand. One more note, you don't want too little workers or else you're GPU won't saturate and it'll train a little slower, so it's a balancing game |
Beta Was this translation helpful? Give feedback.
-
|
In my case, I encountered a problem with missing symbols. As you suggested, maybe changing out unused symbols could help, so does this mean I should add these missing symbols to the unused symbols (and what are they in the vocab.txt file)? I have my own tokenizer for my language, and if I use it, does that mean I need to train from scratch rather than fine-tune? |
Beta Was this translation helpful? Give feedback.
-
youtube tutorial |
Beta Was this translation helpful? Give feedback.
-
|
@jonnytracker check the gradio finetune i have also video tutorial #143 |
Beta Was this translation helpful? Give feedback.
-
|
@JarodMica I have the same issue of out of RAM memory. I have found the cause? and any solutions for fixing it? My case is on supercomputer which has 500GB ram; it still run out of memory every time. |
Beta Was this translation helpful? Give feedback.
-
|
@jpgallegoar I’m trying to train in Spanish as an experiment , let see this take some hours. I just hope the dataset I’m using is okay since I don’t speak Spanish. I’ll let you know soon.
|
Beta Was this translation helpful? Give feedback.
-
|
i just leave to 60k i dont want burn my gpu more... i need make other test also , i try train spanish only for test stuff , like i see you say now working also for you that's great , |
Beta Was this translation helpful? Give feedback.
-
|
Can you send me your model? Perhaps I can keep training it |
Beta Was this translation helpful? Give feedback.
-
|
lol you have 1k sound and i have only 20 hours . thre is not point to compare or train ... |
Beta Was this translation helpful? Give feedback.
-
|
yeah but end result is what matters no? |
Beta Was this translation helpful? Give feedback.
-
|
Hello, I am a native Spanish speaker, I would be honored to help you if you need it. |
Beta Was this translation helpful? Give feedback.
-
|
Given a large dataset, how important is it that the transcription is 1-1 with the source audio? The reason I ask, most of my datasets are built using a Whisper model, and they often do some text compression and correct misspoken words or stutter. Is this TTS-architecture forgiving for those kinds of variations or inconsistencies in transcription, or should I consider using a more verbose Whisper model for creating this dataset? |
Beta Was this translation helpful? Give feedback.
-
|
83C 388W |
Beta Was this translation helpful? Give feedback.
-
|
for me same 83C here @jpgallegoar you have 4090 ? BTW: |
Beta Was this translation helpful? Give feedback.
-
|
I read up to 90 is safe, so 83 should be fine for a few days. I will not do this forever but keep in mind some people mine crypto 24/7 for YEARS and it doesn't break. Same thing |
Beta Was this translation helpful? Give feedback.
-
|
maybe i put in gradio like this you can see your memory and temp in your gpu right now gpu sleep thats why you see 46C |
Beta Was this translation helpful? Give feedback.
-
oh yes I didnt answer I have 4090 @lpscr That looks cool, PR! |
Beta Was this translation helpful? Give feedback.
-
|
Has anyone here tried finetuning the base model on a single speaker dataset? I tried finetuning with a 6 hr English dataset, but I don’t hear any difference after the training. |
Beta Was this translation helpful? Give feedback.
-
|
Change the line I told you to, you're not using the correct model yet
|
Beta Was this translation helpful? Give feedback.
-
Actually it did use my finetuned model and yeah...I don't hear much difference: Here's the audio generated using the base F5-TTS model: https://voca.ro/1hfgaoFTKcIi Here's the audio generated using my finetuned model: https://voca.ro/1gT2MaNzZXW0 The reference audio: https://voca.ro/15snOJ8WmHF5 |
Beta Was this translation helpful? Give feedback.
-
|
The audio should be closer to 15 seconds, at least 10-12. The first mhm part does not get transcribed so for the model, it's a long silence. You're giving a 6 second audio which is 20% silence that's why it's so unnatural. You should use longer audio and splice together the sentences so there's not much silences in between. Either way, the finetuned audio does sound close to the original voice to me, even if the silences didn't dissapear (that can be fixed with a better input audio) |
Beta Was this translation helpful? Give feedback.
-
|
@jpgallegoar The prompt is: Output files are from the finetune model Old reference input audio(6 Seconds) Multiple Reference Input Audio Files Merged (13 Seconds) |
Beta Was this translation helpful? Give feedback.
-
|
I'm facing the same issue at the moment. Finetuning makes no discernable difference. Did you eventually find a way to make it work? I mean, the quality is fine as is, but I was hoping finetuning would make it even better. |
Beta Was this translation helpful? Give feedback.
-
|
After much testing, I'm gonna have to give up on the spanish finetune for now. Anyway, if anyone wants it, here is the model: Link |
Beta Was this translation helpful? Give feedback.
-
|
gracias por compartir, ahi vere como funciona, justo miraba si seria posible hacer un finetuning, en español. estaba pensando si serviria para ello, utilizar audios generados por bark por ejemplo. que uno puede "Crearlos", de cualquier manera es como que no entiendo aun el proceso de finetuning, estaria dando palos de ciego y nose si mi rtx2060 de 12gb sirve para ese proceso. si logras hacer un avance nuevo , estaria feliz de probarlo, y si hay algo que se pueda hacer para colaborar para hacer uno, tambien. Saludos. |
Beta Was this translation helpful? Give feedback.
-
|
jpgallegoar, pregunta desde la absoluta ignorancia, el archivo model_last.pt como lo utilizas? porque veo que en pinokio los modelos los envia al cache del hub y como nombre ilegible, |
Beta Was this translation helpful? Give feedback.
-
hay que hardcodearlo directamente en load_model() de utils_infer.py por ahora. Si en el futuro tenemos varios finetunes decentes, podemos integrarlos en la aplicacion. |
Beta Was this translation helpful? Give feedback.
-
Por ahora creo que el mayor problema son los datos. Si sería posible entrenar el modelo con audios generados por Bark, pero ten en cuenta que la variabilidad de esos datos no sería muy alta, por lo que el finetuning no sería flexible. Se necesita más VRAM para hacer un finetuning, pero puedes contribuir recopilando datasets de alta calidad, por ejemplo. |
Beta Was this translation helpful? Give feedback.
-
|
Has anyone tested training on fp32 vs fp16 vs bf16? Is there a noticeable quality dropoff? Which is the best? |
Beta Was this translation helpful? Give feedback.
-
The truth is 2. and 3. depends only on the quality of your transcriptions. If your transcriptions are 100% and your reference audio, text and gen text is in the domain of your dataset (not training on normal voice and then using cartoon voice with fast speech or things like that), the generated audio will be perfect. On 100h, 1.2 milliok steps, it's really really good. And number 1 depends on the variety of speech patterns of your dataset. If you have only 5 speakers, it won't be able to clone the voices very well, but if you have 100+ speakers with different voices or something like that, it will learn to generalize and clone any voice. (Again, if you train only on normal voices, don't expect the model to clone a very high pitched cartoon voice, for example) |
Beta Was this translation helpful? Give feedback.
-
|
Hi! Just wanted to share some findings from my recent experiments. I tested the impact of different batch sizes on the quality of speech synthesis. The dataset I used is a 35-hour, high-quality, multi-speaker dataset with accurate annotations. I fine-tuned two models with identical configurations, only changing the batch size: one with a batch size of 3000 and the other with 500. Both were trained for 50 epochs using a learning rate of 1e-5. Surprisingly, the model trained with a batch size of 500 produced more accurate speech compared to the one with a batch size of 3000. The 500 batch size run left more than half of my VRAM unused during fine-tuning and took slightly longer, but the improved results made it worth the trade-off. I’m not entirely sure why this happened, as everything else about the models was identical. I plan to dig into the code and experiment further when I have more time, but for now, I thought I’d share these results in case anyone finds them useful. |
Beta Was this translation helpful? Give feedback.
-
|
I just think the higher batch size one needs longer to learn. I am absolutely certain the ceiling of higher batch size is higher than lower batch size (spent hundreds renting H100 / H200 and locally 4090). If you give it enough time, it will improve. For reference. my 50h dataset was trained on 1300 epochs on 12000 batch size. I'm pretty sure it's 99-100% accurate on a new language. |
Beta Was this translation helpful? Give feedback.
-
|
@Alykasym smaller batch size provides more optimization opportunities because the number of steps increases. This is partly why you may see better results compared to a larger batch size. |
Beta Was this translation helpful? Give feedback.
-
When increasing batch size, you should typically increase the learning rate. A common rule is linear scaling: multiply the learning rate by the ratio. it's compensation for reduced gradient noise in larger batches. If you don't adjust it, training might slow down or become unstable. Can be that's why... |
Beta Was this translation helpful? Give feedback.
-
|
I'm finetuning models with F5-TTS via Pinokio but i'm struggling to identify how to use the models i've trained Would a kind person possibly update the Gradio UI for Pinokio and add an ability to automatically find and be able to select any of the finetuned models that have been trained / created to make it easy please? 😊 |
Beta Was this translation helpful? Give feedback.
-
|
@holycowdude The easyest way use ComfyUI with this cudtom node https://github.com/niknah/ComfyUI-F5-TTS |
Beta Was this translation helpful? Give feedback.
-
|
I'm training 200hrs for pt-br reaching 1M steps, using google colab, half with A100 and half with T4, but it still not perfect, it is actually doing a little inference, but have some misspellings, and for numbers, just does not work. Is it possible to finetune it with a new dataset with only numbers and misspells? will it destroy the previous trainings? |
Beta Was this translation helpful? Give feedback.
-
|
No need to regenerate the data. You could just concatenate your audios if spoken by the same speaker. It doesn't matter, if the audio contains 3 spoken sentences or 1 spoken long sentence. Just make the silence padding consistent between sentences when concatenating. You could also aim for a normal distribution of samples between 3-30 secs. |
Beta Was this translation helpful? Give feedback.
-
|
Yes but if he has the long audios, and the transcriptions were for the long audios, it's better to resplit them to avoid unnatural timings and artifacts when rejoining them. |
Beta Was this translation helpful? Give feedback.
-
|
The single splits should be "logical units", i.e. indepent utterances that make sense standalone. But it's okay to have 3 such utterances concatenated together, like:
|
Beta Was this translation helpful? Give feedback.
-
|
Another question, if I want to make it pt-br + en, can I use the same dataset, with both languages? will it be able to speak both? or for multi language is different? pt-br I used the same vocab.txt from original, I want to add english to it as many words in portuguese are english words, specially on tech industry. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, add good amount, though (e.g. LJSpeech). It will have an portugese accent, though. |
Beta Was this translation helpful? Give feedback.
-
|
Hello, I would like to re-finetune the hungarian model again from the original. I collect about 2600 hours of ultra clear audio from about 50 speakers. Unfortunately, the average audio length is around 5 seconds for me as well. Is there an ideal curve for the distribution of sounds? The amount of data is abundant, I can select a data set corresponding to an ideal distribution curve. edit: I think i need to reroll my dataset, it's worse than I thought. |
Beta Was this translation helpful? Give feedback.
-
Unfortunately I don't think anyone has made that test yet. |
Beta Was this translation helpful? Give feedback.
-
|
@jpgallegoar @sch0ngut here is the scripts https://github.com/sarpba/ADCS I no have more time now. It's working, but not too nice (gpt translated) and the last scripts is missing. (trash_dropout, numbers_drouout, csv_maker) I'll continue friday night. under windows need to use WSL. I ran a quick test. I processed 20 hours of audio in 35 minutes. it turned out to be about 15 hour usable data. (2x3090) |
Beta Was this translation helpful? Give feedback.
-
|
Awesome, thank you! |
Beta Was this translation helpful? Give feedback.
-
|
@sarpba What software/library did you use to create that kind of graph for distribution of audio durations? |
Beta Was this translation helpful? Give feedback.
-
|
@campar It's a python script from my Audio Database Creator Scripts https://github.com/sarpba/ADCS/blob/main/statistics_scripts/statistics_histogram.py |
Beta Was this translation helpful? Give feedback.
-
|
Has anyone able to train this on Multiple-4090 GPU's setup (2 or more)?? I am getting this - #728 (comment) |
Beta Was this translation helpful? Give feedback.
-
|
I was able to do it, but it was from Replicate |
Beta Was this translation helpful? Give feedback.
-
|
Has anyone tried parallelized training with multi GPUs? I mean getting parallel performance, not only more VRAM. Is it even possible? |
Beta Was this translation helpful? Give feedback.
-
|
Since NVIDIA's consumer GPUs, such as the 4090 and 3090, do not support P2P communication, they may not provide significant speedup in multi-GPU training. Instead, you could try training with NVIDIA's data center GPUs. Rather than using 4x 4090, you might consider renting a single H100 or A100. @jpgallegoar |
Beta Was this translation helpful? Give feedback.
-
|
@hcsolakoglu Thank you for your answer, that is in fact what I ended up doing. I even rented H200 because I calculated it was the most efficient. I want to spend the same amount of money (or a little bit more) and train in less time via parallelization. Do you think it would work with these server GPUs? |
Beta Was this translation helpful? Give feedback.
-
|
@hcsolakoglu @jpgallegoar theoretically: So there is an increase in speed, just in a different way. My train with 3200 batch size / 2x3090 GPU (NVlik connected) 6400 owerall batch size -> I have 7-8 update/s (underpowered to 280watt) |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the answer, the speedup would be nice for commercial purposes. I did 9600 batch size and the results were very good indeed. Perhaps I am confused and it was speeding up, I will have to test again. |
Beta Was this translation helpful? Give feedback.
-
Makes sense, because (if you use frames for batch limit):
For batch size 800 your maximum sample length per batch is Every sample not fitting in a batch (>8.5 s) would be dropped. Also two samples of 5s duration would need two single batches, as they don't fit into one. In other words: to account for training samples up to 30s minimum batch size would be 2814. On low VRAM hardware, this could still be achieved with AdamW8 (BNB). I assume that's also the reason, why many folks complain that finetunes struggle with longer audio output ... Training batch size to low, relevant training data is disregarded. For large VRAM setups with small training sets (~1k hours) I tried to keep batch size and updates per epoch balanced (with AdamW8 batch sizes >15k are possible with 24GB VRAM, but there will be way less updates per epoch with small sample sets ... Sweet spot should be to keep updates per epoch and batch size balanced) |
Beta Was this translation helpful? Give feedback.
-
|
quick note that might be relevant for anyone trying to finetune with low VRAM: if you're using frame batching, then batch size acts as a maximum frame length for any samples in your dataset. that is, any audio samples with frame length greater than batch size are dropped altogether and are not used during the finetuning. this doesn't matter with like batch size >= 3200 since training samples shouldn't be over 30s anyway, but if you're running, say, 1600 or 2000 batch size (hence probably most relevant to people with low VRAM), you're probably dropping some stuff from your dataset. i suppose this should be obvious in retrospect, but as someone who isn't knowledgeable about what some of these parameters actually mean, i think it's an easy oversight to make (i didn't even notice until i was experimenting with batch size <1000 and suddenly got a division by zero exception somewhere as a consequence of my entire dataset being greater than the batch size...) |
Beta Was this translation helpful? Give feedback.
-
|
@mechasoul the question it's vocos frames or?? just because 256 the default vocos samplerate = 3840 frames / 15 sec |
Beta Was this translation helpful? Give feedback.
-
|
Explained it there: #57 (reply in thread) |
Beta Was this translation helpful? Give feedback.
-
|
Is it possible to finetune this model from the F5TTS_Base checkpoint with around 166 samples of training data? Each clip around 10sec long, normalized, re-sampled to 24000, noise-reduced. At update 1100 the gen still sounds like white noise, and loss is not decreasing. Samples uploaded. Yaml file looks like: hydra: datasets: optim: model: ckpts: custom training samples and metacsv file generated with preapare_csv_wavs.py. Are there any improvements I can make in the YAML file so that this finetuned model can sound as good as the zero-shot examples? Thanks a lot! |
Beta Was this translation helpful? Give feedback.
-
|
Hi @jpgallegoar I tried downloading your .safetensors and vocab spanish version and overwrite them in the 1.08 release. (I changed the file name from model_1200000 to model_1250000 ) It loaded and ran an spanish test but I only got gibberish as audio result. Do you know what could be the reason? Thanks in advance. |
Beta Was this translation helpful? Give feedback.
-
|
check inference readme, especially for custom config setting |
Beta Was this translation helpful? Give feedback.
-
|
I found the config. It's working now. Thank you very much @SWivid . |
Beta Was this translation helpful? Give feedback.
-
|
I tried finetuning the model for Persian data. I changed vocab.txt to include persian characters and used a 'custom' tokenizer with this new vocab.txt. My results weren't promising and it could be because of the lack of data (only 105 hours) or insufficient training (1.2 million steps). batch_size = 4200 Am I missing something else? |
Beta Was this translation helpful? Give feedback.
-
|
What is the length distribution of your dataset? I believe with 105 hours you should get somewhat acceptable results . I am currently training on +- 140 hrs of data, but my data is skewed to lower durations and on lower duration it generates okay (up to 8 seconds no hallucinations, 10-12 sometimes hallucinates), but on longer it struggles. Here is my distribution : 🎵 Total audio files: 60242 📊 Duration statistics: Currently I think about stitching multiple smaller audios by the same speaker into single one and using this in dataset to get more samples with longer durations. However I am very limited with VRAM so my batch_size is maximum 1600, so samples longer than ~17 seconds aren't really used, but that is not issue for you. |
Beta Was this translation helpful? Give feedback.
-
|
I analyzed my dataset and realized my dataset is also skewed on lower durations. Considering 151254 samples in general, I have around 80000 samples with 1-2 sec length, around 56000 samples with 2-4 sec length and the rest of the samples are longer and are 6 sec long at most. My GPU is 4090 RTX with 22GB VRAM which allows for larger batch sizes as you mentioned. Either way, Should I also stitch some samples together or is it okay the way it is? |
Beta Was this translation helpful? Give feedback.
-
|
How do you structure the dataset folder or metadata.csv when you have multiple speakers and want to train a new language? There is no documentation regarding that. Should I just put all the training data into the wavs folder with all the speakers and proceed with the training? Like mentioned in #57 (comment) There are a lot of training data for many different languages on huggingface, they all show a speaker_ID in their database (sql browser), but when it comes to training with those data, there is no information about how to structure the folder or the metadata.csv |
Beta Was this translation helpful? Give feedback.
-
|
-Dataset/ metadata.csv must have path to audio and text, speaker_id is not mandatory and as I understand is not used even if provided. |
Beta Was this translation helpful? Give feedback.
-
|
I think this model trains pretty well on multi-speaker datasets - even with only 100 hours of data. I have recorded 10s of hours of my own speech, quality microphone, no errors in transcripts/audio, and lengths are from 5-20 seconds. I also have some other speakers that have a few hours of audio also, and then I have 400 speakers that have around 20-50 minutes each. In theory, would training on my own speech skew the model away from performing well on the other speakers (my voice would then be maybe 30% of the overall audio)? Or would it just improve more if I add my 10s of hours to the mix? And does shuffling the dataset give any better results when training, when my own recordings are shuffled in with the other 500 speakers? I don't care as much about the voice-cloning part of the tts - I just really want a clear pronunciation in my language, so I don't mind the model being skewed towards my voice, if only it has good pronunciation. |
Beta Was this translation helpful? Give feedback.
-
|
So I am trying to finetune on my dataset which contains the Indian English accent. I have around 60 hours of audio data. Although the inference audio is clear, no noise, the audio that is generated does not resemble English at all. Any ideas so as to why this is happening? I made minimal changes to the hyperparameters while finetuning. And followed the procedure exactly as provided in the documentation. |
Beta Was this translation helpful? Give feedback.
-
|
Hi I was trying to finetune the pretrained weights of hindi model, I changed the path in train.py (checkpoint_paths to the pretrained directory and my config file is hydra: datasets: optim: model: ckpts: the training finish instantly I am not able to understand what mistake I am making this below is the output : (f5tts) root@t1-le-45-gra7:/home/ubuntu/F5-TTS/src/f5_tts/configs# cd ../../.. |
Beta Was this translation helpful? Give feedback.
-
|
Your old checkpoint gets loaded automatically, remember to delete it before rerunning. |
Beta Was this translation helpful? Give feedback.
-
|
Hello! Super awesome model and repo! |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
and others
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Full finetune is currently supported, lora or adapter not yet.
checkpoint_pathto pretrained model dir intest_train.py,model/trainer.pywill load from there to resume. Reuse thevocab.txtunderdata /Emilia_ZH_EN_pinyin(Emilia_ZH_EN_pinyin <- tokenizer = "pinyin"; dataset_name = "Emilia_ZH_EN" intest_train.pysetting)model/dataset.py. Just need e.g. the audio path, text (tokenized, leverageconvert_char_to_pinyinfunc inmodel/utils.pysee script/prepare_xxxx.py), duration of audio in seconds.grad_accumulation_stepscould be used to simulate a large batchsize. Also other settings, e.g. few warmup steps, 1e-4 lr, etc.We didn't specifically experiment with finetuning, so if you get positive results, welcome to share :)
Some helpful issues, #16 #27
Welcome to share your successful results with finetuning, maybe also start a new tutorial doc helping others to get start with it.
Many Thanks !
Beta Was this translation helpful? Give feedback.
All reactions