Storage Issues During F5-TTS Training: How to Reduce Disk Usage?

[sitzbrau]

Storage Issues During F5-TTS Training: How to Reduce Disk Usage?

Hello F5-TTS community,

I've recently started experimenting with fine-tuning F5-TTS and I'm running into some storage issues that I hope someone can help me address.

My Setup

I'm running F5-TTS on a Google Cloud instance (g2-standard-16 with 16 vCPUs, 64 GB Memory) using Ubuntu 22.04.5 LTS.

The Issue

I attempted to train the model with just 89 samples for 500 epochs, but surprisingly, 150GB of free disk space wasn't sufficient to complete the training. The process consumed all available storage before finishing.

Questions

1. Is there a way to reduce disk usage during training by modifying certain parameters in the configuration?
2. What exactly is consuming so much space? Is it checkpoints, logs, or temporary files?
3. Are there any best practices for managing storage when fine-tuning F5-TTS?
4. Could reducing checkpoint frequency or limiting the number of saved checkpoints help?

I'd greatly appreciate any insights or experiences from those who have successfully trained models with limited storage. Are there any configuration parameters I should be tweaking to make better use of disk space?

Thanks in advance for your help!

[SWivid]

#829 (comment)

check
--save_per_updates
--keep_last_n_checkpoints