CoquiTTS is not using GPU when training.

[hjkaddict]

Hi, sorry for noob question.

I installed coqui-tts on my Jetson Xavier and trained the first model with ljspeech dataset just as what coqui tts documentation instructs.However when I run the training script with CUDA_VISIBLE_DEVICES=0 python3 train.py (the script is exactly the same as https://tts.readthedocs.io/en/latest/tutorial_for_nervous_beginners.html), the stdout is showing my training environment is automatically set to CPU. Using --use_cuda true flag didn't help.

Could anyone help me for this?

(coqui-TTS-venv-python3.7) user@pape:/data/TTS$ CUDA_VISIBLE_DEVICES=0 python3 train.py

Setting up Audio Processor...
| > sample_rate:22050
| > resample:False
| > num_mels:80
| > log_func:np.log10
| > min_level_db:-100
| > frame_shift_ms:None
| > frame_length_ms:None
| > ref_level_db:20
| > fft_size:1024
| > power:1.5
| > preemphasis:0.0
| > griffin_lim_iters:60
| > signal_norm:True
| > symmetric_norm:True
| > mel_fmin:0
| > mel_fmax:None
| > pitch_fmin:1.0
| > pitch_fmax:640.0
| > spec_gain:20.0
| > stft_pad_mode:reflect
| > max_norm:4.0
| > clip_norm:True
| > do_trim_silence:True
| > trim_db:45
| > do_sound_norm:False
| > do_amp_to_db_linear:True
| > do_amp_to_db_mel:True
| > do_rms_norm:False
| > db_level:None
| > stats_path:None
| > base:10
| > hop_length:256
| > win_length:1024
| > Found 13100 files in /ssd/data/TTS/recipes/ljspeech/LJSpeech-1.1
Training Environment:
| > Num. of CPUs: 8
| > Num. of Torch Threads: 8
| > Torch seed: 54321
| > Torch CUDNN: True
| > Torch CUDNN deterministic: False
| > Torch CUDNN benchmark: False
Start Tensorboard: tensorboard --logdir=/ssd/data/TTS/run-January-06-2023_08+57PM-42afad5e

Model has 28610257 parameters

EPOCH: 0/1
--> /ssd/data/TTS/run-January-06-2023_08+57PM-42afad5e

[lexkoro]

Did you validate that pytorch ist actually installed with cuda support?

import torch
torch.cuda.is_available()

[hjkaddict]

It returns false unfortunately. Is it supposed to be True for running on GPU? I will try to make it True and report you back. Thanks.

[hjkaddict]

Yes, I needed to install proper pytorch and torchaudio dependencies on Jetson from NVIDIA L4T PyTorch using Docker . Now it works well. Thank you.

[gloomiebloomie]

Any advice on how to do this for 3070 it works fine on 3.10 python and not 3.8 but the model says it needs 3.8?

[ADD-eNavarro]

I have somewhat the same issue, only my CUDA is correctly installed. Actually, I've even changed the default torch and torchaudio with torch+cuda118, as you can see in this text from my last try:

(env_CrearVoz) [path/to/tts]\TTS>pip list
Package Version Editable project location

---

...
tensorboardX 2.6
threadpoolctl 3.1.0
torch 2.0.0+cu118
torchaudio 2.0.1+cu118
tqdm 4.65.0
trainer 0.0.20
TTS 0.12.0 [path/to/tts]\tts
typing_extensions 4.5.0
...

Torch is running with cuda:
(env_CrearVoz) [path/to/tts]\TTS>python
Python 3.9.13 (tags/v3.9.13:6de2ca5, May 17 2022, 16:36:42) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.

import torch
out = torch.fft.rfft(torch.randn(1000).cuda())
print(out.sum())
tensor(-943.4750-204.4260j, device='cuda:0')
quit()

(env_CrearVoz) [path/to/tts]\TTS>set CUDA_VISIBLE_DEVICES="0"

(env_CrearVoz) [path/to/tts]\TTS>python .\recipes\ljspeech\vits_tts\train_vits.py --proxy http://[ProxyIP]:[PORT]

Setting up Audio Processor...
| > sample_rate:22050
| > resample:False
| > num_mels:80
| > log_func:np.log10
| > min_level_db:0
| > frame_shift_ms:None
| > frame_length_ms:None
| > ref_level_db:None
| > fft_size:1024
| > power:None
| > preemphasis:0.0
| > griffin_lim_iters:None
| > signal_norm:None
| > symmetric_norm:None
| > mel_fmin:0
| > mel_fmax:None
| > pitch_fmin:None
| > pitch_fmax:None
| > spec_gain:20.0
| > stft_pad_mode:reflect
| > max_norm:1.0
| > clip_norm:True
| > do_trim_silence:False
| > trim_db:60
| > do_sound_norm:False
| > do_amp_to_db_linear:True
| > do_amp_to_db_mel:True
| > do_rms_norm:False
| > db_level:None
| > stats_path:None
| > base:10
| > hop_length:256
| > win_length:1024
| > Found 13100 files in [path/to/dataset]\LJSpeech-1.1
Training Environment:
| > Num. of CPUs: 8
| > Num. of Torch Threads: 4
| > Torch seed: 54321
| > Torch CUDNN: True
| > Torch CUDNN deterministic: False
| > Torch CUDNN benchmark: False
Start Tensorboard: tensorboard --logdir=[path/to/tts]\TTS\recipes\ljspeech\vits_tts\vits_ljspeech-March-20-2023_02+37PM-14c80dd1

Model has 83059180 parameters

EPOCH: 0/1000
--> [path/to/tts]\TTS\recipes\ljspeech\vits_tts\vits_ljspeech-March-20-2023_02+37PM-14c80dd1

DataLoader initialization
| > Tokenizer:
| > add_blank: True
| > use_eos_bos: False
| > use_phonemes: True
| > phonemizer:
| > phoneme language: en-us
| > phoneme backend: gruut
| > Number of instances : 12969
| > Preprocessing samples
| > Max text length: 188
| > Min text length: 13
| > Avg text length: 100.90014650319993
|
| > Max audio length: 222643.0
| > Min audio length: 24499.0
| > Avg audio length: 144984.29755570978
| > Num. instances discarded samples: 0
| > Batch group size: 160.

TRAINING (2023-03-20 14:37:38)

At this point, training starts, I guess. I get the stft userwarning on return_complex, but since I'm using Python 3.9 it's not stopping the training, although running on CPUs it's unfeasable slow.

The machine is a windows server 2022 (Build 20348.1607), processor: AMD EPYC 7313 16-Core 3GHz with 50GB RAM.

[hjkaddict]

Hi, I see that you installed pytorch and torchaudio again. I also tried that to find a proper pytorch for my machine but didn't work. Because the training machine I am using is Nvidia Jetson Xavier AGX and for using proper pytorch version, I needed to install PyTorch docker Container which Nvidia supports (https://catalog.ngc.nvidia.com/orgs/nvidia/containers/l4t-pytorch).

Below is my pip3 list result.

torch 1.12.0a0+8a1a93a9.nv22.5
torchaudio 0.12.0+2e13884

Which machien are you using? and what does it return when you do

import torch
torch.cuda.is_available()

in python ?

[ADD-eNavarro]

It returns true, since it's working correctly on the virtual environment:
Torch is running with cuda:
(env_CrearVoz) [path/to/tts]\TTS>python
Python 3.9.13 (tags/v3.9.13:6de2ca5, May 17 2022, 16:36:42) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.

import torch
out = torch.fft.rfft(torch.randn(1000).cuda())
print(out.sum())
tensor(-943.4750-204.4260j, device='cuda:0')

It's only when I run the script to train that it won't use the gpu.

[gloomiebloomie]

I would suggest if on windows run it in powershell with administration privileges' it wouldn't work in conda or cmd for me.

[ADD-eNavarro]

Bless you gloomiebloomie!
Running the very same commands on PS (started with admin privileges) made the GPU work, when on a .bat file it wouldn't, no matter what I tried.
Thank you so much.

Hope this helps other Windows users in the future as well.

[Noket]

Following up to save unknown people some time - if you're running TTS via code then there's a boolean hidden when you load preset models. the gpu=True flag was needed to start using hardware acceleration (after CUDA is installed, that is)

tts = TTS("tts_models/en/ljspeech/tacotron2-DDC",gpu=True)

[Cordo-van-Saviour]

If anyone stumbles on this issue while using the cli, the correct use of the flag is --use_cuda true, not True