cannot train gan vocoders on new gpus #1588

neurlang · 2022-05-20T16:26:05Z

neurlang
May 20, 2022

hello, I've seen on Nvidia driver site that cufft fails with inputs of multiplies of 6. Practically all of the gan vocoders use constants that are multiplies of 6. Only thing that works is wavegrad and wavernn on the newest gpus, till the issue is fixed in the next driver and cuda.

It would be a good idea if expert fixes the constants on some fork branch. Or maybe we can change, like 1200 to 1201 so that the constant isn't a multiple of 6, but then trim the dimension back to 1200 after computing, so that we get a compatible models.

p0p4k · 2022-05-21T04:18:49Z

p0p4k
May 21, 2022

Can you point out where you saw this behavior? Any official statement or GitHub link would help.

0 replies

neurlang · 2022-05-21T05:27:27Z

neurlang
May 21, 2022
Author

the exact error is:
RuntimeError: cuFFT error: CUFFT_INTERNAL_ERROR
URL: https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#cufft-11.5.0
Known Issues
FFTs of certain sizes in single and double precision (multiples of size 6) could fail on future devices. This issue will be fixed in an upcoming release.
After fixing files like:
TTS/vocoder/models/univnet_discriminator.py
Change hop_length=120, win_length=600, to hop_length=128, win_length=512,
after this training at least starts

0 replies

neurlang · 2022-05-21T14:42:45Z

neurlang
May 21, 2022
Author

With the following changes, can train on 3090ti:

diff --git a/TTS/config/shared_configs.py b/TTS/config/shared_configs.py
index 3ea49796..2b0f1068 100644
--- a/TTS/config/shared_configs.py
+++ b/TTS/config/shared_configs.py
@@ -113,14 +113,14 @@ class BaseAudioConfig(Coqpit):
     """
 
     # stft parameters
-    fft_size: int = 1024
-    win_length: int = 1024
-    hop_length: int = 256
+    fft_size: int = 512
+    win_length: int = 512
+    hop_length: int = 128
     frame_shift_ms: int = None
     frame_length_ms: int = None
     stft_pad_mode: str = "reflect"
     # audio processing parameters
     sample_rate: int = 22050
     resample: bool = False
     preemphasis: float = 0.0
     ref_level_db: int = 20
diff --git a/TTS/vocoder/configs/fullband_melgan_config.py b/TTS/vocoder/configs/fullband_melgan_config.py
index 2ab83aac..67244447 100644
--- a/TTS/vocoder/configs/fullband_melgan_config.py
+++ b/TTS/vocoder/configs/fullband_melgan_config.py
@@ -91,9 +91,9 @@ class FullbandMelganConfig(BaseGANVocoderConfig):
 
     stft_loss_params: dict = field(
         default_factory=lambda: {
-            "n_ffts": [1024, 2048, 512],
-            "hop_lengths": [120, 240, 50],
-            "win_lengths": [600, 1200, 240],
+            "n_ffts": [512, 640, 800],
+            "hop_lengths": [128, 160, 200],
+            "win_lengths": [512, 640, 800],
         }
     )
 
diff --git a/TTS/vocoder/configs/hifigan_config.py b/TTS/vocoder/configs/hifigan_config.py
index f76bb14c..3ab3fbf9 100644
--- a/TTS/vocoder/configs/hifigan_config.py
+++ b/TTS/vocoder/configs/hifigan_config.py
@@ -123,10 +123,10 @@ class HifiganConfig(BaseGANVocoderConfig):
     l1_spec_loss_params: dict = field(
         default_factory=lambda: {
             "use_mel": True,
             "sample_rate": 22050,
-            "n_fft": 1024,
-            "hop_length": 256,
-            "win_length": 1024,
+            "n_fft": 512,
+            "hop_length": 128,
+            "win_length": 512,
             "n_mels": 80,
             "mel_fmin": 0.0,
             "mel_fmax": None,
diff --git a/TTS/vocoder/configs/melgan_config.py b/TTS/vocoder/configs/melgan_config.py
index dc35b6f8..4208648a 100644
--- a/TTS/vocoder/configs/melgan_config.py
+++ b/TTS/vocoder/configs/melgan_config.py
@@ -91,9 +91,9 @@ class MelganConfig(BaseGANVocoderConfig):
 
     stft_loss_params: dict = field(
         default_factory=lambda: {
-            "n_ffts": [1024, 2048, 512],
-            "hop_lengths": [120, 240, 50],
-            "win_lengths": [600, 1200, 240],
+            "n_ffts": [512, 640, 800],
+            "hop_lengths": [128, 160, 200],
+            "win_lengths": [512, 640, 800],
         }
     )
 
diff --git a/TTS/vocoder/configs/multiband_melgan_config.py b/TTS/vocoder/configs/multiband_melgan_config.py
index 76311353..db439bb5 100644
--- a/TTS/vocoder/configs/multiband_melgan_config.py
+++ b/TTS/vocoder/configs/multiband_melgan_config.py
@@ -132,7 +132,7 @@ class MultibandMelganConfig(BaseGANVocoderConfig):
     use_l1_spec_loss: bool = False
 
     subband_stft_loss_params: dict = field(
-        default_factory=lambda: {"n_ffts": [384, 683, 171], "hop_lengths": [30, 60, 10], "win_lengths": [150, 300, 60]}
+        default_factory=lambda: {"n_ffts": [128, 160, 200], "hop_lengths": [32, 40, 50], "win_lengths": [128, 160, 200]}
     )
 
     # loss weights - overrides
diff --git a/TTS/vocoder/configs/parallel_wavegan_config.py b/TTS/vocoder/configs/parallel_wavegan_config.py
index 7845dd6b..8b92f989 100644
--- a/TTS/vocoder/configs/parallel_wavegan_config.py
+++ b/TTS/vocoder/configs/parallel_wavegan_config.py
@@ -105,9 +105,9 @@ class ParallelWaveganConfig(BaseGANVocoderConfig):
 
     stft_loss_params: dict = field(
         default_factory=lambda: {
-            "n_ffts": [1024, 2048, 512],
-            "hop_lengths": [120, 240, 50],
-            "win_lengths": [600, 1200, 240],
+            "n_ffts": [512, 640, 800],
+            "hop_lengths": [128, 160, 200],
+            "win_lengths": [512, 640, 800],
         }
     )
 
diff --git a/TTS/vocoder/configs/shared_configs.py b/TTS/vocoder/configs/shared_configs.py
index a558cfca..4c4c88f4 100644
--- a/TTS/vocoder/configs/shared_configs.py
+++ b/TTS/vocoder/configs/shared_configs.py
@@ -146,19 +146,19 @@ class BaseGANVocoderConfig(BaseVocoderConfig):
 
     stft_loss_params: dict = field(
         default_factory=lambda: {
-            "n_ffts": [1024, 2048, 512],
-            "hop_lengths": [120, 240, 50],
-            "win_lengths": [600, 1200, 240],
+            "n_ffts": [512, 640, 800],
+            "hop_lengths": [128, 160, 200],
+            "win_lengths": [512, 640, 800],
         }
     )
 
     l1_spec_loss_params: dict = field(
         default_factory=lambda: {
             "use_mel": True,
             "sample_rate": 22050,
-            "n_fft": 1024,
-            "hop_length": 256,
-            "win_length": 1024,
+            "n_fft": 512,
+            "hop_length": 128,
+            "win_length": 512,
             "n_mels": 80,
             "mel_fmin": 0.0,
             "mel_fmax": None,
diff --git a/TTS/vocoder/configs/univnet_config.py b/TTS/vocoder/configs/univnet_config.py
index 67f324cf..9a10dcc8 100644
--- a/TTS/vocoder/configs/univnet_config.py
+++ b/TTS/vocoder/configs/univnet_config.py
@@ -123,9 +123,9 @@ class UnivnetConfig(BaseGANVocoderConfig):
     stft_loss_weight: float = 2.5
     stft_loss_params: Dict = field(
         default_factory=lambda: {
-            "n_ffts": [1024, 2048, 512],
-            "hop_lengths": [120, 240, 50],
-            "win_lengths": [600, 1200, 240],
+            "n_ffts": [512, 640, 800],
+            "hop_lengths": [128, 160, 200],
+            "win_lengths": [512, 640, 800],
         }
     )
     subband_stft_loss_weight: float = 0
@@ -136,10 +136,10 @@ class UnivnetConfig(BaseGANVocoderConfig):
     l1_spec_loss_params: Dict = field(
         default_factory=lambda: {
             "use_mel": True,
             "sample_rate": 22050,
-            "n_fft": 1024,
-            "hop_length": 256,
-            "win_length": 1024,
+            "n_fft": 512,
+            "hop_length": 128,
+            "win_length": 512,
             "n_mels": 80,
             "mel_fmin": 0.0,
             "mel_fmax": None,
diff --git a/TTS/vocoder/layers/losses.py b/TTS/vocoder/layers/losses.py
index 848e292b..4fd28f07 100644
--- a/TTS/vocoder/layers/losses.py
+++ b/TTS/vocoder/layers/losses.py
@@ -39,7 +39,7 @@ class MultiScaleSTFTLoss(torch.nn.Module):
     to spectrograms compared with L1 and Spectral convergence losses.
     It is from ParallelWaveGAN paper https://arxiv.org/pdf/1910.11480.pdf"""
 
-    def __init__(self, n_ffts=(1024, 2048, 512), hop_lengths=(120, 240, 50), win_lengths=(600, 1200, 240)):
+    def __init__(self, n_ffts=(512, 640, 800), hop_lengths=(128, 160, 200), win_lengths=(512, 640, 800)):
         super().__init__()
         self.loss_funcs = torch.nn.ModuleList()
         for n_fft, hop_length, win_length in zip(n_ffts, hop_lengths, win_lengths):
diff --git a/TTS/vocoder/layers/wavegrad.py b/TTS/vocoder/layers/wavegrad.py
index 24b905f9..93686a70 100644
--- a/TTS/vocoder/layers/wavegrad.py
+++ b/TTS/vocoder/layers/wavegrad.py
@@ -64,7 +64,7 @@ class FiLM(nn.Module):
         self.output_conv = weight_norm(self.output_conv)
 
 
[email protected]
+#@torch.jit.script
 def shif_and_scale(x, scale, shift):
     o = shift + scale * x
     return o
diff --git a/TTS/vocoder/models/univnet_discriminator.py b/TTS/vocoder/models/univnet_discriminator.py
index d6b0e5d5..693aa170 100644
--- a/TTS/vocoder/models/univnet_discriminator.py
+++ b/TTS/vocoder/models/univnet_discriminator.py
@@ -12,7 +12,7 @@ LRELU_SLOPE = 0.1
 class SpecDiscriminator(nn.Module):
     """docstring for Discriminator."""
 
-    def __init__(self, fft_size=1024, hop_length=120, win_length=600, use_spectral_norm=False):
+    def __init__(self, fft_size=512, hop_length=128, win_length=512, use_spectral_norm=False):
         super().__init__()
         norm_f = weight_norm if use_spectral_norm is False else spectral_norm
         self.fft_size = fft_size
@@ -51,7 +51,7 @@ class SpecDiscriminator(nn.Module):
 
 class MultiResSpecDiscriminator(torch.nn.Module):
     def __init__(  # pylint: disable=dangerous-default-value
-        self, fft_sizes=[1024, 2048, 512], hop_sizes=[120, 240, 50], win_lengths=[600, 1200, 240], window="hann_window"
+        self, fft_sizes=[512, 640, 800], hop_sizes=[128, 160, 200], win_lengths=[512, 640, 800], window="hann_window"
     ):
 
         super().__init__()
-- 
2.25.1

0 replies

neurlang · 2022-05-21T18:17:43Z

neurlang
May 21, 2022
Author

okay, fft_size 512 should be 768, and win_length 512 should be 768 to avoid empty mel rows
and, likewise set utils/audio.py from 1024 to 768 (forgot about that file)
cherry pick the "Fix GAN optimizer order to main", and start training hifigan from scratch no problem.

0 replies

neurlang · 2022-05-22T04:12:50Z

neurlang
May 22, 2022
Author

There is a problem with hifigan, generated waveform is 133% of the original waveform.

0 replies

neurlang · 2022-05-22T04:17:50Z

neurlang
May 22, 2022
Author

also, the mel ends at row 65 instead of 80. Whole mel is 25% smaller.

0 replies

p0p4k · 2022-05-22T08:36:04Z

p0p4k
May 22, 2022

Interesting find. Thanks.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

cannot train gan vocoders on new gpus #1588

Uh oh!

{{title}}

Uh oh!

Replies: 7 comments

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

cannot train gan vocoders on new gpus #1588

Uh oh!

neurlang May 20, 2022

Replies: 7 comments

Uh oh!

p0p4k May 21, 2022

Uh oh!

neurlang May 21, 2022 Author

Uh oh!

neurlang May 21, 2022 Author

Uh oh!

neurlang May 21, 2022 Author

Uh oh!

neurlang May 22, 2022 Author

Uh oh!

neurlang May 22, 2022 Author

Uh oh!

p0p4k May 22, 2022

neurlang
May 20, 2022

p0p4k
May 21, 2022

neurlang
May 21, 2022
Author

neurlang
May 21, 2022
Author

neurlang
May 21, 2022
Author

neurlang
May 22, 2022
Author

neurlang
May 22, 2022
Author

p0p4k
May 22, 2022