about zero-shot inference

[0913ktg]

Hello p0p4k,

I'm reaching out to you again with a question.

Thanks to your great help, I've successfully trained and inferred the Korean pflow model. During the inference process, I observed a few limitations, but I confirmed that satisfactory voices are synthesized for seen speakers.

I used data from about 3,000 male and female speakers, only utilizing voice files with durations longer than 4.1 seconds. I conducted distributed learning with a batch size of 64 on NVIDIA A100 40G (4 units), completing 160 epochs (500k steps).

However, when synthesizing voices using unseen speakers' voices as prompts, I found that while the voice content is well synthesized, the speakers' voices are not applied to the synthesized sound.

This phenomenon was observed for both male and female speakers, and the inference code was written referring to the synthesis.ipynb (almost identical).

I'm looking into why the speaker's voice characteristics are not applied in zero-shot inference.

If you have experienced the same issue or know anything about it, I would appreciate your help. If there's any additional information I should provide, please comment below.

Thank you.

[0913ktg]

• unseen speaker prompt inference mel-spectrogram
  
• seen speaker prompt inference mel-spectrogram
  

[p0p4k]

So that means the speech prompt encoder is not extracting exact speaker style info, but rather memorizing the seen speaker information. We might have to change the architecture a little bit in that case.

[0913ktg]

Thank you for your response.
I will try to modify it to extract speaker characteristics in comparison with the content of the paper.

If I achieve good results, I will make sure to share them with you.
Thank you once again for your invaluable help.
Have a great day!

[p0p4k]

Maybe take a look at audiobox/voicebox architecture as well.

[yiwei0730]

Maybe take a look at audiobox/voicebox architecture as well.

maybe the naturalspeech 2 speech prompt encoder can help?
but I'm not sure its really be useful, Do you have any suggestions?

NS2 Speech Prompt Encoder
Transformer Layer 6
Attention Heads 8
Hidden Size 512
Conv1D Filter Size 2048
Conv1D Kernel Size 9
Dropout 0.2
Parameters 69M

p-flow Speech-prompted Text Encoder
Phoneme Embedding Dim 192
PreNet Conv Layers 3
PreNet Hidden Dim 192
PreNet Kernel Size 5
PreNet Dropout 0.5
Transformer Layers 6
Transformer Hidden Dim 192
Transformer Feed-forward Hidden Dim 768
Transformer Attention Heads 2
Transformer Dropout 0.1
Prompt Embedding Dim 192
Number of Parameters 3.37M

[p0p4k]

Yes, it can help. We can yank lucidrains's code. Can you do a PR?

[yiwei0730]

I'm glad you think this solution useful, but I work for a company and can't upload the code to GITHUB. I will report back in time if I have any new progress.

[0913ktg]

Hi yiwei0730.
Thank you for your advice.
I'll do some testing and share with you.
Thank you.

[0913ktg]

Hello p0p4k, yiwei0730,

I have incorporated the prompt encoder part from the 'https://github.com/adelacvg/NS2VC' repository to extract prompt features for the text encoder.

The reason I chose this model is that it uses mel spectrograms for training, as opposed to ns2, which utilizes codec representations.

I plan to conduct two experiments: one adding the prompt encoder to the model structure mentioned in the paper, and another incorporating it into p0p4k's structure.

I will share the results once they are available.

[yiwei0730]

If you need training assistance, I may provide some usage support. I think the ZS model with less data is also a good development strategy.
In addition, I have trained 1500hr data (zh+en) on the NS2 github you are using. It seems that the similarity is not good enough when extracting the non-training speaker set.( mos=4, but smos=3.) Not sure if it is because the codec was not used as the first step of training during the original training.

[0913ktg]

Hello,

I have conducted an experiment by adding the ns2 prompt encoder to the P-Flow text encoder. This was applied to both the structure provided by p0p4k and the one presented in the paper, with some noticeable differences observed.

When adding the ns2 prompt encoder to the paper's structure, there was a significant improvement in the clarity of the mel-spectrogram. There was a notable reduction in noise, and the frequency values were more distinct. However, there is an issue where the output is always in a female voice, regardless of the gender of the prompt voice (even when male voice prompts are used).

On the other hand, adding the ns2 prompt encoder to p0p4k's structure resulted in relatively more noise compared to when it was added to the structure from the paper. Additionally, the phenomenon of the output being in a male voice, as before, was observed.

In conclusion, it seems necessary to continue experimenting with the addition of the prompt encoder to the text encoder structure described in the paper and adjusting the parameters accordingly.

If there are any other models you think would be worth trying, please feel free to share.

Thank you.

[0913ktg]

• adding the ns2 prompt encoder to the paper's structure(59epoch, 64batch)
  
• adding the ns2 prompt encoder to p0p4k's structure(59epoch, 64batch)
  

[p0p4k]

Interesting 🤔

[0913ktg]

Is this model zero-shot TTS possible?

[p0p4k]

In pflow blog, the authors say it is possible if we use more data and bigger model size.