bogihsu / tacotron2-pytorch Goto Github PK

Yet another PyTorch implementation of Tacotron 2 with reduction factor and faster training speed.

License: MIT License

Python 94.64% Jupyter Notebook 5.36%

tacotron2-pytorch tacotron2 tacotron pretrained-models reduction-factor pytorch tts text-to-speech ljspeech

tacotron2-pytorch's Introduction

Tacotron2-PyTorch

Yet another PyTorch implementation of Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions. The project is highly based on these. I made some modification to improve speed and performance of both training and inference.

TODO

Add Colab demo.
Update README.
Upload pretrained models.
Compatible with WaveGlow and Hifi-GAN.

Requirements

Python >= 3.5.2
torch >= 1.0.0
numpy
scipy
pillow
inflect
librosa
Unidecode
matplotlib
tensorboardX

Preprocessing

Currently only support LJ Speech. You can modify hparams.py for different sampling rates. prep decides whether to preprocess all utterances before training or online preprocess. pth sepecifies the path to store preprocessed data.

Training

For training Tacotron2, run the following command.

python3 train.py \
    --data_dir=<dir/to/dataset> \
    --ckpt_dir=<dir/to/models>

If you have multiple GPUs, try distributed.launch.

python -m torch.distributed.launch --nproc_per_node <NUM_GPUS> train.py \
    --data_dir=<dir/to/dataset> \
    --ckpt_dir=<dir/to/models>

Note that the training batch size will become <NUM_GPUS> times larger.

For training using a pretrained model, run the following command.

python3 train.py \
    --data_dir=<dir/to/dataset> \
    --ckpt_dir=<dir/to/models> \
    --ckpt_pth=<pth/to/pretrained/model>

For using Tensorboard (optional), run the following command.

python3 train.py \
    --data_dir=<dir/to/dataset> \
    --ckpt_dir=<dir/to/models> \
    --log_dir=<dir/to/logs>

You can find alinment images and synthesized audio clips during training. The text to synthesize can be set in hparams.py.

Inference

For synthesizing wav files, run the following command.

python3 inference.py \
    --ckpt_pth=<pth/to/model> \
    --img_pth=<pth/to/save/alignment> \
    --npy_pth=<pth/to/save/mel> \
    --wav_pth=<pth/to/save/wav> \
    --text=<text/to/synthesize>

Pretrained Model

You can download pretrained models from Realeases. The hyperparameter for training is also in the directory. All the models were trained using 8 GPUs.

Vocoder

A vocoder is not implemented. But the model is compatible with WaveGlow and Hifi-GAN. Check the Colab demo for more information.

References

This project is highly based on the works below.

tacotron2-pytorch's People

Stargazers

Watchers

tacotron2-pytorch's Issues

Set training=self.training in dropout

Line 120 in model.py

Should set training=self.training, otherwise in the evaluation, dropout function is still working.

what is your training speed ?

when reduction equal 3.

waveglow as vocoder

Can I use waveglow, particularly the pretrained one from https://github.com/NVIDIA/waveglow for inference?
Or are some changes required?

Thanks

Does reduction factor > 1 work?

Have you tried the performance with reduction factor > 1? Do you have some samples?

How to combine with WG-WaveNet

Hello @BogiHsu , today i'm reading that paper "WG-WaveNet: Real-Time High-Fidelity Speech Synthesis without GPU" and found
your implemented WG-WaveNet at https://github.com/BogiHsu/WG-WaveNet
i'm very excited and trying to training WG-WaveNet network.
But ldidn't found any source code or solutions for connect Tacotron2 with WG-WaveNet.

In this repo, i can reference demo Vocoder for WaveGlow and Hifi-GAN. but not WG-WaveNet.
Could you add more demo for WG-WaveNet ?

Good alignment but bad prosody

Hi, thanks a lot for this repo.
I tried this code on blizard challenge 2011 data with wavernn as decoder. The alignment is quite good, but the pitch of synthesized speech is wierd and so does the duration. Sometimes, synthesized speech is faster than groud truth. I tried copy synthesis with wavernn decoder, the speech is quite good. I don't know why this happens. Could you give some advice? Attached please find the aligments, synthesized speech and groud truth samples.
Thanks in advance.

samples.zip

WaveGlow scaling

Hi! This is not really an issue, but I've got a question about the tensors mel_outputs and mel_outputs_postnet in the save_mel() function in inference.py.

When running inference I get a tensor of shape [(1, 80, 253)], for example. When I pass this tensor along to my working WaveGlow model (from this project), it takes the mel spectrograms and converts them to audio, but it's just static.

The provided mel tensors from the WaveGlow people have differently scaled magnitude values. Here is a description of a sample vector from one of their mel tensors:

DescribeResult(nobs=760, minmax=(-9.1658077, -4.7563148), mean=-6.8498592, variance=0.56033307, skewness=-0.34983542561531067, kurtosis=-0.3708913831845826)

Here is a description of a sample vector from one of your mel tensors after I've run inference:

DescribeResult(nobs=282, minmax=(0.042787716, 0.4756053), mean=0.22464047, variance=0.0148157105, skewness=0.2994621992111206, kurtosis=-0.9376598291505811)

The number of nobs correlates to the number of milliseconds (I think?) so it shouldn't matter how many there are. The scale for the minmax tuple is off. Have you thought about how to scale this so that the two networks could work together? Any input you can provide would be very helpful.

README duplicate instruction

Hi. In README.md you suggest three ways to run the training:

Standard with just train.py
Using pre-trained models
Using tensorboard

The instructions for 2. and 3. are identical. I think that you need to update suggestion 3. to show which arguments to apply to allow tensorboard usage.

Low power and Reached max decoder steps.

When i train NVIDIA/tacotron2 model, i always get "End with low power." or "Reached max decoder steps".I train your code, the problem is gone.What did you do to solve the "Low power" and "Reached max decoder steps"?