Parallel Tacotron2
Pytorch Implementation of Google's Parallel Tacotron 2: A Non-Autoregressive Neural TTS Model with Differentiable Duration Modeling
Updates
-
2021.05.15: Implementation done. Sanity checks on training and inference. But still the model cannot converge.
I'm waiting for your contribution!Please inform me if you find any mistakes in my implementation or any valuable advice to train the model successfully. See the Implementation Issues section.
Training
Requirements
-
You can install the Python dependencies with
pip3 install -r requirements.txt
-
In addition to that, install fairseq (official document, github) to utilize
LConvBlock.
Datasets
The supported datasets:
- LJSpeech: a single-speaker English dataset consists of 13100 short audio clips of a female speaker reading passages from 7 non-fiction books, approximately 24 hours in total.
- (will be added more)
Preprocessing
After downloading the datasets, set the corpus_path in preprocess.yaml and run the preparation script:
python3 prepare_data.py config/LJSpeech/preprocess.yaml
Then, run the preprocessing script:
python3 preprocess.py config/LJSpeech/preprocess.yaml
Training
Train your model with
python3 train.py -p config/LJSpeech/preprocess.yaml -m config/LJSpeech/model.yaml -t config/LJSpeech/train.yaml
The model cannot converge yet. I'm debugging but it would be boosted if your awesome contribution is ready!
TensorBoard
Use
tensorboard --logdir output/log/LJSpeech
to serve TensorBoard on your localhost.
Implementation Issues
Overall, normalization or activation, which is not suggested in the original paper, is adequately arranged to prevent nan value (gradient) on forward and backward calculations.
Text Encoder
- Use the
FFTBlockof FastSpeech2 for the transformer block of the text encoder. - Use dropout
0.2for theConvBlockof the text encoder. - To restore "proprietary normalization engine",
- Apply the same text normalization as in FastSpeech2.
- Implement
grapheme_to_phonemefunction. (See ./text/init).
Residual Encoder
- Use
80 channelsmel-spectrogrom instead of128-bin. - Regular sinusoidal positional embedding is used in frame-level instead of combinations of three positional embeddings in Parallel Tacotron. As the model depends entirely on unsupervised learning for the position, this choice can be a reason for the fails on model converge.
Duration Predictor & Learned Upsampling (The most important but ambiguous part)
- Use log durations with the prior: there should be at least one frame in total per sequence.
- Use
nn.SiLU()for the swish activation. - When obtaining
WandC, concatenation operation is applied amongS,E, andVafter frame-domain (T domain) broadcasting ofV. As the detailed process is not described in the original paper, this choice can be a reason for the fails on model converge.
Decoder
- Use (Multi-head)
Self-attentionandLConvBlock. - Iterative mel-spectrogram is projected by a linear layer.
- Apply
nn.Tanh()to eachLConvBLockoutput (following activation pattern of decoder part in FastSpeech2).
Loss
- Use optimization & scheduler of FastSpeech2 (which is from Attention is all you need as described in the original paper).
- Base on pytorch-softdtw-cuda (post) for the soft-DTW.
- Implement customized soft-DTW in
model/soft_dtw_cuda.py, reflecting the recursion suggested in the original paper. - In the original soft-DTW, the final loss is not assumed and therefore only
Eis computed. But employed as a loss function, jacobian product is added to return target derivetive ofRw.r.t. inputX. - Currently, the maximum batch size is
6in 24GiB GPU (TITAN RTX) due to space complexity problem in soft-DTW Loss.- In the original paper, a custom differentiable diagonal band operation was implemented and used to solve the complexity of O(T^2), but this part has not been explored in the current implementation yet.
- Implement customized soft-DTW in
- For the stability, mel-spectrogroms are compressed by a sigmoid function before the soft-DTW. If the sigmoid is eliminated, the soft-DTW value is too large, producing nan in the backward.
- Guided attention loss is applied for fast convergence of the attention module in residual encoder.
Citation
@misc{lee2021parallel_tacotron2,
author = {Lee, Keon},
title = {Parallel-Tacotron2},
year = {2021},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/keonlee9420/Parallel-Tacotron2}}
}
69 Dec 17, 2022
13 Dec 5, 2022
14 Sep 13, 2022
17 May 6, 2022
302 Dec 14, 2022
261 Nov 12, 2022
279 Dec 9, 2022
117 Jan 9, 2023
40 Aug 14, 2022
34 Nov 3, 2022
156 Jan 9, 2023
86 Dec 25, 2022
3 Jun 22, 2022
4.1k Jan 3, 2023
66 Nov 28, 2022
235 Dec 26, 2022
103 Dec 23, 2022