HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
Jungil Kong, Jaehyeon Kim, Jaekyoung Bae
In our paper, we proposed HiFi-GAN: a GAN-based model capable of generating high fidelity speech efficiently.
We provide our implementation and pretrained models as open source in this repository.
Abstract : Several recent work on speech synthesis have employed generative adversarial networks (GANs) to produce raw waveforms. Although such methods improve the sampling efficiency and memory usage, their sample quality has not yet reached that of autoregressive and flow-based generative models. In this work, we propose HiFi-GAN, which achieves both efficient and high-fidelity speech synthesis. As speech audio consists of sinusoidal signals with various periods, we demonstrate that modeling periodic patterns of an audio is crucial for enhancing sample quality. A subjective human evaluation (mean opinion score, MOS) of a single speaker dataset indicates that our proposed method demonstrates similarity to human quality while generating 22.05 kHz high-fidelity audio 167.9 times faster than real-time on a single V100 GPU. We further show the generality of HiFi-GAN to the mel-spectrogram inversion of unseen speakers and end-to-end speech synthesis. Finally, a small footprint version of HiFi-GAN generates samples 13.4 times faster than real-time on CPU with comparable quality to an autoregressive counterpart.
Visit our demo website for audio samples.
Pre-requisites
- Python >= 3.6
- Clone this repository.
- Install python requirements. Please refer requirements.txt
- Download and extract the LJ Speech dataset. And move all wav files to
LJSpeech-1.1/wavs
Training
python train.py --config config_v1.json
To train V2 or V3 Generator, replace config_v1.json with config_v2.json or config_v3.json.
Checkpoints and copy of the configuration file are saved in cp_hifigan directory by default.
You can change the path by adding --checkpoint_path option.
Validation loss during training with V1 generator.
Pretrained Model
You can also use pretrained models we provide.
Download pretrained models
Details of each folder are as in follows:
| Folder Name | Generator | Dataset | Fine-Tuned |
|---|---|---|---|
| LJ_V1 | V1 | LJSpeech | No |
| LJ_V2 | V2 | LJSpeech | No |
| LJ_V3 | V3 | LJSpeech | No |
| LJ_FT_T2_V1 | V1 | LJSpeech | Yes (Tacotron2) |
| LJ_FT_T2_V2 | V2 | LJSpeech | Yes (Tacotron2) |
| LJ_FT_T2_V3 | V3 | LJSpeech | Yes (Tacotron2) |
| VCTK_V1 | V1 | VCTK | No |
| VCTK_V2 | V2 | VCTK | No |
| VCTK_V3 | V3 | VCTK | No |
| UNIVERSAL_V1 | V1 | Universal | No |
We provide the universal model with discriminator weights that can be used as a base for transfer learning to other datasets.
Fine-Tuning
- Generate mel-spectrograms in numpy format using Tacotron2 with teacher-forcing.
The file name of the generated mel-spectrogram should match the audio file and the extension should be.npy.
Example:Audio File : LJ001-0001.wav Mel-Spectrogram File : LJ001-0001.npy - Create
ft_datasetfolder and copy the generated mel-spectrogram files into it. - Run the following command.
For other command line options, please refer to the training section.
python train.py --fine_tuning True --config config_v1.json
Inference from wav file
- Make
test_filesdirectory and copy wav files into the directory. - Run the following command.
python inference.py --checkpoint_file [generator checkpoint file path]
Generated wav files are saved in generated_files by default.
You can change the path by adding --output_dir option.
Inference for end-to-end speech synthesis
- Make
test_mel_filesdirectory and copy generated mel-spectrogram files into the directory.
You can generate mel-spectrograms using Tacotron2, Glow-TTS and so forth. - Run the following command.
python inference_e2e.py --checkpoint_file [generator checkpoint file path]
Generated wav files are saved in generated_files_from_mel by default.
You can change the path by adding --output_dir option.
Acknowledgements
We referred to WaveGlow, MelGAN and Tacotron2 to implement this.
![[Question] Quality improvement using different generator parameters](https://avatars.githubusercontent.com/u/25833833?v=4)
![[Question] How to apply on 16k data?](https://avatars.githubusercontent.com/u/5461945?v=4)
the spec has beak point, what do you think the reason of this problem is? I have enlarged the receptive: [1,3,5]-->[1,3,5,7], but the problem still exists.
204 Jul 14, 2022
19 Jun 30, 2022
1k Dec 30, 2022
2 May 14, 2022
1.8k Dec 30, 2022
214 Dec 7, 2022
2 May 17, 2022
135 Dec 18, 2022
38.5k Jan 3, 2023
67 Nov 14, 2022
247 Jan 5, 2023
22 Nov 13, 2022
3.2k Dec 31, 2022
1 Dec 20, 2021
10 Oct 13, 2022
2.2k Jan 9, 2023
11 Nov 17, 2022
5 Dec 28, 2021