the train.py's print is: 46739: -4.754265308 46740: -5.550816059 46741: -4.253830433 46742: -5.338192463 46743: -4.700691700 46744: -5.625311375 46745: -5.753829479 46746: -5.032420158 ......

which value when the second number equals, the chenkpoint is useable?

I have trained a new waveglow model for my language, but I cant load it into Tacotron2 inference.ipynb file to run test. It return this error: AttributeError: 'WN' object has no attribute 'cond_layers'

I tried convert the checkpoint using convert_model.py file in waveglow folder but it still raise the same error

@rafaelvalle Thanks for your sharing! It helps a lot. I find it trains very slow (about 1 epoch/day,batch size=1) when I trained model using my data(about 12h). Could you offer a model which can resume training from it ?

Thanks a lot !!

After days of different approaches, we've decided to finetune the NVIDIA pre-trained model on different language dataset. We're running model on 8 V100-s with 16GB of VRAM. Our dataset was recorded at 48kHz, and then downsampled to 22050. Checkpoint is converted. Additionally, while getting out-of-memory errors on 8 GPUs, we've decreased the batch size to 6, and segment length to 12000. Even now, 3 of 8 GPUs die, and batches are distributed on 5 of them. Training time has been increased drastically.

Any thoughts from collaborators/contributors?

Does anybody have such a problem? When it is trained for 1000k steps with LjSpeech , the "abrupt noise" appears. For example:

The audio file is : LJ001-0007.wav_synthesis_01.zip

My config.json file is:

I used single GPU。

Look forward your help!

Does anyone train this model with single GPU(1080ti) and get good result? In this situation i can only run the model with the batch size 1. Cause I don't have enough GPU...

As the title, I would like to know how to train a new model using another dataset, which have the same structure as LJ Speech dataset. What modifications need to be done for a diffirent language?

I have already trained a WaveGlow model from scratch using LJ Speech dataset and everything worked well during training.

I now try to train a new model using a private dataset that contains only 2 hours of speech. Some audios are inferior to segment_length=16000 (approximatively 10 audios for a dataset composed of 2300 audios). This training is performed in FP32 and, except for batch_size=24, I use the same hyper-parameters than the one in config.json.

Training loss decreases slowly during 120k iterations (which represents a lot of epochs for my small dataset) but further iterations lead to two types of errors:

• NaN loss: I tracked det(W) at each layer to check if the determinant was crossing between positive and negative values, but when loss becomes NaN, all determinants are strictly positive. I don't really know which other term in the loss could cause a NaN issue.
• Jump in the loss: The loss jumps from negative values to positive ones. The loss continues to diverge at next iterations and model forgets what it learned.

I already used this private dataset using WaveNet and everything worked well, which assumes that this dataset is not corrupted.

I tried to decrease the learning rate but instability still persists. Any insight or help to understand these problems would be greatly appreciated.

I implemented a waveglow model in my own project. The codes are almost the same as this repo with some modifications:

1. Upsample the mel-spectrogram to number of groups, so n_mel_channels in WN reduce to 80.
2. Change logdet() in invertable1x1 to det().abs().log() as #35 did because the first few runs I did the loss became nan after thousands of steps.

The n_channels is 256 so the model size is 4~5 times smaller than original. I run model on 2 1080ti using nn.DataParallel with batch size of 8. After about 5k steps the loss is around -6 ~ -7 and I can hear some speech-like sentences from the model outputs. Then the loss value started to go up and down, even over zero, and cannot go any further. I add 24 flows in the model, doesn't help; 32 batch size, the problem still exist. Maybe more steps it will become better, but after 70k steps still cannot see any improvement. Did anyone have similar problems?

I also want to ask about the final loss as reference. In my case, -11 is the smallest value I can get, then the aforementioned problem happened. In #5 @azraelkuan can get a -18 loss value at 56k, is it the normal loss value?

It's a very nice work! I'm study it! But when I'm training the model, I find a problem. May, there is some mistake in mel2samp.py. It can't get a right mel. When I use the "mel_spectrograms" that the NVIDIA gives, the results that "waveglow_old.pt" infers are nice. But ，when I use "python mel2samp.py -f test_files.txt -o . -c config.json" to get mel, the results that "waveglow_old.pt" infers are bad. The voice is from man, not the original woman.

the result of "mel_spectrograms" that the NVIDIA gives:

the result of mel2samp.py

This is my audio. my_results_wav.zip

I have tested NVIDIA/tacotron2+waveglow by using inference.ipynb with pretrained models on single GTX 1080ti. Inference time was 3 times slower than real-time. It spent 24s for generate 7s voice.

Is it possible to generate faster than real-time on single GTX 1080ti. Thank you.

Here is screenshot while generating audio.

Hi NVIDIA/waveglow!

This is a one-off automatically generated pull request from LGTM.com :robot:. You might have heard that we’ve integrated LGTM’s underlying CodeQL analysis engine natively into GitHub. The result is GitHub code scanning!

With LGTM fully integrated into code scanning, we are focused on improving CodeQL within the native GitHub code scanning experience. In order to take advantage of current and future improvements to our analysis capabilities, we suggest you enable code scanning on your repository. Please take a look at our blog post for more information.

This pull request enables code scanning by adding an auto-generated codeql.yml workflow file for GitHub Actions to your repository — take a look! We tested it before opening this pull request, so all should be working :heavy_check_mark:. In fact, you might already have seen some alerts appear on this pull request!

Where needed and if possible, we’ve adjusted the configuration to the needs of your particular repository. But of course, you should feel free to tweak it further! Check this page for detailed documentation.

Questions? Check out the FAQ below!

FAQ

Dear Authors and readers

I would appreciate it if you would give me an answer to my question:

Is it possible to convert the spectrogram image (not the array) to wav (reconstruct the wav audio from the spectrogram in image form)?

When loading the published pre-trained model the dict does not contain neither the optimizer nor the iteration so it will raise a KeyError exception, I changed it to make the iteration 0 if it's not found, and to return the optimizer itself if it is not in the dict

Hello, I have an issue about whether waveglow could be used in a multispeaker setting. For example, could it be trained in VCTK dataset and infer well?
Has anyone tried this? The issues didn't have a definite answer. Thank you very much.

Bumps numpy from 1.13.3 to 1.22.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.