End-2-end speech synthesis with recurrent neural networks

Hello Tiberiu,

I'd love to test TTS-Cube, but unfortunately now i don't have access to a good GPU (and i don't think i could train a TTS on a laptop with a 940mx), do you have a pretrained english model? (it seems you were working on one, but i don't know the current status about that).

Also, do you have an idea what could be the hardware requirements to run the synthesis? For example the nvidia jetson nano seems a nice platform to have a self-hosted TTS, but i'm not sure if it's powerful enough to run TTS-Cube.

Here is my output of training step2

/usr/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  return f(*args, **kwds)
/usr/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  return f(*args, **kwds)
Found 4995 training files and 5 development files
	Rendering devset
		1/5 processing file data/processed/dev/0000001 
		2/5 processing file data/processed/dev/0000002 
		3/5 processing file data/processed/dev/0000003 
		4/5 processing file data/processed/dev/0000004 
		5/5 processing file data/processed/dev/0000005 

Starting epoch 1
Shuffling training data
	1/4995 processing file data/processed/train/0000007
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00,  1.29it/s]
 avg loss=0.9230358004570007 execution time=0.8138909339904785
	2/4995 processing file data/processed/train/0000008
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00,  1.47it/s]
 avg loss=0.8607208132743835 execution time=0.714789867401123
...
avg loss=0.1945137083530426 execution time=0.626471757888794
	17/4995 processing file data/processed/train/0000022
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00,  1.46it/s]
 avg loss=0.0572626106441021 execution time=0.7513647079467773
	18/4995 processing file data/processed/train/0000023
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00,  1.68it/s]
 avg loss=-0.061442214995622635 execution time=0.6261122226715088
	19/4995 processing file data/processed/train/0000024
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00,  1.47it/s]
 avg loss=-0.18586862087249756 execution time=0.7162132263183594
	20/4995 processing file data/processed/train/0000025
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00,  1.68it/s]
 avg loss=0.06383810192346573 execution time=0.6265075206756592
	21/4995 processing file data/processed/train/0000026
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00,  1.67it/s]
 avg loss=-0.20782051980495453 execution time=0.628434419631958
	22/4995 processing file data/processed/train/0000027
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00,  1.46it/s]
 avg loss=-0.31225016713142395 execution time=0.7171187400817871
	23/4995 processing file data/processed/train/0000028
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00,  1.30it/s]
 avg loss=-0.5820147395133972 execution time=0.8073093891143799
	24/4995 processing file data/processed/train/0000029
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00,  1.68it/s]
 avg loss=-0.46214190125465393 execution time=0.6245768070220947
	25/4995 processing file data/processed/train/0000030
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00,  1.46it/s]
 avg loss=-0.7601555585861206 execution time=0.720339298248291
	26/4995 processing file data/processed/train/0000031

First off, this is great work. Can't wait to play around with the code 👍

In the training instructions, I see that you do have multispeaker support. Is it possible to "fine-tune" from an existing checkpoint with another dataset using --resume? Has anyone tried it and see if the results are good?

Hi,

I ran the updated English model in Google Colab, find the code here: https://colab.research.google.com/drive/1L6BYGA0CmQhF6FULWbVr4-yZeKtp4uPK

The synthesis is good, but improvement could be in the pronunciation of the vowels, which sound as if they are underwater, and the consonants, which are a bit sharp. Any idea why this is, or how this could be solved?

Thank you for creating this project, and making it available :)

this is my first github issue, so please forgive me if there are any mistakes. The problem i'm having right now though is simply not understanding what should be contained in a development folder of a training set What I've done. I've downloaded the M-AILABS italian training set, and have splitted the csv in txt files such as every one of them are corresponding to a wav file, and that's for the training set. My question is: what should i put in the other folder? The readme says that there should not be more than 5 files in there, but when i start training with an empty dev folder it gives me an error about a lab file that was not found. I have the same doubt about the g2p thing, but as i'm not going to use that feature that's a secondary thing for me, as well as adding custom things in the lab file which, in fact, i've not added any.

I have trained an encoder on custom data in telugu language for about 4 days but during inference some words are not synthesized and the audio is skipping those words any do you suggest any hyperparameters adjustment or something else to make the synthesizer work correctly. I am using the ljspeech vocoder. and trained it on the last version of the repo before the new g2p pull. and the loss is around 1.8 to 2.3 it remained in that range for the past 20hrs. Thank you

Hello,

Thank you for the wonderful repository.

I read that you're currently training on LJSpeech dataset for english TTS.

Do you have any updates on audio samples?

Also would you be able to provide some rough training stats (number of GPU used, hours need per pass through data, etc).

Thanks again for the awesome repository and open-source effort.

I want to know the general inference time of generating 10sec audio using the vocoder presently i don't have the setup to test it on my own can anyone help thank you

Hello, I recently developed a dataset for voice synthesis in Brazilian Portuguese using my own voice, called TTS-Porguese Corpus, the base has approximately 10 hours of talk, is available at: https://github.com/Edresson/TTS-Portuguese-Corpus

I have already successfully trained the DCTTS model in the dataset.

Is anyone interested in training/adjusting the hyperparameters of the model to get a TTS model in Portuguese?

No idea what the dependencies are for this repo. Apparently it requires python 2 (due to the use of xrange) and is using an old version of scipy ("module scipy.misc has no attribute 'toimage'"). Also, did the folder structure change? Because trainer.py cannot find 'data/processed/train'. I had to manually correct it to '../data/processed/train' (many times because it is hardcoded all over the place).

You may be on to something, but this repo is unusable by others as it is.

Bumps pillow from 6.2.0 to 9.3.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Hello, I've did the required changes to integrate tts cube with melgan vocoder.

The inference is really fast, with wavenet it took 1.5 hour to vocode a few sentences, now it takes literally seconds.

I trained the melgan vocoder for 325 epochs (106170 iterations), it took a day I think and it's already understandable.

Problem is encoder takes so long to train I'm training for days and days and it still says what it wants. I wish a more faster gpu.

It speaks, just not exactly what is in the text file.

The datasets are Japanese and Russian. I want to do a common (multi lingual) model in the future (just for fun).

Is there an interest from others to reproduce my experiment on your dataset?? I can share my code.

I tried using the colab, it works but there is an error in the paths.

After installing the dependencies, code assumes it's in the the repository folder.

add a line to enter the folder %cd TTS-Cube before !git submodule update --init --recursive

or change the paths to include it