• When I tried to load the pretrained model output/LJSpeech/ckpt/900000.pth.tar, I have some errors:

size mismatch for encoder.src_word_emb.weight: copying a param with shape torch.Size([361, 256]) from checkpoint, the shape in current model is torch.Size([151, 256]).

• The code which loads the model from repo

base_config_path = "config/LJSpeech"
prepr_path = f"{base_config_path}/preprocess.yaml"
model_path = f"{base_config_path}/model.yaml"
train_path = f"{base_config_path}/train.yaml"

prepr_config = yaml.load(open(prepr_path, "r"), Loader=yaml.FullLoader)
model_config = yaml.load(open(model_path, "r"), Loader=yaml.FullLoader)
train_config = yaml.load(open(train_path, "r"), Loader=yaml.FullLoader)
configs = (prepr_config, model_config, train_config)
cpkt_path = "output/LJSpeech/ckpt/900000.pth.tar"

def get_model(ckpt_path, configs):
    (preprocess_config, model_config, train_config) = configs
    model = FastSpeech2(preprocess_config, model_config).to(device)
    ckpt = torch.load(ckpt_path)
    model.load_state_dict(ckpt["model"])
    model.eval()
    model.requires_grad_ = False
    return model

Hello, I am trying to train a model using LJSpeech dataset, following steps in README.md and got this error:

$ python3 preprocess.py config/LJSpeech/preprocess.yaml Processing Data ... 0%| | 0/1 [00:00<?, ?it/s] Traceback (most recent call last): File "preprocess.py", line 15, in <module> preprocessor.build_from_path() File "/FastSpeech2/preprocessor/preprocessor.py", line 85, in build_from_path if len(pitch) > 0: UnboundLocalError: local variable 'pitch' referenced before assignment I've already downloaded TextGrid files and put them in '/FastSpeech2/preprocessed_data/LJSpeech/LJSpeech/TextGrid'

Hi, I used Mandarin dataset(BIAOBEI) to train FastSpeech2. The loss of mel and PostNet mel seems no problem. But I find out that the loss of variance_adaptor (Duration Loss, F0 Loss and Energy Loss) is really high.

The following is a part of my log: Epoch [191/1000], Step [115650/608000]: Total Loss: 68.7120, Mel Loss: 0.2892, Mel PostNet Loss: 0.2889, Duration Loss: 2.2572, F0 Loss: 59.7572, Energy Loss: 6.1195; Time Used: 28493.331s, Estimated Time Remaining: 97800.871s.

How could I solve this?

Thank you.

The implementation with frame-level seems more similar to the implementation in the paper. But why the quality is much worse than with phoneme level? And with phoneme level, the pitch is predicted before expanding length, which results in the pitch in a phoneme are the same. I think the pitches in a phoneme sometimes change. But... with phoneme level has excellent performance...

Here : https://github.com/ming024/FastSpeech2/blob/bd4c3413c90c5c1310066b1051d5c67abb75e2fa/modules.py#L48

and then https://github.com/ming024/FastSpeech2/blob/bd4c3413c90c5c1310066b1051d5c67abb75e2fa/modules.py#L50

But as per paper detail the input of Energy predictor should be output of length regulator not the output of pitch predictor. See the fastspeech 2 diagram clearly input of Energy predictor is x output of length regulator without pitch component. Actual code should be like:

 def forward(self, x, duration_target=None, pitch_target=None, energy_target=None, max_length=None):

        duration_prediction = self.duration_predictor(x)
        if duration_target is not None:
            x, mel_pos = self.length_regulator(x, duration_target, max_length)
        else:
            duration_rounded = torch.round(duration_prediction)
            x, mel_pos = self.length_regulator(x, duration_rounded)
        
        pitch_prediction = self.pitch_predictor(x)
        if pitch_target is not None:
            pitch_embedding = self.pitch_embedding(torch.bucketize(pitch_target, self.pitch_bins))
        else:
            pitch_embedding = self.pitch_embedding(torch.bucketize(pitch_prediction, self.pitch_bins))
  
        
        energy_prediction = self.energy_predictor(x)
        if energy_target is not None:
            energy_embedding = self.energy_embedding(torch.bucketize(energy_target, self.energy_bins))
        else:
            energy_embedding = self.energy_embedding(torch.bucketize(energy_prediction, self.energy_bins))

        x = x + pitch_embedding
        x = x + energy_embedding
        
        return x, duration_prediction, pitch_prediction, energy_prediction, mel_pos

I tried to train on my own dataset, but the result is not as good as I expected, and even worse than fastspeech1. I use default setting , phone-level, with hifigan as vocoder How about your results?

hi, i train fastspeech2 on the biaobei dataset. the result seems that the pitch predictor and the duration predictor doesn't work well, but if you input the groud truth, you can get a good result. so does the your demo use the ground truth pitch and energy or use the predictor to predict the pitch and energy? i find other fastspeech2 code on tensorflowTTS and espnet2, they use the structure different from the orginal paper. they use length regulator after all the predictors finished, and pitch and energy predictor works well.

Hey I have a custom datasets, that contains about 12 hours of data, I want to pre-train with multi speaker using LJSSpeech.

Would I take the pretrained FS2 - LJS Speech model Go into model.yml

change to multi_speaker: True speaker: 'universal'

ensure that the preprocessor outputs my data like so ? 6694_70837_000012_000001|6694|{P IY1 S}|peace! name of file | speaker | alignment | utterance?

First thank you, I have solved the issue opened thanks to you support. In my understanding both Melgan (that I have tried) and Waveglow (not run yet) have a female voice. To have a male voice is it necessary to train from scratch the model? Or add support to a specific vocoder?

Thank you.

Hi, I am using MFA to do force alignment on my own dataset. I found that the alignment results is not accurate.

my dataset is 15 hours, maybe not big enough for training am from scratch. after increased my dataset to 30 hours, still not accurate. Do you have some trick to improve the alignment?

I get the following error:

root@75adae8f35d1:/app# python3 synthesize.py --step 300000
|{DH AH0 N EY1 SH AH0 N Z T UH1 R IH2 Z AH0 M M IH1 N AH0 S T ER0 HH AE1 Z AO1 L S OW0 EH0 N K ER1 IH0 JH D AO2 S T R EY1 L Y AH0 N Z T UW1 T EY1 K DH EH1 R HH AA1 L AH0 D EY2 Z W IH0 DH IH1 N DH AH0 K AH1 N T R IY0 DH IH1 S Y IH1 R} |
Traceback (most recent call last):
  File "synthesize.py", line 94, in <module>
    synthesize(model, text, sentence, prefix='step_{}'.format(args.step))
  File "synthesize.py", line 48, in synthesize
    mel, mel_postnet, duration_output, f0_output, energy_output = model(text, src_pos)
  File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 143, in forward
    return self.module(*inputs, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/app/fastspeech2.py", line 33, in forward
    encoder_output, d_target, p_target, e_target, max_length)
  File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/app/modules.py", line 47, in forward
    pitch_embedding = self.pitch_embedding(torch.bucketize(pitch_prediction, self.pitch_bins))
AttributeError: module 'torch' has no attribute 'bucketize'

I'm running in CPU, and I had to modify get_FastSpeech2 like this:

def get_FastSpeech2(num):
    checkpoint_path = os.path.join(hp.checkpoint_path, "checkpoint_{}.pth.tar".format(num))
    model = nn.DataParallel(FastSpeech2())
    if torch.cuda.is_available():
        model.load_state_dict(torch.load(checkpoint_path)['model'])
    else:
        model.load_state_dict(torch.load(checkpoint_path, map_location=torch.device('cpu'))['model'])
    model.requires_grad = False
    model.eval()
    return model

to set map_location to cpu device.

Hello, the last words of my synthesized voice are often unclear. If the sentence is too long, the second half will collapse. How to solve this problem

the preprocess process was running successfully ,but when I running the train code ,it has a error as followed

(torch) zhonghuihang@kdf-X12DAi-N6:~/fastSpeech2-master$ python3 train.py -p config/LJSpeech/preprocess.yaml -m config/LJSpeech/model.yaml -t config/LJSpeech/train.yaml Prepare training ... Number of FastSpeech2 Parameters: 35159361 Traceback (most recent call last): File "/home/zhonghuihang/miniconda3/envs/torch/lib/python3.7/tarfile.py", line 186, in nti s = nts(s, "ascii", "strict") File "/home/zhonghuihang/miniconda3/envs/torch/lib/python3.7/tarfile.py", line 170, in nts return s.decode(encoding, errors) UnicodeDecodeError: 'ascii' codec can't decode byte 0xdf in position 2: ordinal not in range(128)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/zhonghuihang/miniconda3/envs/torch/lib/python3.7/tarfile.py", line 2289, in next tarinfo = self.tarinfo.fromtarfile(self) File "/home/zhonghuihang/miniconda3/envs/torch/lib/python3.7/tarfile.py", line 1095, in fromtarfile obj = cls.frombuf(buf, tarfile.encoding, tarfile.errors) File "/home/zhonghuihang/miniconda3/envs/torch/lib/python3.7/tarfile.py", line 1037, in frombuf chksum = nti(buf[148:156]) File "/home/zhonghuihang/miniconda3/envs/torch/lib/python3.7/tarfile.py", line 189, in nti raise InvalidHeaderError("invalid header") tarfile.InvalidHeaderError: invalid header

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/zhonghuihang/miniconda3/envs/torch/lib/python3.7/site-packages/torch/serialization.py", line 595, in _load return legacy_load(f) File "/home/zhonghuihang/miniconda3/envs/torch/lib/python3.7/site-packages/torch/serialization.py", line 506, in legacy_load with closing(tarfile.open(fileobj=f, mode='r:', format=tarfile.PAX_FORMAT)) as tar,
File "/home/zhonghuihang/miniconda3/envs/torch/lib/python3.7/tarfile.py", line 1593, in open return func(name, filemode, fileobj, **kwargs) File "/home/zhonghuihang/miniconda3/envs/torch/lib/python3.7/tarfile.py", line 1623, in taropen return cls(name, mode, fileobj, **kwargs) File "/home/zhonghuihang/miniconda3/envs/torch/lib/python3.7/tarfile.py", line 1486, in init self.firstmember = self.next() File "/home/zhonghuihang/miniconda3/envs/torch/lib/python3.7/tarfile.py", line 2301, in next raise ReadError(str(e)) tarfile.ReadError: invalid header

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "train.py", line 198, in main(args, configs) File "train.py", line 48, in main vocoder = get_vocoder(model_config, device) File "/home/zhonghuihang/fastSpeech2-master/utils/model.py", line 63, in get_vocoder ckpt = torch.load("/home/zhonghuihang/fastSpeech2-master/hifigan/generator_LJSpeech.pth.tar.zip") File "/home/zhonghuihang/miniconda3/envs/torch/lib/python3.7/site-packages/torch/serialization.py", line 426, in load return _load(f, map_location, pickle_module, **pickle_load_args) File "/home/zhonghuihang/miniconda3/envs/torch/lib/python3.7/site-packages/torch/serialization.py", line 599, in _load raise RuntimeError("{} is a zip archive (did you mean to use torch.jit.load()?)".format(f.name)) RuntimeError: /home/zhonghuihang/fastSpeech2-master/hifigan/generator_LJSpeech.pth.tar.zip is a zip archive (did you mean to use torch.jit.load()?) (torch) zhonghuihang@kdf-X12DAi-N6:~/fastSpeech2-master$ python3 train.py -p config/LJSpeech/preprocess.yaml -m config/LJSpeech/model.yaml -t config/LJSpeech/train.yaml Prepare training ... channel 3: open failed: connect failed: Connection refused channel 4: open failed: connect failed: Connection refused channel 3: open failed: connect failed: Connection refused channel 4: open failed: connect failed: Connection refused channel 3: open failed: connect failed: Connection refused channel 4: open failed: connect failed: Connection refused Number of FastSpeech2 Parameters: 35159361 Removing weight norm... Training: 0%| | 0/900000 [00:00<?, ?it/sTraceback (most recent call last): | 0/197 [00:00<?, ?it/s] File "train.py", line 198, in main(args, configs) File "train.py", line 82, in main output = model(*(batch[2:])) File "/home/zhonghuihang/miniconda3/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in call result = self.forward(*input, **kwargs) File "/home/zhonghuihang/miniconda3/envs/torch/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 150, in forward return self.module(*inputs[0], **kwargs[0]) File "/home/zhonghuihang/miniconda3/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in call result = self.forward(*input, **kwargs) File "/home/zhonghuihang/fastSpeech2-master/model/fastspeech2.py", line 66, in forward output = self.encoder(texts, src_masks) File "/home/zhonghuihang/miniconda3/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in call result = self.forward(*input, **kwargs) File "/home/zhonghuihang/fastSpeech2-master/transformer/Models.py", line 95, in forward enc_output, mask=mask, slf_attn_mask=slf_attn_mask File "/home/zhonghuihang/miniconda3/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in call result = self.forward(*input, **kwargs) File "/home/zhonghuihang/fastSpeech2-master/transformer/Layers.py", line 23, in forward enc_input, enc_input, enc_input, mask=slf_attn_mask File "/home/zhonghuihang/miniconda3/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in call result = self.forward(*input, **kwargs) File "/home/zhonghuihang/fastSpeech2-master/transformer/SubLayers.py", line 39, in forward q = self.w_qs(q).view(sz_b, len_q, n_head, d_k) File "/home/zhonghuihang/miniconda3/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in call result = self.forward(*input, **kwargs) File "/home/zhonghuihang/miniconda3/envs/torch/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 87, in forward return F.linear(input, self.weight, self.bias) File "/home/zhonghuihang/miniconda3/envs/torch/lib/python3.7/site-packages/torch/nn/functional.py", line 1372, in linear output = input.matmul(weight.t()) RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc) Training: 0%| | 1/900000 [00:00<142:38:55, 1.75it/s] Exception ignored in: <function tqdm.del at 0x7ff84f88e4d0> Traceback (most recent call last): File "/home/zhonghuihang/miniconda3/envs/torch/lib/python3.7/site-packages/tqdm/std.py", line 1086, in del File "/home/zhonghuihang/miniconda3/envs/torch/lib/python3.7/site-packages/tqdm/std.py", line 1270, in close File "/home/zhonghuihang/miniconda3/envs/torch/lib/python3.7/site-packages/tqdm/std.py", line 572, in _decr_instances File "/home/zhonghuihang/miniconda3/envs/torch/lib/python3.7/site-packages/tqdm/_monitor.py", line 51, in exit File "/home/zhonghuihang/miniconda3/envs/torch/lib/python3.7/threading.py", line 522, in set File "/home/zhonghuihang/miniconda3/envs/torch/lib/python3.7/threading.py", line 365, in notify_all File "/home/zhonghuihang/miniconda3/envs/torch/lib/python3.7/threading.py", line 348, in notify TypeError: 'NoneType' object is not callable

I am facing a dimension mismatch issue in pitch embedding addition to the encoder output . When I am trying to train the Fastspeech2 model on Hindi data. The Screenshot of the issue is attached as under.

Note :- I have made the necessary changes in the script to adapt it for Hindi dataset

The same code runs for LJSpeech dataset but fails for Hindi Dataset. Kindly help me resolve the issue !

It seems like MFA has changed to new version and your instructions in README is not valid anymore. There is no directory named ./montreal-forced-aligner/bin/mfa_align

I also tried to install MFA folloewd the instruction in https://montreal-forced-aligner.readthedocs.io/en/latest/. When i run the code mfa align raw_data/LJSpeech/ lexicon/librispeech-lexicon.txt english preprocessed_data/LJSpeech . It got the error : `FileNotFoundError: [Errno 2] No such file or directory: '/root/Documents/MFA/LJSpeech/corpus_data/split1/feats.librispeech-lexicon.0.scp'

What should i do next?