hi, folks

I triggered an issue,

when I tried to set '--iter-decode-with-external-reranker ' and '--path checkpoints/nat.pt:at_checkpoints/at.pt '

, do inference with NPD.

Did you meet similar issues,

and do you have any ideas?

It could help me a lot.

Thanks.

Hi, thanks for your awesome work!

When I use the command line here (https://github.com/FLC777/GLAT#train) to train the model on 8*V100 GPUs, the time cost of each epoch increases rapidly (epoch1 10min -> epoch 50 120min). Is there something wrong with my command? I wonder what I am supposed to do.

Thanks very much! hemingkx

Hi, I read your paper and code with great interest. You said in the paper that during glancing, the sampled tokens are replaced with the embeddings from decoder, while the unsampled tokens use encoder output. However, in the code, the unglanced tokens are still masked. Am I missing something?

       glat_info = None
        if glat and tgt_tokens is not None:
            with torch.no_grad():
                with torch_seed(rand_seed):
                    word_ins_out = self.decoder(
                        normalize=False,
                        prev_output_tokens=prev_output_tokens,
                        encoder_out=encoder_out,
                    )
                pred_tokens = word_ins_out.argmax(-1)
                same_num = ((pred_tokens == tgt_tokens) & nonpad_positions).sum(1)
                input_mask = torch.ones_like(nonpad_positions)
                bsz, seq_len = tgt_tokens.size()
                for li in range(bsz):
                    target_num = (((seq_lens[li] - same_num[li].sum()).float()) * glat['context_p']).long()
                    if target_num > 0:
                        input_mask[li].scatter_(dim=0, index=torch.randperm(seq_lens[li])[:target_num].cuda(), value=0)
                input_mask = input_mask.eq(1)
                input_mask = input_mask.masked_fill(~nonpad_positions,False)
                glat_prev_output_tokens = prev_output_tokens.masked_fill(~input_mask, 0) + tgt_tokens.masked_fill(input_mask, 0)
                # this line here
                
                glat_tgt_tokens = tgt_tokens.masked_fill(~input_mask, self.pad)

                prev_output_tokens, tgt_tokens = glat_prev_output_tokens, glat_tgt_tokens

                glat_info = {
                    "glat_accu": (same_num.sum() / seq_lens.sum()).item(),
                    "glat_context_p": glat['context_p'],
                }

Hello, author, while reviewing your code, I found that you did not specify which dataset to use. I read your paper, such as WMT4en de, and downloaded it. However, as for the src dict and tgt dict parameters in preprocess.py, I did not find dict.en.txt and dict.de.txt in the dataset. I dare to bother you, but I hope you can provide a dataset.

@FLC777 Hello, thank you very much for the open source. When I run the glat_CTC script, the model can run and save, but the following errors are always reported from time to time, and then the program stops and kills:

I follow the instructions but I cannot reproduce the results on wmt14 ende. With the provided generation script I only got BLEU4 = 16.65, 48.6/22.0/11.4/6.3 (BP=1.000, ratio=1.014, syslen=65432, reflen=64506) After re-ranking with an AT model, I got BLEU4 = 19.93, 52.2/25.6/14.2/8.3 (BP=1.000, ratio=1.013, syslen=65313, reflen=64506) Still far from 26.55 as reported in the paper. I use the provided training script and train the model on 8 V100 GPUs. The training log shows that the glat accuracy is pretty high at the end of training. epoch 151: 1050 / 1993 loss=3.323, nll_loss=1.371, glat_accu=0.758, glat_keep=0.073, glat_context_p=0.3, word_ins=3.162, length=3.223, ppl=10.01, wps=132588, ups=2.16, wpb=61302.6, bsz=2018.2, num_updates=300000, lr=5.7735e-05, gnorm=1.455, clip=0, train_wall=46, gb_free=6.8, wall=139238 But the valid loss is also very high. valid | epoch 151 | valid on 'valid' subset | loss 7.412 | nll_loss 5.888 | glat_accu 0 | glat_keep 0 | glat_context_p 0 | word_ins 7.252 | length 3.209 | ppl 170.31 | wps 242820 | wpb 41551 | bsz 1500 | num_updates 300000 | best_loss 6.511

I test the inference speed compared autoregressive model with GLAT model, It is no more than 2x speed up actually. There is about 15.3x speed up in your paper (Table 1), pls provide inference scripts. thanks