Hi, I am not sure if you are feeding the right input to the decoder.

(pg. 2) "Given z, the decoder then generates an output sequence (y₁, ..., y_m) of symbols one element at a time. At each step the model is auto-regressive, consuming the previously generated symbols as additional input when generating the next."

I believe your decoder input is a batch of target sequences.

Hi, I tried to force the GPU selection with CUDA_VISIBLE_DEVICES=1 but it pops an error: RuntimeError: cublas runtime error : library not initialized at /py/conda-bld/pytorch_1490903321756/work/torch/lib/THC/THCGeneral.c:387

I think it's related to this: https://discuss.pytorch.org/t/cublas-runtime-error-library-not-initialized-at-data-users-soumith-builder-wheel-pytorch-src-torch-lib-thc-thcgeneral-c-383/1375/8

Hi, I clone your code and run train it on WMT English-German task, but it failed with "RuntimeError: cuda runtime error (2) : out of memory at /opt/conda/conda-bld/pytorch_1502009910772/work/torch/lib/THC/generic/THCStorage.cu:66". I run it on a Tesla K40 which has the same memory capacity of 12GB as your Titan X, and with the default settings. So I don`t know why this happens, do you have any idea? Thanks

I get 98% accuracy after 10 epochs on the multi30k validation set using this 1-layer model:

python train.py -data data/multi30k.atok.low.pt -save_model trained -save_mode best -proj_share_weight -dropout 0.0 -n_layers 1 -n_warmup_steps 40 -epoch 50 -d_inner_hid 1 -d_model 128 -d_word_vec 128 -n_head 4

This is a very small model (note -d_inner_hid 1), which should not get good results at all (98% accuracy is way too high in any case). Generating translations with translate.py produces non-sense. This makes me suspect that there is a problem with the masking code that allows the model to 'cheat' by looking at the target sequence.

I haven't been able to figure out where the problem is, but something seems wrong.

hi, i found that in decoder there is a subsequent mask which mask out the future information here . However, in line 123, you feed in the dec_input(which is the target embeding) at first layer. now check this line and then the MultiHeadAttention moudle's forward function, it has a residual connection and will make dec_input directly reached output, see here. so it doest not use the subsequent mask, which means that the model knows the future. am i correct?

Great work and thanks a lot. I wanted to ask why you do embeddings of the pos encoder? https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/1600401b30eacd3747184827fd40d97752a7627b/transformer/Models.py#L55

I believe the pos encoder should just be added to the input embeddings, like here: https://github.com/Kyubyong/transformer/blob/master/train.py

Let me know, thanks a lot

I can gain 50.8% accuracy on training set and 40.6% accuracy on validation set with WMT14 ch-en,but the translate result can only gain 1% BLEU score,and i find many sentence have the same beginning words or Phrases.Have you test the BLEU score when you got the model? [ Epoch 9 ]

• (Training) ppl: 42.43964, accuracy: 50.856 %, elapse: 98.557 min
• (Validation) ppl: 33.06820, accuracy: 40.699 %, elapse: 0.026 min

https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/20f355eb655bad40195ae302b9d8036716be9a23/transformer/Layers.py#L23

I think the non_pad_mask is not necessary, because processing of padding is done by attn_mask. Why is it necessary?

Currently, the attention mask in the ScaledDotProductAttention is generated in Line 28 in Models.py by: pad_attn_mask = seq_k.data.eq(Constants.PAD).unsqueeze(1) pad_attn_mask = pad_attn_mask.expand(mb_size, len_q, len_k)

Ignoring the batch dimension for an explanation, I assume the generated pad_attn_mask is a matrix of shape (len_q * len_k), then this code will produce the matrix like [A 1], where 1 is an all one submatrix. However, I think the generated attention mask should be like [B 1 // 1 1], where 1 is an all one submatrix and // means line break (sorry I don't know how to type formula in Markdown environments).

Training and validation loss is nan (using commit e21800a6):

$ python3 preprocess.py -train_src data/multi30k/train.en -train_tgt data/multi30k/train.de -valid_src data/multi30k/val.en -valid_tgt data/multi30k/val.de -output data/multi30k/data.pt
$ python3 train.py -data data/multi30k/data.pt -save_model trained -save_model best
[ Epoch 0 ]
  - (Training)   loss:      nan, accuracy: 3.7 %
  - (Validation) loss:      nan, accuracy: 10.0 %
    - [Info] The checkpoint file has been updated.
[ Epoch 1 ]
  - (Training)   loss:      nan, accuracy: 9.09 %
  - (Validation) loss:      nan, accuracy: 9.87 %
[ Epoch 2 ]
  - (Training)   loss:      nan, accuracy: 9.09 %
  - (Validation) loss:      nan, accuracy: 9.83 %
[ Epoch 3 ]
  - (Training)   loss:      nan, accuracy: 9.1 %
  - (Validation) loss:      nan, accuracy: 9.92 %
[ Epoch 4 ]
  - (Training)   loss:      nan, accuracy: 9.09 %
  - (Validation) loss:      nan, accuracy: 9.91 %

Hi, Huang! As far as I know, we hope the pad embedding is a zero-vector, even when it add the position embedding. However, in your new code, the pad embedding is not a zero-vector when the word-embedding add the position embedding. Does it matter? What's more, the encoder output will not multiply the non-pad mask, will this affect the final result? Thanks for your code! Look forward to you reply.

when i run the commond in ubuntu python preprocess.py -lang_src de -lang_trg en -share_vocab -save_data m30k_deen_shr.pkl An error occurs as follows:

` (att) guest1@GPU2:~/zjl/attention-is-all-you-need-pytorch-master$ python preprocess.py -lang_src de -lang_trg en -share_vocab -save_data m30k_deen_shr.pkl Namespace(data_src=None, data_trg=None, keep_case=False, lang_src='de', lang_trg='en', max_len=100, min_word_count=3, save_data='m30k_deen_shr.pkl', share_vocab=True) [Info] Get source language vocabulary size: 5375 [Info] Get target language vocabulary size: 4556 [Info] Merging two vocabulary ... [Info] Get merged vocabulary size: 9521 [Info] Dumping the processed data to pickle file m30k_deen_shr.pkl Traceback (most recent call last): File "preprocess.py", line 337, in main_wo_bpe() File "preprocess.py", line 332, in main_wo_bpe pickle.dump(data,f) File "/home/guest1/anaconda3/envs/att/lib/python3.6/site-packages/dill/_dill.py", line 267, in dump Pickler(file, protocol, **_kwds).dump(obj) File "/home/guest1/anaconda3/envs/att/lib/python3.6/site-packages/dill/_dill.py", line 454, in dump StockPickler.dump(self, obj)

......

File "/home/guest1/anaconda3/envs/att/lib/python3.6/pickle.py", line 476, in save f(self, obj) # Call unbound method with explicit self File "/home/guest1/anaconda3/envs/att/lib/python3.6/site-packages/dill/_dill.py", line 1177, in save_cell f = obj.cell_contents ValueError: Cell is empty `

How can I solve it?

Bumps tensorflow from 1.14.0 to 2.9.3.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

There are doubts in the preprocessing process, is the spatial language model in the code the four datasets of train, test, val, and split in the WMT16 multimodal translation task, and what code command line are these data sets imported through preprocess.py!以及我的preprocess.py的运行结果存在报错情况，如果能解答我的困惑，我将不胜感激！ As well as the running result of my preprocess.py, there are errors. If you can solve my confusion, I would be grateful!

When I run the training code, it does not indicate the specific error location, only the error is: “ValueError: bytes length not a multiple of item size Exception ignored in: 'preshed.bloom.bloom_from_bytes' ValueError: bytes length not a multiple of item size ValueError: bytes length not a multiple of item size Exception ignored in: 'preshed.bloom.bloom_from_bytes' ValueError: bytes length not a multiple of item size ValueError: bytes length not a multiple of item size Exception ignored in: 'preshed.bloom.bloom_from_bytes' ValueError: bytes length not a multiple of item size ValueError: bytes length not a multiple of item size Exception ignored in: 'preshed.bloom.bloom_from_bytes' ValueError: bytes length not a multiple of item size ValueError: bytes length not a multiple of item size Exception ignored in: 'preshed.bloom.bloom_from_bytes' ValueError: bytes length not a multiple of item size ValueError: bytes length not a multiple of item size Exception ignored in: 'preshed.bloom.bloom_from_bytes' ValueError: bytes length not a multiple of item size ValueError: bytes length not a multiple of item size Exception ignored in: 'preshed.bloom.bloom_from_bytes' ValueError: bytes length not a multiple of item size OverflowError: value too large to convert to uint32_t Exception ignored in: 'preshed.bloom.bloom_from_bytes' OverflowError: value too large to convert to uint32_t ValueError: bytes length not a multiple of item size Exception ignored in: 'preshed.bloom.bloom_from_bytes' ValueError: bytes length not a multiple of item size OverflowError: value too large to convert to uint32_t Exception ignored in: 'preshed.bloom.bloom_from_bytes' OverflowError: value too large to convert to uint32_t OverflowError: value too large to convert to uint32_t Exception ignored in: 'preshed.bloom.bloom_from_bytes' OverflowError: value too large to convert to uint32_t OverflowError: value too large to convert to uint32_t Exception ignored in: 'preshed.bloom.bloom_from_bytes' OverflowError: value too large to convert to uint32_t ValueError: bytes length not a multiple of item size Exception ignored in: 'preshed.bloom.bloom_from_bytes' ValueError: bytes length not a multiple of item size ValueError: bytes length not a multiple of item size Exception ignored in: 'preshed.bloom.bloom_from_bytes' ValueError: bytes length not a multiple of item size ValueError: bytes length not a multiple of item size Exception ignored in: 'preshed.bloom.bloom_from_bytes' ValueError: bytes length not a multiple of item size ValueError: bytes length not a multiple of item size Exception ignored in: 'preshed.bloom.bloom_from_bytes' ValueError: bytes length not a multiple of item size OverflowError: value too large to convert to uint32_t Exception ignored in: 'preshed.bloom.bloom_from_bytes' OverflowError: value too large to convert to uint32_t Segmentation fault ”

how should i handle this？

hello, I want to download the WMT'17 by your codes,but I faid,could you tell me how to solve this problem,thank you so much.

the error as following: Already downloaded and extracted http://data.statmt.org/wmt17/translation-task/training-parallel-nc-v12.tgz. Already downloaded and extracted http://data.statmt.org/wmt17/translation-task/dev.tgz. Downloading from http://storage.googleapis.com/tf-perf-public/official_transformer/test_data/newstest2014.tgz to newstest2014.tgz. newstest2014.tgz: 0.00B [00:00, ?B/s] Traceback (most recent call last): File "preprocess.py", line 336, in main() File "preprocess.py", line 187, in main raw_test = get_raw_files(opt.raw_dir, _TEST_DATA_SOURCES) File "preprocess.py", line 100, in get_raw_files src_file, trg_file = download_and_extract(raw_dir, d["url"], d["src"], d["trg"]) File "preprocess.py", line 71, in download_and_extract compressed_file = _download_file(download_dir, url) File "preprocess.py", line 93, in _download_file urllib.request.urlretrieve(url, filename=filename, reporthook=t.update_to) File "/usr/local/lib/python3.7/urllib/request.py", line 247, in urlretrieve with contextlib.closing(urlopen(url, data)) as fp: File "/usr/local/lib/python3.7/urllib/request.py", line 222, in urlopen return opener.open(url, data, timeout) File "/usr/local/lib/python3.7/urllib/request.py", line 531, in open response = meth(req, response) File "/usr/local/lib/python3.7/urllib/request.py", line 641, in http_response 'http', request, response, code, msg, hdrs) File "/usr/local/lib/python3.7/urllib/request.py", line 569, in error return self._call_chain(*args) File "/usr/local/lib/python3.7/urllib/request.py", line 503, in _call_chain result = func(*args) File "/usr/local/lib/python3.7/urllib/request.py", line 649, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 403: Forbidden

if trg_emb_prj_weight_sharing:
            # Share the weight between target word embedding & last dense layer
            self.trg_word_prj.weight = self.decoder.trg_word_emb.weight
if emb_src_trg_weight_sharing:
            self.encoder.src_word_emb.weight = self.decoder.trg_word_emb.weight

The code above want to realize weight share, but I'm confused that the embed layer and the linear layer have different shape of weight. How can this assignment work?