@lyeoni

In the case I want to train an Arabic to Urdu Machine Translation:

• is that attainable using this project?
• what options should be set in training?
• do you suggest another github project?

Hi, @lyeoni ! You have written great tutorials. I really appreciate you) We can improve a little bit with one pretty line. Look, please) Here, we fill first key-value items of stoi, itos by special tokens. I suggest insert this line before cycle. special_tokens = filter(lambda x: x is not None, [self.unk_token, self.bos_token, self.eos_token, self.pad_token]) If we don't set value for self.unk_token and set for self.bos_token, then index in dictionary become wrong. So, we need filter None values before. Input vocab = Vocab(body, bos_token='<bos>'); vocab.build(); vocab.stoi; Wrong Output '<bos>': 1 ' ': 1, 'hi': 2, 'bear': 3, ...

The corresponding package is installed and Data set downloaded，Run vocab.py . The following error occurred： “local variable 'MosesTokenizer' referenced before assignment”

Thanks for your great job! I learned a lot. However, I have a question. I train the model for 7 epochs reaching a train acc of 95.2 and test(validate) acc of 85.2. Is that normal? Could the final test(validate) acc be higher after more epochs? Thanks!

Hi, In cleaning function in the script : nlp-tutorial/news-category-classifcation/preprocessing.py, line 21 is written as text = re.sub(r'[!]{2,}', '?', text) # multiple ?s -> ?. There should be ? in first argument and It should be text = re.sub(r'[?]{2,}', '?', text) # multiple ?s -> ?. Am I correct?

Hi Lyeoni,

First of all, thank you a lot for your work in making these tutorials, which are interesting !

I am trying to run the question-answer-matching tutorial and reproduce your evaluation. Unfortunately, I can't download the Posts.xml file from git lfs as it looks like your subscription doesn't accept download anymore. By any chance, do you have that file hosted somewhere else ? That would allow me to run the evaluation with your trained model.

Thanks a lot and I wish you a nice day ! :-)

Traceback (most recent call last):

File "", line 1, in runfile('D:/nlp-tutorial/neural-machine-translation/nmt/train.py', wdir='D:/nlp-tutorial/neural-machine-translation/nmt')

File "C:\ProgramData\Anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 827, in runfile execfile(filename, namespace)

File "C:\ProgramData\Anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 110, in execfile exec(compile(f.read(), filename, 'exec'), namespace)

File "D:/nlp-tutorial/neural-machine-translation/nmt/train.py", line 254, in trainiters(pairs, encoder, decoder, n_iters)

File "D:/nlp-tutorial/neural-machine-translation/nmt/train.py", line 184, in trainiters train_pairs += [random.choice(train_pairs) for i in range(n_iters%len(train_pairs))]

ZeroDivisionError: integer division or modulo by zero