Hi, Im an nlp rookie, I want to ask u a question, your code extract input(context) in a fixed window in 43th area, and "word sequence" is a sentences list , some words may extract their neighbour words form different sentences, so, is this way harm to the result?

And my training result seems not very well and I didn't change the codes.

If u see this issues, please answer me in your free time. Although my english is poor, I still want to express my gratitude to u.

In this PR, I added a greedy decoder function that generates the decoder input for inference. This is important for translating sentences as we don't know the target input beforehand. In the paper, they mentioned that they ran Beam Search with a k=4. In the greedy approach, k = 1.

1. The variable API has been deprecated. you can remove this. I just provided an example with basic attention model. if you agree with me, retune it. https://pytorch.org/docs/stable/autograd.html#variable-deprecated

2. sequence to sequence model has n_step variable(i guess you mean padding... right? but I just little confused that you used identical terminology even if this is tutorial.

Hi thanks for sharing your codes.

I've had read your seq2seq implementation and I was wondering about the RNN Encode-Decode model.

in the paper, 'Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation'

They say

proposed gating unit

and I couldn't find the new hidden-state activation function in your code.

Do you have any plan to add the proposed activation process? or is it okay to just skip the parts?

thank you so much in advance

In this PR, I added a greedy decoder function that generates the decoder input for inference. This is important for translating sentences as we don't know the target input beforehand. In the paper, they mentioned that they ran Beam Search with a k=4. In the greedy approach, k = 1.

Small mistakes in NNLM:

X = self.C(X) # X : [batch_size, n_step, n_class]
X = X.view(-1, n_step * m) # [batch_size, n_step * n_class]

should be

X = self.C(X) # X : [batch_size, n_step, m]
X = X.view(-1, n_step * m) # [batch_size, n_step * m]

#48

Hi there, I'm Dan from Deepnote. We're a small startup working on a new, jupyter-compatible data science notebook. I was trying out Deepnote with some cool public repos like this one and I thought it might be useful for others too to have a one-click button to run your repo, so I submitted a PR. Hope you find it useful!

Thanks for your work, Dan

`class TextCNN(nn.Module): def init(self): super(TextCNN, self).init()

    self.num_filters_total = num_filters * len(filter_sizes)
    self.W = nn.Parameter(torch.empty(vocab_size, embedding_size).uniform_(-1, 1)).type(dtype)
    self.Weight = nn.Parameter(torch.empty(self.num_filters_total, num_classes).uniform_(-1, 1)).type(dtype)
    self.Bias = nn.Parameter(0.1 * torch.ones([num_classes])).type(dtype)

def forward(self, X):
    embedded_chars = self.W[X] # [batch_size, sequence_length, sequence_length]
    embedded_chars = embedded_chars.unsqueeze(1) # add channel(=1) [batch, channel(=1), sequence_length, embedding_size]

    pooled_outputs = []
    for filter_size in filter_sizes:
        # conv : [input_channel(=1), output_channel(=3), (filter_height, filter_width), bias_option]
        conv = nn.Conv2d(1, num_filters, (filter_size, embedding_size), bias=True)(embedded_chars)
        h = F.relu(conv)
        # mp : ((filter_height, filter_width))
        mp = nn.MaxPool2d((sequence_length - filter_size + 1, 1))
        # pooled : [batch_size(=6), output_height(=1), output_width(=1), output_channel(=3)]
        pooled = mp(h).permute(0, 3, 2, 1)
        pooled_outputs.append(pooled)

    h_pool = torch.cat(pooled_outputs, len(filter_sizes)) # [batch_size(=6), output_height(=1), output_width(=1), output_channel(=3) * 3]
    h_pool_flat = torch.reshape(h_pool, [-1, self.num_filters_total]) # [batch_size(=6), output_height * output_width * (output_channel * 3)]

    model = torch.mm(h_pool_flat, self.Weight) + self.Bias # [batch_size, num_classes]
    return model`

I wonder if it's wrong to create conv inside the loop?

https://github.com/graykode/nlp-tutorial/blob/3b3a80dc63e69935731bcf09c951eb371692af8f/5-1.Transformer/Transformer(Greedy_decoder)-Torch.py#L141

position encoding table should be (src_len, d_model). Why (src_vocab_size, d_model) here?

In Autocomplete We already have

X = tf.placeholder(tf.float32, [None, n_step, n_class]) # [batch_size, n_step, n_class]
Y = tf.placeholder(tf.float32, [None, n_class])         

to guess next missing character

1. How I can customize them to guess more than 1 character ? I don't have any idea about multiplies a tensor by tensor.
2. In outputs, states = tf.nn.dynamic_rnn(cell, X, dtype=tf.float32) Why the shape of states alway (2,), what really 2 mean ? Thank you for sharing the information.

In the code 'Seq2seq-torch.py', i saw u use np.eye,the one-hot representation, to represent embedding, so i change in a normal way ,using nn.Embedding(dict_length,embedding_dim),it can work out. but the loss i got is very high. i wanna ask the differences between this two ways. here are my code and the result.

In Line 209:

optimizer = optim.Adam(model.parameters(), lr=0.001)

In practice, this BERT model is bound to fall into local convergence if the learning rate is 0.001; I think the learning rate should be reduced to 0.0001. The experimental results show that when the learning rate is 0.0001, after about 100 iterations, the loss value will be reduced to 0.1, while if the learning rate is 0.001, the loss value will almost never be less than 2.0.

when lr=0.01

Epoch: 0010 cost = 15.205759
Epoch: 0020 cost = 16.236261
Epoch: 0030 cost = 18.436878
Epoch: 0040 cost = 4.077913
Epoch: 0050 cost = 12.703120
Epoch: 0060 cost = 10.411244
Epoch: 0070 cost = 1.640913
Epoch: 0080 cost = 10.753708
Epoch: 0090 cost = 8.370532
Epoch: 0100 cost = 1.624577
Epoch: 0110 cost = 8.537676
Epoch: 0120 cost = 7.453298
Epoch: 0130 cost = 1.659591
Epoch: 0140 cost = 7.092763
Epoch: 0150 cost = 6.843360
Epoch: 0160 cost = 1.688111
Epoch: 0170 cost = 6.052425
Epoch: 0180 cost = 6.395712
Epoch: 0190 cost = 1.707749
Epoch: 0200 cost = 5.263054
······
Epoch: 5000 cost = 2.523541

when lr=0.0001

Epoch: 0010 cost = 13.998453
Epoch: 0020 cost = 6.168099
Epoch: 0030 cost = 3.504844
Epoch: 0040 cost = 2.312538
Epoch: 0050 cost = 1.723783
Epoch: 0060 cost = 1.412463
Epoch: 0070 cost = 0.930549
Epoch: 0080 cost = 0.671946
Epoch: 0090 cost = 0.745429
Epoch: 0100 cost = 0.139699
Epoch: 0110 cost = 0.187208
Epoch: 0120 cost = 0.075726

Line 202 : optimizer = optim.Adam(model.parameters(), lr=0.001)

In practice, I think the effect of Adam is quite bad. When epoch = 10, cost is 1.6; when epoch = 100 or 1000, cost is still equal to 1.6. So I think we can change Adam to SGD, that is, optimizer = optim.SGD(model.parameters(), lr=0.001)

Here are the effects of using SGD：

Epoch: 0100 cost = 0.047965
Epoch: 0200 cost = 0.020129
Epoch: 0300 cost = 0.012563
Epoch: 0400 cost = 0.009101
Epoch: 0500 cost = 0.007131
Epoch: 0600 cost = 0.005862
Epoch: 0700 cost = 0.004978
Epoch: 0800 cost = 0.004325
Epoch: 0900 cost = 0.003823
Epoch: 1000 cost = 0.003426

Thanks for sharing! Just found out Attention.get_att_weight is calculating attention in a for-loop? this looks rather slow isn't it?

4-2.Seq2Seq(Attention)/Seq2Seq(Attention).ipynb

    def get_att_weight(self, dec_output, enc_outputs):  # get attention weight one 'dec_output' with 'enc_outputs'
        n_step = len(enc_outputs)
        attn_scores = torch.zeros(n_step)  # attn_scores : [n_step]

        for i in range(n_step):
            attn_scores[i] = self.get_att_score(dec_output, enc_outputs[i])

        # Normalize scores to weights in range 0 to 1
        return F.softmax(attn_scores).view(1, 1, -1)

    def get_att_score(self, dec_output, enc_output):  # enc_outputs [batch_size, num_directions(=1) * n_hidden]
        score = self.attn(enc_output)  # score : [batch_size, n_hidden]
        return torch.dot(dec_output.view(-1), score.view(-1))  # inner product make scalar value

Suggested parallel version

    def get_att_weight(self, dec_output, enc_outputs):  # get attention weight one 'dec_output' with 'enc_outputs'
        n_step = len(enc_outputs)
        attn_scores = torch.zeros(n_step,device=self.device)  # attn_scores : [n_step]

        enc_t = self.attn(enc_outputs)
        score = dec_output.transpose(1,0).bmm(enc_t.transpose(1,0).transpose(2,1))
        out1   = score.softmax(-1)
        return out1

calculate attention_score `

Attention

outputs = tf.concat([output[0], output[1]], 2) # output[0] : lstm_fw, output[1] : lstm_bw outputs = tf.transpose(outputs, [1, 0, 2]) # [n_step, batch_size, n_hidden]

只用了最后一个步长的输出

final_hidden_state = outputs[-1] output_all = tf.concat([output[0], output[1]], 2)
final_hidden_state = tf.expand_dims(final_hidden_state, 2)
attn_weights = tf.squeeze(tf.matmul(output_all, final_hidden_state), 2) `

1. in"class Encoder": enc_outputs = self.src_emb(enc_inputs) + self.pos_emb(torch.LongTensor([[1,2,3,4,0]]))

I think it may be: enc_outputs = self.src_emb(enc_inputs) + self.pos_emb(torch.LongTensor([[0,1,2,3,4]]))

2. in"class Decoder": dec_outputs = self.tgt_emb(dec_inputs) + self.pos_emb(torch.LongTensor([[5,1,2,3,4]]))

I think it may be: dec_outputs = self.tgt_emb(dec_inputs) + self.pos_emb(torch.LongTensor([[0,1,2,3,4]]))

In line 16 you use input = input + [0] * (max_len - len(input)) the padding, you use 0, which means the first word 'Lorem'. but it is not the right choose. I think you can change like that

    # word_dict = {w: i for i, w in enumerate(list(set(sentence.split())))}
    # number_dict = {i: w for i, w in enumerate(list(set(sentence.split())))}
    word_dict = {w: i for i, w in enumerate(['PAD']+list(set(sentence.split())))}
    number_dict = {i: w for i, w in enumerate(['PAD']+list(set(sentence.split())))}