class ClfHead(nn.Module):
    """ Classifier Head for the transformer """

    def __init__(self, clf_token, cfg):
        super(ClfHead, self).__init__()
        self.n_embd = cfg.n_embd
        self.clf_token = clf_token
        self.dropout = nn.Dropout2d(cfg.clf_pdrop)  # To reproduce the noise_shape parameter of TF implementation
        self.linear = nn.Linear(cfg.n_embd, 1)
        nn.init.normal_(self.linear.weight, std=0.02)
        nn.init.normal_(self.linear.bias, 0)

    def forward(self, h, x):
        # Classification logits
        clf_h = h.view(-1, self.n_embd)
        flat = x[:, :, :, 0].contiguous().view(-1)
        clf_h = clf_h[flat == self.clf_token, :]
        clf_h = clf_h.view(-1, x.size(1), self.n_embd, 1)
        clf_h = self.dropout(clf_h)
        clf_h = clf_h.view(-1, self.n_embd)
        clf_logits = self.linear(clf_h)
        return clf_logits.view(-1, x.size(1))

Here the self.dropout(clf_h) essentially removes the representation of a sentence and its conclusion, there is remote chance (0.2*0.2) that both representations get removed for a given data item. I am confused on how this aids training .

Dear guys,

I did some experiments on text generation with the pretrained LM model. I made a PR so you can see the changes: https://github.com/huggingface/pytorch-openai-transformer-lm/pull/35 I have some questions regarding the results.

1. The generation quality is very poor. The model can not generate grammatical sentences, let alone long coherent sentences. Here are some snippets: Input some beginning words: I love you , " you said . first . last click ... game ' keep ' ' the zer that

Input some beginning words: Once upon a time . " freyja , freyja freyja freyja freyja freyja freyja freyja freyja freyja freyja freyja freyja freyja freyja freyja freyja

Input some beginning words: Everytime . the . - holding . - " nothing in . very ... out . " grin .

Input some beginning words: I feel very royal . please . at , very , ' ! deserving ... ' something , family , had

2. At each step, the top 5 candidates for next token are dominated by the most frequent tokens, e.g. ",", "and", "the", "was", but also have some infrequent tokens, e.g. "-", "f". When these infrequent tokens show up, they are irrelevant with the sentence context. I don't know why.

3. As the output layer also have weights for 512 position embeddings, the output dim is 40478 (word indices) + 512 (position indices). The logits for these 512 indices are usually much larger than 40478 word indices, so I have to mask them before softmax. I think this is a bit strange because during pretraining the correct labels are always within the 40478 indices.

The paper reported a very low ppl of 18.4 on the BooksCorpus. I thought the pretrained model should be a very strong LM model able to generate high quality text. The results confused me. Can you give me some advice? Is it because deep transformer lm is inherently not good at the generation task, or due to some hidden bug in my code?

• Da Xiao

Hi,

I am new to PyTorch (but still more at ease with it than TF) so I thought to experiment with @thomwolf 's implementation in this repo (thanks for sharing it!!)

I would like to try out the code to perform binary text classification of text snippets, similar to the classification tasks such as the Corpus of Linguistic Acceptability (CoLA) and the Stanford Sentiment Treebank (SST-2) in the original reference.

These are the steps that I think are needed to get the code working (but I am not sure that these are correct and/or exhaustive):

1. Create two sets snippets_val.csv and snippets_test.csv containing two columns, text (string) and class (an int equal to 0 or 1).
2. In datasets.py create two new functions:
   • _snippets returning two lists st, y, and
   • snippets defined with different values of n_train and n_validand whose return statement looks like return (trX, trY), (vaX, vaY), (teX, )
3. In train.py, rewrite transform_roc into a transform_snippet that doesn't use [delimiter] and takes only one argument in input <- somewhat tricky to me can anyone provide some guidance?
4. In train.py, in the encoding bit and afterwards:
   • modify the tuple in output of encode_dataset to match the output of the function of snippets redefined above.
   • get rid of encoder['_delimiter_'] = len(encoder)
   • set n_special = 2 as we got rid of ['_delimiter_']
   • get rid of the vars containing 2 and 3 in their name (?) e.g. in the definition of n_ctx <- somewhat tricky to me can anyone provide some guidance?
5. In train.py:
   • modify the call to dh_model to use ('classification', 2) instead of 'multiple_choice'
   • use (unless it's bugged!) ClassificationLossCompute instead of MultipleChoiceLossCompute
6. In analysis.py:
   • create a new function snippets so to invoke _snippets (from datasets.py) and read in snippets_test.csv and adjust its call to _snippets so to take into account that it outputs two lists (not 4)
7. Modify imports in train.py coherently with all of the above.

Does all of the above make sense as a plan, or can somebody fill missing bits or provide an alternative list of "sub-steps" ? Also, can someone provide some guidance on how to rewrite transform_roc (comments on the original code would be fantastic, I am glad to annotate the original function and contribute to the repo as a result of this!)

Thanks to anyone patiently reading this!

In attention class, you have the following code for masking. I understand the logic for pre training, but in fine tuning if we dont include language model loss we should have a check here for not applying the mask. Do we have to always apply the masking because the model was trained that way, is there an intuitive idea for this, because I dont see a necessity to do it experimentally

This is the line I am talking about w = w * self.b + -1e9 * (1 - self.b) # TF implem method: mask_attn_weights

https://github.com/huggingface/pytorch-openai-transformer-lm/blob/master/model_py.py#L77-L84

The reference implementation and paper uses tensorflow, in which channel dimension is represented last. But in pytorch, the channel dimension is represented as dim=1 after the batch. So does the equation need to be reversed? Is this done somewhere else in the code?

so in pytorch it should look like: matmul(v, matmul(k.t(), q))

example (tensorflow):

q size [attn_depth=3, feature_depth=2]
k size [attn_depth=3, feature_depth=2]
v size [attn_depth=3, feature_depth=2]

weights = matmul(q, k.t()) -> [3, 3]
result = matmul(weights, v) -> [3, 2]

example (pytorch) with error:

q size [2, 3]
k size [2, 3]
v size [2, 3]

weights = matmul(q, k.t()) -> [2, 2]
result = matmul(weights, v) -> [2, 3]

Notice the dimensionality of the weights has been over-reduced [2,2] instead of [3,3]

example (pytorch) with correction:

q size [2, 3]
k size [2, 3]
v size [2, 3]

weights = matmul(k.t(), q) -> [3, 3]
result = matmul(v, weights) -> [2, 3]

Apologies if this is handled correctly somewhere, limited on time atm for a thorough read.

Reproducing the specific behavior of the classifier dropout of the original OpenAI implementation of the article. The details of the this patch can be found in issue #11.

Hi!

First, I would like to thank you for your translation of the implementation of this paper.

I have read the code and managed to run it but I am not able to find how to load pre-trained weights into a LMHead to get a general English language model. Did the openAI guys not release the weights of this final layer?

I thought the performance would be better starting the finetuning process from a pre-trained LMHead.

The target of an optimizer should contain clf_head (new task-specific output matrix) in addition to model (Transformer encoder). The code might fail to do that, right?

So there's the TransformerModel's forward method, and I just can't get a hold of the position embedding part (and might be wrong about others). So, as far as I can tell, step-by-step it goes like this:

1. Reshape our input to have 3 dimensions -> [ ? x sequences (?) x tokens (512) ]
2. Get the individual token embeddings -> [ ? x sequences (?) x tokens (512) x emb_dim (768) ]
3. Sum up those embeddings along axis 2 (summing token embeddings element-wise for each sequence?) -> [ ? x sequences x emb_dim (768) ]
4. Shouldn't we have [ sequences x tokens (512) x emb_dim (768) ] here?

def forward(self, x):
        x = x.view(-1, x.size(-2), x.size(-1))
        e = self.embed(x)
        # Add the position information to the input embeddings
        h = e.sum(dim=2)
        for block in self.h:
            h = block(h)
        return h

My questions are:

• What are x , e, and h tensors' axes?
• How can a sum of an internal part add positional information to our token embeddings?
• How is that operation equivalent to the paper's, where the position embedding is an external, learned matrix which is added to the token embeddings?

Thank you in advance!

Hello. Sorry, but i can't understand how this function work. In my tests in most cases the result is equal to original token parameter value. https://github.com/openai/finetune-transformer-lm/blob/master/text_utils.py#L49

From the model definition, vocab is used to define the size of embedding.

   vocab = n_vocab + n_special + n_ctx

I am guessing that the n_ctx here is used for the position embedding, but still not clear.

In my case, I sometimes run into the following shape error if n_ctx is very large.

Traceback (most recent call last):
  File "/home/vimos/git/QA/pytorch-openai-transformer-lm/train.py", line 413, in <module>
    load_openai_pretrained_model(dh_model.transformer, n_ctx=n_ctx, n_special=n_special)
  File "/home/vimos/git/QA/pytorch-openai-transformer-lm/model_pytorch.py", line 402, in load_openai_pretrained_model
    assert model.embed.weight.shape == init_params[0].shape
AssertionError: (torch.Size([41140, 768]), (40993, 768))

Can anybody explain the code? Should I restrict n_ctx to a value? Thanks!

Similarity Head and Loss function were tested on the STS-B dataset, achieving nearly the same performance as reported (82.45% PC relative to the 82% in the paper). I can provide the necessary changed code for loading the dataset and reproducing my results if wanted.

I am trying to use this repository to train a language model with an additional input. My data looks like this:

┌─────────┬─────┬────┬───┐
│side info│start│The │cat│
└─────────┴─────┴────┴───┘

The labels look like this

┌────┬───┬─────┐
│The │cat│meows│
└────┴───┴─────┘

Since my objective is quite different from the original training script I implemented the training from scratch but I noticed that it takes much more time than a simple LSTM model to become somewhat decent and the results are not fully concise language even after 15 epochs on 2 million sentences. I am getting outputs that look like this

Gold label: In most cases , accurate results can only be achieved after a laborious and expensive trial and error process .

Output: only most accurate cases can be achieved after a laborious error and process results In trial and expensive suit.

Currently I am using a small model with 4 layers and 2 heads each.

I randomly initialized the position encodings and multiplied them by 0.1 to match the variance of my word embeddings.

Any ideas what I could have missed?

Here is some of my code

batch_size = 32
n_epochs = 100
max_len = 120

embeddings, emb_weights = load_embeddings(data_path+'de.en.fr.ka.tok.60000.shuf.vec',max_len)
train_dataset = SortedSentenceDataset(data_path+'train.txt', 200000, max_len, embeddings, 'avg',device)
train_sampler = train_dataset.get_sampler(batch_size)
train_loader = DataLoader(train_dataset, batch_size=1, sampler=train_sampler)
dev_dataset = SortedSentenceDataset(data_path+'valid.txt', 1000, max_len, embeddings, 'avg',device)
dev_sampler = dev_dataset.get_sampler(batch_size)
dev_loader = DataLoader(dev_dataset, batch_size=1, sampler=dev_sampler)

args = DEFAULT_CONFIG
args.n_embd = emb_weights.size(1)
# Constraint: embedding size % number of heads = 0
args.n_head = 2
args.n_layer = 4
model = load_model(args, emb_weights)

model.to(device)

criterion = torch.nn.CrossEntropyLoss()

optimizer = OpenAIAdam(model.parameters(),
                           lr=6.25e-3,
                           schedule='warmup_linear',
                           warmup=0.02,
                           t_total=n_epochs*len(train_dataset)*20,
                           b1=0.9,
                           b2=0.999,
                           e=1e-8,
                           l2=0.01,
                           vector_l2='store_true',
                           max_grad_norm=1)

best = 1000
for epoch in range(n_epochs):
    do_epoch(train_loader)
    val_loss = eval(dev_loader)
    print('Validation loss: {}'.format(val_loss))
    if val_loss < best:
        best = val_loss
        print('Saving model')
        torch.save(model.state_dict(),"context-at-each-layer-checkpoint-{}k{}e4b.pt".format(len(train_dataset)//1000,n_epochs))
    print(' '.join(generate(train_dataset,max_len,embeddings)))

I'd like to use GPT to encode my dataset and use the representations further for the task of question generation. I have problems with understanding the code and the name of the arguments in the train.py file (in main). Could anyone direct me to some examples (I already search online) or possibly post some here?

Cheers

Hi all,

I am trying to train a new dataset with a similar structure to rocstories. It has a story part, 2 options and one correct option. I just added a new function in datasets.py but this is not enough. I am not able to train. Has anyone done that and can provide me with some suggestions?

Thanks in advance.

From the ConvAI slides, it sounds like the Hugging Face submission was based off of this model -- is the code for your ConvAI system available somewhere to take a look at? Thanks!

I know that n_vocab is the total number of tokens in encoder dictionary. But when I saw vocab = n_vocab + n_special + n_ctx, I was confused, maybe n_special is the for start,delimiter and classify. But what is n_ctx? Why add these 3 things? (why there is little comment about variables and functions....Is there somewhere else to see the explanation of the codes?) I am new to learn about the transformer.