Nice paper! Thanks for sharing the code. I was trying to reproduce your results. It would be great if you could share the best hyperparameter for each GLUE task? For example with the command: $ python NLI_KD_training.py

For RTE I was able to get following results:

acc = 0.6216666666666667
eval_loss = 1.3263624415118644

BUT

With only change at line 34 to argv = get_predefine_argv('glue', 'MRPC', 'finetune_student') I got:

acc = 0.28289855072463765
acc_and_f1 = 0.14144927536231883
eval_loss = 3.820818234373022
f1 = 0.0

Hey, I am trying to reproduce your results, and am interested in training several students with different number of hidden layers. I want to submit the student predictions on GLUE website. I have been able too train student models with PKD-skip procedure.

My question is, how do I make predictions from the student model? I guess I should change the run_glue_benchmark somehow. Any help in this regard will be appreciated.

I Found a bug, when we set fp16=False, and train RTE with kd.cls, got this problem. and the traceback is：

Traceback (most recent call last):
  File "NLI_KD_training.py", line 288, in <module>
    loss.backward()
  File "/home/vernon/anaconda3/lib/python3.6/site-packages/torch/tensor.py", line 102, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/vernon/anaconda3/lib/python3.6/site-packages/torch/autograd/__init__.py", line 90, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: Function AddBackward0 returned an invalid gradient at index 1 - expected type torch.cuda.HalfTensor but got torch.cuda.FloatTensor

Thank you for your code. However, when i run code setting only finetune teacher(BERT-base)

# run simple fune-tuning *teacher* by uncommenting below cmd
    argv = get_predefine_argv('glue', 'RTE', 'finetune_teacher')

In argument_parser.py,

    elif mode == 'glue':
        argv = [
                '--task_name', task_name,
                '--bert_model', 'bert-base-uncased',
                '--max_seq_length', '128',
                '--train_batch_size', '32',
                '--learning_rate', '2e-5',
                '--num_train_epochs', '4',
                '--eval_batch_size', '32',
                '--log_every_step', '1',
                '--output_dir', os.path.join(HOME_DATA_FOLDER, f'outputs/KD/{task_name}/teacher_12layer'),
                '--do_train', 'True',
                '--do_eval', 'True',
                '--fp16', 'True',
            ]
        if train_type == 'finetune_teacher':
            argv += [
                '--student_hidden_layers', '12',
                '--kd_model', 'kd',
                '--do_eval', 'True',
                '--alpha', '0.0',    # alpha = 0 is equivalent to fine-tuning for KD
            ]

Result

12/16/2019 06:48:04 - INFO - __main__ -   ***** Eval results *****
12/16/2019 06:48:04 - INFO - __main__ -     acc = 0.5983333333333334
12/16/2019 06:48:04 - INFO - __main__ -     eval_loss = 1.6177796708776595

what is the reason..?

File 'NLI_KD_training.py',line 204, max_grad_norms=1.0) TypeError: init() got an unexpected keyword argument 'max_grad_norm=1.0' According to https://nvidia.github.io/apex/optimizers.html, the apex has been updated to a new release that removed this parameter. If just ignore this parameters, will the performance be influenced?

Hi, I am trying to extend your research and compute accuracy for all GLUE tasks, in that I am kind of stuck with STS-B. Since it is a regression task, where all do you think I should make the changes to get the numbers?

I am Impressed by your crucial work. But I encounter some issues when I reproduce this project, may I ask a question that: What's the version of python and pytorch of this project?

Thanks for sharing your work! I'm training a new dataset (classification task just like glue dataset) with following steps and wanted to make sure whether I'm doing it right.

1. get finetuned BERT (pytorch_model.bin) on dataset and put it into pretrained directory

2. run NLI_KD_training.py with below to get encoder.pkl, classifier.pkl

# run simple fune-tuning *teacher* by uncommenting below cmd
argv = get_predefine_argv('glue', 'RTE', 'finetune_teacher')

3. run run_glue_benchmark.py to get teacher's prediction for KD

4. run NLI_KD_training.py with below to distil knowledge from teacher to student model

    # run Patient Teacher by uncommenting below cmd
    argv = get_predefine_argv('glue', 'RTE', 'kd.cls')

I'm confused since doing step 2 redundantly finetunes teacher (bert base model) which was already done in step 1. Is it correct to place finetuned version into pretrained directory? or should I just use plain pytorch_model.bin?

Thanks a lot for your impressive work and I want to reproduce the results in the paper. Now I have a question: where can I get the pretrained model (bert-base-uncased)? Thank you!

Hi,

Thank you for your interesting work! I just wondering why don`t you used the pooler for only KD.Full and if you use the pooler, did you initialize the pooler with BERT_teacher weight and bias?

Thank you, Sincerely,

Hi,

Thank you for your interesting work! I have just started to learn BERT and distillation recently. I have some general questions regarding this topic.

1. I want to compare the performance of BERT with different model size (transformer block number). Is it necessary to do distillation? If I just train a BERT with 6 Layers without distillation, does the performance look bad?

2. Do you have to do pretraining every time you change the layer number of BERT? Is it possible to just remove some layers in an existing pre-trained model and finetune on tasks?

3. Why BERT has 12 blocks? Not 11 or 13 etc. ? I couldn't find any explanation.

Thanks, ZLK

First, thank you for releasing your code.

I am trying to reproduce results of your paper. I am running NLI_KD_training.py for MRPC with DEBUG=True.

The setting I am running is argv = get_predefine_argv('glue', 'MRPC', 'finetune_teacher').

After completing the training for 4 epochs, I get following results :

05/10/2020 19:09:30 - INFO - __main__ -   ***** Eval results *****
05/10/2020 19:09:30 - INFO - __main__ -     acc = 0.27942028985507245
05/10/2020 19:09:30 - INFO - __main__ -     acc_and_f1 = 0.13971014492753622
05/10/2020 19:09:30 - INFO - __main__ -     eval_loss = 3.8775325307139643
05/10/2020 19:09:30 - INFO - __main__ -     f1 = 0.0

Also the eval_log has the following :

epoch,acc,loss
1,0.8259803921568627,0.35975449818831223
2,0.8700980392156863,0.3205762528456174
3,0.8774509803921569,0.3944101127294394
4,0.8578431372549019,0.4749428268808585

-- which means training is probably correct but there is something wrong with test evaluation.

I have referred to the hyperparameter files that are provided in results_summary but I am not sure what might be wrong.