First of all, I would thank you for your work. I have read you paper about "BERT-of-Theseus", and found it very amazing and good written. After Fine tuning BERT on the MRPC GLUE task, I tried to compress it using 'BERT-of-Theseus' technique, But unfortunately getting the following error: "ImportError: cannot import name 'BERT_PRETRAINED_MODEL_ARCHIVE_MAP'". I wish if you have any idea to fix this! thank you a lot.

Hi, I read this paper and the idea is interesting.

However, after I run your code, I cannot achieve the performance in the paper. For instance, the MRPC dataset is used to finetuning the teacher model, then the model is loaded and used to distill the dark knowledge to the student model. But the final performance of student on the MRPC dataset is 85.2, which is much lower than the result in the paper, i.e., 89.0.

I do not know why that happen, would you please help me out?

Best.

Hi, thanks for your great work. According to the paper description, the predecessor module weights are frozen after fine-tuned on the task data( including embedding & output classifier). The code, however, if my understanding is correct, the fine-tuned predecessor weights are not frozen, instead, the loss can BP to the corresponding parameters. So, which pattern is supposed to be correct? Thanks in advance.

Hi

I read your paper and found it very interesting, I was wondering if you had any ablation results which compared the performance of a 6 layer Bert-of-Theseus (compressed from 12 layer BERT) against the performance of a 6 layer BERT trained from scratch? If not do you have any intuition for whether the module replacement of a larger model would surpass the performance of that same smaller model trained from scratch?

Many thanks David

First, thank you for the amazing work. I have already use this method to train a small Bert(6 hidden layers) from the Bert-base(12 hidden layers), and in my own dataset the inference speed almost double but the accuracy goes down from 84.5% to 82.8%. So I'm wondering if I can use Roberta-large(24 hidden layers) as the predecessor model and set the number of hidden layers for scc_layer to 8?

Because in modeling_bert_of_theseus.py I found such code lines:

def init(self, config, scc_n_layer=6):

    super(BertEncoder, self).__init__()
    self.prd_n_layer = config.num_hidden_layers
    self.scc_n_layer = scc_n_layer
    assert self.prd_n_layer % self.scc_n_layer == 0

the config.json of the model which has been compressed as follows： { "_num_labels": 20, "architectures": [ "BertForSequenceClassification" ], "attention_probs_dropout_prob": 0.1, "bos_token_id": null, "directionality": "bidi", "do_sample": false, "early_stopping": false, "eos_token_ids": null, "finetuning_task": "mytask", "hidden_act": "gelu", "hidden_dropout_prob": 0.1, "hidden_size": 768, "id2label": { "0": "LABEL_0", "1": "LABEL_1", "2": "LABEL_2", "3": "LABEL_3", "4": "LABEL_4", "5": "LABEL_5", "6": "LABEL_6", "7": "LABEL_7", "8": "LABEL_8", "9": "LABEL_9", "10": "LABEL_10", "11": "LABEL_11", "12": "LABEL_12", "13": "LABEL_13", "14": "LABEL_14", "15": "LABEL_15", "16": "LABEL_16", "17": "LABEL_17", "18": "LABEL_18", "19": "LABEL_19" }, "initializer_range": 0.02, "intermediate_size": 3072, "is_decoder": false, "is_encoder_decoder": false, "label2id": { "LABEL_0": 0, "LABEL_1": 1, "LABEL_10": 10, "LABEL_11": 11, "LABEL_12": 12, "LABEL_13": 13, "LABEL_14": 14, "LABEL_15": 15, "LABEL_16": 16, "LABEL_17": 17, "LABEL_18": 18, "LABEL_19": 19, "LABEL_2": 2, "LABEL_3": 3, "LABEL_4": 4, "LABEL_5": 5, "LABEL_6": 6, "LABEL_7": 7, "LABEL_8": 8, "LABEL_9": 9 }, "layer_norm_eps": 1e-12, "length_penalty": 1.0, "max_length": 20, "max_position_embeddings": 512, "min_length": 0, "model_type": "bert", "no_repeat_ngram_size": 0, "num_attention_heads": 12, "num_beams": 1, "num_hidden_layers": 12, "num_return_sequences": 1, "output_attentions": false, "output_hidden_states": true, "output_past": true, "pad_token_id": null, "pooler_fc_size": 768, "pooler_num_attention_heads": 12, "pooler_num_fc_layers": 3, "pooler_size_per_head": 128, "pooler_type": "first_token_transform", "pruned_heads": {}, "repetition_penalty": 1.0, "temperature": 1.0, "top_k": 50, "top_p": 1.0, "torchscript": false, "type_vocab_size": 2, "use_bfloat16": false, "vocab_size": 21128 }

If I will use this model ,the config.json will be loading, and these params shoule be modified? As I know, the num_hidden_layers is not 12.

Hi, thanks for your share of Theseus. To my understanding, Theseus can support prune depth of the network， I wonder if it can also support prune width of the network?

Would you provide the checkpoint of the predecessor model on the RTE dataset? I cannot obtain the score as same as your BERT-base in the paper with the mentioned hyperparameter set.

I got ImportError: cannot import name 'BERT_PRETRAINED_MODEL_ARCHIVE_MAP' from 'transformers.modeling_bert' when trying to follow the demo in README. It seems BERT-of-Theseus is using an older version of transformers.

how to draw the cover picture? excuse me, for the serious curious about your logo of BERT-of-Theseus. With some advanced technology or just use Photoshop?

After training like that:

# For compression with a replacement scheduler
export GLUE_DIR=glue_script/glue_data
export TASK_NAME=MRPC

python ./run_glue.py \
  --model_name_or_path /home/bert-base \
  --task_name $TASK_NAME \
  --do_train \
  --do_eval \
  --do_lower_case \
  --data_dir "$GLUE_DIR/$TASK_NAME" \
  --max_seq_length 128 \
  --per_gpu_train_batch_size 32 \
  --per_gpu_eval_batch_size 32 \
  --learning_rate 2e-5 \
  --save_steps 50 \
  --num_train_epochs 15 \
  --output_dir result/ \
  --evaluate_during_training \
  --replacing_rate 0.3 \
  --scheduler_type linear \
  --scheduler_linear_k 0.0006

I convert the result model to Hugging Face with convert_to_hf_ckpt.py. And I try to export to onnx:

output = torch.onnx.export(model,
                               org_dummy_input,
                               MODEL_ONNX_PATH,
                               verbose=True,
                               operator_export_type=OPERATOR_EXPORT_TYPE,
                               opset_version=12,
                               input_names=['input_ids', 'attention_mask', 'token_type_ids'], 
                               output_names=['last_hidden_state', 'pooler_output'],  
                               do_constant_folding=True,
                               dynamic_axes={"input_ids": {0: "batch_size", 1: 'seq_length'},
                                             "token_type_ids": {0: "batch_size", 1: 'seq_length'},
                                             "attention_mask": {0: "batch_size", 1: 'seq_length'},
                                             "pooler_output": {0: "batch_size"},
                                             "last_hidden_state": {0: "batch_size", 1: 'seq_length'}}
                               )

The result is float32 ONNX. I try to convert to float 16 onnx:

python3 -m onnxruntime_tools.transformers.optimizer --input  onnx/bert_fp32.onnx --output onnx/bert_fp16.onnx --float16

However when do inference for float16 onnx, it comes errors:

  session = InferenceSession(model_file, providers=['CUDAExecutionProvider'])
  File "/usr/local/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 283, in __init__
    self._create_inference_session(providers, provider_options, disabled_optimizers)
  File "/usr/local/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 310, in _create_inference_session
    sess = C.InferenceSession(session_options, self._model_path, True, self._read_config_from_model)
onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Load model from /home/model_compression/BERT-of-Theseus/onnx/bert_fp16.onnx failed:This is an invalid model. Error: Duplicate definition of name (last_hidden_state).

Have you try convert the pytorch model to float 16 onnx ？ Why the half-precision operation makes the Duplicate definition of name (last_hidden_state). By the way, float32 onnx and run successfully without errors.

Hi, I cannot reproduce the CoLA score as same as the one on paper. I followed the HuggingFace's repo to train a predecessor model with Matthew correlation score of 55.76. However, the hightest score of the successor model I got is 35.82. Could you provide the hyperparameter set for training on the CoLA dataset?

What does “max_length” mean in config.json of successor? I set max_seq_length=128 when I Run compression, but the "max_length" in config.json of successor is 20.