Source code for "FastBERT: a Self-distilling BERT with Adaptive Inference Time".

Weijie Liu

Last update: Jan 2, 2023

Related tags

Overview

FastBERT

Source code for "FastBERT: a Self-distilling BERT with Adaptive Inference Time".

Good News

2021/10/29 - Code: Code of FastPLM is released on both Pypi and Github.

2021/09/08 - Paper: Journal version of FastBERT (FastPLM) is accepted by IEEE TNNLS. "An Empirical Study on Adaptive Inference for Pretrained Language Model".

2020/07/05 - Update: Pypi version of FastBERT has been launched. Please see fastbert-pypi.

Install fastbert with pip

$ pip install fastbert

Requirements

python >= 3.4.0, Install all the requirements with pip.

$ pip install -r requirements.txt

Quick start on the Chinese Book review dataset

Download the pre-trained Chinese BERT parameters from here, and save it to the models directory with the name of "Chinese_base_model.bin".

Run the following command to validate our FastBERT with Speed=0.5 on the Book review datasets.

$ CUDA_VISIBLE_DEVICES="0" python3 -u run_fastbert.py \
        --pretrained_model_path ./models/Chinese_base_model.bin \
        --vocab_path ./models/google_zh_vocab.txt \
        --train_path ./datasets/douban_book_review/train.tsv \
        --dev_path ./datasets/douban_book_review/dev.tsv \
        --test_path ./datasets/douban_book_review/test.tsv \
        --epochs_num 3 --batch_size 32 --distill_epochs_num 5 \
        --encoder bert --fast_mode --speed 0.5 \
        --output_model_path  ./models/douban_fastbert.bin

Meaning of each option.

usage: --pretrained_model_path Path to initialize model parameters.
       --vocab_path Path to the vocabulary.
       --train_path Path to the training dataset.
       --dev_path Path to the validating dataset.
       --test_path Path to the testing dataset.
       --epochs_num The epoch numbers of fine-tuning.
       --batch_size Batch size.
       --distill_epochs_num The epoch numbers of the self-distillation.
       --encoder The type of encoder.
       --fast_mode Whether to enable the fast mode of FastBERT.
       --speed The Speed value in the paper.
       --output_model_path Path to the output model parameters.

Test results on the Book review dataset.

Test results at fine-tuning epoch 3 (Baseline): Acc.=0.8688;  FLOPs=21785247744;
Test results at self-distillation epoch 1     : Acc.=0.8698;  FLOPs=6300902177;
Test results at self-distillation epoch 2     : Acc.=0.8691;  FLOPs=5844839008;
Test results at self-distillation epoch 3     : Acc.=0.8664;  FLOPs=5170940850;
Test results at self-distillation epoch 4     : Acc.=0.8664;  FLOPs=5170940327;
Test results at self-distillation epoch 5     : Acc.=0.8664;  FLOPs=5170940327;

Quick start on the English Ag.news dataset

Download the pre-trained English BERT parameters from here, and save it to the models directory with the name of "English_uncased_base_model.bin".

Download the ag_news.zip from here, and then unzip it to the datasets directory.

Run the following command to validate our FastBERT with Speed=0.5 on the Ag.news datasets.

$ CUDA_VISIBLE_DEVICES="0" python3 -u run_fastbert.py \
        --pretrained_model_path ./models/English_uncased_base_model.bin \
        --vocab_path ./models/google_uncased_en_vocab.txt \
        --train_path ./datasets/ag_news/train.tsv \
        --dev_path ./datasets/ag_news/test.tsv \
        --test_path ./datasets/ag_news/test.tsv \
        --epochs_num 3 --batch_size 32 --distill_epochs_num 5 \
        --encoder bert --fast_mode --speed 0.5 \
        --output_model_path  ./models/ag_news_fastbert.bin

Test results on the Ag.news dataset.

Test results at fine-tuning epoch 3 (Baseline): Acc.=0.9447;  FLOPs=21785247744;
Test results at self-distillation epoch 1     : Acc.=0.9308;  FLOPs=2172009009;
Test results at self-distillation epoch 2     : Acc.=0.9311;  FLOPs=2163471246;
Test results at self-distillation epoch 3     : Acc.=0.9314;  FLOPs=2108341649;
Test results at self-distillation epoch 4     : Acc.=0.9314;  FLOPs=2108341649;
Test results at self-distillation epoch 5     : Acc.=0.9314;  FLOPs=2108341649;

Datasets

More datasets can be downloaded from here.

Other implementations

There are some other excellent implementations of FastBERT.

BitVoyage/FastBERT (Pytorch): https://github.com/BitVoyage/FastBERT

Acknowledgement

This work is funded by 2019 Tencent Rhino-Bird Elite Training Program. Work done while this author was an intern at Tencent.

If you use this code, please cite this paper:

@inproceedings{weijie2020fastbert,
  title={{FastBERT}: a Self-distilling BERT with Adaptive Inference Time},
  author={Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Haotang Deng, Qi Ju},
  booktitle={Proceedings of ACL 2020},
  year={2020}
}

Comments

实验效果疑惑

Hello，感谢你杰出的工作。我在glue的蚂蚁金服语义相似度语料上进行试验，finetune_epochs取20，distill_epochs取10，learning_rate取2e-5，dev_speed取0.5，最终蒸馏后在dev上的dev_acc始终在0.725徘徊。若想让蒸馏后的dev_acc达到0.9，是不是要增大训练epoch，还是有别的影响因素呢？感谢解答！

opened by Vincent131499 3
Calculation about the FLOP of full-connected layer.

First, thanks for you work, it's very useful for inference BERT-like modes. Hope your paper get published soon. And something i'm confused about is the FLOP of dense layer, which is in the section 4.1. As far as i know, the FLOPs of fully connect layer with bias = 2 * I * O

I=input neuron numbers, O=output neuron numbers.

For the Fully-connect layer 128 ->128, FLOPs = 2 * 128 * 128 = 32,768 And in the Table 1, the answer is 4.2M, which is much higher than i got. Can you release your method to calculate the answer ?

opened by AllenChennn 2
ModuleNotFoundError: No module named 'uer.encoders.synt_encoder'

win10+pycharm，运行FastBERT\pypi\examples\single_sentence_classification\test.py，出错： ModuleNotFoundError: No module named 'uer.encoders.synt_encoder'。在windows命令行执行没有问题。

opened by baiziyuandyufei 1
修正无验证集输入时的代码报错问题

作者您好，我在使用您封装好的FastBERT进行单句子文本分类的实验时，在未输入验证集的情况下，在一个epoch训练结束时，报错“TypeError: cannot unpack non-iterable float object”。位置位于fastbert.py文件436行，我发现为变量赋值问题导致，具体请查阅PR。

opened by aestheticisma 1
DistillBERT (3layer)

Thanks for releasing the great repo.

Could you share the DistillBERT (3layer) model and the DistillBERT (1layer)? It is very helpful for me.

Thanks!

Best, Deming Ye

opened by YeDeming 0
I'm curious about the reason for making self-attention for each classifier layer.

First of all, thanks for your kind offer.

What do you think is the reason for self-attention for each classifier layer?

The paper also says that it does self-attention in 128 dimensions.

What do you think is the difference from deriving a result without self-attention with a only hidden size of 768 dimensions?

opened by wonbeeny 0
I'm curious why you set segment_embedding's first dimension to 3.

First of all, thanks for your kind offer.

Why did you set segment_embedding's first dimension to 3 ??

This is on path [FastBERT/uer/layers/embeddings.py] (line 18)

Is this part flexible depending on the model architecture? The paper does not have this content, so I asked a question.

opened by wonbeeny 1
关于inference的效率问题

https://github.com/autoliuweijie/FastBERT/blob/5f9e98bc87b577487771f9dffceb386b5c0107d3/run_fastbert.py#L132 我觉得推理性能慢不是因为nozero。看代码实现，实际上相当于每过一层transformer encoder，就在当前这个batch剔除掉过于简单的样本 ,也就是batchsize变得更小，然而只要有一个样本到达最后一层，耗时都会比原来bert要多。

有没有办法能够更灵活的调度需要计算的样本，比如建立一个pool，进入到第10层之后的都放到一个池子里，一起调度，让每一层计算的batchsize固定，这样充分利用显卡资源的话推理起来应该会快很多。

opened by Youarerare 1

Source code for "FastBERT: a Self-distilling BERT with Adaptive Inference Time".

Related tags

Overview

FastBERT

Good News

Requirements

Quick start on the Chinese Book review dataset

Quick start on the English Ag.news dataset

Datasets

Other implementations

Acknowledgement

Comments

Owner

Weijie Liu

PaddleRobotics is an open-source algorithm library for robots based on Paddle, including open-source parts such as human-robot interaction, complex motion control, environment perception, SLAM positioning, and navigation.

Source-to-Source Debuggable Derivatives in Pure Python

Code for the prototype tool in our paper "CoProtector: Protect Open-Source Code against Unauthorized Training Usage with Data Poisoning".

Graph Transformer Architecture. Source code for

Open source repository for the code accompanying the paper 'Non-Rigid Neural Radiance Fields Reconstruction and Novel View Synthesis of a Deforming Scene from Monocular Video'.

Source code for Acorn, the precision farming rover by Twisted Fields

[CVPR2021] The source code for our paper 《Removing the Background by Adding the Background: Towards Background Robust Self-supervised Video Representation Learning》.

Open source code for Paper "A Co-Interactive Transformer for Joint Slot Filling and Intent Detection"

Source code of "Hold me tight! Influence of discriminative features on deep network boundaries"

The open source code of SA-UNet: Spatial Attention U-Net for Retinal Vessel Segmentation.

Source code, datasets and trained models for the paper Learning Advanced Mathematical Computations from Examples (ICLR 2021), by François Charton, Amaury Hayat (ENPC-Rutgers) and Guillaume Lample

Source Code for DialogBERT: Discourse-Aware Response Generation via Learning to Recover and Rank Utterances (https://arxiv.org/pdf/2012.01775.pdf)

Implementation of the paper "Language-agnostic representation learning of source code from structure and context".

source code and pre-trained/fine-tuned checkpoint for NAACL 2021 paper LightningDOT

Code to reproduce the experiments in the paper "Transformer Based Multi-Source Domain Adaptation" (EMNLP 2020)

Source code for the GPT-2 story generation models in the EMNLP 2020 paper "STORIUM: A Dataset and Evaluation Platform for Human-in-the-Loop Story Generation"

The source code for the Cutoff data augmentation approach proposed in this paper: "A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation".

PGPortfolio: Policy Gradient Portfolio, the source code of "A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem"(https://arxiv.org/pdf/1706.10059.pdf).

Source code for "OmniPhotos: Casual 360° VR Photography"