Revisiting Self-Training for Few-Shot Learning of Language Model.

Last update: Nov 19, 2022

Related tags

Deep Learning SFLM

Overview

SFLM

This is the implementation of the paper Revisiting Self-Training for Few-Shot Learning of Language Model. SFLM is short for self-training for few-shot learning of language model.

Requirements

To run our code, please install all the dependency packages by using the following command:

pip install -r requirements.txt

Preprocess

The original data can be found from LM-BFF. To generate data for the few-shot experiments, please run the below command:

python tools/generate_data.py

The original data shall be in ./data/original, and the sampled data will be in ./data/few-shot/$K-$MU-$SEED. Please refer to ./tools/generate_data.py for more options.

Train

Our code can be run as the below example:

python3 run.py \
  --task_name SST-2 \
  --data_dir data/few-shot/SST-2/16-4-100 \
  --do_train \
  --do_eval \
  --do_predict \
  --evaluate_during_training \
  --model_name_or_path roberta-base \
  --few_shot_type prompt-demo \
  --num_k 16 \
  --max_seq_length 256 \
  --per_device_train_batch_size 2 \
  --per_device_eval_batch_size 16 \
  --gradient_accumulation_steps 4 \
  --learning_rate 1e-5 \
  --max_steps 1000 \
  --logging_steps 100 \
  --eval_steps 100 \
  --num_train_epochs 0 \
  --output_dir result/SST-2-16-4-100 \
  --save_logit_dir result/SST-2-16-4-100 \
  --seed 100 \
  --template "*cls**sent_0*_It_was*mask*.*sep+*" \
  --mapping "{'0':'terrible','1':'great'}" \
  --num_sample 16 \
  --threshold 0.95 \
  --lam1 0.5 \
  --lam2 0.1

Most arguments are the same as LM-BFF, and the same manual prompts are used in our experiments. We list additional arguments used in SFLM:

threshold: The threshold used to filter out low-confidence samples for self-training loss
lam1: The weight of self-training loss
lam2: The weight of self-supervised loss

Citation

Please cite our paper if you use SFLM in your work:

@inproceedings{chen2021revisit,        
    title={Revisiting Self-Training for Few-Shot Learning of Language Model},         
    author={Chen, Yiming and Zhang, Yan and Zhang, Chen and Lee, Grandee and Cheng, Ran and Li, Haizhou},         
    booktitle={EMNLP},        
    year={2021},
}

Acknowledgements

Code is implemented based on LM-BFF. We would like to thank the authors of LM-BFF for making their code public.

[EMNLP 2021] Distantly-Supervised Named Entity Recognition with Noise-Robust Learning and Language Model Augmented Self-Training

RoSTER The source code used for Distantly-Supervised Named Entity Recognition with Noise-Robust Learning and Language Model Augmented Self-Training, p

60 Dec 30, 2022

Comments

Will the re-implemented PET/iPET code be open-sourced?

Hi, Yiming, Thanks for your impressive work! I am able to reproduce your result with SFLM. But I noticed there was a re-implemented PET/iPET in Section 4.2 and Table 1 and the paper claimed that it had a better performance than the official code. Therefore, it could be helpful to me to reproduce the baseline too. Will the re-implemented PET/iPET code be open-sourced?

Regards, Ray

opened by RayOnFire 0

Revisiting Self-Training for Few-Shot Learning of Language Model.

Related tags

Overview

SFLM

Requirements

Preprocess

Train

Citation

Acknowledgements

You might also like...

[EMNLP 2021] Distantly-Supervised Named Entity Recognition with Noise-Robust Learning and Language Model Augmented Self-Training

Pytorch Implementation for CVPR2018 Paper: Learning to Compare: Relation Network for Few-Shot Learning

SAS: Self-Augmentation Strategy for Language Model Pre-training

Revisiting Contrastive Methods for Unsupervised Learning of Visual Representations. [2021]

Few-shot Learning of GPT-3

Library of various Few-Shot Learning frameworks for text classification

Few-Shot Graph Learning for Molecular Property Prediction

Few-shot Relation Extraction via Bayesian Meta-learning on Relation Graphs

Adaptive Prototype Learning and Allocation for Few-Shot Segmentation (CVPR 2021)

Comments

Will the re-implemented PET/iPET code be open-sourced?

Owner

The code is for the paper "A Self-Distillation Embedded Supervised Affinity Attention Model for Few-Shot Segmentation"

Few-NERD: Not Only a Few-shot NER Dataset

An original implementation of "Noisy Channel Language Model Prompting for Few-Shot Text Classification"

Code for 'Self-Guided and Cross-Guided Learning for Few-shot segmentation. (CVPR' 2021)'

Implementation of the paper "Self-Promoted Prototype Refinement for Few-Shot Class-Incremental Learning"

True Few-Shot Learning with Language Models

Pytorch implementation of the paper "Optimization as a Model for Few-Shot Learning"

The source code for Generating Training Data with Language Models: Towards Zero-Shot Language Understanding.

Byte-based multilingual transformer TTS for low-resource/few-shot language adaptation.

Revisiting Video Saliency: A Large-scale Benchmark and a New Model (CVPR18, PAMI19)