The implementation for "Comprehensive Knowledge Distillation with Causal Intervention".

Xiang Deng

Last update: Nov 3, 2022

Related tags

Deep Learning CID

Overview

Comprehensive Knowledge Distillation with Causal Intervention

This repository is a PyTorch implementation of "Comprehensive Knowledge Distillation with Causal Intervention". The code is modified from CRD, and the pretrained teachers (except WRN-40-4) are also downloaded from CRD.

Requirements

The code was tested on

Python 3.6
torch 1.2.0
torchvision 0.4.0

Evaluation

To evaluate our pre-trained light-weight student networks, first download the folder "pretrained_student_model" from CID models into the "save" folder, then simply run the command below to evaluate these light-weight students:

run evaluate_scripts.sh

Training

To train students from scratch by distilling knowledge from teacher networks with CID, first download the pretrained teacher folder "models" from CID models into the "save" folder, and then simply run the command below to compress large models to smaller ones:

run train_scripts.sh

Citation

If you find this code helpful, you may consider citing this paper:

@inproceedings{deng2021comprehensive,
  title={Comprehensive Knowledge Distillation with Causal Intervention},
  author={Deng, Xiang and Zhang, Zhongfei},
  booktitle = {Proceedings of the 30th Annual Conference on Neural Information Processing Systems},
  year={2021}
}

Comments

a question about the publication time of the paper

I think CID is a nice work in knowledge distillation. This paper has been accepted by NeurIPS2021. Do you have any plans to upload the paper to arxiv?

opened by lijincm 0

Implementation of Geometric Vector Perceptron, a simple circuit for 3d rotation equivariance for learning over large biomolecules, in Pytorch. Idea proposed and accepted at ICLR 2021

Geometric Vector Perceptron Implementation of Geometric Vector Perceptron, a simple circuit with 3d rotation equivariance for learning over large biom

59 Nov 24, 2022

Official implementation of AAAI-21 paper "Label Confusion Learning to Enhance Text Classification Models"

Description: This is the official implementation of our AAAI-21 accepted paper Label Confusion Learning to Enhance Text Classification Models. The str

101 Nov 25, 2022

Official PyTorch implementation for paper Context Matters: Graph-based Self-supervised Representation Learning for Medical Images

Context Matters: Graph-based Self-supervised Representation Learning for Medical Images Official PyTorch implementation for paper Context Matters: Gra

49 Nov 23, 2022

PyTorch implementation of "Conformer: Convolution-augmented Transformer for Speech Recognition" (INTERSPEECH 2020)

PyTorch implementation of Conformer: Convolution-augmented Transformer for Speech Recognition. Transformer models are good at capturing content-based

565 Jan 4, 2023

An essential implementation of BYOL in PyTorch + PyTorch Lightning

Essential BYOL A simple and complete implementation of Bootstrap your own latent: A new approach to self-supervised Learning in PyTorch + PyTorch Ligh

48 Sep 27, 2022

The official implementation of NeMo: Neural Mesh Models of Contrastive Features for Robust 3D Pose Estimation [ICLR-2021]. https://arxiv.org/pdf/2101.12378.pdf

NeMo: Neural Mesh Models of Contrastive Features for Robust 3D Pose Estimation [ICLR-2021] Release Notes The offical PyTorch implementation of NeMo, p

76 Nov 23, 2022

A PyTorch re-implementation of the paper 'Exploring Simple Siamese Representation Learning'. Reproduced the 67.8% Top1 Acc on ImageNet.

Exploring simple siamese representation learning This is a PyTorch re-implementation of the SimSiam paper on ImageNet dataset. The results match that

72 Nov 9, 2022

PyTorch implementation of "A Full-Band and Sub-Band Fusion Model for Real-Time Single-Channel Speech Enhancement."

FullSubNet This Git repository for the official PyTorch implementation of "A Full-Band and Sub-Band Fusion Model for Real-Time Single-Channel Speech E

357 Jan 4, 2023

Pytorch implementation of PCT: Point Cloud Transformer

PCT: Point Cloud Transformer This is a Pytorch implementation of PCT: Point Cloud Transformer.

265 Dec 22, 2022

The implementation for "Comprehensive Knowledge Distillation with Causal Intervention".

Related tags

Overview

Comprehensive Knowledge Distillation with Causal Intervention

Requirements

Evaluation

Training

Citation

You might also like...

Implementation of Geometric Vector Perceptron, a simple circuit for 3d rotation equivariance for learning over large biomolecules, in Pytorch. Idea proposed and accepted at ICLR 2021

Official implementation of AAAI-21 paper "Label Confusion Learning to Enhance Text Classification Models"

Official PyTorch implementation for paper Context Matters: Graph-based Self-supervised Representation Learning for Medical Images

PyTorch implementation of "Conformer: Convolution-augmented Transformer for Speech Recognition" (INTERSPEECH 2020)

An essential implementation of BYOL in PyTorch + PyTorch Lightning

The official implementation of NeMo: Neural Mesh Models of Contrastive Features for Robust 3D Pose Estimation [ICLR-2021]. https://arxiv.org/pdf/2101.12378.pdf

A PyTorch re-implementation of the paper 'Exploring Simple Siamese Representation Learning'. Reproduced the 67.8% Top1 Acc on ImageNet.

PyTorch implementation of "A Full-Band and Sub-Band Fusion Model for Real-Time Single-Channel Speech Enhancement."

Pytorch implementation of PCT: Point Cloud Transformer

Comments

a question about the publication time of the paper

Owner

Xiang Deng

Numenta Platform for Intelligent Computing is an implementation of Hierarchical Temporal Memory (HTM), a theory of intelligence based strictly on the neuroscience of the neocortex.

PyTorch implementation of neural style transfer algorithm

PyTorch implementation of DeepDream algorithm

The project is an official implementation of our CVPR2019 paper "Deep High-Resolution Representation Learning for Human Pose Estimation"

Image-to-Image Translation with Conditional Adversarial Networks (Pix2pix) implementation in keras

Python implementation of cover trees, near-drop-in replacement for scipy.spatial.kdtree

Home repository for the Regularized Greedy Forest (RGF) library. It includes original implementation from the paper and multithreaded one written in C++, along with various language-specific wrappers.

Implementation of Restricted Boltzmann Machine (RBM) and its variants in Tensorflow

A fast Evolution Strategy implementation in Python

🌳 A Python-inspired implementation of the Optimum-Path Forest classifier.