Awesome Pruning

A curated list of neural network pruning and related resources. Inspired by awesome-deep-vision, awesome-adversarial-machine-learning, awesome-deep-learning-papers and Awesome-NAS.

Please feel free to pull requests or open an issue to add papers.

Type of Pruning
2020 Venues
2019 Venues
2018 Venues
2017 Venues
2016 Venues
2015 Venues

Type of Pruning

Type	`F`	`W`	`Other`
Explanation	Filter pruning	Weight pruning	other types

2020

Title	Venue	Type	Code
HYDRA: Pruning Adversarially Robust Neural Networks	NeurIPS	`W`	PyTorch(Author)
Logarithmic Pruning is All You Need	NeurIPS	`W`	-
Directional Pruning of Deep Neural Networks	NeurIPS	`W`	-
Movement Pruning: Adaptive Sparsity by Fine-Tuning	NeurIPS	`W`	PyTorch(Author)
Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpot	NeurIPS	`W`	PyTorch(Author)
Neuron Merging: Compensating for Pruned Neurons	NeurIPS	`F`	PyTorch(Author)
Neuron-level Structured Pruning using Polarization Regularizer	NeurIPS	`F`	PyTorch(Author)
SCOP: Scientific Control for Reliable Neural Network Pruning	NeurIPS	`F`	-
Storage Efficient and Dynamic Flexible Runtime Channel Pruning via Deep Reinforcement Learning	NeurIPS	`F`	-
The Generalization-Stability Tradeoff In Neural Network Pruning	NeurIPS	`F`	PyTorch(Author)
Pruning Filter in Filter	NeurIPS	`Other`	PyTorch(Author)
Position-based Scaled Gradient for Model Quantization and Pruning	NeurIPS	`Other`	PyTorch(Author)
Bayesian Bits: Unifying Quantization and Pruning	NeurIPS	`Other`	-
Pruning neural networks without any data by iteratively conserving synaptic flow	NeurIPS	`Other`	PyTorch(Author)
EagleEye: Fast Sub-net Evaluation for Efficient Neural Network Pruning	ECCV (Oral)	`F`	PyTorch(Author)
DSA: More Efficient Budgeted Pruning via Differentiable Sparsity Allocation	ECCV	`F`	-
DHP: Differentiable Meta Pruning via HyperNetworks	ECCV	`F`	PyTorch(Author)
Meta-Learning with Network Pruning	ECCV	`W`	-
Accelerating CNN Training by Pruning Activation Gradients	ECCV	`W`	-
DA-NAS: Data Adapted Pruning for Efficient Neural Architecture Search	ECCV	`Other`	-
Differentiable Joint Pruning and Quantization for Hardware Efficiency	ECCV	`Other`	-
Channel Pruning via Automatic Structure Search	IJCAI	`F`	PyTorch(Author)
Adversarial Neural Pruning with Latent Vulnerability Suppression	ICML	`W`	-
Proving the Lottery Ticket Hypothesis: Pruning is All You Need	ICML	`W`	-
Soft Threshold Weight Reparameterization for Learnable Sparsity	ICML	`WF`	Pytorch(Author)
Network Pruning by Greedy Subnetwork Selection	ICML	`F`	-
Operation-Aware Soft Channel Pruning using Differentiable Masks	ICML	`F`	-
DropNet: Reducing Neural Network Complexity via Iterative Pruning	ICML	`F`	-
Towards Efficient Model Compression via Learned Global Ranking	CVPR (Oral)	`F`	Pytorch(Author)
HRank: Filter Pruning using High-Rank Feature Map	CVPR (Oral)	`F`	Pytorch(Author)
Neural Network Pruning with Residual-Connections and Limited-Data	CVPR (Oral)	`F`	-
Multi-Dimensional Pruning: A Unified Framework for Model Compression	CVPR (Oral)	`WF`	-
DMCP: Differentiable Markov Channel Pruning for Neural Networks	CVPR (Oral)	`F`	TensorFlow(Author)
Group Sparsity: The Hinge Between Filter Pruning and Decomposition for Network Compression	CVPR	`F`	PyTorch(Author)
Few Sample Knowledge Distillation for Efficient Network Compression	CVPR	`F`	-
Discrete Model Compression With Resource Constraint for Deep Neural Networks	CVPR	`F`	-
Structured Compression by Weight Encryption for Unstructured Pruning and Quantization	CVPR	`W`	-
Learning Filter Pruning Criteria for Deep Convolutional Neural Networks Acceleration	CVPR	`F`	-
APQ: Joint Search for Network Architecture, Pruning and Quantization Policy	CVPR	`F`	-
Comparing Rewinding and Fine-tuning in Neural Network Pruning	ICLR (Oral)	`WF`	TensorFlow(Author)
A Signal Propagation Perspective for Pruning Neural Networks at Initialization	ICLR (Spotlight)	`W`	-
ProxSGD: Training Structured Neural Networks under Regularization and Constraints	ICLR	`W`	TF+PT(Author)
One-Shot Pruning of Recurrent Neural Networks by Jacobian Spectrum Evaluation	ICLR	`W`	-
Lookahead: A Far-sighted Alternative of Magnitude-based Pruning	ICLR	`W`	PyTorch(Author)
Dynamic Model Pruning with Feedback	ICLR	`WF`	-
Provable Filter Pruning for Efficient Neural Networks	ICLR	`F`	-
Data-Independent Neural Pruning via Coresets	ICLR	`W`	-
AutoCompress: An Automatic DNN Structured Pruning Framework for Ultra-High Compression Rates	AAAI	`F`	-
DARB: A Density-Aware Regular-Block Pruning for Deep Neural Networks	AAAI	`Other`	-
Pruning from Scratch	AAAI	`Other`	-

2019

Title	Venue	Type	Code
Network Pruning via Transformable Architecture Search	NeurIPS	`F`	PyTorch(Author)
Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks	NeurIPS	`F`	PyTorch(Author)
Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask	NeurIPS	`W`	TensorFlow(Author)
One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers	NeurIPS	`W`	-
Global Sparse Momentum SGD for Pruning Very Deep Neural Networks	NeurIPS	`W`	PyTorch(Author)
AutoPrune: Automatic Network Pruning by Regularizing Auxiliary Parameters	NeurIPS	`W`	-
Model Compression with Adversarial Robustness: A Unified Optimization Framework	NeurIPS	`Other`	PyTorch(Author)
MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning	ICCV	`F`	PyTorch(Author)
Accelerate CNN via Recursive Bayesian Pruning	ICCV	`F`	-
Adversarial Robustness vs Model Compression, or Both?	ICCV	`W`	PyTorch(Author)
Learning Filter Basis for Convolutional Neural Network Compression	ICCV	`Other`	-
Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration	CVPR (Oral)	`F`	PyTorch(Author)
Towards Optimal Structured CNN Pruning via Generative Adversarial Learning	CVPR	`F`	PyTorch(Author)
Centripetal SGD for Pruning Very Deep Convolutional Networks with Complicated Structure	CVPR	`F`	PyTorch(Author)
On Implicit Filter Level Sparsity in Convolutional Neural Networks, Extension1, Extension2	CVPR	`F`	PyTorch(Author)
Structured Pruning of Neural Networks with Budget-Aware Regularization	CVPR	`F`	-
Importance Estimation for Neural Network Pruning	CVPR	`F`	PyTorch(Author)
OICSR: Out-In-Channel Sparsity Regularization for Compact Deep Neural Networks	CVPR	`F`	-
Partial Order Pruning: for Best Speed/Accuracy Trade-off in Neural Architecture Search	CVPR	`Other`	TensorFlow(Author)
Variational Convolutional Neural Network Pruning	CVPR	-	-
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks	ICLR (Best)	`W`	TensorFlow(Author)
Rethinking the Value of Network Pruning	ICLR	`F`	PyTorch(Author)
Dynamic Channel Pruning: Feature Boosting and Suppression	ICLR	`F`	TensorFlow(Author)
SNIP: Single-shot Network Pruning based on Connection Sensitivity	ICLR	`W`	TensorFLow(Author)
Dynamic Sparse Graph for Efficient Deep Learning	ICLR	`F`	CUDA(3rd)
Collaborative Channel Pruning for Deep Networks	ICML	`F`	-
Approximated Oracle Filter Pruning for Destructive CNN Width Optimization github	ICML	`F`	-
EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis4	ICML	`W`	PyTorch(Author)
COP: Customized Deep Model Compression via Regularized Correlation-Based Filter-Level Pruning	IJCAI	`F`	Tensorflow(Author)

2018

Title	Venue	Type	Code
Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers	ICLR	`F`	TensorFlow(Author), PyTorch(3rd)
To prune, or not to prune: exploring the efficacy of pruning for model compression	ICLR	`W`	-
Discrimination-aware Channel Pruning for Deep Neural Networks	NeurIPS	`F`	TensorFlow(Author)
Frequency-Domain Dynamic Pruning for Convolutional Neural Networks	NeurIPS	`W`	-
Learning Sparse Neural Networks via Sensitivity-Driven Regularization	NeurIPS	`WF`	-
Amc: Automl for model compression and acceleration on mobile devices	ECCV	`F`	TensorFlow(3rd)
Data-Driven Sparse Structure Selection for Deep Neural Networks	ECCV	`F`	MXNet(Author)
Coreset-Based Neural Network Compression	ECCV	`F`	PyTorch(Author)
Constraint-Aware Deep Neural Network Compression	ECCV	`W`	SkimCaffe(Author)
A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers	ECCV	`W`	Caffe(Author)
PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning	CVPR	`F`	PyTorch(Author)
NISP: Pruning Networks using Neuron Importance Score Propagation	CVPR	`F`	-
CLIP-Q: Deep Network Compression Learning by In-Parallel Pruning-Quantization	CVPR	`W`	-
“Learning-Compression” Algorithms for Neural Net Pruning	CVPR	`W`	-
Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks	IJCAI	`F`	PyTorch(Author)
Accelerating Convolutional Networks via Global & Dynamic Filter Pruning	IJCAI	`F`	-

2017

Title	Venue	Type	Code
Pruning Filters for Efficient ConvNets	ICLR	`F`	PyTorch(3rd)
Pruning Convolutional Neural Networks for Resource Efficient Inference	ICLR	`F`	TensorFlow(3rd)
Net-Trim: Convex Pruning of Deep Neural Networks with Performance Guarantee	NeurIPS	`W`	TensorFlow(Author)
Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon	NeurIPS	`W`	PyTorch(Author)
Runtime Neural Pruning	NeurIPS	`F`	-
Designing Energy-Efficient Convolutional Neural Networks using Energy-Aware Pruning	CVPR	`F`	-
ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression	ICCV	`F`	Caffe(Author), PyTorch(3rd)
Channel pruning for accelerating very deep neural networks	ICCV	`F`	Caffe(Author)
Learning Efficient Convolutional Networks Through Network Slimming	ICCV	`F`	PyTorch(Author)

2016

Title	Venue	Type	Code
Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding	ICLR (Best)	`W`	Caffe(Author)
Dynamic Network Surgery for Efficient DNNs	NeurIPS	`W`	Caffe(Author)

2015

Title	Venue	Type	Code
Learning both Weights and Connections for Efficient Neural Networks	NeurIPS	`W`	PyTorch(3rd)

EfficientDNNs

Embedded-Neural-Network

awesome-AutoML-and-Lightweight-Models

Model-Compression-Papers

knowledge-distillation-papers

Network-Speed-and-Compression

Greetings,

We are the authors of the said paper/code, and we thank you for your inclusion; it has certainly generated some traffic for us.

However, it might be worth noting that our pruning granularity is not F (filter level). Our algorithm prunes at a Grouped Kernel level, which is — to the best of our knowledge — the most fine-grained approach under the constraint of outputting a densely structured pruned network, much like channel or filter prunings.

Since pushing the pruning freedom further while remaining structured is probably our most important contribution, we'd appreciate a simple fix (and maybe a new type category if you're feeling generous — as we'd certainly welcome more adaptations on the grouped kernel pruning framework). Thanks!

enhancement

pruning weights by L1 regularization

Hi,

I would like to bring to your attention our paper on weight pruning accepted by ICLR 2020.

paper url: https://openreview.net/forum?id=HygpthEtvr code url: https://github.com/optyang/proxsgd

Thank you for considering to include it into the database!

Best,

Yang

opened by optyang 3
Fix some links and types, add bulky downloader
EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis W -> F

Designing Energy-Efficient Convolutional Neural Networks using Energy-Aware Pruning F -> W

Add bulky downloader which allows to automatically download most of the papers.
opened by ArmandXiao 2
Only Train Once: A One-Shot Neural Network Training And Pruning Framework

Conf: NeurIPS 2021 Paper: Only Train Once: A One-Shot Neural Network Training And Pruning Framework Code: https://github.com/tianyic/only_train_once Type: 'F'

opened by tianyic 2
CHIP: CHannel Independence-based Pruning for Compact Neural Networks

Conf: NeurIPS 2021 Paper: CHIP: CHannel Independence-based Pruning for Compact Neural Networks Code: https://github.com/Eclipsess/CHIP_NeurIPS2021 Type: 'F'

opened by Eclipsess 1
A new pruning paper in ICML2021

Thanks for your great project that helps us meet many valuable pruning papers.

The issue is a kindly remind of the new pruning paper published on ICML 2021: Accelerate CNNs from Three Dimensions: A Comprehensive Pruning Framework.

Thanks again.

opened by cheerss 1
The code of SCOP (NeurIPS 2020) has been released.

Hi Yang,

Thanks for the awesome paper list for network pruning! Here are our papers and codes about pruning.

The code of NeurIPS 2020 paper 'SCOP: Scientific Control for Reliable Neural Network Pruning' has been released and the link is
https://github.com/huawei-noah/Pruning/tree/master/SCOP_NeurIPS2020. The code is written with Pytorch.

What's more, another paper on AAAI 2020, Reborn filters: Pruning convolutional neural networks with limited data, proposes a filter pruning method to reduce the computation cost. The paper link is https://ojs.aaai.org/index.php/AAAI/article/view/6058

Thanks!

opened by yehuitang 1
Add an IJCAI2019 paper

Just a kind remind of filter-level pruning method: COP: Customized Deep Model Compression via Regularized Correlation-Based Filter-Level Pruning and code

opened by cheerss 1
Add paper - ECCV 2020 Oral EagleEye

Hi,

I would like to introduce our simple and fast filter pruning work accepted by ECCV 2020 Oral.

paper url: https://arxiv.org/abs/2007.02491 code url: https://github.com/anonymous47823493/EagleEye

Best,

Bailin Li

opened by bezorro 1
add new paper

Hello, please kindly introduce our one new paper:

Channel Pruning via Automatic Structure Search (IJCAI'2020). Link: https://arxiv.org/abs/2001.08565 Code: https://github.com/lmbxmu/ABCPruner

Besides, our previous work, i.e., HRank: Filter Pruning using High-Rank Feature Map (CVPR'2020), has been accepted as Oral Paper.

Thanks.

opened by lmbxmu 1
Add code and workshop links for implicit filter level sparsity

Hi Yang Thanks for maintaining this list. The CVPR 2019 paper 'On Implicit Filter Level Sparsity in Convolutional Neural Networks' also has 2 workshop versions, with a few additional results. Implicit Filter Sparsification In Convolutional Neural Networks (https://arxiv.org/abs/1905.04967), ODML-CDNNR Workshop, ICML 2019 Emergence of Implicit Filter Sparsity in Convolutional Neural Networks (https://openreview.net/forum?id=rylVvNS3hE), Deep Phenomena Workshop, ICML 2019

Also, an example of using the implicit sparsity for speedup is available in the following repository https://github.com/mehtadushy/SelecSLS-Pytorch as part of the model (https://github.com/mehtadushy/SelecSLS-Pytorch/blob/master/models/selecsls.py#L280)

I would appreciate if you could update the list to include these!

opened by mehtadushy 1
A wrong type label

Thanks for your collection. However, "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks"(ICLR2019) is a weight-level pruning algorithm. As the paper says, it only tries unstructured pruning and the structured pruning is in its plan.

opened by cheerss 1
Author code links for multiple papers added
Code links for two papers added:

Winning the Lottery Ahead of Time: Efficient Early Network Pruning (ICML 2022).

An Operator Theoretic View On Pruning Deep Neural Networks (ICLR 2022).
opened by kiryteo 0
New pruning type for our ICLR22 paper: Grouped Kernel Pruning — a densely structured pruning granularity with better pruning freedom than filter/channel methods.

Greetings,

We are the authors of the said paper/code, and we thank you for your inclusion; it has certainly generated some traffic for us.

However, it might be worth noting that our pruning granularity is not F (filter level). Our algorithm prunes at a Grouped Kernel level, which is — to the best of our knowledge — the most fine-grained approach under the constraint of outputting a densely structured pruned network, much like channel or filter prunings.

Since pushing the pruning freedom further while remaining structured is probably our most important contribution, we'd appreciate a simple fix (and maybe a new type category if you're feeling generous — as we'd certainly welcome more adaptations on the grouped kernel pruning framework). Thanks!
enhancement

opened by choH 6
Sparse Networks from Scratch via Bregman Iterations

Hi,

we have a new paper on training sparse networks from scratch via regularization and Bregman Iterations.

Paper url: https://arxiv.org/abs/2105.04319 GitHub: https://github.com/TimRoith/BregmanLearning

Since it is currently under review for a journal I want to ask if you even consider non-conference paper for this repository. If not please ignore this issue.

Best,

Tim

opened by TimRoith 1