Group Fisher Pruning for Practical Network Compression(ICML2021)

Related tags

Deep Learning pruning network-compression
Overview

Group Fisher Pruning for Practical Network Compression (ICML2021)

By Liyang Liu*, Shilong Zhang*, Zhanghui Kuang, Jing-Hao Xue, Aojun Zhou, Xinjiang Wang, Yimin Chen, Wenming Yang, Qingmin Liao, Wayne Zhang

Updates

  • All one stage models of Detection has been released (21/6/2021)

NOTES

All models about detection has been released. The classification models will be released later, because we want to refactor all our code into a Hook , so that it can become a more general tool for all tasks in OpenMMLab.

We will continue to improve this method and apply it to more other tasks, such as segmentation and pose.

The layer grouping algorithm is implemtated based on the AutoGrad of Pytorch, If you are not familiar with this feature and you can read Chinese, then these materials may be helpful to you.

  1. AutoGrad in Pytorch

  2. Hook of MMCV

Introduction

1. Compare with state-of-the-arts.

2. Can be applied to various complicated structures and various tasks.

3. Boosting inference speed on GPU under same flops.

Get Started

1. Creat a basic environment with pytorch 1.3.0 and mmcv-full

Due to the frequent changes of the autograd interface, we only guarantee the code works well in pytorch==1.3.0.

  1. Creat the environment
conda create -n open-mmlab python=3.7 -y
conda activate open-mmlab
  1. Install PyTorch 1.3.0 and corresponding torchvision.
conda install pytorch=1.3.0 cudatoolkit=10.0 torchvision=0.2.2 -c pytorch
  1. Build the mmcv-full from source with pytorch 1.3.0 and cuda 10.0

Please use gcc-5.4 and nvcc 10.0

 git clone https://github.com/open-mmlab/mmcv.git
 cd mmcv
 MMCV_WITH_OPS=1 pip install -e .

2. Install the corresponding codebase in OpenMMLab.

e.g. MMdetection

pip install mmdet==2.13.0

3. Pruning the model.

e.g. Detection

cd detection

Modify the load_from as the path to the baseline model in of xxxx_pruning.py

# for slurm train
sh tools/slurm_train.sh PATITION_NAME JOB_NAME configs/retina/retina_pruning.py work_dir
# for slurm_test
sh tools/slurm_test.sh PATITION_NAME JOB_NAME configs/retina/retina_pruning.py PATH_CKPT --eval bbox
# for torch.dist
# sh tools/dist_train.sh configs/retina/retina_pruning.py 8

4. Finetune the model.

e.g. Detection

cd detection

Modify the deploy_from as the path to the pruned model in custom_hooks of xxxx_finetune.py

# for slurm train
sh tools/slurm_train.sh PATITION_NAME JOB_NAME configs/retina/retina_finetune.py work_dir
# for slurm test
sh tools/slurm_test.sh PATITION_NAME JOB_NAME configs/retina/retina_fintune.py PATH_CKPT --eval bbox
# for torch.dist
# sh tools/dist_train.sh configs/retina/retina_finetune.py 8

Models

Detection

Method Backbone Baseline(mAP) Finetuned(mAP) Download
RetinaNet R-50-FPN 36.5 36.5 Baseline/Pruned/Finetuned
ATSS* R-50-FPN 38.1 37.9 Baseline/Pruned/Finetuned
PAA* R-50-FPN 39.0 39.4 Baseline/Pruned/Finetuned
FSAF R-50-FPN 37.4 37.4 Baseline/Pruned/Finetuned

* indicate with no Group Normalization in heads.

Classification

Coming soon.

Please cite our paper in your publications if it helps your research.

@InProceedings{liu2021group,
  title = {Group Fisher Pruning for Practical Network Compression},
  author =       {Liu, Liyang and Zhang, Shilong and Kuang, Zhanghui and Zhou, Aojun and Xue, Jing-Hao and Wang, Xinjiang and Chen, Yimin and Yang, Wenming and Liao, Qingmin and Zhang, Wayne},
  booktitle = {Proceedings of the 38th International Conference on Machine Learning},
  year = {2021},
  series = {Proceedings of Machine Learning Research},
  month = {18--24 Jul},
  publisher ={PMLR},
}
Comments
  • How prune the FC layer?

    How prune the FC layer?

    Hi, I tried to complete the codes of pruning faster rcnn, but I don't know how to prune the FC layer in roi_head. I have finished the twostage_wrapper function as shown below twostage But I got error. error Could you help me see where the problem is? Thanks!

    opened by mls1999725 8
  • How compute the speedup on GPUs?

    How compute the speedup on GPUs?

    Hi, I wonder how to compute speedup on GPUs? Is it the total inference time of CPU divide by that on GPU? If so, I wonder how to calculate the total time on the CPU by mmdetection, do I need to put all the data on the CPU for retraining? Or just need to put the data on the CPU during inference? tempsnip Looking forward to reply！

    opened by mls1999725 4
  • How compute the acts of FC layer?

    How compute the acts of FC layer?

    Hi, I'm trying to complete the pruning codes of FC layer. I wonder how to compute the acts of FC layer, is it n times oc where n is the batch size and oc is the channel numbers of output feature?

    opened by mls1999725 4
  • The batch size and number of gpu during pruning.

    The batch size and number of gpu during pruning.

    Hi, I basically completed the pruning codes of faster-rcnn. I wonder the batch size and the number of gpu during pruning faster-rcnn, is it 16 on 8 2080ti which is same as all detection models? Thank you very much!

    opened by mls1999725 3
  • An issue about running the code

    An issue about running the code

    I create a virtual environment, open-mmlab, with python=3.7, pytorch=1.3.0, cudatoolkit=10.0, mmcv=1.3.16, mmdet=2.13.0. However, it still can not run.

    File "tools/train.py", line 16, in
    from mmdet.apis import set_random_seed, train_detector ImportError: libcudart.so.10.1: cannot open shared object file: No such file or directory

    opened by MaltoseFlower 2
  • Finetuning speed is too slow.

    Finetuning speed is too slow.

    Hi, I'm very interested in your work. When I fintune the RetinaNet of your pruned model, the finetuning process will last for 2 days, which is very different from the 7 hours in the paper, as shown in the figure below. image One 1080ti is used and I didn't use slurm. The command for finetuning I used is python -u tools/train.py configs/retina/retina_finetune.py --work-dir ./ --gpu-ids 0

    opened by mls1999725 2
  • Some questions about the paper and code

    Some questions about the paper and code

    Thanks for the great work, I have some questions about this code

    1. In the paper you mentioned coupled channels, I understand what it means is that C2/C5 (you described in Figure 4) receive the same input. Is there any other meaning besides this?
    截屏2022-01-10 上午10 30 32 2. For this common concat operation in yolo, can your code still handle it normally？Because the input comes from two ancestors, and it seems that the number of channels cannot correspond to the function **def construct_outchannel_masks(self)** to construct the mask， 截屏2022-01-10 上午10 35 54 Hope to get your reply.
    opened by syswyl 1
  • Why does the gradient multiply the input?

    Why does the gradient multiply the input?

    https://github.com/jshilong/FisherPruning/blob/b28ea9185f1269ef4597303ccb6be3300c7e1b93/detection/tools/fisher_pruning.py#L432 How it work if we have different in_channels and out_channels?

    opened by yumath 0
  • Resume pruning

    Resume pruning

    Hi, thanks for this awesome work. I'm wondering if there is an option to continue pruning, I tried setting pruning=True and deploy_from=checkpoint to achieve it, but it didn't work. And then I modify the code in https://github.com/jshilong/FisherPruning/blob/b28ea9185f1269ef4597303ccb6be3300c7e1b93/detection/tools/fisher_pruning.py#L113, making model loading mask and checkpoint when pruning. At the same time I comment the code in https://github.com/jshilong/FisherPruning/blob/b28ea9185f1269ef4597303ccb6be3300c7e1b93/detection/tools/fisher_pruning.py#L133 to prevent model sets mask to 1. Although it can obtain information about the pruned channels, it still prunes them according to the initial model.

    Is there anything else I need to change?

    opened by Janspiry 0
  • The batch size and number of gpu during pruning.

    The batch size and number of gpu during pruning.

    Hi, I basically completed the pruning codes of faster-rcnn. I wonder the batch size and the number of gpu during pruning faster-rcnn, is it 16 on 8 2080ti which is same as all detection models? Thank you very much!

    opened by mls1999725 0
  • The batch size and number of gpu during pruning.

    The batch size and number of gpu during pruning.

    Hi, I basically completed the pruning codes of faster-rcnn. I wonder the batch size and the number of gpu during pruning faster-rcnn, is it 16 on 8 2080ti which is same as all detection models? Thank you very much!

    opened by mls1999725 0
  • Question on Algorithm 1 (Layer Grouping)

    Question on Algorithm 1 (Layer Grouping)

    In ResNeXt blocks, conv2 is a GConv and should be grouped with its child conv3. However, if conv2 is not in any group while we assign a group for conv3, e.g. conv2 is behind conv3 in \mathbb{L}, conv3 will not be assigned to the same group with conv2. Instead, it will be assigned to a new group. The conv2 will not be assigned to this group since conv2 and conv3 have no shared parent. image When we modify the make_groups function according to this algorithm, it indeed fails to group the conv2&conv3 in ResNeXt50 blocks though the found ancestors are correct.

    Could u please help check this problem?

    Now in this repo, we tentatively change this function to image and it works.

    opened by Ben-Louis 2
  • The finetuned ATSS model file is bad.

    The finetuned ATSS model file is bad.

    Hi, the given finetuned ATSS model can't be tested, the error is as follows, maybe the file is incomplete. And I have tested finetuned models of RetinaNet and PAA, they are normal. wrong size

    bug 
    opened by mls1999725 2
Owner
Shilong Zhang
Shilong Zhang
GitHub
A Closer Look at Structured Pruning for Neural Network Compression

A Closer Look at Structured Pruning for Neural Network Compression Code used to reproduce experiments in https://arxiv.org/abs/1810.04622. To prune, w

Bayesian and Neural Systems Group 140 Dec 5, 2022
Official implementation of the ICML2021 paper "Elastic Graph Neural Networks"

ElasticGNN This repository includes the official implementation of ElasticGNN in the paper "Elastic Graph Neural Networks" [ICML 2021]. Xiaorui Liu, W

liuxiaorui 34 Dec 4, 2022
YOLOv5 Series Multi-backbone, Pruning and quantization Compression Tool Box.

YOLOv5-Compression Update News Requirements 环境安装 pip install -r requirements.txt Evaluation metric Visdrone Model mAP mAP@50 Parameters(M) GFLOPs FPS@

ZhangYuan 719 Jan 2, 2023
Revisiting Discriminator in GAN Compression: A Generator-discriminator Cooperative Compression Scheme (NeurIPS2021)

Revisiting Discriminator in GAN Compression: A Generator-discriminator Cooperative Compression Scheme (NeurIPS2021) Overview Prerequisites Linux Pytho

Shaojie Li 34 Mar 31, 2022
An Image compression simulator that uses Source Extractor and Monte Carlo methods to examine the post compressive effects different compression algorithms have.

ImageCompressionSimulation An Image compression simulator that uses Source Extractor and Monte Carlo methods to examine the post compressive effects o

James Park 1 Dec 11, 2021
This code reproduces the results of the paper, "Measuring Data Leakage in Machine-Learning Models with Fisher Information"

Fisher Information Loss This repository contains code that can be used to reproduce the experimental results presented in the paper: Awni Hannun, Chua

Facebook Research 43 Dec 30, 2022
Official PyTorch code for CVPR 2020 paper "Deep Active Learning for Biased Datasets via Fisher Kernel Self-Supervision"

Deep Active Learning for Biased Datasets via Fisher Kernel Self-Supervision https://arxiv.org/abs/2003.00393 Abstract Active learning (AL) aims to min

Denis 29 Nov 21, 2022
A curated list of neural network pruning resources.

A curated list of neural network pruning and related resources. Inspired by awesome-deep-vision, awesome-adversarial-machine-learning, awesome-deep-learning-papers and Awesome-NAS.

Yang He 1.7k Jan 9, 2023
Code for PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning

PackNet: https://arxiv.org/abs/1711.05769 Pretrained models are available here: https://uofi.box.com/s/zap2p03tnst9dfisad4u0sfupc0y1fxt Datasets in Py

Arun Mallya 216 Jan 5, 2023
Network Pruning That Matters: A Case Study on Retraining Variants (ICLR 2021)

Network Pruning That Matters: A Case Study on Retraining Variants (ICLR 2021)

Duong H. Le 18 Jun 13, 2022
Ensemble Knowledge Guided Sub-network Search and Fine-tuning for Filter Pruning

Ensemble Knowledge Guided Sub-network Search and Fine-tuning for Filter Pruning This repository is official Tensorflow implementation of paper: Ensemb

Seunghyun Lee 12 Oct 18, 2022
Code for paper: Group-CAM: Group Score-Weighted Visual Explanations for Deep Convolutional Networks

Group-CAM By Zhang, Qinglong and Rao, Lu and Yang, Yubin [State Key Laboratory for Novel Software Technology at Nanjing University] This repo is the o

zhql 98 Nov 16, 2022
BC3407-Group-5-Project - BC3407 Group Project With Python

BC3407-Group-5-Project As the world struggles to contain the ever-changing varia

null 1 Jan 26, 2022
Code of paper "CDFI: Compression-Driven Network Design for Frame Interpolation", CVPR 2021

CDFI (Compression-Driven-Frame-Interpolation) [Paper] (Coming soon...) | [arXiv] Tianyu Ding*, Luming Liang*, Zhihui Zhu, Ilya Zharkov IEEE Conference

Tianyu Ding 95 Dec 4, 2022
Block Sparse movement pruning

Movement Pruning: Adaptive Sparsity by Fine-Tuning Magnitude pruning is a widely used strategy for reducing model size in pure supervised learning; ho

Hugging Face 54 Dec 20, 2022
This repo contains the official implementations of EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis

EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis This repo contains the official implementations of EigenDamage: Structured Prunin

Chaoqi Wang 107 Apr 20, 2022
Learned Token Pruning for Transformers

LTP: Learned Token Pruning for Transformers Check our paper for more details. Installation We follow the same installation procedure as the original H

Sehoon Kim 52 Dec 29, 2022
Learnable Motion Coherence for Correspondence Pruning

Learnable Motion Coherence for Correspondence Pruning Yuan Liu, Lingjie Liu, Cheng Lin, Zhen Dong, Wenping Wang Project Page Any questions or discussi

liuyuan 41 Nov 30, 2022
A Python implementation of the Locality Preserving Matching (LPM) method for pruning outliers in image matching.

LPM_Python A Python implementation of the Locality Preserving Matching (LPM) method for pruning outliers in image matching. The code is established ac

AoxiangFan 11 Nov 7, 2022