This is the pytorch implementation for the paper: Generalizable Mixed-Precision Quantization via Attribution Rank Preservation, which is accepted to ICCV2021.

Last update: Sep 2, 2022

Related tags

Deep Learning GMPQ

Overview

GMPQ: Generalizable Mixed-Precision Quantization via Attribution Rank Preservation

This is the pytorch implementation for the paper: Generalizable Mixed-Precision Quantization via Attribution Rank Preservation, which is accepted to ICCV2021. This repo contains searching the quantization policy via attribution preservation on small datasets including CIFAR-10, Cars, Flowers, Aircraft, Pets and Food, and finetuning on largescale dataset like ImageNet using our proposed GMPQ.

Quick Start

Prerequisites

python>=3.5
pytorch>=1.1.0
torchvision>=0.3.0
other packages like numpy and sklearn

Dataset

If you already have the ImageNet dataset for pytorch, you could create a link to data folder and use it:

# prepare dataset, change the path to your own
ln -s /path/to/imagenet/ data/

If you don't have the ImageNet, you can use the following script to download it: https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh

For small datasets which we search the quantization policy on, please follow the official instruction:

Searching the mixed-precision quantization policy

For a specific small dataset, you should first pretrain a full-precision model to provide supervision for attribution rank consistency preservation and save it to pretrain_model.pth.tar.

After that, you can start searching the quantization policy. Take ResNet18 and CIFAR-10 for example:

CUDA_VISIBLE_DEVICES=0,1 python search_attention.py \
-a mixres18_w2346a2346  -fa qresnet18_cifar  --epochs 25  --pretrained pretrain_model.pth.tar --aw 40 \
--dataname cifar10 --expname cifar10_resnet18  --cd 0.0003   --step-epoch 10    \
--batch-size 256   --lr 0.1   --lra 0.01 -j 16  \
  path/to/cifar10 \

It also supports other network architectures like ResNet50 and other small datasets like Cars, Flowers, Aircraft, Pets and Food.

Finetuning on ImageNet

After searching, you can get the optimal quantization policy, with the checkpoint arch_checkpoint.pth.tar. You can run the following command to finetune and evaluate the performance on ImageNet dataset.


CUDA_VISIBLE_DEVICES=0,1 python main.py     \
 -a qresnet18                 \
 --ac arch_checkpoint.pth.tar \
 -c checkpoints/train_resnet18   \
 --data_name imagenet          \
 --data path/to/imagenet           \
 --epochs 100                     \
 --pretrained pretrained.pth.tar
 --lr 0.01                    \
 --gpu_id 1,2,3     \
 --train_batch_per_gpu 192              \
 --wd 4e-5                       \
 --workers 32                    \

DiffQ performs differentiable quantization using pseudo quantization noise. It can automatically tune the number of bits used per weight or group of weights, in order to achieve a given trade-off between model size and accuracy.

Differentiable Model Compression via Pseudo Quantization Noise DiffQ performs differentiable quantization using pseudo quantization noise. It can auto

145 Dec 30, 2022

This is the pytorch implementation of the paper - Axiomatic Attribution for Deep Networks.

Integrated Gradients This is the pytorch implementation of "Axiomatic Attribution for Deep Networks". The original tensorflow version could be found h

150 Dec 23, 2022

PyTorch implementation of VAGAN: Visual Feature Attribution Using Wasserstein GANs

PyTorch implementation of VAGAN: Visual Feature Attribution Using Wasserstein GANs This code aims to reproduce results obtained in the paper "Visual F

93 Aug 17, 2022

DCSL - Generalizable Crowd Counting via Diverse Context Style Learning

DCSL Generalizable Crowd Counting via Diverse Context Style Learning Requirement

3 Jun 13, 2022

git《Self-Attention Attribution: Interpreting Information Interactions Inside Transformer》(AAAI 2021) GitHub:

Self-Attention Attribution This repository contains the implementation for AAAI-2021 paper Self-Attention Attribution: Interpreting Information Intera

60 Dec 29, 2022

Keep CALM and Improve Visual Feature Attribution

Keep CALM and Improve Visual Feature Attribution Jae Myung Kim1*, Junsuk Choe1*, Zeynep Akata2, Seong Joon Oh1† * Equal contribution † Corresponding a

90 Dec 7, 2022

Text mining project; Using distilBERT to predict authors in the classification task authorship attribution.

DistilBERT-Text-mining-authorship-attribution Dataset used: https://www.kaggle.com/azimulh/tweets-data-for-authorship-attribution-modelling/version/2

1 Jan 13, 2022

[ICLR 2021] "CPT: Efficient Deep Neural Network Training via Cyclic Precision" by Yonggan Fu, Han Guo, Meng Li, Xin Yang, Yining Ding, Vikas Chandra, Yingyan Lin

CPT: Efficient Deep Neural Network Training via Cyclic Precision Yonggan Fu, Han Guo, Meng Li, Xin Yang, Yining Ding, Vikas Chandra, Yingyan Lin Accep

26 Oct 25, 2022

Code of paper "Compositionally Generalizable 3D Structure Prediction"

Compositionally Generalizable 3D Structure Prediction In this work, We bring in the concept of compositional generalizability and factorizes the 3D sh

30 Dec 17, 2022

Comments

How to compute BOPs of ResNet-50

Thanks for a great job! I have a question regarding the BOPs of ResNet-50. How to compute BOPs of ResNet-50? In my implementation, the BOPs of ResNet-50 is 4187.3G, which is different from the number in the paper (3952.6G).

opened by liujingcs 0
What is the argument --pretrained pretrained.pth.tar?

Thanks for your exciting work. When running your code, I am not sure the input "pretrained.pth.tar" in the main.py when fine-tuning on imagenet. It seems the weight of pretrained full precision of Resnet18 on imagenet. Is the is right?.

opened by cuong-pv 1

This is the pytorch implementation for the paper: Generalizable Mixed-Precision Quantization via Attribution Rank Preservation, which is accepted to ICCV2021.

Related tags

Overview

GMPQ: Generalizable Mixed-Precision Quantization via Attribution Rank Preservation

Quick Start

Prerequisites

Dataset

Searching the mixed-precision quantization policy

Finetuning on ImageNet

You might also like...

DiffQ performs differentiable quantization using pseudo quantization noise. It can automatically tune the number of bits used per weight or group of weights, in order to achieve a given trade-off between model size and accuracy.

This is the pytorch implementation of the paper - Axiomatic Attribution for Deep Networks.

PyTorch implementation of VAGAN: Visual Feature Attribution Using Wasserstein GANs

DCSL - Generalizable Crowd Counting via Diverse Context Style Learning

git《Self-Attention Attribution: Interpreting Information Interactions Inside Transformer》(AAAI 2021) GitHub:

Keep CALM and Improve Visual Feature Attribution

Text mining project; Using distilBERT to predict authors in the classification task authorship attribution.

[ICLR 2021] "CPT: Efficient Deep Neural Network Training via Cyclic Precision" by Yonggan Fu, Han Guo, Meng Li, Xin Yang, Yining Ding, Vikas Chandra, Yingyan Lin

Code of paper "Compositionally Generalizable 3D Structure Prediction"

Comments

How to compute BOPs of ResNet-50

What is the argument --pretrained pretrained.pth.tar?

Owner

Quantization library for PyTorch. Support low-precision and mixed-precision quantization, with hardware implementation through TVM.

BitPack is a practical tool to efficiently save ultra-low precision/mixed-precision quantized models.

This is an official implementation of the paper "Distance-aware Quantization", accepted to ICCV2021.

A PyTorch Extension: Tools for easy mixed precision and distributed training in Pytorch

A PyTorch Extension: Tools for easy mixed precision and distributed training in Pytorch

Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation. In CVPR 2022.

Pytorch implementation of the paper "Enhancing Content Preservation in Text Style Transfer Using Reverse Attention and Conditional Layer Normalization"

Official PyTorch Implementation of Rank & Sort Loss [ICCV2021]

EdMIPS: Rethinking Differentiable Search for Mixed-Precision Neural Networks

MicRank is a Learning to Rank neural channel selection framework where a DNN is trained to rank microphone channels.