Distilling Knowledge via Knowledge Review, CVPR 2021

Overview

ReviewKD

Distilling Knowledge via Knowledge Review

Pengguang Chen, Shu Liu, Hengshuang Zhao, Jiaya Jia

This project provides an implementation for the CVPR 2021 paper "Distilling Knowledge via Knowledge Review"

CIFAR-100 Classification

Please refer to CIFAR-100 for more details.

ImageNet Classification

Please refer to ImageNet for more details.

COCO Detection

Coming soon

COCO Instance Segmentation

Coming soon

Citation

Please consider citing ReviewKD in your publications if it helps your research.

@inproceedings{chen2021reviewkd,
    title={Distilling Knowledge via Knowledge Review},
    author={Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia},
    booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
    year={2021},
}
Issues
  • Questions about detection pretrained weights

    Questions about detection pretrained weights

    I want to make sure that the file mv2-r50.pth in the detection pretrained weights you provided contains both teacher's and student's weights.

    Thank you!

    opened by Coldfire93 6
  • Realization of the knowledge review

    Realization of the knowledge review

    Hi, thanks for your great job! I wrote a kr version using paddle, could you please help see is there any problems? thank you!

    https://github.com/littletomatodonkey/code_scipts/blob/main/knowledge_review/knowledge_review.py

    I used conv_1x1 for all the channel transform and adaptative avg pool for the size transform.

    opened by littletomatodonkey 2
  • Can we find the teacher_weights somewhere?

    Can we find the teacher_weights somewhere?

    When I run your scripts "reviewKD.sh" and "baseline.sh" in Cifar100. There's FileNotFoundError:

    FileNotFoundError: [Errno 2] No such file or directory: 'checkpoints/cifar100_wrn-40-2__baseline1_best.pt'
    Namespace(T=4.0, batch_size=128, ce_loss_weight=1.0, dataset='cifar100', epochs=240, gamma=0.1, kd_loss_weight=5.0, kd_warm_up=20.0, kl_loss_weight=1.0, lr=0.1, lr_adjust_step=[150, 180, 210], model='wrn-40-1', resume='', seed=148, suffix='reviewkd1', teacher='wrn-40-2', teacher_weight='checkpoints/cifar100_wrn-40-2__baseline1_best.pt', test=False, use_kl=False, wd=0.0005)
    

    Where could I find those weights or can you release the related teacher weights so that we can download and better configure our experiment environment.

    opened by Luodian 2
  • Log file of loss values

    Log file of loss values

    Hi Author, thanks for your excellent work. I want to ask whether you can release a log file that includes loss values. Based on this file, I can check what‘s the loss change? It is better for the detection model. It would be the best for retinanet. Thank you!

    opened by hdjsjyl 2
  • When would you like to release the official code?

    When would you like to release the official code?

    Thanks for your contributions, can you tell us the time you would like to release the code?

    opened by HUALIMUGU 1
  • Where is the mobilenet baseline from

    Where is the mobilenet baseline from

    Hi, thanks for your great job! Where is the mobilenet baseline from? I train the mobilenet for 100epochs and the top1-acc is 69.4%, which seems higher than that provided in the article(68.8%).

    opened by littletomatodonkey 1
  • Knowledge distillation on RetinaNet

    Knowledge distillation on RetinaNet

    Hi authors, thanks for the great work. But the repository only includes object detectors on Faster RCNN. I want to know when the knowledge distillation of the object detector based on RetinaNet will be released? Thank you!

    opened by hdjsjyl 1
  • about teacher net

    about teacher net

    Thank you very much for your work!

    I have noticed that before distillation, the teacher networks are loaded with a pre-trained model. Is the teacher network fixed during distillation, I didn't find where this part of the code (like detach or i.requires_grad = False)

    opened by yyuxin 1
  • add paddlepaddle review-kd

    add paddlepaddle review-kd

    att.

    opened by littletomatodonkey 0
  • loss

    loss

    null

    opened by zf6578 1
  • A new re-implementation for KnowledgeReview

    A new re-implementation for KnowledgeReview

    @akuxcw @littletomatodonkey Nice work !!!

    Based on this repos, I tried a new implementation ZJCV/KnowledgeReview. From the training results of cifar100, KR does achieve very excellent functions. For resnet50, the distillation results even exceed the teacher network

    arch_s | top1 | top5 | arch_t | top1 | top5 | dataset | lambda | top1 | top5 -- | -- | -- | -- | -- | -- | -- | -- | -- | -- MobileNetv2 | 80.620 | 95.820 | ResNet50 | 83.540 | 96.820 | CIFAR100 | 7.0 | 83.370 | 96.810 MobileNetv2 | 80.620 | 95.820 | ResNet152 | 85.490 | 97.590 | CIFAR100 | 8.0 | 84.530 | 97.470 MobileNetv2 | 80.620 | 95.820 | ResNeXt_32x8d | 85.720 | 97.650 | CIFAR100 | 6.0 | 84.520 | 97.470 ResNet18 | 80.540 | 96.040 | ResNet50 | 83.540 | 96.820 | CIFAR100 | 10.0 | 83.130 | 96.350 ResNet50 | 83.540 | 96.820 | ResNet152 | 85.490 | 97.590 | CIFAR100 | 6.0 | 86.240 | 97.610 ResNet50 | 83.540 | 96.820 | ResNeXt_32x8d | 85.720 | 97.650 | CIFAR100 | 6.0 | 86.220 | 97.490

    opened by zjykzj 3
  • preact setting

    preact setting

    Line 222, ReviewKD/CIFAR-100/train.py t_features, t_pred = teacher(images, is_feat = True, preact=True)

    Line 76, ReviewKD/CIFAR-100/model/reviewkd.py student_features = self.student(x,is_feat=True)

    Why preact is set different value in these two positions? One is True, the other is False (default value) thanks!

    opened by fingerk28 4
  • Reproduced ImageNet result with torchdistill and questions about your baselines

    Reproduced ImageNet result with torchdistill and questions about your baselines

    Thank you for open-sourcing your project!

    Reproduced ImageNet result

    I reimplemented the knowledge review method in a model-agnostic way with torchdistill. Using the reimplemented method, I successfully reproduced the ImageNet result for a pair of ResNet-18 (student, 71.64% accuracy) and ResNet-34 (teacher) as shown here. Hope this helps if you further study knowledge distillation.

    Implementation of baselines for object detection

    In your paper (Table 4), Hinton et al.'s knowledge distillation and FitNet methods are used as baselines for object detection. The KD method and the 2nd stage of FitNet method trains student model in end-to-end manner with teacher's final output. I was wondering how you could implement these methods for object detection models. Could you please publish the code as well?

    From my understanding (as discussed here), such methods cannot be directly applied to object detection models like R-CNNs since they return different number of bounding boxes and class probabilities depending on both 1) an input image and 2) learnt model parameters. Thus, the shapes of outputs from teacher model may not match those from student model. Even when the shapes match like after several epochs of training, the order of the teacher's predicted objects in the input image may not be aligned with that of the student's predicted objects.

    opened by yoshitomo-matsubara 6
Owner
DV Lab
Deep Vision Lab
DV Lab
When Age-Invariant Face Recognition Meets Face Age Synthesis: A Multi-Task Learning Framework (CVPR 2021 oral)

MTLFace This repository contains the PyTorch implementation and the dataset of the paper: When Age-Invariant Face Recognition Meets Face Age Synthesis

Hzzone 82 Nov 16, 2021
(CVPR 2021) Back-tracing Representative Points for Voting-based 3D Object Detection in Point Clouds

BRNet Introduction This is a release of the code of our paper Back-tracing Representative Points for Voting-based 3D Object Detection in Point Clouds,

null 70 Nov 22, 2021
Scale-aware Automatic Augmentation for Object Detection (CVPR 2021)

SA-AutoAug Scale-aware Automatic Augmentation for Object Detection Yukang Chen, Yanwei Li, Tao Kong, Lu Qi, Ruihang Chu, Lei Li, Jiaya Jia [Paper] [Bi

Jia Research Lab 139 Nov 25, 2021
(CVPR 2021) ST3D: Self-training for Unsupervised Domain Adaptation on 3D Object Detection

ST3D Code release for the paper ST3D: Self-training for Unsupervised Domain Adaptation on 3D Object Detection, CVPR 2021 Authors: Jihan Yang*, Shaoshu

CVMI Lab 158 Nov 24, 2021
Code for the head detector (HeadHunter) proposed in our CVPR 2021 paper Tracking Pedestrian Heads in Dense Crowd.

Head Detector Code for the head detector (HeadHunter) proposed in our CVPR 2021 paper Tracking Pedestrian Heads in Dense Crowd. The head_detection mod

Ramana Subramanyam 44 Nov 24, 2021
Source code of our TPAMI'21 paper Dual Encoding for Video Retrieval by Text and CVPR'19 paper Dual Encoding for Zero-Example Video Retrieval.

Dual Encoding for Video Retrieval by Text Source code of our TPAMI'21 paper Dual Encoding for Video Retrieval by Text and CVPR'19 paper Dual Encoding

null 60 Nov 8, 2021
Automatically download multiple papers by keywords in CVPR

CVFPaperHelper Automatically download multiple papers by keywords in CVPR Install mkdir PapersToRead cd PaperToRead pip install requests tqdm git clon

null 44 Nov 22, 2021
Code for generating synthetic text images as described in "Synthetic Data for Text Localisation in Natural Images", Ankush Gupta, Andrea Vedaldi, Andrew Zisserman, CVPR 2016.

SynthText Code for generating synthetic text images as described in "Synthetic Data for Text Localisation in Natural Images", Ankush Gupta, Andrea Ved

Ankush Gupta 1.7k Nov 29, 2021
Source code of RRPN ---- Arbitrary-Oriented Scene Text Detection via Rotation Proposals

Paper source Arbitrary-Oriented Scene Text Detection via Rotation Proposals https://arxiv.org/abs/1703.01086 News We update RRPN in pytorch 1.0! View

null 421 Nov 21, 2021
Multi-Oriented Scene Text Detection via Corner Localization and Region Segmentation

This is the official implementation of "Multi-Oriented Scene Text Detection via Corner Localization and Region Segmentation". For more details, please

Pengyuan Lyu 301 Nov 18, 2021
Implementation of our paper 'PixelLink: Detecting Scene Text via Instance Segmentation' in AAAI2018

Code for the AAAI18 paper PixelLink: Detecting Scene Text via Instance Segmentation, by Dan Deng, Haifeng Liu, Xuelong Li, and Deng Cai. Contributions

null 763 Nov 23, 2021
A tool for extracting text from scanned documents (via OCR), with user-defined post-processing.

The project is based on older versions of tesseract and other tools, and is now superseded by another project which allows for more granular control o

Maxim 31 Jul 24, 2021
Detect text blocks and OCR poorly scanned PDFs in bulk. Python module available via pip.

doc2text doc2text extracts higher quality text by fixing common scan errors Developing text corpora can be a massive pain in the butt. Much of the tex

Joe Sutherland 1.3k Nov 27, 2021
Learning Camera Localization via Dense Scene Matching, CVPR2021

This repository contains code of our CVPR 2021 paper - "Learning Camera Localization via Dense Scene Matching" by Shitao Tang, Chengzhou Tang, Rui Hua

tangshitao 49 Nov 13, 2021
Code release for our paper, "SimNet: Enabling Robust Unknown Object Manipulation from Pure Synthetic Data via Stereo"

SimNet: Enabling Robust Unknown Object Manipulation from Pure Synthetic Data via Stereo Thomas Kollar, Michael Laskey, Kevin Stone, Brijen Thananjeyan

null 40 Nov 26, 2021
Code for paper "Role-based network embedding via structural features reconstruction with degree-regularized constraint"

Role-based network embedding via structural features reconstruction with degree-regularized constraint Train python main.py --dataset brazil-flights

wang zhang 1 Oct 21, 2021
Official implementation of "An Image is Worth 16x16 Words, What is a Video Worth?" (2021 paper)

An Image is Worth 16x16 Words, What is a Video Worth? paper Official PyTorch Implementation Gilad Sharir, Asaf Noy, Lihi Zelnik-Manor DAMO Academy, Al

null 181 Nov 18, 2021
Code for AAAI 2021 paper: Sequential End-to-end Network for Efficient Person Search

This repository hosts the source code of our paper: [AAAI 2021]Sequential End-to-end Network for Efficient Person Search. SeqNet achieves the state-of

Zj Li 161 Nov 26, 2021