Multi-task Self-supervised Object Detection via Recycling of Bounding Box Annotations (CVPR 2019)
To make better use of given limited labels, we propose a novel object detection approach that takes advantage of both multi-task learning (MTL) and self-supervised learning (SSL). We propose a set of auxiliary tasks that help improve the accuracy of object detection.
Here is a guide to the source code.
Reference
If you are willing to use this code or cite the paper, please refer the following:
@inproceedings{lee2019multi,
author = {Wonhee Lee and Joonil Na and Gunhee Kim},
title = {Multi-task Self-supervised Object Detection via Recycling of Bounding Box Annotations},
booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
year = {2019}
}
CVPR Poster [PPT][PDF]
Introduction [PPT][PDF]
Multi-task Learning
Multi-task learning (MTL) aims at jointly training multiple relevant tasks with less annotations to improve the performance of each task.
[1] An Overview of Multi-Task Learning in Deep Neural Networks
[2] Mask R-CNN
Self-supervised Learning
Self-supervised learning (SSL) aims at training the model from the annotations generated by itself with no additional human effort.
[3] Learning Representations for Automatic Colorization
[4] Unsupervised learning of visual representations by solving jigsaw puzzles
Annotation Reuse
Reusing labels of one task is not only helpful to create new tasks and their labels but also capable of improving the performance of the main task through pretraining. Our work focuses on recycling bounding box labels for object detection.
[5] Look into Person: Self-supervised Structure-sensitive Learning and A New Benchmark for Human Parsing
[6] Mix-and-Match Tuning for Self-Supervised Semantic Segmentation
Our approach
The key to our approach is to propose a set of auxiliary tasks that are relevant but not identical to object detection. They create their own labels by recycling the bounding box labels (e.g. annotations of the main task) in an SSL manner while regarding the bounding box as metadata. Then these auxiliary tasks are jointly trained with the object detection model in an MTL way.
Approach
Overall architecture
It shows how the object detector (i.e. main task model) such as Faster R-CNN makes a prediction for a given proposal box (red) with assistance of three auxiliary tasks at inference. The auxiliary task models (shown in the bottom right) are almost identical to the main task predictor except no box regressor. The refinement of detection prediction (shown in right) is also collectively done by cooperation of the main and auxiliary task models. K is the number of categories.
3 auxiliary tasks
This is an example of how to generate labels of auxiliary tasks via recycling of GT bounding boxes.
- The multi-object soft label assigns the area portions occupied by each class’s GT boxes within a window.
- The closeness label scores the distances from the center of the GT box to those of other GT boxes.
- The foreground label is a binary mask between foreground and background.
Results
We empirically validate that our approach effectively improves detection performance on various architectures and datasets. We test two state-of-the-art region proposal object detectors, including Faster R-CNN and R-FCN, with three CNN backbones of ResNet-101, InceptionResNet-v2, and MobileNet on two benchmark datasets of PASCAL VOC and COCO.
Qualitative results
Qualitative comparison of detection results between baseline (left) and our approach (right) in each set. We divide the errors into five categories (Localization, Classification, Redundancy, Background, False Negative). Our approach often improves the baseline’s detection by correcting several false negatives and false positives such as background, similar object and redundant detection.
202 Jan 3, 2023
4 Jul 12, 2021
16 Oct 26, 2022
![[CVPR 21] Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting, IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021.](https://github.com/AyanKumarBhunia/Self-Supervised-Learning-for-Sketch/raw/main/sample_images/outline.png)
44 Dec 12, 2022
![[CVPR 2021]](https://github.com/VITA-Group/CV_LTH_Pre-training/raw/main/Figs/Teaser.png)
59 Dec 28, 2022
92 Nov 30, 2022
6 Nov 25, 2022
52 Dec 24, 2022
0 Sep 28, 2022
469 Dec 26, 2022
5 Dec 10, 2022
4 Nov 27, 2022
147 Dec 5, 2022
390 Dec 31, 2022
88 Dec 21, 2022
266 Nov 24, 2022
1.5k Dec 27, 2022
23 Nov 5, 2022