Introdunction
This is the official implementation of the paper "Query2Label: A Simple Transformer Way to Multi-Label Classification".
Abstract
This paper presents a simple and effective approach to solving the multi-label classification problem. The proposed approach leverages Transformer decoders to query the existence of a class label. The use of Transformer is rooted in the need of extracting local discriminative features adaptively for different labels, which is a strongly desired property due to the existence of multiple objects in one image. The built-in cross-attention module in the Transformer decoder offers an effective way to use label embeddings as queries to probe and pool class-related features from a feature map computed by a vision backbone for subsequent binary classifications. Compared with prior works, the new framework is simple, using standard Transformers and vision backbones, and effective, consistently outperforming all previous works on five multi-label classification data sets, including MS-COCO, PASCAL VOC, NUS-WIDE, and Visual Genome. Particularly, we establish 91.3% mAP on MS-COCO. We hope its compact structure, simple implementation, and superior performance serve as a strong baseline for multi-label classification tasks and future studies.
Results on MS-COCO:
Quick start
-
(optional) Star this repo.
-
Clone this repo:
git clone [email protected]:SlongLiu/query2labels.git
cd query2labels
- Install
cuda,PyTorchandtorchvision.
Please make sure they are compatible. We test our models on two envs and other configs may also work:
cuda==11, torch==1.9.0, torchvision==0.10.0, python==3.7.3
or
cuda==10.2, torch==1.6.0, torchvision==0.7.0, python==3.7.3
- Install other needed packages.
pip install -r requirments.txt
- Data preparation.
Download MS-COCO 2014 and modify the path in lib/dataset/cocodataset.py: line 24, 25.
- Download pretrained models.
You could download pretrained models from this link. See more details below.
- Run!
python q2l_infer.py -a modelname --config /path/to/json/file --resume /path/to/pkl/file [other args]
e.g.
python q2l_infer.py -a 'Q2L-R101-448' --config "pretrained/Q2L-R101-448/config_new.json" -b 16 --resume 'pretrained/Q2L-R101-448/checkpoint.pkl'
pretrianed model
| Modelname | mAP | link(Tsinghua-cloud) |
|---|---|---|
| Q2L-R101-448 | 84.9 | this link |
| Q2L-R101-576 | 86.5 | this link |
| Q2L-TResL-448 | 87.3 | this link |
| Q2L-TResL_22k-448 | 89.2 | this link |
| Q2L-SwinL-384 | 90.5 | this link |
| Q2L-CvT_w24-384 | 91.3 | this link |
Training
Training scripts will be available later.
BibTex
@misc{liu2021query2label,
title={Query2Label: A Simple Transformer Way to Multi-Label Classification},
author={Shilong Liu and Lei Zhang and Xiao Yang and Hang Su and Jun Zhu},
year={2021},
eprint={2107.10834},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
Acknowledgement
We thank the authors of ASL, TResNet, detr, CvT, and Swin-Transformer for their great works and codes. Thanks to @mrT23 for sharing training tricks and providing a useful script for training.
![关于Swin-L[37]的实验结果](https://avatars.githubusercontent.com/u/37521721?v=4)
101 Nov 25, 2022
49 Nov 23, 2022
272 Dec 28, 2022
201 Dec 21, 2022
364 Dec 14, 2022
111 Dec 31, 2022
35 Dec 6, 2022
59 Dec 17, 2022
307 Jan 3, 2023
243 Dec 26, 2022
153 Dec 14, 2022
367 Dec 27, 2022
1k Dec 31, 2022
59 Sep 25, 2022
29 Sep 2, 2022
109 Dec 23, 2022
574 Jan 2, 2023
115 Dec 28, 2022