SEAM
The implementation of Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentaion.
You can also download the repository from https://gitee.com/hibercraft/SEAM
Abstract
Image-level weakly supervised semantic segmentation is a challenging problem that has been deeply studied in recentyears. Most of advanced solutions exploit class activation map (CAM). However, CAMs can hardly serve as the object mask due to the gap between full and weak supervisions. In this paper, we propose a self-supervised equivariant attention mechanism (SEAM) to discover additional supervision and narrow the gap. Our method is based on the observation that equivariance is an implicit constraint in fully supervised semantic segmentation, whose pixel-level labels take the same spatial transformation as the input images during data augmentation. However, this constraint is lost on the CAMs trained by image-level supervision. Therefore, we propose consistency regularization on predicted CAMs from various transformed images to provide self-supervision for network learning. Moreover, we propose a pixel correlation module (PCM), which exploits context appearance information and refines the prediction of current pixel by its similar neighbors, leading to further improvement on CAMs consistency. Extensive experiments on PASCAL VOC 2012 dataset demonstrate our method outperforms state-of-the-art methods using the same level of supervision.
Thanks to the work of jiwoon-ahn, the code of this repository borrow heavly from his AffinityNet repository, and we follw the same pipeline to verify the effectiveness of our SEAM.
Requirements
- Python 3.6
- pytorch 0.4.1, torchvision 0.2.1
- CUDA 9.0
- 4 x GPUs (12GB)
Usage
Installation
- Download the repository.
git clone https://github.com/YudeWang/SEAM.git
- Install python dependencies.
pip install -r requirements.txt
-
Download model weights from google drive or baidu cloud (with code 6nmo), including ImageNet pretrained models and our training results.
-
Download PASCAL VOC 2012 devkit (follow instructions in http://host.robots.ox.ac.uk/pascal/VOC/voc2012/#devkit). It is suggested to make a soft link toward downloaded dataset.
ln -s $your_dataset_path/VOCdevkit/VOC2012 VOC2012
- (Optional) The image-level labels have already been given in
voc12/cls_label.npy. If you want to regenerate it (which is unnecessary), please download the annotation of VOC 2012 SegmentationClassAug training set (containing 10582 images), which can be download here and place them all asVOC2012/SegmentationClassAug/xxxxxx.png. Then run the code
cd voc12
python make_cls_labels.py --voc12_root VOC2012
SEAM step
- SEAM training
python train_SEAM.py --voc12_root VOC2012 --weights $pretrained_model --session_name $your_session_name
- SEAM inference.
python infer_SEAM.py --weights $SEAM_weights --infer_list [voc12/val.txt | voc12/train.txt | voc12/train_aug.txt] --out_cam $your_cam_dir --out_crf $your_crf_dir
- SEAM step evaluation. We provide python mIoU evaluation script
evaluation.py, or you can use official development kit. Here we suggest to show the curve of mIoU with different background score.
python evaluation.py --list VOC2012/ImageSets/Segmentation/[val.txt | train.txt] --predict_dir $your_cam_dir --gt_dir VOC2012/SegmentationClass --comment $your_comments --type npy --curve True
Random walk step
The random walk step keep the same with AffinityNet repository.
- Train AffinityNet.
python train_aff.py --weights $pretrained_model --voc12_root VOC2012 --la_crf_dir $your_crf_dir_4.0 --ha_crf_dir $your_crf_dir_24.0 --session_name $your_session_name
- Random walk propagation
python infer_aff.py --weights $aff_weights --infer_list [voc12/val.txt | voc12/train.txt] --cam_dir $your_cam_dir --voc12_root VOC2012 --out_rw $your_rw_dir
- Random walk step evaluation
python evaluation.py --list VOC2012/ImageSets/Segmentation/[val.txt | train.txt] --predict_dir $your_rw_dir --gt_dir VOC2012/SegmentationClass --comment $your_comments --type png
Pseudo labels retrain
Pseudo label retrain on DeepLabv1. Code is available here.
Citation
Please cite our paper if the code is helpful to your research.
@InProceedings{Wang_2020_CVPR_SEAM,
author = {Yude Wang and Jie Zhang and Meina Kan and Shiguang Shan and Xilin Chen},
title = {Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation},
booktitle = {Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
year = {2020}
}
Reference
[1] J. Ahn and S. Kwak. Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
and we can also calculate the background as:
but, how can we use the two result to calculate the loss? how can we generate the ground truth? Is img(m, n) = c (the true label) the ground truth?
![cam_full_arr[k+1] = v out of bounds](https://avatars.githubusercontent.com/u/54871708?v=4)
120 Jan 5, 2023
83 Jan 4, 2023
99 Jan 6, 2023
224 Dec 28, 2022
231 Dec 27, 2022
111 Nov 27, 2022
595 Dec 27, 2022
62 Oct 10, 2022
323 Nov 10, 2022
84 Jan 5, 2022
215 Dec 7, 2022
646 Nov 11, 2022
933 Dec 29, 2022
1.1k Jan 2, 2023
2 Jul 14, 2022
684 Jan 6, 2023
81 Jan 1, 2023
3 Sep 2, 2022
81 Dec 1, 2022