Complementary Patch for Weakly Supervised Semantic Segmentation, ICCV21 (poster)

Last update: Dec 12, 2022

Related tags

Overview

CPN (ICCV2021)

This is an implementation of Complementary Patch for Weakly Supervised Semantic Segmentation, which is accepted by ICCV2021 poster.

This implementation is based on SEAM and IRN.

Abstract

Weakly Supervised Semantic Segmentation (WSSS) based on image-level labels has been greatly advanced by exploiting the outputs of Class Activation Map (CAM) to generate the pseudo labels for semantic segmentation. However, CAM merely discovers seeds from a small number of regions, which may be insufficient to serve as pseudo masks for semantic segmentation. In this paper, we formulate the expansion of object regions in CAM as an increase in information. From the perspective of information theory, we propose a novel Complementary Patch (CP) Representation and prove that the information of the sum of the CAMs by a pair of input images with complementary hidden (patched) parts, namely CP Pair, is greater than or equal to the information of the baseline CAM. Therefore, a CAM with more information related to object seeds can be obtained by narrowing down the gap between the sum of CAMs generated by the CP Pair and the original CAM. We propose a CP Network (CPN) implemented by a triplet network and three regularization functions. To further improve the quality of the CAMs, we propose a Pixel-Region Correlation Module (PRCM) to augment the contextual information by using object-region relations between the feature maps and the CAMs. Experimental results on the PASCAL VOC 2012 datasets show that our proposed method achieves a new state-of-the-art in WSSS, validating the effectiveness of our CP Representation and CPN.

Prerequisite

The requirements are in requirements.txt. However, the settings are not limited to it (CUDA 11.0, Pytorch 1.7 for one RTX3090). Besides,the batch size could be even larger like 8 or 16 if you have sufficient GPU resources, which you may get higher performance than the paper reported.
The pretrained_weight for the initialization of ResNet38 and well-trained CPN is here in BaiDuYun, and the code is y6h4, or you could find them in Google Drive, which is here.
PASCAL VOC 2012 devkit with expanded version, which includes 10582 training samples.

Usage

Train the CPN to obtain the weight, which will be saved in "CPN/CPN". Remember to set the VOC12 and pre-trained weight path in the script.
```
python train_cpn.py
```
Generate the foreground seeds of CAM (without background) using the weight or the well-trained CPN, the results is in out_cam.
```
python infer_cam.py 
```
Evaluate the CAM by selecting the background. Remember to set the data path of VOC in this script.
```
python evaluation_cam.py
```

Implementation of results in paper

I suggest to use the IRN and the for the second expansion of the CAM. Although you can directly use the old version of AffinityNet, you may take long time to find the parameters to generate the CAM that reaches the reported performance. You can directly use the well-trained weights from IRN to generated the mask for segmentation.
For the segmentation model, we use the DeepLab here.

Acknowledgement

Great thanks to the code of the SEAM and IRN.

Comments

Result is not the same as the paper. Is this still under construction?

Hi, I've tested the val set by your pretrained model. But I think the threshold is not set? And the result is about 55 mIoU. So is this repo still under construction? Which threshold should I set? Thanks very much!

opened by ZhangYushan3 7
Why I get a very lower score based on the advice set?

I get the mIOU score only have 15.156095005115391, using the train_aug dataset training and using validation dataset to evaluate it. But the SEAM can get a high score just as the paper description.

opened by Jamie-obj 5
mxnet version

Thank you very much for your work, your code is very helpful to me. When I use your code for training, the following errors occurs: I guess it's due to the incompatibility of the MXNET I installed with yours. Could you tell me what version of MXNET is on your machine?

opened by bbb-x1 2
Got some trouble in grid_strategy

Does hide_patch really work? It has two options to cut the image into patches of size 224 or 448. I got several questions. first: Images are of different sizes in the voc12 dataset，When the patch_size is 224, only four image patches can be obtained for per image, which is not same as shown in Figure 1. And there are no other sizes here. And the second one is 'grid_size = grid_sizes[random.randint(0, len(grid_sizes) - 1)]' could not reach the result of random at all.

opened by lions-heart 1
some question about CAM

I`m a fresher in CV. So there's a question about cam.Is it the final segmentation result？Is the final evaluation result calculated by cam and ground truth?

opened by lions-heart 1
Asking for help
Hi! Thank you for your code! But I have some problems with the code.

In the step of train the CPN, I don't understand the meaning of assert args in the cpn file network == "network.resnet38 cls" code at line 89 of the train_cpn.py。 About ilsvrc-cls_rna-a1_cls1000_ep-0001.params not the initial weight file for train resnet38 _CPN? Or should I first train network.resnet38_cls with ilsvrc-cls_rna-a1_cls1000_ep-0001.params to get xxx.pth, and then use the pth to train resnet38_CPN?

Is it convenient for you to upload code that the expansion of the CAM using IRN to GitHub？ I would be very grateful！！！😊
opened by luckyjjj29 0

Owner

Ferenas

Fly higher, everyone will see it

GitHub

A Comprehensive Analysis of Weakly-Supervised Semantic Segmentation in Different Image Domains (IJCV submission)

wsss-analysis The code of: A Comprehensive Analysis of Weakly-Supervised Semantic Segmentation in Different Image Domains, arXiv pre-print 2019 paper.

48 Dec 18, 2022

Context Decoupling Augmentation for Weakly Supervised Semantic Segmentation

Context Decoupling Augmentation for Weakly Supervised Semantic Segmentation The code of: Context Decoupling Augmentation for Weakly Supervised Semanti

54 Dec 12, 2022

Discriminative Region Suppression for Weakly-Supervised Semantic Segmentation

Discriminative Region Suppression for Weakly-Supervised Semantic Segmentation (AAAI 2021) Official pytorch implementation of our paper: Discriminative

74 Dec 27, 2022

Anti-Adversarially Manipulated Attributions for Weakly and Semi-Supervised Semantic Segmentation (CVPR 2021)

Anti-Adversarially Manipulated Attributions for Weakly and Semi-Supervised Semantic Segmentation Input Image Initial CAM Successive Maps with adversar

110 Dec 7, 2022

Code for the paper One Thing One Click: A Self-Training Approach for Weakly Supervised 3D Semantic Segmentation, CVPR 2021.

One Thing One Click One Thing One Click: A Self-Training Approach for Weakly Supervised 3D Semantic Segmentation (CVPR2021) Code for the paper One Thi

44 Dec 12, 2022

DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervision

The Official PyTorch Implementation of DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervision

3 Oct 15, 2021

Reducing Information Bottleneck for Weakly Supervised Semantic Segmentation (NeurIPS 2021)

Reducing Information Bottleneck for Weakly Supervised Semantic Segmentation (NeurIPS 2021) The implementation of Reducing Infromation Bottleneck for W

81 Dec 16, 2022

The PyTorch implementation of DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervision.

DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervision The PyTorch implementation of DiscoBox: Weakly Supe

1 Oct 23, 2021

Perturbed Self-Distillation: Weakly Supervised Large-Scale Point Cloud Semantic Segmentation (ICCV2021)

Perturbed Self-Distillation: Weakly Supervised Large-Scale Point Cloud Semantic Segmentation (ICCV2021) This is the implementation of PSD (ICCV 2021),

12 Dec 12, 2022

Contrastive learning of Class-agnostic Activation Map for Weakly Supervised Object Localization and Semantic Segmentation (CVPR 2022)

CCAM (Unsupervised) Code repository for our paper "CCAM: Contrastive learning of Class-agnostic Activation Map for Weakly Supervised Object Localizati

113 Dec 27, 2022

[CVPR'22] Weakly Supervised Semantic Segmentation by Pixel-to-Prototype Contrast

wseg Overview The Pytorch implementation of Weakly Supervised Semantic Segmentation by Pixel-to-Prototype Contrast. [arXiv] Though image-level weakly

96 Dec 30, 2022

Patch Rotation: A Self-Supervised Auxiliary Task for Robustness and Accuracy of Supervised Models

Patch-Rotation(PatchRot) Patch Rotation: A Self-Supervised Auxiliary Task for Robustness and Accuracy of Supervised Models Submitted to Neurips2021 To

4 Jul 12, 2021

HyperSeg: Patch-wise Hypernetwork for Real-time Semantic Segmentation Official PyTorch Implementation

: We present a novel, real-time, semantic segmentation network in which the encoder both encodes and generates the parameters (weights) of the decoder. Furthermore, to allow maximal adaptivity, the weights at each decoder block vary spatially. For this purpose, we design a new type of hypernetwork, composed of a nested U-Net for drawing higher level context features

182 Dec 14, 2022

A pytorch-version implementation codes of paper: "BSN++: Complementary Boundary Regressor with Scale-Balanced Relation Modeling for Temporal Action Proposal Generation"

BSN++: Complementary Boundary Regressor with Scale-Balanced Relation Modeling for Temporal Action Proposal Generation A pytorch-version implementation

11 Oct 8, 2022