Discriminative Region Suppression for Weakly-Supervised Semantic Segmentation

Related tags

Deep Learning DRS
Overview

Discriminative Region Suppression for Weakly-Supervised Semantic Segmentation (AAAI 2021)

Official pytorch implementation of our paper: Discriminative Region Suppression for Weakly-Supervised Semantic Segmentation [Paper], Beomyoung Kim, Sangeun Han, and Junmo Kim, AAAI 2021

PWC PWC

We propose the discriminative region suppression (DRS) module that is a simple yet effective method to expand object activation regions. DRS suppresses the attention on discriminative regions and spreads it to adjacent non-discriminative regions, generating dense localization maps.

DRS module

Setup

  1. Dataset Preparing

    # dataset structure
    VOC2012/
        --- Annotations/
        --- ImageSets/
        --- JPEGImages/
        --- SegmentationClassAug/
        --- saliency_map/
        --- refined_pseudo_segmentation_labels/
    
  2. Requirements pip install -r requirements.txt

Training & Pseudo Segmentation Labels Generation

  • step1 : training the classifier with DRS modules
  • step2 : training the refinement network for the localization maps refinement
  • step3 : pseudo segmentation labels generation
# all-in-one
bash run.sh 
Model pretrained
VGG-16 with the learnable DRS DRS_learnable/best.pth
Refinement network Refine_DRS_learnable/best.pth
Pseudo Segmentation Labels refined_pseudo_segmentation_labels/

Training the DeepLab-V2 using pseudo labels

We adopt the DeepLab-V2 pytorch implementation from https://github.com/kazuto1011/deeplab-pytorch.

cd DeepLab-V2-PyTorch/

# motify the dataset path (DATASET.ROOT)
vi configs/voc12.yaml

# 1. training the DeepLab-V2 using pseudo labels
bash train.sh

# 2. evaluation the DeepLab-V2
bash eval.sh
Model mIoU mIoU + CRF pretrained
DeepLab-V2 with ResNet-101 69.4% 70.4% [link]
  • Note that the pretrained weight path ./DeepLab-V2-Pytorch/data/models/Deeplabv2_pseudo_segmentation_labels/deeplabv2_resnet101_msc/train_cls/checkpoint_final.pth
  • According to the DeepLab-V2 pytorch implementation we used, we requires an initial weights [download].

Citation

We hope that you find this work useful. If you would like to acknowledge us, please, use the following citation:

@inproceedings{kim2021discriminative,
    title={Discriminative Region Suppression for Weakly-Supervised Semantic Segmentation},
    author={Kim, Beomyoung and Han, Sangeun and Kim, Junmo},
    year={2021},
    booktitle={AAAI Conference on Artificial Intelligence},
}
Comments
  • About mIOU

    About mIOU

    Thank your brilliant Idea! I've trained a deeplabv3+ by your official pseudo labels and your network. However, I evaluate the mIOU of val_cls.txt that only achieve about 67.1 without dCRF. All of the hyperparameter is arranged to such as yours. I wanna know why my mIOU is lower than yours. Thank your for your reply. Best wishes with you.

    opened by callmelaoyi 12
  • Model in ResNet50

    Model in ResNet50

    Hi, thanks for your great work! I was wondering if DRS could be adapted in RESNET50. Have you ever tried it on ResNet? I have tried DRS in some layers buy found limited improvement. Do you have any suggestion?

    opened by Ferenas 4
  • Questions about DeeplabV2 pretrained weight

    Questions about DeeplabV2 pretrained weight

    Greetings,

    Thank you for your code and paper, they give us a novel way to process the relationship with most descriptive part and the other object part. When I read your code, I find some question, would you answer them for me? I find that the pretrained link in your deeplabv2 table turn into the download for a checkpoint file with name "final checkpoint.pth", is that truly the pretrained model for deeplabv2 or just the final checkpoint of your deeplabv2 training? Another reason let me think like this is that I find the pretrained path in your config file is "data/models/coco/deeplabv1_resnet101/caffemodel/deeplabv1_resnet101-coco.pth". Is it the same checkpoint file with the "checkpoint.pth"?

    opened by Unrealluver 4
  • Question on performance gap

    Question on performance gap

    Hi, Thank you for your code and paper. I've already tried your code, but I cannot achieve the same performance as the paper. Would you please help me figure out where the problem is? I retrained DeepLabV2 with your shared refined pseudo segmentation labels. But the performance on val only achieves 64.8 and 65.7(after crf). All hyper-parameters follow your provided code. Could you please point out the possible mistake? Thank you!

    opened by ghost 3
  • The performance is lower than reported ?

    The performance is lower than reported ?

    Amazing Work! When I used the provided code and followed the method in your paper, I only got val 69.45 with learnable DRS. How can I get the same performance as reported in your paper?

    opened by SunGuoYing 3
  • Confused about Training the DeepLab-V3+

    Confused about Training the DeepLab-V3+

    Thank you for a great job! "In contrast, DeepLab-V3 does not require the COCO-pretrained weight due to the recent large memory GPUs and Synchronized BatchNorm" confused me. Is deeplab-V3+ is trained from scratch without any pre-trained weight? Then I have a question about the link. The link of Deeplab-V3+ with ResNet-101 is pre-trained weight? or it's the model to test and get the results, 71%.

    Model | mIoU | mIoU + CRF | pretrained -- | -- | -- | -- DeepLab-V2 with ResNet-101 | 69.4% | 70.4% | [link] DeepLab-V3+ with ResNet-101 | 70.4% | 71.0% | [link]

    I'm looking forward to hearing from you.
    opened by xinyuaning 2
  • What is the 'refined_pseudo_segmentation_labels/' in data structure

    What is the 'refined_pseudo_segmentation_labels/' in data structure

    Hi qjadud,

    I've got all the other data in the data structure shown in README file but I can not find the dir name 'refined_pseudo_segmentation_labels/'. Would you mind help me with it?

    opened by LucasZhan 2
  • The pretrained network for Deeplabv2

    The pretrained network for Deeplabv2

    As the pretrained network you provide for deeplab v2, https://drive.google.com/file/d/1Wj8Maj9KGQgwtDfvIp8FChsdAIgDvliT/view, the name of it is deeplabv1_resnet101-coco.pth, is it correct?

    opened by Zhengyang1995 1
  • White regions in 'Pseudo Segmentation Labels'

    White regions in 'Pseudo Segmentation Labels'

    Hey, thanks for your work. When I see your result in Pseudo Segmentation Labels, I found there are some white regions, which I guess is generated because of the saliency. Since these regions belong to no categories, what did you do for it when you train a fully supervised segmentation network, like Deeplab2 or 3? Does these regions regarded as background or categories nearby?

    opened by Zhengyang1995 1
  • Question about validate() function in train_cls.py

    Question about validate() function in train_cls.py

    Hey, thank you so much for your interesting work. When I run with train_cls.py, there is a error occurs in validate() function. In the last of the function, mIoU is made between pred_map and gt_map. The pred map has the size of HW1, and each pixel value in the map shows the prediction labels. However, the gt_map is HW3, which is the map read directly from the 'SegmentationAug' dir, it is a RGB image. Could you please tell the convert method from the RGB into the Seg label? Thank you so much for your time.

    opened by Zhengyang1995 1
  • CVE-2007-4559 Patch

    CVE-2007-4559 Patch

    Patching CVE-2007-4559

    Hi, we are security researchers from the Advanced Research Center at Trellix. We have began a campaign to patch a widespread bug named CVE-2007-4559. CVE-2007-4559 is a 15 year old bug in the Python tarfile package. By using extract() or extractall() on a tarfile object without sanitizing input, a maliciously crafted .tar file could perform a directory path traversal attack. We found at least one unsantized extractall() in your codebase and are providing a patch for you via pull request. The patch essentially checks to see if all tarfile members will be extracted safely and throws an exception otherwise. We encourage you to use this patch or your own solution to secure against CVE-2007-4559. Further technical information about the vulnerability can be found in this blog.

    If you have further questions you may contact us through this projects lead researcher Kasimir Schulz.

    opened by TrellixVulnTeam 0
  • The performance of CAM

    The performance of CAM

    Hey, maybe this is question is not strongly related to this proj, but I will appreciate a lot if you could provide any advice for me. The question is, do you have any idea about the miou performance for a classification network based on vgg (13 layers as backbone and with other 3 extra_conv). When I try to reproduce your code ( The first step, train_cls.py), I remove the saliency and computing the background by a given threshold. I find that the miou drops. This is acceptable since saliency provides much useful information. But when I try to remove other module in the vgg.py, to make it as a classification network, the best CAM I could get is among 42. You could find that in PSA, same backbone network(vgg16) could reach 48+. At first, I wonder it is caused by the different parameters in the last three conv layers, but when I keep them as the same, I still can't reach 48. Do you have any idea about the limitation? If so, I will appreciate a lot for your sincere help!

    opened by Zhengyang1995 0
  • A question about  about the validation code

    A question about about the validation code

    Thank you for providing such a convenient code, but I have some questions about it. On the line 75 of eval.py, you directly use the image after crop for performance verification. Is this accurate? Another question is why not apply ms+flip to inference?

    opened by Italy2006 5
  • Questions about Deeplab retrain part

    Questions about Deeplab retrain part

    Greetings!

    I read your paper again and again recently, and I really learned a lot from your work. For the retrain part, I met some problems, would you like to help to answer?

    At first, I noticed that your results in Table.4 were devided into 3 parts, the subtitle of first part is Segmentation Network : DeepLab-Large-FOV (VGG-16), the 2st part is Segmentation Network : DeepLab-Large-FOV (ResNet-101), the 3rd part is Segmentation Network : DeepLab-ASPP (ResNet-101).

    My first question is that your deeplab retrained script(/DRS/DeepLab-V2-PyTorch/train.sh) indicate which part in Table.4 we mentioned before? What about other two parts' code? I could not find a VGG-Backboned model in DeepLab-V2-PyTorch folder.

    opened by Unrealluver 2
Owner
Beom
Computer Vision & Deep Learning
Beom
PyTorch implementation of ''Background Activation Suppression for Weakly Supervised Object Localization''.

Background Activation Suppression for Weakly Supervised Object Localization PyTorch implementation of ''Background Activation Suppression for Weakly S

null 35 Jan 6, 2023
Weakly Supervised Dense Event Captioning in Videos, i.e. generating multiple sentence descriptions for a video in a weakly-supervised manner.

WSDEC This is the official repo for our NeurIPS paper Weakly Supervised Dense Event Captioning in Videos. Description Repo directories ./: global conf

Melon(Xuguang Duan) 96 Nov 1, 2022
Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentation, CVPR 2018

Learning Pixel-level Semantic Affinity with Image-level Supervision This code is deprecated. Please see https://github.com/jiwoon-ahn/irn instead. Int

Jiwoon Ahn 337 Dec 15, 2022
[ICCV 2021] Official Pytorch implementation for Discriminative Region-based Multi-Label Zero-Shot Learning SOTA results on NUS-WIDE and OpenImages

Discriminative Region-based Multi-Label Zero-Shot Learning (ICCV 2021) [arXiv][Project page >> coming soon] Sanath Narayan*, Akshita Gupta*, Salman Kh

Akshita Gupta 54 Nov 21, 2022
[ICCV 2021] Official Pytorch implementation for Discriminative Region-based Multi-Label Zero-Shot Learning SOTA results on NUS-WIDE and OpenImages

Discriminative Region-based Multi-Label Zero-Shot Learning (ICCV 2021) [arXiv][Project page >> coming soon] Sanath Narayan*, Akshita Gupta*, Salman Kh

Akshita Gupta 54 Nov 21, 2022
A Comprehensive Analysis of Weakly-Supervised Semantic Segmentation in Different Image Domains (IJCV submission)

wsss-analysis The code of: A Comprehensive Analysis of Weakly-Supervised Semantic Segmentation in Different Image Domains, arXiv pre-print 2019 paper.

Lyndon Chan 48 Dec 18, 2022
Context Decoupling Augmentation for Weakly Supervised Semantic Segmentation

Context Decoupling Augmentation for Weakly Supervised Semantic Segmentation The code of: Context Decoupling Augmentation for Weakly Supervised Semanti

null 54 Dec 12, 2022
Anti-Adversarially Manipulated Attributions for Weakly and Semi-Supervised Semantic Segmentation (CVPR 2021)

Anti-Adversarially Manipulated Attributions for Weakly and Semi-Supervised Semantic Segmentation Input Image Initial CAM Successive Maps with adversar

Jungbeom Lee 110 Dec 7, 2022
Code for the paper One Thing One Click: A Self-Training Approach for Weakly Supervised 3D Semantic Segmentation, CVPR 2021.

One Thing One Click One Thing One Click: A Self-Training Approach for Weakly Supervised 3D Semantic Segmentation (CVPR2021) Code for the paper One Thi

null 44 Dec 12, 2022
DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervision

The Official PyTorch Implementation of DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervision

Shiyi Lan 3 Oct 15, 2021
Reducing Information Bottleneck for Weakly Supervised Semantic Segmentation (NeurIPS 2021)

Reducing Information Bottleneck for Weakly Supervised Semantic Segmentation (NeurIPS 2021) The implementation of Reducing Infromation Bottleneck for W

Jungbeom Lee 81 Dec 16, 2022
The PyTorch implementation of DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervision.

DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervision The PyTorch implementation of DiscoBox: Weakly Supe

Shiyi Lan 1 Oct 23, 2021
Perturbed Self-Distillation: Weakly Supervised Large-Scale Point Cloud Semantic Segmentation (ICCV2021)

Perturbed Self-Distillation: Weakly Supervised Large-Scale Point Cloud Semantic Segmentation (ICCV2021) This is the implementation of PSD (ICCV 2021),

null 12 Dec 12, 2022
Contrastive learning of Class-agnostic Activation Map for Weakly Supervised Object Localization and Semantic Segmentation (CVPR 2022)

CCAM (Unsupervised) Code repository for our paper "CCAM: Contrastive learning of Class-agnostic Activation Map for Weakly Supervised Object Localizati

Computer Vision Insitute, SZU 113 Dec 27, 2022
[CVPR'22] Weakly Supervised Semantic Segmentation by Pixel-to-Prototype Contrast

wseg Overview The Pytorch implementation of Weakly Supervised Semantic Segmentation by Pixel-to-Prototype Contrast. [arXiv] Though image-level weakly

Ye Du 96 Dec 30, 2022
OrienMask: Real-time Instance Segmentation with Discriminative Orientation Maps

OrienMask This repository implements the framework OrienMask for real-time instance segmentation. It achieves 34.8 mask AP on COCO test-dev at the spe

null 45 Dec 13, 2022
[CVPR22] Official codebase of Semantic Segmentation by Early Region Proxy.

RegionProxy Figure 2. Performance vs. GFLOPs on ADE20K val split. Semantic Segmentation by Early Region Proxy Yifan Zhang, Bo Pang, Cewu Lu CVPR 2022

Yifan 54 Nov 29, 2022
A Robust Non-IoU Alternative to Non-Maxima Suppression in Object Detection

Confluence: A Robust Non-IoU Alternative to Non-Maxima Suppression in Object Detection 1. 介绍 用以替代 NMS,在所有 bbox 中挑选出最优的集合。 NMS 仅考虑了 bbox 的得分,然后根据 IOU 来

null 44 Sep 15, 2022
Official implementation of Protected Attribute Suppression System, ICCV 2021

Official implementation of Protected Attribute Suppression System, ICCV 2021

Prithviraj Dhar 6 Jan 1, 2023