Contrastive learning of Class-agnostic Activation Map for Weakly Supervised Object Localization and Semantic Segmentation (CVPR 2022)

Overview

CCAM (Unsupervised)

Code repository for our paper "CCAM: Contrastive learning of Class-agnostic Activation Map for Weakly Supervised Object Localization and Semantic Segmentation" in CVPR 2022.

The repository includes full training, evaluation, and visualization codes on CUB-200-2011, ILSVRC2012, and PASCAL VOC2012 datasets.

We provide the extracted class-agnostic bounding boxes (on CUB-200-2011 and ILSVRC2012) and background cues (on PASCAL VOC12) at here.

Dependencies

  • Python 3
  • PyTorch 1.7.1
  • OpenCV-Python
  • Numpy
  • Scipy
  • MatplotLib
  • Yaml
  • Easydict

Dataset

CUB-200-2011

You will need to download the images (JPEG format) in CUB-200-2011 dataset at here. Make sure your data/CUB_200_2011 folder is structured as follows:

├── CUB_200_2011/
|   ├── images
|   ├── images.txt
|   ├── bounding_boxes.txt
|   ...
|   └── train_test_split.txt

You will need to download the images (JPEG format) in ILSVRC2012 dataset at here. Make sure your data/ILSVRC2012 folder is structured as follows:

ILSVRC2012

├── ILSVRC2012/ 
|   ├── train
|   ├── val
|   ├── val_boxes
|   |   ├——val
|   |   |   ├—— ILSVRC2012_val_00050000.xml
|   |   |   ├—— ...
|   ├── train.txt
|   └── val.txt

PASCAL VOC2012

You will need to download the images (JPEG format) in PASCAL VOC2012 dataset at here. Make sure your data/VOC2012 folder is structured as follows:

├── VOC2012/
|   ├── Annotations
|   ├── ImageSets
|   ├── SegmentationClass
|   ├── SegmentationClassAug
|   └── SegmentationObject

For WSOL task

please refer to the directory of './WSOL'

cd WSOL

For WSSS task

please refer to the directory of './WSSS'

cd WSSS

Comparison with CAM

CUSTOM DATASET

As CCAM is an unsupervised method, it can be applied to various scenarios, like ReID, Saliency detection, or skin lesion detection. We provide an example to apply CCAM on your custom dataset like 'Market-1501'.

cd CUSTOM

Reference

If you are using our code, please consider citing our paper.

@article{xie2022contrastive,
  title={Contrastive learning of Class-agnostic Activation Map for Weakly Supervised Object Localization and Semantic Segmentation},
  author={Xie, Jinheng and Xiang, Jianfeng and Chen, Junliang and Hou, Xianxu and Zhao, Xiaodong and Shen, Linlin},
  journal={arXiv preprint arXiv:2203.13505},
  year={2022}
}
Comments
  • Questions about the experiments

    Questions about the experiments

    Hello! Thank you for sharing this very well organized code. Additionally, I wonder if you could clarify a few points for me:

    • F.relu (used in inference_CCAM.py) is commonly used during inference of multi-class CAMs to squash negatively contributing pixels and unrelated pixels together. However, in a binary case, the 0 means uncertain if background or foreground, and it does give you some additional information, no? Why did you choose to go like this, instead of simply checking ccam > 0.0 or taking the sigmoid function and checking for sigmoid(ccam) > 0.5?
    • a) Which hyper-parameters did you use to train PoolNet? Was all of them set to default? b) Did you produce the .lst file yourself, pointing out to the locations of the saliency maps produced by inference_CRF.py? c) Did you use only images/pseudo saliency maps in train.txt, or all files listed in train_aug.txt?
    • I noticed CCAM has a negative effect on the IoU of classes chair, diningtable and sofa (image). From these, only chair's IoU degeneration is reported in the paper, but not discussed. I guess you ran out of space? Do you have any insights on why that happened?

    Thank you very much once again, cheers.

    image

    opened by lucasdavid 4
  • For WSSS task

    For WSSS task

    Thanks for your working! But when i runing the third step of the WSSS task , there is 'No module named 'core.puzzle_utils''. i can't find this module in the 'core' folder. Please help me.

    opened by YYDS-cc 3
  • AttributeError: 'AxesSubplot' object has no attribute 'shape'

    AttributeError: 'AxesSubplot' object has no attribute 'shape'

    Sorry to bother you again.When i use train_CCAM.py in CUSTOM to train a model using custom data, a got an EEOR:

    `/data/yunzi/CCAM/CUSTOM$ OMP_NUM_THREADS=16 CUDA_VISIBLE_DEVICES=5 python train_CCAM.py --tag CCAM_Maket1501_MOCO --batch_size 1 --pretrained mocov2 --alpha 0.25 [i] CCAM_Maket1501_MOCO

    [i] mean values is [0.485, 0.456, 0.406] [i] std values is [0.229, 0.224, 0.225] [i] train_transform is Compose( Resize(size=(256, 128), interpolation=bilinear, max_size=None, antialias=None) RandomHorizontalFlip(p=0.5) ToTensor() Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)) ) [i] #train data

    [i] log_iteration : 2,587 [i] val_iteration : 12,936 [i] max_iteration : 129,360 Loading unsupervised mocov2 pretrained parameters!

    [i] Architecture is resnet50 [i] Total Params: 23.54M

    Traceback (most recent call last): File "train_CCAM.py", line 229, in visualize_heatmap(args.tag, images.clone().detach(), ccam, 0, iteration) File "/data/yunzi/CCAM/CUSTOM/utils.py", line 49, in visualize_heatmap for j in range(axes.shape[0]): AttributeError: 'AxesSubplot' object has no attribute 'shape'`

    Do you have the same problem?

    opened by roseyunzi 2
  • ModuleNotFoundError: No module named 'cmapy'

    ModuleNotFoundError: No module named 'cmapy'

    Hi,thanks for the work. But when i run/data/yunzi/CCAM/WSOL$ OMP_NUM_THREADS=16 CUDA_VISIBLE_DEVICES=0 python train_CCAM_CUB.py --experiment CCAM_CUB_MOCO --lr 0.0001 --batch_size 16 --pretrained mocov2 --alpha 0.75

    I got a error :ModuleNotFoundError: No module named 'cmapy' where can i import this file?

    opened by roseyunzi 2
  • In SimMaxLoss, why rank=rank-1 ?

    In SimMaxLoss, why rank=rank-1 ?

    class SimMaxLoss(nn.Module): def init(self, metric='cos', alpha=0.25, reduction='mean'): super(SimMaxLoss, self).init() self.metric = metric self.alpha = alpha self.reduction = reduction

    def forward(self, embedded_bg):
        """
        :param embedded_fg: [N, C]
        :param embedded_bg: [N, C]
        :return:
        """
        if self.metric == 'l2':
            raise NotImplementedError
    
        elif self.metric == 'cos':
            sim = cos_simi(embedded_bg, embedded_bg)  #(N,N)
            loss = -torch.log(sim)
            loss[loss < 0] = 0
            _, indices = sim.sort(descending=True, dim=1)
            _, rank = indices.sort(dim=1)
            rank = rank - 1
            rank_weights = torch.exp(-rank.float() * self.alpha)
            loss = loss * rank_weights
        else:
            raise NotImplementedError
    
        if self.reduction == 'mean':
            return torch.mean(loss)
        elif self.reduction == 'sum':
            return torch.sum(loss)
    
    opened by YingLv1106 3
  • Top-1 Loc, Top-5 Loc and backbone( DenseNet161 and EfficientNet-B7 )

    Top-1 Loc, Top-5 Loc and backbone( DenseNet161 and EfficientNet-B7 )

    Hello! First of all, you did a great job! Congratulations! Second, I have three questions I'd like to ask you about running the code.

    1. First, in the case of WSOL, I didn't find the test file test.py in the code. Is the result on the CUB-200-2011 test set and imagenet-1k validation set mentioned in this paper calculated by def test in train_ccam_cub. py? 屏幕截图 2022-07-25 171210

    2. Secondly, the three evaluation Top-1 Loc, Top-5 Loc mentioned in the paper were not found in the code to calculate Top-1 Loc and Top-5 Loc, so there was no Top-1 Loc and Top-5 Loc in the running result of my CUB-200-2011 dataset. How to solve this problem? 屏幕截图 2022-07-25 171357(1)

    3. Thirdly, the C2AM(Ours) in Table1 uses backbone DenseNet161 and Efficientnet-B7. I did not find the part of the code downloading and replacing these two backbone networks in the code, so that the results in C2AM(Ours) in Table1 could not be reproduced. How to solve this problem? 屏幕截图 2022-07-25 171357(2)

    opened by rjy-fighting 15
  • the  backbone of CAM is shared witch CCAM?

    the backbone of CAM is shared witch CCAM?

    the backbone of CAM and CCAM is the same or different one? when i use the ccam to refine the CAM, should i start a new network to generate the ccam separately?

    opened by YuYue26 13
Owner
Computer Vision Insitute, SZU
Computer Vision Insitute, Shenzhen University
Computer Vision Insitute, SZU
Codes for TS-CAM: Token Semantic Coupled Attention Map for Weakly Supervised Object Localization.

TS-CAM: Token Semantic Coupled Attention Map for Weakly SupervisedObject Localization This is the official implementaion of paper TS-CAM: Token Semant

vasgaowei 110 Nov 7, 2022
PyTorch implementation of ''Background Activation Suppression for Weakly Supervised Object Localization''.

Background Activation Suppression for Weakly Supervised Object Localization PyTorch implementation of ''Background Activation Suppression for Weakly S

null 32 Nov 24, 2022
Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentation, CVPR 2018

Learning Pixel-level Semantic Affinity with Image-level Supervision This code is deprecated. Please see https://github.com/jiwoon-ahn/irn instead. Int

Jiwoon Ahn 336 Nov 30, 2022
Many Class Activation Map methods implemented in Pytorch for CNNs and Vision Transformers. Including Grad-CAM, Grad-CAM++, Score-CAM, Ablation-CAM and XGrad-CAM

Class Activation Map methods implemented in Pytorch pip install grad-cam ⭐ Tested on many Common CNN Networks and Vision Transformers. ⭐ Includes smoo

Jacob Gildenblat 6.4k Dec 7, 2022
Weakly Supervised Dense Event Captioning in Videos, i.e. generating multiple sentence descriptions for a video in a weakly-supervised manner.

WSDEC This is the official repo for our NeurIPS paper Weakly Supervised Dense Event Captioning in Videos. Description Repo directories ./: global conf

Melon(Xuguang Duan) 96 Nov 1, 2022
Normalization Matters in Weakly Supervised Object Localization (ICCV 2021)

Normalization Matters in Weakly Supervised Object Localization (ICCV 2021) 99% of the code in this repository originates from this link. ICCV 2021 pap

Jeesoo Kim 10 Feb 1, 2022
Anti-Adversarially Manipulated Attributions for Weakly and Semi-Supervised Semantic Segmentation (CVPR 2021)

Anti-Adversarially Manipulated Attributions for Weakly and Semi-Supervised Semantic Segmentation Input Image Initial CAM Successive Maps with adversar

Jungbeom Lee 110 Dec 7, 2022
Code for the paper One Thing One Click: A Self-Training Approach for Weakly Supervised 3D Semantic Segmentation, CVPR 2021.

One Thing One Click One Thing One Click: A Self-Training Approach for Weakly Supervised 3D Semantic Segmentation (CVPR2021) Code for the paper One Thi

null 43 Nov 30, 2022
[CVPR'22] Weakly Supervised Semantic Segmentation by Pixel-to-Prototype Contrast

wseg Overview The Pytorch implementation of Weakly Supervised Semantic Segmentation by Pixel-to-Prototype Contrast. [arXiv] Though image-level weakly

Ye Du 93 Nov 30, 2022
Hybrid CenterNet - Hybrid-supervised object detection / Weakly semi-supervised object detection

Hybrid-Supervised Object Detection System Object detection system trained by hybrid-supervision/weakly semi-supervision (HSOD/WSSOD): This project is

null 4 Mar 21, 2022
Rational Activation Functions - Replacing Padé Activation Units

Rational Activations - Learnable Rational Activation Functions First introduce as PAU in Padé Activation Units: End-to-end Learning of Activation Func

ml-research@TUDarmstadt 38 Nov 22, 2022
FactSeg: Foreground Activation Driven Small Object Semantic Segmentation in Large-Scale Remote Sensing Imagery (TGRS)

FactSeg: Foreground Activation Driven Small Object Semantic Segmentation in Large-Scale Remote Sensing Imagery by Ailong Ma, Junjue Wang*, Yanfei Zhon

Kingdrone 42 Dec 7, 2022
The official implementation of CVPR 2021 Paper: Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation.

Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation This repository is the official implementation of CVPR 2021 paper:

null 9 Nov 14, 2022
Code for weakly supervised segmentation of a single class

SingleClassRL Implementation of weak single object segmentation from paper "Regularized Loss for Weakly Supervised Single Class Semantic Segmentation"

null 16 Nov 14, 2022
Official Pytorch Implementation of 'Learning Action Completeness from Points for Weakly-supervised Temporal Action Localization' (ICCV-21 Oral)

Learning-Action-Completeness-from-Points Official Pytorch Implementation of 'Learning Action Completeness from Points for Weakly-supervised Temporal A

Pilhyeon Lee 64 Nov 8, 2022
DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervision

The Official PyTorch Implementation of DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervision

Shiyi Lan 3 Oct 15, 2021
The PyTorch implementation of DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervision.

DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervision The PyTorch implementation of DiscoBox: Weakly Supe

Shiyi Lan 1 Oct 23, 2021
[CVPR 2022] Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels

Using Unreliable Pseudo Labels Official PyTorch implementation of Semi-Supervised Semantic Segmentation Using Unreliable Pseudo Labels, CVPR 2022. Ple

Haochen Wang 262 Nov 30, 2022
Scribble-Supervised LiDAR Semantic Segmentation, CVPR 2022 (ORAL)

Scribble-Supervised LiDAR Semantic Segmentation Dataset and code release for the paper Scribble-Supervised LiDAR Semantic Segmentation, CVPR 2022 (ORA

null 97 Dec 5, 2022