Semi-Supervised Semantic Segmentation via Adaptive Equalization Learning, NeurIPS 2021 (Spotlight)

Related tags

Deep Learning semi-supervised-learning semantic-segmentation neurips-2021
Overview

Semi-Supervised Semantic Segmentation via Adaptive Equalization Learning, NeurIPS 2021 (Spotlight)

Abstract

Due to the limited and even imbalanced data, semi-supervised semantic segmentation tends to have poor performance on some certain categories, e.g., tailed categories in Cityscapes dataset which exhibits a long-tailed label distribution. Existing approaches almost all neglect this problem, and treat categories equally. Some popular approaches such as consistency regularization or pseudo-labeling may even harm the learning of under-performing categories, that the predictions or pseudo labels of these categories could be too inaccurate to guide the learning on the unlabeled data. In this paper, we look into this problem, and propose a novel framework for semi-supervised semantic segmentation, named adaptive equalization learning (AEL). AEL adaptively balances the training of well and badly performed categories, with a confidence bank to dynamically track category-wise performance during training. The confidence bank is leveraged as an indicator to tilt training towards under-performing categories, instantiated in three strategies: 1) adaptive Copy-Paste and CutMix data augmentation approaches which give more chance for under-performing categories to be copied or cut; 2) an adaptive data sampling approach to encourage pixels from under-performing category to be sampled; 3) a simple yet effective re-weighting method to alleviate the training noise raised by pseudo-labeling. Experimentally, AEL outperforms the state-of-the-art methods by a large margin on the Cityscapes and Pascal VOC benchmarks under various data partition protocols. For more details, please refer to our NeurIPS paper (arxiv).

image

Installation

Check INSTALL.md for installation instructions.

Training and Evaluation

For example, perform training and evaluation with 1/2 data parttition on Cityscapes dataset.

cd experiments/cityscapes_2
bash train.sh

For other partition protocols, change n_sup in config.yaml.

TODO

  • Other SOTA semi-supervised segmentation methods
Comments
  • Bug in forward

    Bug in forward

    Hi, I am trying to run your code, and find a bug.

    As you can see, the batch_size of images_unsup_strong is 1, which is not allowed in BatchNorm2d when the model is on train mode. https://github.com/hzhupku/SemiSeg-AEL/blob/main/train.py#L351

    The information for the bug is

    File "../../train.py", line 497, in <module>
    main()
  File "../../train.py", line 134, in main
    labeled_epoch, model_teacher, trainloader_unsup, criterion_cons, class_criterion, cutmix_bank)
  File "../../train.py", line 344, in train
    preds_student_unsup = model(images_unsup_strong)
  File "/mnt/cache/share/spring/conda_envs/miniconda3/envs/s0.3.4/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/cache/share/spring/conda_envs/miniconda3/envs/s0.3.4/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 705, in forward
    output = self.module(*inputs[0], **kwargs[0])
  File "/mnt/cache/share/spring/conda_envs/miniconda3/envs/s0.3.4/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/lustre/wanghaochen/semiseg/semseg/models/model_helper.py", line 48, in forward
    pred_head = self.decoder([f1, f2,feat1, feat2])
  File "/mnt/cache/share/spring/conda_envs/miniconda3/envs/s0.3.4/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/lustre/wanghaochen/semiseg/semseg/models/decoder.py", line 54, in forward
    aspp_out = self.aspp(x4)
  File "/mnt/cache/share/spring/conda_envs/miniconda3/envs/s0.3.4/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/lustre/wanghaochen/semiseg/semseg/models/base.py", line 46, in forward
    feat1 = F.upsample(self.conv1(x), size=(h, w), mode='bilinear', align_corners=True)
  File "/mnt/cache/share/spring/conda_envs/miniconda3/envs/s0.3.4/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/cache/share/spring/conda_envs/miniconda3/envs/s0.3.4/lib/python3.6/site-packages/torch/nn/modules/container.py", line 119, in forward
    input = module(input)
  File "/mnt/cache/share/spring/conda_envs/miniconda3/envs/s0.3.4/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/cache/share/spring/conda_envs/miniconda3/envs/s0.3.4/lib/python3.6/site-packages/torch/nn/modules/batchnorm.py", line 539, in forward
    bn_training, exponential_average_factor, self.eps)
  File "/mnt/cache/share/spring/conda_envs/miniconda3/envs/s0.3.4/lib/python3.6/site-packages/torch/nn/functional.py", line 2147, in batch_norm
    _verify_batch_size(input.size())
  File "/mnt/cache/share/spring/conda_envs/miniconda3/envs/s0.3.4/lib/python3.6/site-packages/torch/nn/functional.py", line 2114, in _verify_batch_size
    raise ValueError("Expected more than 1 value per channel when training, got input size {}".format(size))
ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 256, 1, 1])
    opened by Haochen-Wang409 6
  • the results on pascal voc look strange

    the results on pascal voc look strange

    according to table 6，supervised baseline with different label ratio has achieved good results. and none of the other methods seemed to work as their paper presented. it is strange that the results are very different between with others paper. for example, in PixelSSL, there are significant decrease of miou with diff label ratio. PixelSSL

    opened by CuberrChen 4
  • Multi-scale evaluation?

    Multi-scale evaluation?

    Hi, it seems that you use multi-scale evaluation: https://github.com/hzhupku/SemiSeg-AEL/blob/main/eval.py#L37 Are the results in your papers under this setting? How about previous SOTA methods?

    opened by Haochen-Wang409 2
  • Question about ignore_cat

    Question about ignore_cat

    Hi @hzhupku, thanks for sharing. I have a question when I read your code: https://github.com/hzhupku/SemiSeg-AEL/blob/747c7972a1aed589a8a62cdd98c3d1836c609735/train.py#L193 Can you explain why you set ignore_cat to [0, 1, 2, 8, 10], and it seems like be same for pascal voc and cityscapes?

    opened by lliuz 2
  • Could you provide the splits?

    Could you provide the splits?

    Hi, thanks for your great work. I wonder if you could provide splits (which images are considered as labeled data) for both PASCAL VOC and Cityscapes? Because even if the random seed is set, there may be a certain uncertainty when sampling.

    opened by Haochen-Wang409 2
  • Question for your few supervision splits in VOC?

    Question for your few supervision splits in VOC?

    I can't reproduce your results based on the aug set of the ratio of 1/32 and 1/16.

    In 1/16(662), with the partitioning protocol of the CPS, I only have 75.64 with 4*v100 GPUs, where every single GPU with batch size = 16, and 513x513 as input resolution. It is far lower than your reported results in your paper.

    For these two ratios, do you split your data based on the entire aug dataset, or the high-quality set

    Can you provide your splits?

    Cheers

    opened by ghost 1
  • Question about Computational Resources

    Question about Computational Resources

    Hello! It would be very beneficial if you mention your used computational resources (Number and model of GPU/TPU's used). Looking forward to hearing from you.

    opened by mradul2 1
  • Typo in paper

    Typo in paper

    Hello, thank you for your awesome works!

    Typo is on page 9, section 4.7 Qualitative Results second line,

    """ We compare the proposed AEL with ground-truth, supervsied baseline and our basic framework described in Section 3.2. """

    I think supervsied should be supervised :) Ah, I downloaded pdf file on arxiv today.

    Thank you. Jihwan Eom

    opened by JihwanEom 1
  • Ohem and Aux_loss for pascal voc?

    Ohem and Aux_loss for pascal voc?

    Hi, sorry to bother you. I found ohem and aux_loss are in the config for pascal voc: https://github.com/hzhupku/SemiSeg-AEL/blob/main/experiments/pascal_4/config.yaml#L61 https://github.com/hzhupku/SemiSeg-AEL/blob/main/experiments/pascal_4/config.yaml#L70

    However, these two tricks are not mentioned in your paper.

    opened by Haochen-Wang409 1
  • Data / Resnet

    Data / Resnet

    Hi,

    Is there a link to the data? I registered and downloaded the gtfine dataset from the Cityscapes website. Is that the right one -it provides a bunch of subfolders for cities? The download did not contain a datalist - fine_train.txt and fine_test.txt. How do we obtain those? Also, is there a link to the resnet model?

    opened by dd1github 1
  • About acp and acm

    About acp and acm

    Hello How are you? Thanks for contributing to this project. I have some question about acp and acm. What do they mean in configuration? What does the "number" in acp or acm configuration mean?

    image

    opened by rose-jinyang 1
  • Question about hyperparameters of Cutout.

    Question about hyperparameters of Cutout.

    Hi~ Could you please release the config of MT which was listed in your paper. It's also great to teach me how to set n_holes and length in cutout. Thanks a lot ~ Looking forward to your reply.

    opened by suilin0432 0
  • Epoch for cityscapes

    Epoch for cityscapes

    Thanks for your great work! I found that in your paper, your model was trained 18k iteration for cityscapes with batch size 16. As the train set for cityscapes is around 3k. Therefore, the total epoch is to be 18k * 16 / 3k = 96. However, the number of epoch in your config is 200. So which one is correct?

    opened by Haochen-Wang409 0
Owner
Hanzhe Hu
focusing on computer vision
Hanzhe Hu
GitHub
Semi-supervised Semantic Segmentation with Directional Context-aware Consistency (CVPR 2021)

Semi-supervised Semantic Segmentation with Directional Context-aware Consistency (CAC) Xin Lai*, Zhuotao Tian*, Li Jiang, Shu Liu, Hengshuang Zhao, Li

Jia Research Lab 137 Dec 14, 2022
Anti-Adversarially Manipulated Attributions for Weakly and Semi-Supervised Semantic Segmentation (CVPR 2021)

Anti-Adversarially Manipulated Attributions for Weakly and Semi-Supervised Semantic Segmentation Input Image Initial CAM Successive Maps with adversar

Jungbeom Lee 110 Dec 7, 2022
Semi-supervised Semantic Segmentation with Directional Context-aware Consistency (CVPR 2021)

Semi-supervised Semantic Segmentation with Directional Context-aware Consistency (CAC) Xin Lai*, Zhuotao Tian*, Li Jiang, Shu Liu, Hengshuang Zhao, Li

DV Lab 137 Dec 14, 2022
[CVPR 2021] Semi-Supervised Semantic Segmentation with Cross Pseudo Supervision

TorchSemiSeg [CVPR 2021] Semi-Supervised Semantic Segmentation with Cross Pseudo Supervision by Xiaokang Chen1, Yuhui Yuan2, Gang Zeng1, Jingdong Wang

Chen XiaoKang 387 Jan 8, 2023
[ICCV 2021] A Simple Baseline for Semi-supervised Semantic Segmentation with Strong Data Augmentation

[ICCV 2021] A Simple Baseline for Semi-supervised Semantic Segmentation with Strong Data Augmentation

CodingMan 45 Dec 12, 2022
Semantic Segmentation for Real Point Cloud Scenes via Bilateral Augmentation and Adaptive Fusion (CVPR 2021)

Semantic Segmentation for Real Point Cloud Scenes via Bilateral Augmentation and Adaptive Fusion (CVPR 2021) This repository is for BAAF-Net introduce

null 90 Dec 29, 2022
Pytorch Implementation for NeurIPS (oral) paper: Pixel Level Cycle Association: A New Perspective for Domain Adaptive Semantic Segmentation

Pixel-Level Cycle Association This is the Pytorch implementation of our NeurIPS 2020 Oral paper Pixel-Level Cycle Association: A New Perspective for D

null 87 Oct 19, 2022
Reducing Information Bottleneck for Weakly Supervised Semantic Segmentation (NeurIPS 2021)

Reducing Information Bottleneck for Weakly Supervised Semantic Segmentation (NeurIPS 2021) The implementation of Reducing Infromation Bottleneck for W

Jungbeom Lee 81 Dec 16, 2022
The official implementation of Equalization Loss v1 & v2 (CVPR 2020, 2021) based on MMDetection.

The Equalization Losses for Long-tailed Object Detection and Instance Segmentation This repo is official implementation CVPR 2021 paper: Equalization

Jingru Tan 129 Dec 16, 2022
[NeurIPS 2021 Spotlight] Code for Learning to Compose Visual Relations

Learning to Compose Visual Relations This is the pytorch codebase for the NeurIPS 2021 Spotlight paper Learning to Compose Visual Relations. Demo Imag

Nan Liu 88 Jan 4, 2023
Code for our paper Domain Adaptive Semantic Segmentation with Self-Supervised Depth Estimation

CorDA Code for our paper Domain Adaptive Semantic Segmentation with Self-Supervised Depth Estimation Prerequisite Please create and activate the follo

Qin Wang 60 Nov 30, 2022
Implementation of "Semi-supervised Domain Adaptive Structure Learning"

Semi-supervised Domain Adaptive Structure Learning - ASDA This repo contains the source code and dataset for our ASDA paper. Illustration of the propo

null 3 Dec 13, 2021
Shape-aware Semi-supervised 3D Semantic Segmentation for Medical Images

SASSnet Code for paper: Shape-aware Semi-supervised 3D Semantic Segmentation for Medical Images(MICCAI 2020) Our code is origin from UA-MT You can fin

klein 125 Jan 3, 2023
ST++: Make Self-training Work Better for Semi-supervised Semantic Segmentation

ST++ This is the official PyTorch implementation of our paper: ST++: Make Self-training Work Better for Semi-supervised Semantic Segmentation. Lihe Ya

Lihe Yang 147 Jan 3, 2023
[CVPR 2022] Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels

Using Unreliable Pseudo Labels Official PyTorch implementation of Semi-Supervised Semantic Segmentation Using Unreliable Pseudo Labels, CVPR 2022. Ple

Haochen Wang 268 Dec 24, 2022
[cvpr22] Perturbed and Strict Mean Teachers for Semi-supervised Semantic Segmentation

PS-MT [cvpr22] Perturbed and Strict Mean Teachers for Semi-supervised Semantic Segmentation by Yuyuan Liu, Yu Tian, Yuanhong Chen, Fengbei Liu, Vasile

Yuyuan Liu 132 Jan 3, 2023
Prototypical Pseudo Label Denoising and Target Structure Learning for Domain Adaptive Semantic Segmentation (CVPR 2021)

Prototypical Pseudo Label Denoising and Target Structure Learning for Domain Adaptive Semantic Segmentation (CVPR 2021, official Pytorch implementatio

Microsoft 247 Dec 25, 2022
Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentation, CVPR 2018

Learning Pixel-level Semantic Affinity with Image-level Supervision This code is deprecated. Please see https://github.com/jiwoon-ahn/irn instead. Int

Jiwoon Ahn 337 Dec 15, 2022
PyTorch implementation for our NeurIPS 2021 Spotlight paper "Long Short-Term Transformer for Online Action Detection".

Long Short-Term Transformer for Online Action Detection Introduction This is a PyTorch implementation for our NeurIPS 2021 Spotlight paper "Long Short

null 77 Dec 16, 2022