ST++: Make Self-training Work Better for Semi-supervised Semantic Segmentation

Related tags

Deep Learning semi-supervised-learning self-training semantic-segmentation semi-supervised-segmentation
Overview

ST++

This is the official PyTorch implementation of our paper:

ST++: Make Self-training Work Better for Semi-supervised Semantic Segmentation.
Lihe Yang, Wei Zhuo, Lei Qi, Yinghuan Shi and Yang Gao.

Getting Started

Data Preparation

Pre-trained Model

ResNet-50 | ResNet-101 | DeepLabv2-ResNet-101

Dataset

Pascal | Augmented Masks | Cityscapes | Class Mapped Masks

File Organization

├── ./pretrained
    ├── resnet50.pth
    ├── resnet101.pth
    └── deeplabv2_resnet101_coco_pretrained.pth
    
├── [Your Pascal Path]
    ├── JPEGImages
    └── SegmentationClass    # replace the official folder with above augmented masks 
    
├── [Your Cityscapes Path]
    ├── gtFine               # replace the official folder with above class mapped masks 
    └── leftImg8bit

Training and Testing

export semi_setting='pascal/1_8/split_0'

CUDA_VISIBLE_DEVICES=0,1 python -W ignore main.py \
  --dataset pascal --data-root [Your Pascal Path] \
  --batch-size 16 --backbone resnet50 --model deeplabv3plus \
  --labeled-id-path dataset/splits/$semi_setting/labeled.txt \
  --unlabeled-id-path dataset/splits/$semi_setting/unlabeled.txt \
  --pseudo-mask-path outdir/pseudo_masks/$semi_setting \
  --save-path outdir/models/$semi_setting

This script is for our ST framework. To run ST++, add --plus --reliable-id-path outdir/reliable_ids/$semi_setting.

Acknowledgement

The DeepLabv2 MS COCO pre-trained model is borrowed and converted from AdvSemiSeg. The image partitions are borrowed from Context-Aware-Consistency and PseudoSeg. Part of the training hyper-parameters and network structures are adapted from PyTorch-Encoding. The strong data augmentations are borrowed from MoCo v2 and PseudoSeg.

Thanks a lot for their great works!

Citation

If you find this project useful, please consider citing:

@article{yang2021st++,
  title={ST++: Make Self-training Work Better for Semi-supervised Semantic Segmentation},
  author={Yang, Lihe and Zhuo, Wei and Qi, Lei and Shi, Yinghuan and Gao, Yang},
  journal={arXiv preprint arXiv:2106.05095},
  year={2021}
}
Comments
  • About comparison with other sota methods

    About comparison with other sota methods

    Sorry for bothering you again! First, I have re-trained several models, i.e., CPS, U2PL, and AEL. But my results are far below their reported results. Among these SOTA methods with public codes, I can only reproduce expected results based on your ST++ (e.g. 74.15 for 1/16 pascal). Appreciate it! I found there are several different settings within former methods, i.e., stronger backbone(resnet_stem), auxiliary decoders, larger crop_size, OHEM loss, Sync_bn, and several training tricks, which are extremely unfair for comparisons. Then, I wonder whether the reviewers ask you to compare yours with these methods? Is it necessary to re-train your ST++ with the same settings? Just for curiosity.......

    opened by Luffy03 9
  • Unable to generate pseudo label?

    Unable to generate pseudo label?

    作者你好，我在windos下pycharm复现您代码的时候，无论在VOC还是自己数据集下都无法生成伪标签， pred = Image.fromarray(pred.squeeze(0).numpy().astype(np.uint8), mode='P')会报维度3>2的错误。我的pytorch版本为1.6.0 , python=3.6.13，pillow=8.3.1，numpy=1.16.3，下面是完整信息： 很感激期待您的回复！

    Namespace(backbone='resnet101', batch_size=2, crop_size=321, data_root='./ywDataset/', dataset='pascal', epochs=2, labeled_id_path='./dataset/splits/pascal/1_8/split_1/labeled.txt', lr=0.000125, model='deeplabv3plus', plus=None, pseudo_mask_path='./outdir/pseudo_masks/pascal/1_8/split_1', reliable_id_path='./outdir/pseudo_masks/1_8/split_1', save_path='./outdir/models/pascal/1_8/split_1', unlabeled_id_path='./dataset/splits/pascal/1_8/split_1/unlabeled.txt')

    ================> Total stage 1/3: Supervised training on labeled images (SupOnly)

    Params: 59.3M

    ==> Epoch 0, learning rate = 0.0001 previous best = 0.00 Loss: 0.712: 100%|██████████| 5/5 [00:44<00:00, 8.92s/it] mIOU: 33.15: 100%|██████████| 10/10 [00:05<00:00, 1.99it/s]

    ==> Epoch 1, learning rate = 0.0001 previous best = 33.15 Loss: 0.589: 100%|██████████| 5/5 [00:42<00:00, 8.56s/it] mIOU: 33.15: 100%|██████████| 10/10 [00:05<00:00, 1.79it/s]

    ================> Total stage 2/3: Pseudo labeling all unlabeled images 0%| | 0/25 [00:07<?, ?it/s] Traceback (most recent call last): File "D:/ST-PlusPlus-master/main.py", line 359, in main(args) File "D:/ST-PlusPlus-master/main.py", line 94, in main label(best_model, dataloader, args) File "D:/ST-PlusPlus-master/main.py", line 331, in label pred = Image.fromarray(pred.squeeze(0).numpy().astype(np.uint8), mode='P') File "C:\Users\admin\anaconda3\envs\pytorch\lib\site-packages\PIL\Image.py", line 2840, in fromarray raise ValueError(f"Too many dimensions: {ndim} > {ndmax}.") ValueError: Too many dimensions: 3 > 2.

    opened by QiJiawei1 6
  • About the `evaluation bug'

    About the `evaluation bug'

    Hi, I'm a beginner in semantic segmentation, and thanks for your great work. It seems ST++ is the only method which select `reliable DT mask' for unlabeled images.

    I'm interested in the `evaluation bug' metioned on Table.1 image Could you give me a brief introduction about it?

    opened by Haochen-Wang409 6
  • Supervised training

    Supervised training

    Hello, thank you very much for your work!

    I'd like to ask what is the purpose of doubling the training ids when the number of labeled images is less than 200?

    image

    opened by ltb801 4
  • Pascal VOC with split_0, 1/16 setting get 55.17 mAP?

    Pascal VOC with split_0, 1/16 setting get 55.17 mAP?

    We use your code to reproduce the experimental results in your paper. Our settings are pascal_batchsize_16_resnet50_deeplabv3plus_pascal_1_16_split_0 but get 55.17mAP In your paper, it's about 73 mAP with ST++

    opened by djh-dzxw 3
  • Results about Cityscapes

    Results about Cityscapes

    Congrats on CVPR first, nice work! The detailed results about the cityscapes are not reported in the preprint. I wonder that have you compared your results with other sota methods on the cityscapes dataset?

    opened by Luffy03 3
  • Question about the reported results

    Question about the reported results

    Although this work is very insightful, I have some questions about the report results. After running the code for several times, I found that the biggest gap in baseline（Stage 1）is 0.47 of mIoU. This value is even greater than the ST++'s improvement over ST in some data splits. So what is your principle for those reported results?

    image

    opened by 4-0-4-notfound 3
  • Using the best model to do pseudo labeling

    Using the best model to do pseudo labeling

    Hi thanks for your work.

    I noticed that you use the best checkpoint to do the pseudo labeling

    https://github.com/LiheYoung/ST-PlusPlus/blob/561d68fdd452b6764cccabac1472e43f23366078/main.py#L129

    The best model is selected by the evaluation on the val set with ground truth label. Do you think this cause an information leaking?

    Although the ground truth doesn’t explicitly involve in the training. it does provide some information for the model checkpoint selection.

    When we train our model, we should make sure that we can touch nothing about the val set.

    Have you tried to train your method without this selection? Say just use the final checkpoint?

    thanks.

    opened by geocccc 2
  • Why

    Why "Re-initialize S" in Algorithm2?

    Hi! Thanks a lot for your great work!

    In Algo2, after computing score $s_i$, ranking by the scores to get $D^{u_1}$ and $D^{u_2}$, there is a line saying "Re-initialize S".

    Does it mean to re-initialize the weight of the student model to the initialized random parameters? Why the re-initialization is needed? I think I did not find the relevant words in the text.

    Thanks!

    opened by yd-yin 2
  • pseudo- labels生成之后，更新labelled data的问题

    pseudo- labels生成之后，更新labelled data的问题

    你好，谢谢你的工作。我有个小问题，请问pseudo- labels生成，是实时就加入 原来的labelled data里面，保持一直的训练模式，还是人为地需要把生成的pseudo- labels 放入原来的labelled data，再开始重新训练模型。

    那如果是人为地加入pseudo-labels, 是不是就应该可以理解成分阶段循环训练模型。ô 请问是不是有end-to-end, 就是pseudo-labels generating 和 re-train模型合并的，就不分阶段，实时地进行生成伪标签和重新训练模型

    opened by XiaoyanQian 2
  • 关于cutout

    关于cutout

    感谢您精彩的工作。我有以下几个问题，希望可以得到您的解答。

    1. 是否可以解释一下代码中cutout的实现呢？这两个变量分别代表什么意思呢？

    erase_w = int(np.sqrt(size / ratio)) erase_h = int(np.sqrt(size * ratio))

    1. 请问是否有尝试过用cutmix代替cutout的实验呢？
    opened by Renp1ngs 2
Owner
Lihe Yang
Master student at Nanjing University, Computer Vision
Lihe Yang
GitHub
Code for the paper One Thing One Click: A Self-Training Approach for Weakly Supervised 3D Semantic Segmentation, CVPR 2021.

One Thing One Click One Thing One Click: A Self-Training Approach for Weakly Supervised 3D Semantic Segmentation (CVPR2021) Code for the paper One Thi

null 44 Dec 12, 2022
Shape-aware Semi-supervised 3D Semantic Segmentation for Medical Images

SASSnet Code for paper: Shape-aware Semi-supervised 3D Semantic Segmentation for Medical Images(MICCAI 2020) Our code is origin from UA-MT You can fin

klein 125 Jan 3, 2023
Semi-supervised Semantic Segmentation with Directional Context-aware Consistency (CVPR 2021)

Semi-supervised Semantic Segmentation with Directional Context-aware Consistency (CAC) Xin Lai*, Zhuotao Tian*, Li Jiang, Shu Liu, Hengshuang Zhao, Li

Jia Research Lab 137 Dec 14, 2022
Anti-Adversarially Manipulated Attributions for Weakly and Semi-Supervised Semantic Segmentation (CVPR 2021)

Anti-Adversarially Manipulated Attributions for Weakly and Semi-Supervised Semantic Segmentation Input Image Initial CAM Successive Maps with adversar

Jungbeom Lee 110 Dec 7, 2022
Semi-supervised Semantic Segmentation with Directional Context-aware Consistency (CVPR 2021)

Semi-supervised Semantic Segmentation with Directional Context-aware Consistency (CAC) Xin Lai*, Zhuotao Tian*, Li Jiang, Shu Liu, Hengshuang Zhao, Li

DV Lab 137 Dec 14, 2022
[CVPR 2021] Semi-Supervised Semantic Segmentation with Cross Pseudo Supervision

TorchSemiSeg [CVPR 2021] Semi-Supervised Semantic Segmentation with Cross Pseudo Supervision by Xiaokang Chen1, Yuhui Yuan2, Gang Zeng1, Jingdong Wang

Chen XiaoKang 387 Jan 8, 2023
[ICCV 2021] A Simple Baseline for Semi-supervised Semantic Segmentation with Strong Data Augmentation

[ICCV 2021] A Simple Baseline for Semi-supervised Semantic Segmentation with Strong Data Augmentation

CodingMan 45 Dec 12, 2022
[CVPR 2022] Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels

Using Unreliable Pseudo Labels Official PyTorch implementation of Semi-Supervised Semantic Segmentation Using Unreliable Pseudo Labels, CVPR 2022. Ple

Haochen Wang 268 Dec 24, 2022
[cvpr22] Perturbed and Strict Mean Teachers for Semi-supervised Semantic Segmentation

PS-MT [cvpr22] Perturbed and Strict Mean Teachers for Semi-supervised Semantic Segmentation by Yuyuan Liu, Yu Tian, Yuanhong Chen, Fengbei Liu, Vasile

Yuyuan Liu 132 Jan 3, 2023
Learning recognition/segmentation models without end-to-end training. 40%-60% less GPU memory footprint. Same training time. Better performance.

InfoPro-Pytorch The Information Propagation algorithm for training deep networks with local supervision. (ICLR 2021) Revisiting Locally Supervised Lea

null 78 Dec 27, 2022
Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentation, CVPR 2018

Learning Pixel-level Semantic Affinity with Image-level Supervision This code is deprecated. Please see https://github.com/jiwoon-ahn/irn instead. Int

Jiwoon Ahn 337 Dec 15, 2022
The Self-Supervised Learner can be used to train a classifier with fewer labeled examples needed using self-supervised learning.

Published by SpaceML • About SpaceML • Quick Colab Example Self-Supervised Learner The Self-Supervised Learner can be used to train a classifier with

SpaceML 92 Nov 30, 2022
Code for our paper Domain Adaptive Semantic Segmentation with Self-Supervised Depth Estimation

CorDA Code for our paper Domain Adaptive Semantic Segmentation with Self-Supervised Depth Estimation Prerequisite Please create and activate the follo

Qin Wang 60 Nov 30, 2022
Self-supervised Augmentation Consistency for Adapting Semantic Segmentation (CVPR 2021)

Self-supervised Augmentation Consistency for Adapting Semantic Segmentation This repository contains the official implementation of our paper: Self-su

Visual Inference Lab @TU Darmstadt 132 Dec 21, 2022
Perturbed Self-Distillation: Weakly Supervised Large-Scale Point Cloud Semantic Segmentation (ICCV2021)

Perturbed Self-Distillation: Weakly Supervised Large-Scale Point Cloud Semantic Segmentation (ICCV2021) This is the implementation of PSD (ICCV 2021),

null 12 Dec 12, 2022
[CVPR 2021] "The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models" Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Michael Carbin, Zhangyang Wang

The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models Codes for this paper The Lottery Tickets Hypo

VITA 59 Dec 28, 2022
Unified Pre-training for Self-Supervised Learning and Supervised Learning for ASR

UniSpeech The family of UniSpeech: UniSpeech (ICML 2021): Unified Pre-training for Self-Supervised Learning and Supervised Learning for ASR UniSpeech-

Microsoft 282 Jan 9, 2023
Recall Loss for Semantic Segmentation (This repo implements the paper: Recall Loss for Semantic Segmentation)

Recall Loss for Semantic Segmentation (This repo implements the paper: Recall Loss for Semantic Segmentation) Download Synthia dataset The model uses

null 32 Sep 21, 2022
Official code for "Simpler is Better: Few-shot Semantic Segmentation with Classifier Weight Transformer. ICCV2021".

Simpler is Better: Few-shot Semantic Segmentation with Classifier Weight Transformer. ICCV2021. Introduction We proposed a novel model training paradi

Lucas 103 Dec 14, 2022