Semi-Supervised Semantic Segmentation with Pixel-Level Contrastive Learning from a Class-wise Memory Bank

Iñigo Alonso Ruiz

Last update: Dec 15, 2022

Related tags

Deep Learning SemiSeg-Contrastive

Overview

This repository provides the official code for replicating experiments from the paper: Semi-Supervised Semantic Segmentation with Pixel-Level Contrastive Learning from a Class-wise Memory Bank which as been accepted as an oral paper in the IEEE International Conference on Computer Vision (ICCV) 2021.

This code is based on ClassMix code

Semi-Supervised Semantic Segmentation with Pixel-Level Contrastive Learning from a Class-wise Memory Bank

Prerequisites

CUDA/CUDNN
Python3
Packages found in requirements.txt

Contact

If any question, please either open a github issue or contact via email to: [email protected]

Datasets

Create a folder outsite the code folder:

mkdir ../data/

Cityscapes

mkdir ../data/CityScapes/

Download the dataset from (Link).

Download the files named 'gtFine_trainvaltest.zip', 'leftImg8bit_trainvaltest.zip' and extract in ../data/Cityscapes/

Pascal VOC 2012

mkdir ../data/VOC2012/

Download the dataset from (Link).

Download the file 'training/validation data' under 'Development kit' and extract in ../data/VOC2012/

GTA5

mkdir ../data/GTA5/

Download the dataset from (Link). Unzip all the datasets parts to create an structure like this:

../data/GTA5/images/val/*.png
../data/GTA5/images/train/*.png
../data/GTA5/labels/val/*.png
../data/GTA5/labels/train/*.png

Then, reformat the label images from colored images to training ids. For that, execute this:

python3 utils/translate_labels.py

Experiments

Here there are some examples for replicating the experiments from the paper. Implementation details are specified in the paper (section 4.2) any modification could potentially affect to the final result.

Semi-Supervised

Search here for the desired configuration:

ls ./configs/

For example, for this configuration:

Dataset: CityScapes
% of labels: 1/30
Pretrain: COCO
Split: 0
Network: Deeplabv2

Execute:

python3 trainSSL.py --config ./configs/configSSL_city_1_30_split0_COCO.json

Another example, for this configuration:

Dataset: CityScapes
% of labels: 1/30
Pretrain: imagenet
Split: 0
Network: Deeplabv3+

Execute:

python3 trainSSL.py --config ./configs/configSSL_city_1_30_split0_v3.json

For example, for this configuration:

Dataset: PASCAL VOC
% of labels: 1/50
Pretrain: COCO
Split: 0

Execute:

python3 trainSSL.py --config ./configs/configSSL_pascal_1_50_split0_COCO.json

For replicating paper experiments, just execute the training of the specific set-up to replicate. We already provide all the configuration files used in the paper. For modifying them and a detail description of all the parameters in the configuration files, check this example:

Configuration File Description

2 for random splits "labeled_samples": 744, # Number of labeled samples to use for supervised learning. The rest will be use without labels. Options: any integer "input_size": "512,512" # Image crop size Options: any integer tuple } }, "seed": 5555, # seed for randomization. Options: any integer "ignore_label": 250, # ignore label value. Options: any integer "utils": { "save_checkpoint_every": 10000, # The model will be saved every this number of iterations. Options: any integer "checkpoint_dir": "../saved/DeepLab", # Path to save the models. Options: any path "val_per_iter": 1000, # The model will be evaluated every this number of iterations. Options: any integer "save_best_model": true # Whether to use teacher model for generating the psuedolabels. The student model wil obe used otherwise. Options: boolean } }">

{
  "model": "DeepLab", # Network architecture. Options: Deeplab
  "version": "2", # Version of the network architecture. Options: {2, 3} for deeplabv2 and deeplabv3+
  "dataset": "cityscapes", # Dataset to use. Options: {"cityscapes", "pascal"}

  "training": { 
    "batch_size": 5, # Batch size to use. Options: any integer
    "num_workers": 3, # Number of cpu workers (threads) to use for laoding the dataset. Options: any integer
    "optimizer": "SGD", # Optimizer to use. Options: {"SGD"}
    "momentum": 0.9, # momentum for SGD optimizer, Options: any float 
    "num_iterations": 100000, # Number of iterations to train. Options: any integer
    "learning_rate": 2e-4, # Learning rate. Options: any float
    "lr_schedule": "Poly", # decay scheduler for the learning rate. Options: {"Poly"}
    "lr_schedule_power": 0.9, # Power value for the Poly scheduler. Options: any float
    "pretraining": "COCO", # Pretraining to use. Options: {"COCO", "imagenet"}
    "weight_decay": 5e-4, # Weight decay. Options: any float
    "use_teacher_train": true, # Whether to use the teacher network to generate pseudolabels. Use student otherwise. Options: boolean. 
    "save_teacher_test": false, # Whether to save the teacher network as the model for testing. Use student otherwise. Options: boolean. 
    
    "data": {
      "split_id_list": 0, # Data splits to use. Options: {0, 1, 2} for pre-computed splits. N >2 for random splits
      "labeled_samples": 744, # Number of labeled samples to use for supervised learning. The rest will be use without labels. Options: any integer
      "input_size": "512,512" # Image crop size  Options: any integer tuple
    }

  },
  "seed": 5555, # seed for randomization. Options: any integer
  "ignore_label": 250, # ignore label value. Options: any integer

  "utils": {
    "save_checkpoint_every": 10000,  # The model will be saved every this number of iterations. Options: any integer
    "checkpoint_dir": "../saved/DeepLab", # Path to save the models. Options: any path
    "val_per_iter": 1000, # The model will be evaluated every this number of iterations. Options: any integer
    "save_best_model": true # Whether to use teacher model for generating the psuedolabels. The student model wil obe used otherwise. Options: boolean
  }
}

Memory Restrictions

All experiments have been run in an NVIDIA Tesla V100. To try to fit the training in a smaller GPU, try to follow this tips:

Reduce batch_size from the configuration file
Reduce input_size from the configuration file
Instead of using trainSSL.py use trainSSL_less_memory.py which optimized labeled and unlabeled data separate steps.

For example, for this configuration:

Dataset: PASCAL VOC
% of labels: 1/50
Pretrain: COCO
Split: 0
Batch size: 8
Crop size: 256x256 Execute:

python3 trainSSL_less_memory.py --config ./configs/configSSL_pascal_1_50_split2_COCO_reduced.json

Semi-Supervised Domain Adaptation

Experiments for domain adaptation from GTA5 dataset to Cityscapes.

For example, for configuration:

% of labels: 1/30
Pretrain: Imagenet
Split: 0

Execute:

python3 trainSSL_domain_adaptation_targetCity.py --config ./configs/configSSL_city_1_30_split0_imagenet.json

Evaluation

The training code will evaluate the training model every some specific number of iterations (modify the parameter val_per_iter in the configuration file).

Best evaluated model will be printed at the end of the training.

For every training, several weights will be saved under the path specified in the parameter checkpoint_dir of the configuration file.

One model every save_checkpoint_every (see configuration file) will be saved, plus the best evaluated model.

So, the model has trained we can already know the performance.

For a later evaluation, just execute the next command specifying the model to evaluate in the model-path argument:

python3 evaluateSSL.py --model-path ../saved/DeepLab/best.pth

Citation

If you find this work useful, please consider citing:

@inproceedings{alonso2021semi,
  title={Semi-Supervised Semantic Segmentation with Pixel-Level Contrastive Learning from a Class-wise Memory Bank},
  author={Alonso, I{\~n}igo and Sabater, Alberto and Ferstl, David and Montesano, Luis and Murillo, Ana C},
  booktitle={Proceedings of the IEEE International Conference on Computer Vision},
  year={2021}
}

License

Thi code is released under the Apache 2.0 license. Please see the LICENSE file for more information.

Comments

Report some bugs
Hi @Shathe, nice work but I find some bugs:

The entropy loss compute in a wrong way: https://github.com/Shathe/SemiSeg-Contrastive/blob/381e182579d286185816356085a497e6d7f14ac5/trainSSL.py#L590-L591 https://github.com/Shathe/SemiSeg-Contrastive/blob/381e182579d286185816356085a497e6d7f14ac5/trainSSL.py#L41-L43 valid_mask has shape of [B, 1, H, W], while loss_image in L42 has shape of [B, H, W], so the multiply will broadcast in a wrong way and results a tensor in L43 with [B, B, H, W].

Some parameters are optimized multiple times in a iteration. https://github.com/Shathe/SemiSeg-Contrastive/blob/381e182579d286185816356085a497e6d7f14ac5/model/deeplabv3.py#L336-L342 The .modules() function returns all modules recursively, and .parameters() returns the params in the modules, so the parameters in modules will be optimized multiple times.

Despite these problems, I successfully reproduced your work. However, fixing the above bugs may need to adjust the hyperparameters, I hope you can update the new hyperparameters and results if possible.
opened by lliuz 1
Performance gap and issue in code

Hi there,

Thanks for the work! I trained this SSL model for semi-supervised domain adaptation semantic segmentation. I used the label ratio of 1/30, trained on the GTA5 dataset as the full labelled dataset and Cityscapes as the partially labelled dataset. Evaluated on Cityscapes val set. I kept all configs as the same as the descriptions in your supplementary material. Here are the results I got from my training on a single NVIDIA GeForce RTX 3090:

| label ratio | mIoU | Road | Sidewalk | Building | Wall | Fence | Pole | Traffic light | Traffic sign | Vegetartion | Terrain | Sky | Person | Rider | Car | Truck | Bus | Train | Motorcycle | Bicycle | |:-----------:|:----:|:----:|:--------:|:--------:|:----:|:-----:|:----:|---------------|:------------:|:-----------:|:-------:|:----:|:------:|:-----:|:----:|:-----:|:----:|:-----:|:----------:|:-------:| | 1/30 | 51.8 | 94.8 | 65.7 | 85.4 | 369 | 32.6 | 35.3 | 38.8 | 47.5 | 86.1 | 47.1 | 89.4 | 61.5 | 35.4 | 86.2 | 33.9 | 0.59 | 21.68 | 29.55 | 56.51 |

============================================================================================ Besides, there is a variable best_mIoU_improved which is not defined but used to compare with best_mIoU in the code. Is that any improvement made here?

Thanks.

opened by suikei-wang 0
About the feature vectors from teacher and student

Dear author,

I am confused about the feature vectors used in contrastive learning. The teacher's feature vectors are simply obtained using the projection head g. Why use the projection head g and the prediction head q to obtain the feature vectors of student? Can we just use only g in the student model?

opened by ccccly 0
Questions about the paper
Hi,

Thanks for the interesting work. I have some questions about the paper (https://openaccess.thecvf.com/content/ICCV2021/papers/Alonso_Semi-Supervised_Semantic_Segmentation_With_Pixel-Level_Contrastive_Learning_From_a_Class-Wise_ICCV_2021_paper.pdf)

I do not find any explaination about the S_{c, \theta} of Eq.(8) in Section 4.2

In the experiments, the reference number of tables seems to be wrong, e.g., Table 4.3.1 and Table 4.4
opened by HeimingX 0
Time question

Hello! Thank you for your code implementation.

As mentioned in the paper, all experiments were performed on a single NVIDIA Tesla V100 GPU, it would be very helpful if you mention approximately how much time was required for the model training per experiment.

Looking forward to hearing from you soon.

opened by mradul2 1
Question about the contrastive loss
I am wondering why the memory are normalized for the input of selector, but the features are not. https://github.com/Shathe/SemiSeg-Contrastive/blob/dc53c6da64f91c1ebce320f6e2ce3cc9f3df7414/contrastive_losses.py#L36-L37 https://github.com/Shathe/SemiSeg-Contrastive/blob/dc53c6da64f91c1ebce320f6e2ce3cc9f3df7414/contrastive_losses.py#L47-L48

I am very confused why permute is needed here, which causes a inconsistent with Eq 8 in the paper for memory part. https://github.com/Shathe/SemiSeg-Contrastive/blob/dc53c6da64f91c1ebce320f6e2ce3cc9f3df7414/contrastive_losses.py#L58
opened by lliuz 0

Owner

Iñigo Alonso Ruiz

PhD student (University of Zaragoza)

GitHub

ISBI 2022: Cross-level Contrastive Learning and Consistency Constraint for Semi-supervised Medical Image.

Cross-level Contrastive Learning and Consistency Constraint for Semi-supervised Medical Image Introduction This repository contains the PyTorch implem

25 Nov 9, 2022

Pytorch Implementation for NeurIPS (oral) paper: Pixel Level Cycle Association: A New Perspective for Domain Adaptive Semantic Segmentation

Pixel-Level Cycle Association This is the Pytorch implementation of our NeurIPS 2020 Oral paper Pixel-Level Cycle Association: A New Perspective for D

87 Oct 19, 2022

Pixel-wise segmentation on VOC2012 dataset using pytorch.

PiWiSe Pixel-wise segmentation on the VOC2012 dataset using pytorch. FCN SegNet PSPNet UNet RefineNet For a more complete implementation of segmentati

378 Dec 30, 2022

Retinal Vessel Segmentation with Pixel-wise Adaptive Filters (ISBI 2022)

Retinal Vessel Segmentation with Pixel-wise Adaptive Filters (ISBI 2022) Introdu

14 Oct 27, 2022

Official code of Retinal Vessel Segmentation with Pixel-wise Adaptive Filters and Consistency Training

Official code of Retinal Vessel Segmentation with Pixel-wise Adaptive Filters and Consistency Training (ISBI 2022)

7 Feb 10, 2022

[CVPR'22] Weakly Supervised Semantic Segmentation by Pixel-to-Prototype Contrast

wseg Overview The Pytorch implementation of Weakly Supervised Semantic Segmentation by Pixel-to-Prototype Contrast. [arXiv] Though image-level weakly

96 Dec 30, 2022

The Dual Memory is build from a simple CNN for the deep memory and Linear Regression fro the fast Memory

Simple-DMA a simple Dual Memory Architecture for classifications. based on the paper Dual-Memory Deep Learning Architectures for Lifelong Learning of

1 Jan 27, 2022

Implementation of Transformer in Transformer, pixel level attention paired with patch level attention for image classification, in Pytorch

Transformer in Transformer Implementation of Transformer in Transformer, pixel level attention paired with patch level attention for image c

272 Dec 23, 2022

This is an official implementation of "Polarized Self-Attention: Towards High-quality Pixel-wise Regression"

Polarized Self-Attention: Towards High-quality Pixel-wise Regression This is an official implementation of: Huajun Liu, Fuqiang Liu, Xinyi Fan and Don

212 Jan 8, 2023

Code for "PVNet: Pixel-wise Voting Network for 6DoF Pose Estimation" CVPR 2019 oral

Good news! We release a clean version of PVNet: clean-pvnet, including how to train the PVNet on the custom dataset. Use PVNet with a detector. The tr

722 Dec 27, 2022

Tools to create pixel-wise object masks, bounding box labels (2D and 3D) and 3D object model (PLY triangle mesh) for object sequences filmed with an RGB-D camera.

Tools to create pixel-wise object masks, bounding box labels (2D and 3D) and 3D object model (PLY triangle mesh) for object sequences filmed with an RGB-D camera. This project prepares training and testing data for various deep learning projects such as 6D object pose estimation projects singleshotpose, as well as object detection and instance segmentation projects.

305 Dec 16, 2022

Semi-Supervised Semantic Segmentation with Pixel-Level Contrastive Learning from a Class-wise Memory Bank

Related tags

Overview

Semi-Supervised Semantic Segmentation with Pixel-Level Contrastive Learning from a Class-wise Memory Bank

Prerequisites

Contact

Datasets

Cityscapes

Pascal VOC 2012

GTA5

Experiments

Semi-Supervised

Configuration File Description

Memory Restrictions

Semi-Supervised Domain Adaptation

Evaluation

Citation

License

Comments

Report some bugs

Performance gap and issue in code

About the feature vectors from teacher and student

Questions about the paper

Time question

Question about the contrastive loss

Owner

Iñigo Alonso Ruiz

ISBI 2022: Cross-level Contrastive Learning and Consistency Constraint for Semi-supervised Medical Image.

Pytorch Implementation for NeurIPS (oral) paper: Pixel Level Cycle Association: A New Perspective for Domain Adaptive Semantic Segmentation

Pixel-wise segmentation on VOC2012 dataset using pytorch.

Retinal Vessel Segmentation with Pixel-wise Adaptive Filters (ISBI 2022)

Official code of Retinal Vessel Segmentation with Pixel-wise Adaptive Filters and Consistency Training

[CVPR'22] Weakly Supervised Semantic Segmentation by Pixel-to-Prototype Contrast

The Dual Memory is build from a simple CNN for the deep memory and Linear Regression fro the fast Memory

Implementation of Transformer in Transformer, pixel level attention paired with patch level attention for image classification, in Pytorch

This is an official implementation of "Polarized Self-Attention: Towards High-quality Pixel-wise Regression"

Code for "PVNet: Pixel-wise Voting Network for 6DoF Pose Estimation" CVPR 2019 oral

Tools to create pixel-wise object masks, bounding box labels (2D and 3D) and 3D object model (PLY triangle mesh) for object sequences filmed with an RGB-D camera.

Shape-aware Semi-supervised 3D Semantic Segmentation for Medical Images

Semi-supervised Semantic Segmentation with Directional Context-aware Consistency (CVPR 2021)

Anti-Adversarially Manipulated Attributions for Weakly and Semi-Supervised Semantic Segmentation (CVPR 2021)

Semi-supervised Semantic Segmentation with Directional Context-aware Consistency (CVPR 2021)

[CVPR 2021] Semi-Supervised Semantic Segmentation with Cross Pseudo Supervision

ST++: Make Self-training Work Better for Semi-supervised Semantic Segmentation

[ICCV 2021] A Simple Baseline for Semi-supervised Semantic Segmentation with Strong Data Augmentation

[CVPR 2022] Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels