Pytorch implementation of SenFormer: Efficient Self-Ensemble Framework for Semantic Segmentation

Last update: Dec 26, 2022

Related tags

Overview

SenFormer: Efficient Self-Ensemble Framework for Semantic Segmentation

Efficient Self-Ensemble Framework for Semantic Segmentation by Walid Bousselham, Guillaume Thibault, Lucas Pagano, Archana Machireddy, Joe Gray, Young Hwan Chang and Xubo Song.

This repository contains the official Pytorch implementation of training & evaluation code and the pretrained models for SenFormer.

💾 Code Snippet (SenFormer)| ⌨️ Code Snippet (FPNT)| 📜 Paper | 论文

🔨 Installation

Conda environment

Clone this repository and enter it: git clone [email protected]:WalBouss/SenFormer.git && cd SenFormer.
Create a conda environment conda create -n senformer python=3.8, and activate it conda activate senformer.
Install Pytorch and torchvision conda install pytorch==1.7.1 torchvision==0.8.2 cudatoolkit=10.2 -c pytorch — (you may also switch to other version by specifying the version number).
Install MMCV library pip install mmcv-full==1.4.0
Install MMSegmentation library by running pip install -e . in SenFormer directory.
Install other requirements pip install timm einops

Here is a full script for setting up a conda environment to use SenFormer (with CUDA 10.2 and pytorch 1.7.1):

conda create -n senformer python=3.8
conda activate senformer
conda install pytorch==1.7.1 torchvision==0.8.2 cudatoolkit=10.2 -c pytorch

git clone [email protected]:WalBouss/SenFormer.git && cd SenFormer
pip install mmcv-full==1.4.0
pip install -e .
pip install timm einops

Datasets

For datasets preparations please refer to MMSegmentation guidelines.

Pretrained weights

ResNet pretrained weights will be automatically downloaded before training.

For Swin Transformer ImageNet pretrained weights, you can either:

run bash tools/download_swin_weights.sh in SenFormer project to download all Swin Transformer pretrained weights (it will place weights under pretrain/ folder ).
download desired backbone weights here: Swin-T, Swin-S, Swin-B, Swin-L and place them under pretrain/ folder.
download weights from official repository then, convert them to mmsegmentation format following mmsegmentation guidelines.

🎯 Model Zoo

SenFormer models with ResNet and Swin's backbones and ADE20K, COCO-Stuff 10K, Pascal Context and Cityscapes.

ADE20K

Backbone	mIoU	mIoU (MS)	#params	FLOPs	Resolution	Download
ResNet-50	44.6	45.6	144M	179G	512x512	model	config
ResNet-101	46.5	47.0	163M	199G	512x512	model	config
Swin-Tiny	46.0	46.4	144M	179G	512x512	model	config
Swin-Small	49.2	50.4	165M	202G	512x512	model	config
Swin-Base	51.8	53.2	204M	242G	640x640	model	config
Swin-Large	53.1	54.2	314M	546G	640x640	model	config

COCO-Stuff 10K

Backbone	mIoU	mIoU (MS)	#params	Resolution	Download
ResNet-50	39.0	39.7	144M	512x512	model	config
ResNet-101	39.6	40.6	163M	512x512	model	config
Swin-Large	49.1	50.1	314M	512x512	model	config

Pascal Context

Backbone	mIoU	mIoU (MS)	#params	Resolution	Download
ResNet-50	53.2	54.3	144M	480x480	model	config
ResNet-101	55.1	56.6	163M	480x480	model	config
Swin-Large	62.4	64.0	314M	480x480	model	config

Cityscapes

Backbone	mIoU	mIoU (MS)	#params	Resolution	Download
ResNet-50	78.8	80.1	144M	512x1024	model	config
ResNet-101	80.3	81.4	163M	512x1024	model	config
Swin-Large	82.2	83.3	314M	512x1024	model	config

🔭 Inference

Download one checkpoint weights from above, for example SenFormer with ResNet-50 backbone on ADE20K:

Inference on a dataset

# Single-gpu testing
python tools/test.py senformer_configs/senformer/ade20k/senformer_fpnt_r50_512x512_160k_ade20k.py /path/to/checkpoint_file

# Multi-gpu testing
./tools/dist_test.sh senformer_configs/senformer/ade20k/senformer_fpnt_r50_512x512_160k_ade20k.py /path/to/checkpoint_file <GPU_NUM>

# Multi-gpu, multi-scale testing
tools/dist_test.sh senformer_configs/senformer/ade20k/senformer_fpnt_r50_512x512_160k_ade20k.py /path/to/checkpoint_file <GPU_NUM> --aug-test

Inference on custom data

To generate segmentation maps for your own data, run the following command:

python demo/image_demo.py ${IMAGE_FILE} ${CONFIG_FILE} ${CHECKPOINT_FILE}

Run python demo/image_demo.py --help for additional options.

🔩 Training

Follow above instructions to download ImageNet pretrained weights for backbones and run one of the following command:

# Single-gpu training
python tools/train.py path/to/model/config 

# Multi-gpu training
./tools/dist_train.sh path/to/model/config <GPU_NUM>

For example to train SenFormer with a ResNet-50 as backbone on ADE20K:

# Single-gpu training
python tools/train.py senformer_configs/senformer/ade20k/senformer_fpnt_r50_512x512_160k_ade20k.py 

# Multi-gpu training
./tools/dist_train.sh senformer_configs/senformer/ade20k/senformer_fpnt_r50_512x512_160k_ade20k.py <GPU_NUM>

Note that the default learning rate and training schedule is for an effective batch size of 16, (e.g. 8 GPUs & 2 imgs/gpu).

⭐ Acknowledgement

This code is build using MMsegmentation library as codebase and uses timm and einops as well.

📚 Citation

If you find this repository useful, please consider citing our work 📝 and giving a star 🌟 :

@article{bousselham2021senformer,
  title={Efficient Self-Ensemble Framework for Semantic Segmentation},
  author={Walid Bousselham, Guillaume Thibault, Lucas Pagano, Archana Machireddy, Joe Gray, Young Hwan Chang, Xubo Song},
  journal={arXiv preprint arXiv:2111.13280},
  year={2021}
}

Comments

How to train custom data?

Hi! @WalBouss I use 'swin_small_patch4_window7_224.pth' pretrained model to train my data. It can run ,but I got acc = 0 and loss = 0, like this picture.

I want to ask how to train custom data. where I should modify?

thank you!

opened by FrankFang0813 5
coco stuff

hi, Im student in taiwan

Does the model not support for "coco-stuff" yet ? the file : SenFormer\mmseg\core\evaluation\classesnames which only have three different types of dataset,

dataset_aliases = { 'cityscapes': ['cityscapes'], 'ade': ['ade', 'ade20k'], 'voc': ['voc', 'pascal_voc', 'voc12', 'voc12aug'] }

...?

opened by kevinwei0410 3
The performances of pre-trained weights are lower than the paper

Hi, I notice that the performances of pre-trained weights you provided are lower than the paper. The results in the latest paper are obviously better. Do you have the plan to release the matched pre-trained weights?

opened by ydhongHIT 3
CVE-2007-4559 Patch

Patching CVE-2007-4559

Hi, we are security researchers from the Advanced Research Center at Trellix. We have began a campaign to patch a widespread bug named CVE-2007-4559. CVE-2007-4559 is a 15 year old bug in the Python tarfile package. By using extract() or extractall() on a tarfile object without sanitizing input, a maliciously crafted .tar file could perform a directory path traversal attack. We found at least one unsantized extractall() in your codebase and are providing a patch for you via pull request. The patch essentially checks to see if all tarfile members will be extracted safely and throws an exception otherwise. We encourage you to use this patch or your own solution to secure against CVE-2007-4559. Further technical information about the vulnerability can be found in this blog.

If you have further questions you may contact us through this projects lead researcher Kasimir Schulz.

opened by TrellixVulnTeam 0
mmcv version
Hello @WalBouss thanks for your great work. but could you really edit the dependencies part in the README concerning the pytorch and mmcv-full versions, since it took me a while to adjust the dependencies specifically on windows.

pytorch

conda install pytorch==1.7.0 torchvision==0.8.0 torchaudio==0.7.0 cudatoolkit=10.2 -c pytorch

mmcv-full

pip install mmcv-full==1.4.0 -f https://download.openmmlab.com/mmcv/dist/cu102/torch1.7.0/index.html and this was references from mmcv's official repo https://github.com/open-mmlab/mmcv
Thanks in advance.
opened by alien19 0

Pytorch implementation of SenFormer: Efficient Self-Ensemble Framework for Semantic Segmentation

Related tags

Overview

SenFormer: Efficient Self-Ensemble Framework for Semantic Segmentation

🔨 Installation

Conda environment

Datasets

Pretrained weights

🎯 Model Zoo

ADE20K

COCO-Stuff 10K

Pascal Context

Cityscapes

🔭 Inference

Inference on a dataset

Inference on custom data

🔩 Training

⭐ Acknowledgement

📚 Citation

Comments

How to train custom data?

coco stuff

The performances of pre-trained weights are lower than the paper

CVE-2007-4559 Patch

Patching CVE-2007-4559

mmcv version

Owner

Recall Loss for Semantic Segmentation (This repo implements the paper: Recall Loss for Semantic Segmentation)

An efficient 3D semantic segmentation framework for Urban-scale point clouds like SensatUrban, Campus3D, etc.

Official implementation of "SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers"

Official and maintained implementation of the paper "OSS-Net: Memory Efficient High Resolution Semantic Segmentation of 3D Medical Data" [BMVC 2021].

Implementation for paper: Self-Regulation for Semantic Segmentation

TorchDistiller - a collection of the open source pytorch code for knowledge distillation, especially for the perception tasks, including semantic segmentation, depth estimation, object detection and instance segmentation.

Code for the ICCV 2021 Workshop paper: A Unified Efficient Pyramid Transformer for Semantic Segmentation.

Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentation, CVPR 2018

zeus is a Python implementation of the Ensemble Slice Sampling method.

Code for our paper Domain Adaptive Semantic Segmentation with Self-Supervised Depth Estimation

Self-supervised Augmentation Consistency for Adapting Semantic Segmentation (CVPR 2021)

Code for the paper One Thing One Click: A Self-Training Approach for Weakly Supervised 3D Semantic Segmentation, CVPR 2021.

ST++: Make Self-training Work Better for Semi-supervised Semantic Segmentation

Perturbed Self-Distillation: Weakly Supervised Large-Scale Point Cloud Semantic Segmentation (ICCV2021)

PyTorch implementation of CVPR 2020 paper (Reference-Based Sketch Image Colorization using Augmented-Self Reference and Dense Semantic Correspondence) and pre-trained model on ImageNet dataset

Segmentation in Style: Unsupervised Semantic Image Segmentation with Stylegan and CLIP

Mae segmentation - Reproduction of semantic segmentation using masked autoencoder (mae)

Efficient-GlobalPointer - Pytorch Efficient GlobalPointer

HyperSeg: Patch-wise Hypernetwork for Real-time Semantic Segmentation Official PyTorch Implementation