DecoupledNet is semantic segmentation system which using heterogeneous annotations

Hyeonwoo Noh

Last update: Sep 22, 2021

Related tags

Deep Learning DecoupledNet

Overview

DecoupledNet: Decoupled Deep Neural Network for Semi-supervised Semantic Segmentation

Created by Seunghoon Hong, Hyeonwoo Noh and Bohyung Han at POSTECH

Acknowledgements: Thanks to Yangqing Jia and the BVLC team for creating Caffe.

Introduction

DecoupledNet is semantic segmentation system which using heterogeneous annotations. Based on pre-trained classification network, DecoupledNet fine-tune the segmentation network with very small amount of segmentation annotations and obtains excellent results on semantic segmentation task.

Detailed description of the system will be provided by our technical report [arXiv tech report] http://arxiv.org/abs/1506.04924

Citation

If you're using this code in a publication, please cite our papers.

@article{hong2015decoupled,
  title={Decoupled Deep Neural Network for Semi-supervised Semantic Segmentation},
  author={Hong, Seunghoon and Noh, Hyeonwoo and Han, Bohyung},
  journal={arXiv preprint arXiv:1506.04924},
  year={2015}
}

Pre-trained Model

If you need model definition and pre-trained model only, you can download them from following location: 0. caffe for DecoupledNet: https://github.com/HyeonwooNoh/caffe 0. DecoupledNet [Full annotation] : 0. [prototxt] (http://cvlab.postech.ac.kr/research/decouplednet/model/DecoupledNet_Full_anno/DecoupledNet_Full_anno_inference_deploy.prototxt) 0. [caffemodel] (http://cvlab.postech.ac.kr/research/decouplednet/model/DecoupledNet_Full_anno/DecoupledNet_Full_anno_inference.caffemodel) 0. DecoupledNet [25 annotations] : 0. [prototxt] (http://cvlab.postech.ac.kr/research/decouplednet/model/DecoupledNet_25_anno/DecoupledNet_25_anno_inference_deploy.prototxt) 0. [caffemodel] (http://cvlab.postech.ac.kr/research/decouplednet/model/DecoupledNet_25_anno/DecoupledNet_25_anno_inference.caffemodel) 0. DecoupledNet [10 annotations] : 0. [prototxt] (http://cvlab.postech.ac.kr/research/decouplednet/model/DecoupledNet_10_anno/DecoupledNet_10_anno_inference_deploy.prototxt) 0. [caffemodel] (http://cvlab.postech.ac.kr/research/decouplednet/model/DecoupledNet_10_anno/DecoupledNet_10_anno_inference.caffemodel) 0. DecoupledNet [5 annotations] : 0. [prototxt] (http://cvlab.postech.ac.kr/research/decouplednet/model/DecoupledNet_5_anno/DecoupledNet_5_anno_inference_deploy.prototxt) 0. [caffemodel] (http://cvlab.postech.ac.kr/research/decouplednet/model/DecoupledNet_5_anno/DecoupledNet_5_anno_inference.caffemodel)

Licence

This software is being made available for research purpose only. Check LICENSE file for details.

System Requirements

This software is tested on Ubuntu 14.04 LTS (64bit).

Prerequisites 0. MATLAB (tested with 2014b on 64-bit Linux) 0. prerequisites for caffe(http://caffe.berkeleyvision.org/installation.html#prequequisites)

Installing DecoupledNet

By running "setup.sh" you can download all the necessary file for training and inference including: 0. caffe: you need modified version of caffe which support DeconvNet - https://github.com/HyeonwooNoh/caffe.git 0. data: data used for training 0. model: caffemodel of trained DecoupledNet and caffemodel of pre-trained classification network

Training DecoupledNet

Training scripts are included in ./training/ directory

To train DecoupledNet with various setting, you can run following scripts 0. 001_convert_classification_network_to_fp_bp_network.sh: * converting classification network to make forward-backward propagation possible (this converted model is prerequisite for DecoupledNet training) 0. 002_train_seg_Full_anno.sh: * training DecoupledNet with full segmentation annotations 0. 003_train_seg_25_anno.sh: * training DecoupledNet with 25 segmentation annotations per class 0. 004_train_seg_10_anno.sh: * training DecoupledNet with 10 segmentation annotations per class 0. 005_train_seg_5_anno.sh: * training DecoupledNet with 5 segmentation annotations per class

DecoupledNet Inference

Inference scripts are included in ./inference/ directory.

Run run_demo.m to run DecoupledNet on VOC2012 test data.

run_demo.m script will run DecoupledNet trained in various settings (Full, 25, 10, 5 annotations): 0. DecoupledNet-Full (66.6 mean I/U on PASCAL VOC 2012 Test) 0. DecoupledNet-25 (62.5 mean I/U on PASCAL VOC 2012 Test) 0. DecoupledNet-10 (58.7 mean I/U on PASCAL VOC 2012 Test) 0. DecoupledNet-5 (54.7 mean I/U on PASCAL VOC 2012 Test)

Runtime type annotations for the shape, dtype etc. of PyTorch Tensors.

torchtyping Type annotations for a tensor's shape, dtype, names, ... Turn this: def batch_outer_product(x: torch.Tensor, y: torch.Tensor) - torch.Ten

1.2k Jan 3, 2023

Synthetic LiDAR sequential point cloud dataset with point-wise annotations

SynLiDAR dataset: Learning From Synthetic LiDAR Sequential Point Cloud This is official repository of the SynLiDAR dataset. For technical details, ple

78 Dec 27, 2022

[BMVC2021] The official implementation of "DomainMix: Learning Generalizable Person Re-Identification Without Human Annotations"

DomainMix [BMVC2021] The official implementation of "DomainMix: Learning Generalizable Person Re-Identification Without Human Annotations" [paper] [de

17 Dec 20, 2022

68 keypoint annotations for COFW test data

68 keypoint annotations for COFW test data This repository contains manually annotated 68 keypoints for COFW test data (original annotation of CFOW da

31 Dec 6, 2022

MACE is a deep learning inference framework optimized for mobile heterogeneous computing platforms.

4.7k Dec 29, 2022

Deep generative modeling for time-stamped heterogeneous data, enabling high-fidelity models for a large variety of spatio-temporal domains.

Neural Spatio-Temporal Point Processes [arxiv] Ricky T. Q. Chen, Brandon Amos, Maximilian Nickel Abstract. We propose a new class of parameterizations

75 Dec 19, 2022

Comments

Inference problem?

I think there exit some problem in inference process. You forward once to get a cls score and bp to pool5 and then forward to whole deconv process. You have calculate softmax when you get the result. But I think the softmax score doesn't work for segmentation. e.g. 21 channel, 1st channel is background and 7th channel contain true object. The probability of true object seems smaller than that of background channel when calculate by softmax, so that the result map is all 0(max of all channels). How do you solve this problem?

Thanks.

opened by terrychenism 7
Reproducing resulting model from DecoupledNet
While trying to train DecoupledNet on my own dataset that targets a specific group of classes (mostly humans), I tried to retrain the network using the Full Annotation scripts, changed with my own dataset. Since the results of segmentation with my retrained network were worse than those of DecoupledNet without specialized dataset, I tried to reproduce the DecoupledNet model using the VOC2012 dataset, i.e. using the Full annotation scripts as is.

I let it run for 17500 iterations and got results that are getting similar, but still pretty far in term of segmentation quality. See illustrations below with a segmentation on a test image of mine (soccer player). The heatmap represents the human segmentation.

Result using original Full Annotation DecoupledNet model

Result after 17500 iterations retraining (Full annotation scripts / VOC 2012 dataset):

As you can see, there's a lot more noise, and it's far from being as smooth. I thought: maybe it hasn't fully converged so I should just train more, so I finetuned for 17500 more iterations, but that didn't seem to improve.

Result after 35000 iterations training:

Did you do any further processing to reach the model you got or should I be able to get a similar model with the scripts you provided? Do you have any idea why my results would be so different?

I trained on a Titan X on Ubuntu 14.04 and the installation from caffe of DecoupledNet, without modification.
opened by xionluhnis 5
GPU memory requirements
I am trying to fine-tune DecoupledNet with a specific dataset of mine but am failing to do so with Cuda. The CPU version works, but it's not realistic given the time it takes, so I wonder if you had any clear requirements about the GPU card that I may have missed.

I unsuccessfully tried the example of 002_train_seg_Full_anno, with a NVIDIA K20 (6GB).

DecoupledNet_Full_anno is ~1.6GB

fp_bp_model is ~1GB

Here is the training error.

For my dataset, I tried to split it (full ~20GB) into 1~2GB slices to train on, but I couldn't and got cuda out of memory. I even tried using a batch size of 1 with a set of only 100 pictures, and it still failed because of memory.

What GPU did you use for training? Shouldn't 6GB be sufficient? Is caffe replicating the model in memory needlessly? What makes it use so much memory?

Thanks for any hint and/or hardware requirement.

Note: I tried training on two different machines, each one with a Nvidia K20 (6GB), the result is the same.
opened by xionluhnis 5
num_output of deconv2_2

In the prototxt files (e.g. 002_train_seg_Full_anno.prototxt) deconv2_2 is defined to have 64 output channels (num_output), while 128 would be expected if it is a mirror of the feedforward pathway.

I haven't tested the results of changing this, but it would be interesting to know.

opened by williford 0

DecoupledNet is semantic segmentation system which using heterogeneous annotations

Related tags

Overview

DecoupledNet: Decoupled Deep Neural Network for Semi-supervised Semantic Segmentation

Introduction

Citation

Pre-trained Model

Licence

System Requirements

Installing DecoupledNet

Training DecoupledNet

DecoupledNet Inference

You might also like...

Runtime type annotations for the shape, dtype etc. of PyTorch Tensors.

Synthetic LiDAR sequential point cloud dataset with point-wise annotations

[BMVC2021] The official implementation of "DomainMix: Learning Generalizable Person Re-Identification Without Human Annotations"

68 keypoint annotations for COFW test data

MACE is a deep learning inference framework optimized for mobile heterogeneous computing platforms.

Deep generative modeling for time-stamped heterogeneous data, enabling high-fidelity models for a large variety of spatio-temporal domains.

Scalable Graph Neural Networks for Heterogeneous Graphs

DVG-Face: Dual Variational Generation for Heterogeneous Face Recognition, TPAMI 2021

Radar-to-Lidar: Heterogeneous Place Recognition via Joint Learning

Comments

Inference problem?

Reproducing resulting model from DecoupledNet

GPU memory requirements

num_output of deconv2_2

Owner

Hyeonwoo Noh

STEAL - Learning Semantic Boundaries from Noisy Annotations (CVPR 2019)

BoxInst: High-Performance Instance Segmentation with Box Annotations

This is the unofficial code of Deep Dual-resolution Networks for Real-time and Accurate Semantic Segmentation of Road Scenes. which achieve state-of-the-art trade-off between accuracy and speed on cityscapes and camvid, without using inference acceleration and extra data

Mae segmentation - Reproduction of semantic segmentation using masked autoencoder (mae)

Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentation, CVPR 2018

Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT

Official implementation for the paper: "Multi-label Classification with Partial Annotations using Class-aware Selective Loss"

Product-based-recommendation-system - A product based recommendation system which uses Machine learning algorithm such as KNN and cosine similarity

Segmentation in Style: Unsupervised Semantic Image Segmentation with Stylegan and CLIP

TorchDistiller - a collection of the open source pytorch code for knowledge distillation, especially for the perception tasks, including semantic segmentation, depth estimation, object detection and instance segmentation.