Implementation of DropLoss for Long-Tail Instance Segmentation in Pytorch

Overview

[AAAI 2021]DropLoss for Long-Tail Instance Segmentation

[AAAI 2021] DropLoss for Long-Tail Instance Segmentation
Ting-I Hsieh*, Esther Robb*, Hwann-Tzong Chen, Jia-Bin Huang.
Association for the Advancement of Artificial Intelligence (AAAI), 2021

Image Figure: Measuring the performance tradeoff. Comparison between rare, common, and frequent categories AP for baselines and our method. We visualize the tradeoff for ‘common vs. frequent’ and ‘rare vs. frequent’as a Pareto frontier, where the top-right position indicates an ideal tradeoff between objectives. DropLoss achieves an improved tradeoff between object categories, resulting in higher overall AP.

This project is a pytorch implementation of DropLoss for Long-Tail Instance Segmentation. DropLoss improves long-tail instance segmentation by adaptively removing discouraging gradients to infrequent classes. A majority of the code is modified from facebookresearch/detectron2 and tztztztztz/eql.detectron2.

Progress

  • Training code.
  • Evaluation code.
  • LVIS v1.0 datasets.
  • Provide checkpoint model.

Installation

Requirements

  • Linux or macOS with Python = 3.7
  • PyTorch = 1.4 and torchvision that matches the PyTorch installation. Install them together at pytorch.org to make sure of this
  • OpenCV (optional but needed for demos and visualization)

Build Detectron2 from Source

gcc & g++ ≥ 5 are required. ninja is recommended for faster build.

After installing them, run:

python -m pip install 'git+https://github.com/facebookresearch/detectron2.git'
# (add --user if you don't have permission)

# Or, to install it from a local clone:
git clone https://github.com/facebookresearch/detectron2.git
python -m pip install -e detectron2


# Or if you are on macOS
CC=clang CXX=clang++ ARCHFLAGS="-arch x86_64" python -m pip install ......

Remove the latest fvcore package and install an older version:

pip uninstall fvcore
pip install fvcore==0.1.1.post200513

LVIS Dataset

Following the instructions of README.md to set up the LVIS dataset.

Training

To train a model with 8 GPUs run:

cd /path/to/detectron2/projects/DropLoss
python train_net.py --config-file configs/droploss_mask_rcnn_R_50_FPN_1x.yaml --num-gpus 8

Evaluation

Model evaluation can be done similarly:

cd /path/to/detectron2/projects/DropLoss
python train_net.py --config-file configs/droploss_mask_rcnn_R_50_FPN_1x.yaml --eval-only MODEL.WEIGHTS /path/to/model_checkpoint

Citing DropLoss

If you use DropLoss, please use the following BibTeX entry.

@inproceedings{DBLP:conf/aaai/Ting21,
  author 	= {Hsieh, Ting-I and Esther Robb and Chen, Hwann-Tzong and Huang, Jia-Bin},
  title     = {DropLoss for Long-Tail Instance Segmentation},
  booktitle = {Proceedings of the Workshop on Artificial Intelligence Safety 2021
               (SafeAI 2021) co-located with the Thirty-Fifth {AAAI} Conference on
               Artificial Intelligence {(AAAI} 2021), Virtual, February 8, 2021},
  year      = {2021}
  }
You might also like...
PyTorch code for the paper
PyTorch code for the paper "FIERY: Future Instance Segmentation in Bird's-Eye view from Surround Monocular Cameras"

FIERY This is the PyTorch implementation for inference and training of the future prediction bird's-eye view network as described in: FIERY: Future In

This is an official implementation for "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" on Object Detection and Instance Segmentation.

Swin Transformer for Object Detection This repo contains the supported code and configuration files to reproduce object detection results of Swin Tran

This is the official implementation of the paper
This is the official implementation of the paper "Object Propagation via Inter-Frame Attentions for Temporally Stable Video Instance Segmentation".

[CVPRW 2021] - Object Propagation via Inter-Frame Attentions for Temporally Stable Video Instance Segmentation

Keras implementation of PersonLab for Multi-Person Pose Estimation and Instance Segmentation.

PersonLab This is a Keras implementation of PersonLab for Multi-Person Pose Estimation and Instance Segmentation. The model predicts heatmaps and vari

YolactEdge: Real-time Instance Segmentation on the Edge
YolactEdge: Real-time Instance Segmentation on the Edge

YolactEdge, the first competitive instance segmentation approach that runs on small edge devices at real-time speeds. Specifically, YolactEdge runs at up to 30.8 FPS on a Jetson AGX Xavier (and 172.7 FPS on an RTX 2080 Ti) with a ResNet-101 backbone on 550x550 resolution images.

the code used for the preprint Embedding-based Instance Segmentation of Microscopy Images.
the code used for the preprint Embedding-based Instance Segmentation of Microscopy Images.

EmbedSeg Introduction This repository hosts the version of the code used for the preprint Embedding-based Instance Segmentation of Microscopy Images.

Learning RGB-D Feature Embeddings for Unseen Object Instance Segmentation
Learning RGB-D Feature Embeddings for Unseen Object Instance Segmentation

Unseen Object Clustering: Learning RGB-D Feature Embeddings for Unseen Object Instance Segmentation Introduction In this work, we propose a new method

[CVPR2021 Oral] End-to-End Video Instance Segmentation with Transformers
[CVPR2021 Oral] End-to-End Video Instance Segmentation with Transformers

VisTR: End-to-End Video Instance Segmentation with Transformers This is the official implementation of the VisTR paper: Installation We provide instru

Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers [CVPR 2021]
Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers [CVPR 2021]

Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers [BCNet, CVPR 2021] This is the official pytorch implementation of BCNet built on

Comments
  • Update README.md

    Update README.md

    Thanks for your contribution!

    If you're sending a large PR (e.g., >50 lines), please open an issue first about the feature / bug, and indicate how you want to contribute. See more at https://detectron2.readthedocs.io/notes/contributing.html#pull-requests about how we handle PRs.

    Before submitting a PR, please run dev/linter.sh to lint the code.

    opened by e-271 0
  • Update README.md

    Update README.md

    Thanks for your contribution!

    If you're sending a large PR (e.g., >50 lines), please open an issue first about the feature / bug, and indicate how you want to contribute. See more at https://detectron2.readthedocs.io/notes/contributing.html#pull-requests about how we handle PRs.

    Before submitting a PR, please run dev/linter.sh to lint the code.

    opened by e-271 0
  • About the implementation of eq(6)

    About the implementation of eq(6)

    Hi @timy90022 , Thanks for sharing your excellent work. When I try to understand the implementation of Eq(6), I find a bit difference between the implementation and the Eq(6) in the paper.

     def exclude_func_and_ratio(self):
            # instance-level weight
            bg_ind = self.n_c
            weight = (self.gt_classes != bg_ind)
    
            gt_classes    = self.gt_classes[weight]
            # exclude_ratio = \mu_{f_j}}
            exclude_ratio = torch.mean((self.freq_info[gt_classes] < self.lambda_).float())
            # weight = E(r)
            weight = weight.float().view(self.n_i, 1).expand(self.n_i, self.n_c)
    
            return weight, exclude_ratio
    
    
        def threshold_func(self):
            # class-level weight
            weight = self.pred_class_logits.new_zeros(self.n_c)
            # weight = T_{lambda}(f_j)
            weight[self.freq_info < self.lambda_] = 1
            weight = weight.view(1, self.n_c).expand(self.n_i, self.n_c)
    
            # fg = E(r)
            # ratio = \mu_{f_j}
            fg, ratio = self.exclude_func_and_ratio()
            # bg =  1 - E(r)
            bg = 1 - fg
            # random = (1-E(r)) * rand
            random = torch.rand_like(bg) * bg
    
            random = torch.where(random>ratio, torch.ones_like(random), torch.zeros_like(random))
            # weight = { [ (1-E(r)) * rand > \mu_{f_j} ? 1 : 0 ] + E(r) } * T_{\lambda}(f_j)
            weight = (random + fg) * weight
    
            return weight
    
    
        def drop_loss(self):
    
            self.n_i, self.n_c = self.pred_class_logits.size()
    
            def expand_label(pred, gt_classes):
                target = pred.new_zeros(self.n_i, self.n_c + 1)
                target[torch.arange(self.n_i), gt_classes] = 1
                return target[:, :self.n_c]
    
            target = expand_label(self.pred_class_logits, self.gt_classes)
            # drop_w =  1 - { [ (1-E(r)) * rand > \mu_{f_j} ? 1 : 0 ] + E(r) } * T_{\lambda}(f_j) * (1-y_j)
            # When E(r) = 1, drop_w = 1 - T_{\lambda}(f_j) * (1-y_j)
            # When E(r) = 0, drop_w = 1 - { [ (1-E(r)) * rand > \mu_{f_j} ? 1 : 0 ] } * T_{\lambda}(f_j) * (1-y_j) ????
                            # when rand > \mu_{f_j}, drop_w = 1 - T_{\lambda}(f_j) * (1-y_j)
                            # when rand <= \mu_{f_j}, drop_w = 1
            self.drop_w = 1 - self.threshold_func() * (1 - target)
    
            self.cls_loss = F.binary_cross_entropy_with_logits(self.pred_class_logits, target,
                                                          reduction='none')
            return torch.sum(self.cls_loss * self.drop_w) / self.n_i
    
    # When E(r) = 0, drop_w = 1 - [ rand > \mu_{f_j} ? 1 : 0 ] *  T_{\lambda}(f_j) * (1-y_j) 
    # when rand > \mu_{f_j}, drop_w = 1 - T_{\lambda}(f_j) * (1-y_j)???????
    # when rand <= \mu_{f_j}, drop_w = 1
    

    The implementation is inconsistent with the otherwise condition in Eq(6) .

    drop_w = 1 - T_{\lambda}(f_j) * (1-y_j)
    

    Could you please explain that? Is there any misunderstanding about the code?

    opened by xizero00 0
Owner
Tim
Tim
Implementation of "Distribution Alignment: A Unified Framework for Long-tail Visual Recognition"(CVPR 2021)

Implementation of "Distribution Alignment: A Unified Framework for Long-tail Visual Recognition"(CVPR 2021)

null 105 Nov 7, 2022
Official implementation for CVPR 2021 paper: Adaptive Class Suppression Loss for Long-Tail Object Detection

Adaptive Class Suppression Loss for Long-Tail Object Detection This repo is the official implementation for CVPR 2021 paper: Adaptive Class Suppressio

CASIA-IVA-Lab 67 Dec 4, 2022
A scientific and useful toolbox, which contains practical and effective long-tail related tricks with extensive experimental results

Bag of tricks for long-tailed visual recognition with deep convolutional neural networks This repository is the official PyTorch implementation of AAA

Yong-Shun Zhang 181 Dec 28, 2022
OpenLT: An open-source project for long-tail classification

OpenLT: An open-source project for long-tail classification Supported Methods for Long-tailed Recognition: Cross-Entropy Loss Focal Loss (ICCV'17) Cla

Ming Li 37 Sep 15, 2022
Leveraging Instance-, Image- and Dataset-Level Information for Weakly Supervised Instance Segmentation

Leveraging Instance-, Image- and Dataset-Level Information for Weakly Supervised Instance Segmentation This paper has been accepted and early accessed

Yun Liu 39 Sep 20, 2022
TorchDistiller - a collection of the open source pytorch code for knowledge distillation, especially for the perception tasks, including semantic segmentation, depth estimation, object detection and instance segmentation.

This project is a collection of the open source pytorch code for knowledge distillation, especially for the perception tasks, including semantic segmentation, depth estimation, object detection and instance segmentation.

yifan liu 147 Dec 3, 2022
A coin flip game in which you can put the amount of money below or equal to 1000 and then choose heads or tail

COIN_FLIPPY ##This is a simple example package. You can use Github-flavored Markdown to write your content. Coinflippy A coin flip game in which you c

null 2 Dec 26, 2021
[ArXiv 2021] Data-Efficient Instance Generation from Instance Discrimination

InsGen - Data-Efficient Instance Generation from Instance Discrimination Data-Efficient Instance Generation from Instance Discrimination Ceyuan Yang,

GenForce: May Generative Force Be with You 93 Dec 25, 2022
Fast, modular reference implementation of Instance Segmentation and Object Detection algorithms in PyTorch.

Faster R-CNN and Mask R-CNN in PyTorch 1.0 maskrcnn-benchmark has been deprecated. Please see detectron2, which includes implementations for all model

Facebook Research 9k Jan 4, 2023
The PyTorch implementation of DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervision.

DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervision The PyTorch implementation of DiscoBox: Weakly Supe

Shiyi Lan 1 Oct 23, 2021