Implementation of DropLoss for Long-Tail Instance Segmentation in Pytorch

Last update: Dec 2, 2022

Overview

[AAAI 2021]DropLoss for Long-Tail Instance Segmentation

[AAAI 2021] DropLoss for Long-Tail Instance Segmentation
Ting-I Hsieh*, Esther Robb*, Hwann-Tzong Chen, Jia-Bin Huang.
Association for the Advancement of Artificial Intelligence (AAAI), 2021

Figure: Measuring the performance tradeoff. Comparison between rare, common, and frequent categories AP for baselines and our method. We visualize the tradeoff for ‘common vs. frequent’ and ‘rare vs. frequent’as a Pareto frontier, where the top-right position indicates an ideal tradeoff between objectives. DropLoss achieves an improved tradeoff between object categories, resulting in higher overall AP.

This project is a pytorch implementation of DropLoss for Long-Tail Instance Segmentation. DropLoss improves long-tail instance segmentation by adaptively removing discouraging gradients to infrequent classes. A majority of the code is modified from facebookresearch/detectron2 and tztztztztz/eql.detectron2.

Progress

Training code.
Evaluation code.
LVIS v1.0 datasets.
Provide checkpoint model.

Installation

Requirements

Linux or macOS with Python = 3.7
PyTorch = 1.4 and torchvision that matches the PyTorch installation. Install them together at pytorch.org to make sure of this
OpenCV (optional but needed for demos and visualization)

Build Detectron2 from Source

gcc & g++ ≥ 5 are required. ninja is recommended for faster build.

After installing them, run:

python -m pip install 'git+https://github.com/facebookresearch/detectron2.git'
# (add --user if you don't have permission)

# Or, to install it from a local clone:
git clone https://github.com/facebookresearch/detectron2.git
python -m pip install -e detectron2


# Or if you are on macOS
CC=clang CXX=clang++ ARCHFLAGS="-arch x86_64" python -m pip install ......

Remove the latest fvcore package and install an older version:

pip uninstall fvcore
pip install fvcore==0.1.1.post200513

LVIS Dataset

Following the instructions of README.md to set up the LVIS dataset.

Training

To train a model with 8 GPUs run:

cd /path/to/detectron2/projects/DropLoss
python train_net.py --config-file configs/droploss_mask_rcnn_R_50_FPN_1x.yaml --num-gpus 8

Evaluation

Model evaluation can be done similarly:

cd /path/to/detectron2/projects/DropLoss
python train_net.py --config-file configs/droploss_mask_rcnn_R_50_FPN_1x.yaml --eval-only MODEL.WEIGHTS /path/to/model_checkpoint

Citing DropLoss

If you use DropLoss, please use the following BibTeX entry.

@inproceedings{DBLP:conf/aaai/Ting21,
  author 	= {Hsieh, Ting-I and Esther Robb and Chen, Hwann-Tzong and Huang, Jia-Bin},
  title     = {DropLoss for Long-Tail Instance Segmentation},
  booktitle = {Proceedings of the Workshop on Artificial Intelligence Safety 2021
               (SafeAI 2021) co-located with the Thirty-Fifth {AAAI} Conference on
               Artificial Intelligence {(AAAI} 2021), Virtual, February 8, 2021},
  year      = {2021}
  }

You might also like...

PyTorch code for the paper "FIERY: Future Instance Segmentation in Bird's-Eye view from Surround Monocular Cameras"

FIERY This is the PyTorch implementation for inference and training of the future prediction bird's-eye view network as described in: FIERY: Future In

406 Dec 24, 2022

This is an official implementation for "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" on Object Detection and Instance Segmentation.

Swin Transformer for Object Detection This repo contains the supported code and configuration files to reproduce object detection results of Swin Tran

1.4k Dec 30, 2022

This is the official implementation of the paper "Object Propagation via Inter-Frame Attentions for Temporally Stable Video Instance Segmentation".

[CVPRW 2021] - Object Propagation via Inter-Frame Attentions for Temporally Stable Video Instance Segmentation

6 May 3, 2022

Keras implementation of PersonLab for Multi-Person Pose Estimation and Instance Segmentation.

PersonLab This is a Keras implementation of PersonLab for Multi-Person Pose Estimation and Instance Segmentation. The model predicts heatmaps and vari

160 Dec 21, 2022

YolactEdge: Real-time Instance Segmentation on the Edge

YolactEdge, the first competitive instance segmentation approach that runs on small edge devices at real-time speeds. Specifically, YolactEdge runs at up to 30.8 FPS on a Jetson AGX Xavier (and 172.7 FPS on an RTX 2080 Ti) with a ResNet-101 backbone on 550x550 resolution images.

1.1k Jan 6, 2023

the code used for the preprint Embedding-based Instance Segmentation of Microscopy Images.

EmbedSeg Introduction This repository hosts the version of the code used for the preprint Embedding-based Instance Segmentation of Microscopy Images.

88 Dec 25, 2022

Learning RGB-D Feature Embeddings for Unseen Object Instance Segmentation

Unseen Object Clustering: Learning RGB-D Feature Embeddings for Unseen Object Instance Segmentation Introduction In this work, we propose a new method

132 Dec 13, 2022

[CVPR2021 Oral] End-to-End Video Instance Segmentation with Transformers

VisTR: End-to-End Video Instance Segmentation with Transformers This is the official implementation of the VisTR paper: Installation We provide instru

687 Jan 7, 2023

Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers [CVPR 2021]

Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers [BCNet, CVPR 2021] This is the official pytorch implementation of BCNet built on

434 Dec 1, 2022

Comments

Update README.md

Thanks for your contribution!

If you're sending a large PR (e.g., >50 lines), please open an issue first about the feature / bug, and indicate how you want to contribute. See more at https://detectron2.readthedocs.io/notes/contributing.html#pull-requests about how we handle PRs.

Before submitting a PR, please run dev/linter.sh to lint the code.

opened by e-271 0
Update README.md

Thanks for your contribution!

If you're sending a large PR (e.g., >50 lines), please open an issue first about the feature / bug, and indicate how you want to contribute. See more at https://detectron2.readthedocs.io/notes/contributing.html#pull-requests about how we handle PRs.

Before submitting a PR, please run dev/linter.sh to lint the code.

opened by e-271 0

About the implementation of eq(6)

Hi @timy90022 ， Thanks for sharing your excellent work. When I try to understand the implementation of Eq(6), I find a bit difference between the implementation and the Eq(6) in the paper.

 def exclude_func_and_ratio(self):
        # instance-level weight
        bg_ind = self.n_c
        weight = (self.gt_classes != bg_ind)

        gt_classes    = self.gt_classes[weight]
        # exclude_ratio = \mu_{f_j}}
        exclude_ratio = torch.mean((self.freq_info[gt_classes] < self.lambda_).float())
        # weight = E(r)
        weight = weight.float().view(self.n_i, 1).expand(self.n_i, self.n_c)

        return weight, exclude_ratio


    def threshold_func(self):
        # class-level weight
        weight = self.pred_class_logits.new_zeros(self.n_c)
        # weight = T_{lambda}(f_j)
        weight[self.freq_info < self.lambda_] = 1
        weight = weight.view(1, self.n_c).expand(self.n_i, self.n_c)

        # fg = E(r)
        # ratio = \mu_{f_j}
        fg, ratio = self.exclude_func_and_ratio()
        # bg =  1 - E(r)
        bg = 1 - fg
        # random = (1-E(r)) * rand
        random = torch.rand_like(bg) * bg

        random = torch.where(random>ratio, torch.ones_like(random), torch.zeros_like(random))
        # weight = { [ (1-E(r)) * rand > \mu_{f_j} ? 1 : 0 ] + E(r) } * T_{\lambda}(f_j)
        weight = (random + fg) * weight

        return weight


    def drop_loss(self):

        self.n_i, self.n_c = self.pred_class_logits.size()

        def expand_label(pred, gt_classes):
            target = pred.new_zeros(self.n_i, self.n_c + 1)
            target[torch.arange(self.n_i), gt_classes] = 1
            return target[:, :self.n_c]

        target = expand_label(self.pred_class_logits, self.gt_classes)
        # drop_w =  1 - { [ (1-E(r)) * rand > \mu_{f_j} ? 1 : 0 ] + E(r) } * T_{\lambda}(f_j) * (1-y_j)
        # When E(r) = 1, drop_w = 1 - T_{\lambda}(f_j) * (1-y_j)
        # When E(r) = 0, drop_w = 1 - { [ (1-E(r)) * rand > \mu_{f_j} ? 1 : 0 ] } * T_{\lambda}(f_j) * (1-y_j) ????
                        # when rand > \mu_{f_j}, drop_w = 1 - T_{\lambda}(f_j) * (1-y_j)
                        # when rand <= \mu_{f_j}, drop_w = 1
        self.drop_w = 1 - self.threshold_func() * (1 - target)

        self.cls_loss = F.binary_cross_entropy_with_logits(self.pred_class_logits, target,
                                                      reduction='none')
        return torch.sum(self.cls_loss * self.drop_w) / self.n_i

# When E(r) = 0, drop_w = 1 - [ rand > \mu_{f_j} ? 1 : 0 ] *  T_{\lambda}(f_j) * (1-y_j) 
# when rand > \mu_{f_j}, drop_w = 1 - T_{\lambda}(f_j) * (1-y_j)???????
# when rand <= \mu_{f_j}, drop_w = 1

The implementation is inconsistent with the otherwise condition in Eq(6) .

drop_w = 1 - T_{\lambda}(f_j) * (1-y_j)

Could you please explain that? Is there any misunderstanding about the code?

opened by xizero00 0

Owner

Tim

GitHub

Implementation of "Distribution Alignment: A Unified Framework for Long-tail Visual Recognition"(CVPR 2021)

105 Nov 7, 2022

Official implementation for CVPR 2021 paper: Adaptive Class Suppression Loss for Long-Tail Object Detection

Adaptive Class Suppression Loss for Long-Tail Object Detection This repo is the official implementation for CVPR 2021 paper: Adaptive Class Suppressio

67 Dec 4, 2022

A scientific and useful toolbox, which contains practical and effective long-tail related tricks with extensive experimental results

Bag of tricks for long-tailed visual recognition with deep convolutional neural networks This repository is the official PyTorch implementation of AAA

181 Dec 28, 2022

OpenLT: An open-source project for long-tail classification

OpenLT: An open-source project for long-tail classification Supported Methods for Long-tailed Recognition: Cross-Entropy Loss Focal Loss (ICCV'17) Cla

37 Sep 15, 2022

Leveraging Instance-, Image- and Dataset-Level Information for Weakly Supervised Instance Segmentation

Leveraging Instance-, Image- and Dataset-Level Information for Weakly Supervised Instance Segmentation This paper has been accepted and early accessed

39 Sep 20, 2022

TorchDistiller - a collection of the open source pytorch code for knowledge distillation, especially for the perception tasks, including semantic segmentation, depth estimation, object detection and instance segmentation.

This project is a collection of the open source pytorch code for knowledge distillation, especially for the perception tasks, including semantic segmentation, depth estimation, object detection and instance segmentation.

147 Dec 3, 2022

The PyTorch implementation of DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervision.

DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervision The PyTorch implementation of DiscoBox: Weakly Supe

1 Oct 23, 2021