Official PyTorch Implementation of Rank & Sort Loss [ICCV2021]

Overview

Rank & Sort Loss for Object Detection and Instance Segmentation

The official implementation of Rank & Sort Loss. Our implementation is based on mmdetection.

Rank & Sort Loss for Object Detection and Instance Segmentation,
Kemal Oksuz, Baris Can Cam, Emre Akbas, Sinan Kalkan, ICCV 2021 (Oral Presentation). (arXiv pre-print)

Summary

What is Rank & Sort (RS) Loss? Rank & Sort (RS) Loss supervises object detectors and instance segmentation methods to (i) rank the scores of the positive anchors above those of negative anchors, and at the same time (ii) sort the scores of the positive anchors with respect to their localisation qualities.

Benefits of RS Loss on Simplification of Training. With RS Loss, we significantly simplify training: (i) Thanks to our sorting objective, the positives are prioritized by the classifier without an additional auxiliary head (e.g. for centerness, IoU, mask-IoU), (ii) due to its ranking-based nature, RS Loss is robust to class imbalance, and thus, no sampling heuristic is required, and (iii) we address the multi-task nature of visual detectors using tuning-free task-balancing coefficients.

Benefits of RS Loss on Improving Performance. Using RS Loss, we train seven diverse visual detectors only by tuning the learning rate, and show that it consistently outperforms baselines: e.g. our RS Loss improves (i) Faster R-CNN by ~3 box AP and aLRP Loss (ranking-based baseline) by ~2 box AP on COCO dataset, (ii) Mask R-CNN with repeat factor sampling by 3.5 mask AP (~7 AP for rare classes) on LVIS dataset.

How to Cite

Please cite the paper if you benefit from our paper or the repository:

@inproceedings{RSLoss,
       title = {Rank & Sort Loss for Object Detection and Instance Segmentation},
       author = {Kemal Oksuz and Baris Can Cam and Emre Akbas and Sinan Kalkan},
       booktitle = {International Conference on Computer Vision (ICCV)},
       year = {2021}
}

Specification of Dependencies and Preparation

  • Please see get_started.md for requirements and installation of mmdetection.
  • Please refer to introduction.md for dataset preparation and basic usage of mmdetection.

Trained Models

Here, we report minival results in terms of AP and oLRP.

Multi-stage Object Detection

RS-R-CNN

Backbone Epoch Carafe MS train box AP box oLRP Log Config Model
ResNet-50 12 39.6 67.9 log config model
ResNet-50 12 + 40.8 66.9 log config model
ResNet-101-DCN 36 [480,960] 47.6 61.1 log config model
ResNet-101-DCN 36 + [480,960] 47.7 60.9 log config model

RS-Cascade R-CNN

Backbone Epoch box AP box oLRP Log Config Model
ResNet-50 12 41.3 66.6 Coming soon

One-stage Object Detection

Method Backbone Epoch box AP box oLRP Log Config Model
RS-ATSS ResNet-50 12 39.9 67.9 log config model
RS-PAA ResNet-50 12 41.0 67.3 log config model

Multi-stage Instance Segmentation

RS-Mask R-CNN on COCO Dataset

Backbone Epoch Carafe MS train mask AP box AP mask oLRP box oLRP Log Config Model
ResNet-50 12 36.4 40.0 70.1 67.5 log config model
ResNet-50 12 + 37.3 41.1 69.4 66.6 log config model
ResNet-101 36 [640,800] 40.3 44.7 66.9 63.7 log config model
ResNet-101 36 + [480,960] 41.5 46.2 65.9 62.6 log config model
ResNet-101-DCN 36 + [480,960] 43.6 48.8 64.0 60.2 log config model
ResNeXt-101-DCN 36 + [480,960] 44.4 49.9 63.1 59.1 Coming Soon config model

RS-Mask R-CNN on LVIS Dataset

Backbone Epoch MS train mask AP box AP mask oLRP box oLRP Log Config Model
ResNet-50 12 [640,800] 25.2 25.9 Coming Soon Coming Soon Coming Soon Coming soon Coming soon

One-stage Instance Segmentation

RS-YOLACT

Backbone Epoch mask AP box AP mask oLRP box oLRP Log Config Model
ResNet-50 55 29.9 33.8 74.7 71.8 log config model

RS-SOLOv2

Backbone Epoch mask AP mask oLRP Log Config Model
ResNet-34 36 32.6 72.7 Coming soon Coming soon Coming soon
ResNet-101 36 39.7 66.9 Coming soon Coming soon Coming soon

Running the Code

Training Code

The configuration files of all models listed above can be found in the configs/ranksort_loss folder. You can follow get_started.md for training code. As an example, to train Faster R-CNN with our RS Loss on 4 GPUs as we did, use the following command:

./tools/dist_train.sh configs/ranksort_loss/ranksort_faster_rcnn_r50_fpn_1x_coco.py 4

Test Code

The configuration files of all models listed above can be found in the configs/ranksort_loss folder. You can follow get_started.md for test code. As an example, first download a trained model using the links provided in the tables below or you train a model, then run the following command to test an object detection model on multiple GPUs:

./tools/dist_test.sh configs/ranksort_loss/ranksort_faster_rcnn_r50_fpn_1x_coco.py ${CHECKPOINT_FILE} 4 --eval bbox 

and use the following command to test an instance segmentation model on multiple GPUs:

./tools/dist_test.sh configs/ranksort_loss/ranksort_mask_rcnn_r50_fpn_1x_coco.py ${CHECKPOINT_FILE} 4 --eval bbox segm 

You can also test a model on a single GPU with the following example command:

python tools/test.py configs/ranksort_loss/ranksort_faster_rcnn_r50_fpn_1x_coco.py ${CHECKPOINT_FILE} 4 --eval bbox 

Details for Rank & Sort Loss Implementation

Below is the links to the files that can be useful to check out the details of the implementation:

Comments
  • RuntimeError: invalid argument 1: cannot perform reduction function min on tensor with no elements because the operation does not have an identity at /pytorch/aten/src/THC/generic/THCTensorMathReduce.cu:64

    RuntimeError: invalid argument 1: cannot perform reduction function min on tensor with no elements because the operation does not have an identity at /pytorch/aten/src/THC/generic/THCTensorMathReduce.cu:64

    I trained on my custom dataset with 4 classes; when I used RPNHead, it worked normally. However, when I used rpn_head is RankBasedRPNHead and got an error: AssertionError: The num_classes (1) in RankBasedRPNHead of MMDataParallel does not matches the length of CLASSES 4) in CocoDataset.

    I used the same config and just only edited num_classes from 80 to 4 to fit my dataset.

    Thanks.

    opened by trongthuan205 8
  • Can not reproduce results in your paper

    Can not reproduce results in your paper

    Hi, I downloaded your codes, and train ATSS-R50 and RankLoss-ATSS-R50, the results are:

    algorithm | exp1 mAP | exp2 mAP -- | -- | -- ATSS | 0.396 | 0.393 RS-ATSS | 0.393 | 0.392

    The mAP of RS-ATSS is lower than ATSS, however, in the paper, RS-ATSS is 0.399 , ATSS is 0.393。 I wonder why can not reproduce the results, is there something wrong?

    opened by AIZOOTech 7
  • train errror

    train errror

    ranking_loss, sorting_loss = self.loss_rank.apply(flat_preds, flat_labels) File "/media/ymhj/data/kxp/zkxt/RankSortLoss-main/mmdet/models/losses/ranking_losses.py", line 18, in forward threshold_logit = torch.min(fg_logits)-delta_RS RuntimeError: m

    opened by Muke6 5
  • About adjusting the learning rate

    About adjusting the learning rate

    Hi author, thank you for your excellent work! I want to train my own datasets with rs_cascade_rcnn, so I'm wondering how should I set a reasonable learning rate (only one GPU, batch_size=2). Looking forward to your reply!

    opened by sanmulab 4
  • why using delta_RS

    why using delta_RS

    Hello

    In the file ranking_losses.py, I am a little confused about the following codes.

    if delta_RS > 0: fg_relations=torch.clamp(fg_relations/(2delta_RS)+0.5,min=0,max=1) bg_relations=torch.clamp(bg_relations/(2delta_RS)+0.5,min=0,max=1) else: fg_relations = (fg_relations >= 0).float() bg_relations = (bg_relations >= 0).float()

    I do not quite understand why you used delta_RS and what is the bias number 0.5 used for. It seems to me that when you add 0.5, it will make some negative logits to be positive. Does it make sense?

    Thanks

    opened by liyunsheng13 4
  • about the implement of sort error

    about the implement of sort error

    according the paper, I can compute the $L_{ij}$ as following defined in Eq. 9: 微信截图_20211231155145 I notice that the implement of sort error in your code is as following:

    # Compute sorting error on example ii
    sorting_error[ii] = current_sorting_error - target_sorting_error
    

    why not multiply the sort pmf? as following

    sorting_error[ii] = (current_sorting_error - target_sorting_error)*missorted_examples/sorting_pmf_denom
    
    opened by FL77N 3
  • RuntimeError: expected scalar type Half but found Float

    RuntimeError: expected scalar type Half but found Float

    Hello author! I seem to have a new problem now: I want to use half-precision fp16 for training, so I add the line fp16 = dict(loss_scale=512.) to the config file, but I get an error: . Does your codebase currently not support half-precision training?

    error: File "/home/ubuntu/cs_logdet/rankSortLoss-main/mmdet/models/necks/fpn_carafe.py", line 256, in forward upsample_feat = self.upsample_modulesi - 1 File "/home/ubuntu/.conda/envs/testlogdet/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl return forward_call(*input, **kwargs) File "/home/ubuntu/.conda/envs/testlogdet/lib/python3.7/site-packages/mmcv/ops/carafe.py", line 285, in forward x = self.feature_reassemble(x, mask) File "/home/ubuntu/.conda/envs/testlogdet/lib/python3.7/site-packages/mmcv/ops/carafe.py", line 277, in feature_reassemble x = carafe(x, mask, self.up_kernel, self.up_group, self.scale_factor) File "/home/ubuntu/.conda/envs/testlogdet/lib/python3.7/site-packages/mmcv/ops/carafe.py", line 139, in forward scale_factor=scale_factor) RuntimeError: expected scalar type Half but found Float

    opened by sanmulab 2
  • About the blance of loss_bbox and loss_rank_sort

    About the blance of loss_bbox and loss_rank_sort

    I notice that loss_bbox is weighted to be equal to the sum of loss_rank and loss_sort. If not, how is the performance? What's the intuition of this action? And is there any other reference to do so? https://github.com/kemaloksuz/RankSortLoss/blob/b41f64df70b1f8e3260e5c04015e0c7b59b339d9/mmdet/models/roi_heads/bbox_heads/convfc_bbox_head.py#L287-L289

    opened by jcdubron 2
  • RankSort loss computation over batch

    RankSort loss computation over batch

    At this line the predictions are flattened discarding the batch dimension. vectorize_labels also do the same. I guess the ranking and sorting of positives/negatives are done across image instances in the batch. Is this done intentional? I think instance-wise RankSort loss computation is more meaningful for the batch case.

    opened by xonobo 2
  • LVIS validation issue

    LVIS validation issue

    Hi, I met an issue when I try to do validation on LVIS v1 dataset. I follow your work and use 4 V100 to do the validation. However, the validation speed is limited to 0.7 img/second ,it is very slow. Is that normal and could you give me a hint about your validation speed? It will be so helpful! Thank you for sharing your work :)

    opened by xinqiaozhao 2
  • How to apply RankSort Loss on SOLO model?

    How to apply RankSort Loss on SOLO model?

    Hi Kemal,

    Thanks for sharing your code. I tried to implement RS Loss for SOLO mode through descriptions in your paper, i.e.

    use Dice Loss as the continuous ground truth labels.

    However, I found that since mask branch and category branch are disentangled in SOLO model, there is no one-one correspondence in mask predictions and category predictions. Therefore, I cannot figure out how to regress category predictions with their corresponding mask dice loss.

    Best regards, Jiahua Wang

    opened by usherbob 1
Owner
Kemal Oksuz
Kemal Oksuz
Much faster than SORT(Simple Online and Realtime Tracking), a little worse than SORT

QSORT QSORT(Quick + Simple Online and Realtime Tracking) is a simple online and realtime tracking algorithm for 2D multiple object tracking in video s

Yonghye Kwon 8 Jul 27, 2022
This is the pytorch implementation for the paper: Generalizable Mixed-Precision Quantization via Attribution Rank Preservation, which is accepted to ICCV2021.

GMPQ: Generalizable Mixed-Precision Quantization via Attribution Rank Preservation This is the pytorch implementation for the paper: Generalizable Mix

null 18 Sep 2, 2022
MicRank is a Learning to Rank neural channel selection framework where a DNN is trained to rank microphone channels.

MicRank: Learning to Rank Microphones for Distant Speech Recognition Application Scenario Many applications nowadays envision the presence of multiple

Samuele Cornell 20 Nov 10, 2022
Implement of "Training deep neural networks via direct loss minimization" in PyTorch for 0-1 loss

This is the implementation of "Training deep neural networks via direct loss minimization" published at ICML 2016 in PyTorch. The implementation targe

Cuong Nguyen 1 Jan 18, 2022
An implementation for the loss function proposed in Decoupled Contrastive Loss paper.

Decoupled-Contrastive-Learning This repository is an implementation for the loss function proposed in Decoupled Contrastive Loss paper. Requirements P

Ramin Nakhli 71 Dec 4, 2022
Recall Loss for Semantic Segmentation (This repo implements the paper: Recall Loss for Semantic Segmentation)

Recall Loss for Semantic Segmentation (This repo implements the paper: Recall Loss for Semantic Segmentation) Download Synthia dataset The model uses

null 32 Sep 21, 2022
Yolov5 + Deep Sort with PyTorch

딥소트 수정중 Yolov5 + Deep Sort with PyTorch Introduction This repository contains a two-stage-tracker. The detections generated by YOLOv5, a family of obj

null 1 Nov 26, 2021
Official implementation of our paper "LLA: Loss-aware Label Assignment for Dense Pedestrian Detection" in Pytorch.

LLA: Loss-aware Label Assignment for Dense Pedestrian Detection This project provides an implementation for "LLA: Loss-aware Label Assignment for Dens

null 35 Dec 6, 2022
Official pytorch implementation of "Feature Stylization and Domain-aware Contrastive Loss for Domain Generalization" ACMMM 2021 (Oral)

Feature Stylization and Domain-aware Contrastive Loss for Domain Generalization This is an official implementation of "Feature Stylization and Domain-

null 22 Sep 22, 2022
Official implementation of Influence-balanced Loss for Imbalanced Visual Classification in PyTorch.

Official implementation of Influence-balanced Loss for Imbalanced Visual Classification in PyTorch.

Seulki Park 70 Jan 3, 2023
Official PyTorch code for Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling (HCFlow, ICCV2021)

Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling (HCFlow, ICCV2021) This repository is the official P

Jingyun Liang 159 Dec 30, 2022
Official PyTorch code for Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling (HCFlow, ICCV2021)

Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling (HCFlow, ICCV2021) This repository is the official P

Jingyun Liang 159 Dec 30, 2022
Official PyTorch code for Mutual Affine Network for Spatially Variant Kernel Estimation in Blind Image Super-Resolution (MANet, ICCV2021)

Mutual Affine Network for Spatially Variant Kernel Estimation in Blind Image Super-Resolution (MANet, ICCV2021) This repository is the official PyTorc

Jingyun Liang 139 Dec 29, 2022
Pytorch based library to rank predicted bounding boxes using text/image user's prompts.

pytorch_clip_bbox: Implementation of the CLIP guided bbox ranking for Object Detection. Pytorch based library to rank predicted bounding boxes using t

Sergei Belousov 50 Nov 27, 2022
(Personalized) Page-Rank computation using PyTorch

torch-ppr This package allows calculating page-rank and personalized page-rank via power iteration with PyTorch, which also supports calculation on GP

Max Berrendorf 69 Dec 3, 2022
Official implementation of "A Unified Objective for Novel Class Discovery", ICCV2021 (Oral)

A Unified Objective for Novel Class Discovery This is the official repository for the paper: A Unified Objective for Novel Class Discovery Enrico Fini

Enrico Fini 118 Dec 26, 2022
This is an official implementation of the paper "Distance-aware Quantization", accepted to ICCV2021.

PyTorch implementation of DAQ This is an official implementation of the paper "Distance-aware Quantization", accepted to ICCV2021. For more informatio

CV Lab @ Yonsei University 36 Nov 4, 2022
An efficient and effective learning to rank algorithm by mining information across ranking candidates. This repository contains the tensorflow implementation of SERank model. The code is developed based on TF-Ranking.

SERank An efficient and effective learning to rank algorithm by mining information across ranking candidates. This repository contains the tensorflow

Zhihu 44 Oct 20, 2022
Open source implementation of AceNAS: Learning to Rank Ace Neural Architectures with Weak Supervision of Weight Sharing

AceNAS This repo is the experiment code of AceNAS, and is not considered as an official release. We are working on integrating AceNAS as a built-in st

Yuge Zhang 6 Sep 7, 2022