null

Overview

CP-Cluster

Confidence Propagation Cluster aims to replace NMS-based methods as a better box fusion framework in 2D/3D Object detection, Instance Segmentation:

[Confidence Propagation Cluster: Unleash the Full Potential of Object Detectors](arxivlink to do),
Yichun Shen*, Wanli Jiang*, Zhen Xu, Rundong Li, Junghyun Kwon, Siyi Li,

Contact: [email protected]. Welcome for any questions and comments!

Abstract

It’s been a long history that most object detection methods obtain objects by using the non-maximum suppression(NMS) and its improved versions like Soft-NMS to remove redundant bounding boxes. We challenge those NMS-based methods from three aspects: 1) The bounding box with highest confidence value may not be the true positive having the biggest overlap with the ground-truth box. 2) Not only suppression is required for redundant boxes, but also confidence enhancement is needed for those true positives. 3) Sorting candidate boxes by confidence values is not necessary so that full parallelism is achievable.

Inspired by belief propagation (BP), we propose the Confidence Propagation Cluster (CP-Cluster) to replace NMS-based methods, which is fully parallelizable as well as better in accuracy. In CP-Cluster, we borrow the message passing mechanism from BP to penalize redundant boxes and enhance true positives simultaneously in an iterative way until convergence. We verified the effectiveness of CP-Cluster by applying it to various mainstream detectors such as FasterRCNN, SSD, FCOS, YOLOv3, YOLOv5, Centernet etc. Experiments on MS COCO show that our plug and play method, without retraining detectors, is able to steadily improve average mAP of all those state-of-the-art models with a clear margin from 0.2 to 1.9 respectively when compared with NMS-based methods.

Highlights

  • Better accuracy: Compared with all previous NMS-based methods, CP-Cluster manages to achieve better accuracy

  • Fully parallelizable: No box sorting is required, and each candidate box can be handled separately when propagating confidence messages

Main results

Detectors from MMDetection on COCO val/test-dev

Method NMS Soft-NMS CP-Cluster
FRcnn-fpn50 38.4 / 38.7 39.0 / 39.2 39.2 / 39.4
Yolov3 33.5 / 33.5 33.6 / 33.6 34.1 / 34.1
Retina-fpn50 37.4 / 37.7 37.5 / 37.9 38.1 / 38.4
FCOS-X101 42.7 / 42.8 42.7 / 42.8 42.9 / 43.1
AutoAssign-fpn50 40.4 / 40.6 40.5 / 40.7 41.0 / 41.2

Yolov5(v6 model) on COCO val

Model NMS Soft-NMS CP-Cluster
Yolov5s 37.2 37.4 37.5
Yolov5m 45.2 45.3 45.5
Yolov5l 48.8 48.8 49.1
Yolov5x 50.7 50.8 51.0
Yolov5s_1280 44.5 50.8 44.8
Yolov5m_1280 51.1 51.1 51.3
Yolov5l_1280 53.6 53.7 53.8
Yolov5x_1280 54.7 54.8 55.0

Replace maxpooling with CP-Cluster for Centernet(Evaluated on COCO test-dev), where "flip_scale" means flip and multi-scale augmentations

Model maxpool Soft-NMS CP-Cluster
dla34 37.3 38.1 39.2
dla34_flip_scale 41.7 40.6 43.3
hg_104 40.2 40.6 41.1
hg_104_flip_scale 45.2 44.3 46.6

Instance Segmentation(MASK-RCNN, 3X models) from MMDetection on COCO test-dev

Box/Mask AP NMS Soft-NMS CP-Cluster
MRCNN_R50 41.5/37.7 42.0/37.8 42.1/38.0
MRCNN_R101 43.1/38.8 43.6/39.0 43.6/39.1
MRCNN_X101 44.6/40.0 45.2/40.2 45.2/40.2

Integrate into MMCV

Clone the mmcv repo from https://github.com/shenyi0220/mmcv (Cut down by 9/28/2021 from main branch with no extra modifications)

Copy the implementation of "cp_cluster_cpu" in src/nms.cpp to the mmcv nms code("mmcv/ops/csrc/pytorch/nms.cpp")

Borrow the "soft_nms_cpu" API by calling "cp_cluster_cpu" rather than orignal Soft-NMS implementations, so that modify the code like below:

@@ -186,8 +186,8 @@ Tensor softnms(Tensor boxes, Tensor scores, Tensor dets, float iou_threshold,
   if (boxes.device().is_cuda()) {
     AT_ERROR("softnms is not implemented on GPU");
   } else {
-    return softnms_cpu(boxes, scores, dets, iou_threshold, sigma, min_score,
-                       method, offset);
+    return cp_cluster_cpu(boxes, scores, dets, iou_threshold, min_score,
+                          offset, 0.8, 3);
   }
 }

Compile mmcv with source code

MMCV_WITH_OPS=1 pip install -e .

Reproduce Object Detection and Instance Segmentation in MMDetection

Make sure that the MMCV with CP-Cluster has been successfully installed.

Download code from https://github.com/shenyi0220/mmdetection (Cut down by 9/26/2021 from main branch with some config file modifications to call Soft-NMS/CP-Cluster), and install all the dependancies accordingly.

Download models from model zoo

Run below command to reproduce Faster-RCNN-r50-fpn-2x:

python tools/test.py ./configs/faster_rcnn/faster_rcnn_r50_fpn_2x_coco.py ./checkpoints/faster_rcnn_r50_fpn_2x_coco_bbox_mAP-0.384_20200504_210434-a5d8aa15.pth --eval bbox

To check original metrics with NMS, you can switch the model config back to use default NMS.

To check Soft-NMS metrics, just re-compile with mmcv without CP-Cluster modifications.

Reproduce Yolov5

Make sure that the MMCV with CP-Cluster has been successfully installed.

Download code from https://github.com/shenyi0220/yolov5 (Cut down by 11/9/2021 from main branch, replacing the default torchvision.nms with CP-Cluster from mmcv), and install all the dependancies accordingly.

Run below command to reproduce the CP-Cluster exp with yolov5s-v6

python val.py --data coco.yaml --conf 0.001 --iou 0.6 --weights yolov5s.pt --batch-size 32

License

For the time being, this implementation is published with NVIDIA proprietary license, and the only usage of the source code is to reproduce the experiments of CP-Cluster. For any possible commercial use and redistribution of the code, pls contact [email protected]

Citation

If you find this project useful for your research, please use the following BibTeX entry.

Comments
  • question in yolov5

    question in yolov5

    How to replace the default torchvision.nms with CP-Cluster from mmcv? `i = torchvision.ops.nms(boxes, scores, iou_thres) # NMS

        if i.shape[0] > max_det:  # limit detections
            i = i[:max_det]
        if merge and (1 < n < 3E3):  # Merge NMS (boxes merged using weighted mean)
            # update boxes as boxes(i,4) = weights(i,n) * boxes(n,4)
            iou = box_iou(boxes[i], boxes) > iou_thres  # iou matrix
            weights = iou * scores[None]  # box weights
            x[i, :4] = torch.mm(weights, x[:, :4]).float() / weights.sum(1, keepdim=True)  # merged boxes
            if redundant:
                i = i[iou.sum(1) > 1]  # require redundancy
    
        output[xi] = x[i]
        if (time.time() - t) > time_limit:
            LOGGER.warning(f'WARNING: NMS time limit {time_limit}s exceeded')
            break  # time limit exceeded`
    

    I found above in utils/general.py where i think should be modified,but i have difficulty in how to do this because the return type is not the same.Would you please provide a detail code?thanks very much.

    opened by 535484159 10
  • 在自己的代码上尝试

    在自己的代码上尝试

    大佬您好,请问我没有用ultralytics的代码

    keep = nms(

                #     detections_class[:, :4],
                #     detections_class[:, 4] * detections_class[:, 5],
                #     nms_thres
                # )
    

    dets , keep = soft_nms(detections_class[:, :4], detections_class[:, 4] * detections_class[:, 5], nms_thres, sigma=0.5, min_score=0.001, method='linear', offset=0) 上面nms是torchvision,下面的是您的cp-cluster,请问这样更改就行了吗,而且我并没有用到dets这个参数。期待您的回复。

    opened by Autismab 5
  • yolov5在coco-val数据集上测试

    yolov5在coco-val数据集上测试

    image @shenyi0220 非常感谢您的工作,我在复现CP-cluster在yolov5上测试coco数据集的过程中发现yolov5自带计算出来的[email protected]和pycocotools计算出来的AP50不一致,请问您在论文表中取的是上图红框中的哪一个呢?如果使用yolov5自带计算出来的map无法达到您论文中的精度,恳请您帮忙指导 image

    opened by River-Cold 3
  • mmcv安装问题

    mmcv安装问题

    File "/yolov5-main/utils/general.py", line 31, in from mmcv.ops import soft_nms, nms File "/yolov5-main/mmcv-main/mmcv/ops/init.py", line 2, in from .active_rotated_filter import active_rotated_filter File "/yolov5-main/mmcv-main/mmcv/ops/active_rotated_filter.py", line 8, in ext_module = ext_loader.load_ext( File "/yolov5-main/mmcv-main/mmcv/utils/ext_loader.py", line 13, in load_ext ext = importlib.import_module('mmcv.' + name) File "/anaconda3/envs/python/lib/python3.8/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) ImportError: /yolov5-main/mmcv-main/mmcv/_ext.cpython-38-x86_64-linux-gnu.so: undefined symbol: _Z3nmsN2at6TensorES0_fi 大佬您好,我用的您的这个版本Yolov5:https://github.com/shenyi0220/yolov5以及mmcv: https://github.com/shenyi0220/mmcv,已经将mmcv/ops/csrc/pytorch/cpu/nms.cpp和mmcv/ops/csrc/pytorch/nms.cpp进行了替换,执行MMCV_WITH_OPS=1 pip install -e .安装mmcv提示成功安装1.4.4版本,但是运行val.py文件提示如上错误,我的pythorch版本是1.9.0,cuda11.1。

    opened by pipnc 3
  • About implementation?

    About implementation?

    Hi, thank u for your awesome job! Could u please tell me how to apply this technique to the yolov5 project? Or can u provide a demo tutorial? Thank you very much.

    opened by David-19940718 2
  • Questions about how to use it for fasterrcnn which is based on detectron2.

    Questions about how to use it for fasterrcnn which is based on detectron2.

    Your work is very constructive. I see that the framework you are using for fasterrcnn based on mmdetection, so how do I migrate the code to 101x-C4 backbone for fasterrcnn based on detectron2?

    opened by Simon-Stma 1
  • About the parameters of CP-Cluster on COCO test-dev

    About the parameters of CP-Cluster on COCO test-dev

    I saw the examples about COCO val in README. Could you please provide the parameters(iou_threshold and min_score) of CP-Cluster on COCO test-dev? Thx!

    opened by RangeKing 5
  • Yolov5测试问题

    Yolov5测试问题

    Traceback (most recent call last): File "val.py", line 26, in from models.common import DetectMultiBackend File "/Users/yangpeng/Desktop/yolov5-main/models/common.py", line 22, in from utils.datasets import exif_transpose, letterbox File "/Users/yangpeng/Desktop/yolov5-main/utils/datasets.py", line 28, in from utils.augmentations import Albumentations, augment_hsv, copy_paste, letterbox, mixup, random_perspective File "/Users/yangpeng/Desktop/yolov5-main/utils/augmentations.py", line 12, in from utils.general import LOGGER, check_version, colorstr, resample_segments, segment2box File "/Users/yangpeng/Desktop/yolov5-main/utils/general.py", line 31, in from mmcv.ops import soft_nms, nms File "/Users/yangpeng/mmcv/mmcv/ops/init.py", line 2, in from .active_rotated_filter import active_rotated_filter File "/Users/yangpeng/mmcv/mmcv/ops/active_rotated_filter.py", line 8, in ext_module = ext_loader.load_ext( File "/Users/yangpeng/mmcv/mmcv/utils/ext_loader.py", line 13, in load_ext ext = importlib.import_module('mmcv.' + name) File "/opt/anaconda3/envs/yp_env/lib/python3.8/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) ImportError: dlopen(/Users/yangpeng/mmcv/mmcv/_ext.cpython-38-darwin.so, 0x0002): symbol not found in flat namespace '__Z3nmsN2at6TensorES0_fi'

    opened by YangPeng61790312 7
  • Install Error in MMCV

    Install Error in MMCV

    I tried to install cp-cluster in mmcv. After adding and replacing your code in mmcv, I ran the script MMCV_WITH_OPS=1 pip install -e . However, some errors occured. There are some details about errors below.

      ERROR: Command errored out with exit status 1:
       command: /data1/ljh/anaconda3/envs/open-mmlab/bin/python /data1/ljh/anaconda3/envs/open-mmlab/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpk836o1xj
           cwd: /data1/ljh/anaconda3/envs/open-mmlab
      Complete output (6 lines):
      running egg_info
      writing src/rapidfuzz.egg-info/PKG-INFO
      writing dependency_links to src/rapidfuzz.egg-info/dependency_links.txt
      writing requirements to src/rapidfuzz.egg-info/requires.txt
      writing top-level names to src/rapidfuzz.egg-info/top_level.txt
      error: package directory 'src/rapidfuzz' does not exist
    
    WARNING: Discarding file:///data1/ljh/anaconda3/envs/open-mmlab. Command errored out with exit status 1: /data1/ljh/anaconda3/envs/open-mmlab/bin/python /data1/ljh/anaconda3/envs/open-mmlab/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpk836o1xj Check the logs for full command output.
    ERROR: Command errored out with exit status 1: /data1/ljh/anaconda3/envs/open-mmlab/bin/python /data1/ljh/anaconda3/envs/open-mmlab/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpk836o1xj Check the logs for full command output.
    

    How can I solve this problem?

    opened by pd162 6
Owner
Yichun Shen
Yichun Shen
Null safe support for Python

Null Safe Python Null safe support for Python. Installation pip install nullsafe Quick Start Dummy Class class Dummy: pass Normal Python code: o =

Paaksing 13 Nov 17, 2022
Ingest GreyNoise.io malicious feed for CVE-2021-44228 and apply null routes

log4j-nullroute Quick script to ingest IP feed from greynoise.io for log4j (CVE-2021-44228) and null route bad addresses. Works w/Cisco IOS-XE and Ari

Ryan 5 Sep 12, 2022