Sparse R-CNN: End-to-End Object Detection with Learnable Proposals, CVPR2021

Hi! I have a problem about the evaluation. I know that we will get the top 100 or 300 boxes in the inference, however, there must be some no-object boxes. I don't clear the process how to only keep the object boxes. The postprocess of boxes I find is:

if results.has("pred_boxes"):
     output_boxes = results.pred_boxes
 elif results.has("proposal_boxes"):
     output_boxes = results.proposal_boxes
 else:
     output_boxes = None
 assert output_boxes is not None, "Predictions must contain boxes!"
 
 output_boxes.scale(scale_x, scale_y)
 output_boxes.clip(results.image_size)

 results = results[output_boxes.nonempty()]

请教下：RuntimeError: [enforce fail at /pytorch/third_party/gloo/gloo/transport/tcp/http://device.cc:208] ifa != nullptr. Unable to find interface for: [10.16.32.68]，这是什么问题，在运行github训练代码时：python projects/SparseRCNN/train_net.py --num-gpus 4
--config-file projects/SparseRCNN/configs/sparsercnn.res50.100pro.3x.yaml
--eval-only MODEL.WEIGHTS path/to/model.pth

Hi When I implement the code with yaml file sparsercnn.res50.100pro.3x.yaml and sparsercnn.res50.300pro.3x.yaml, I can only get mAP 40.8 and 42.5 respectively, both of which are about 2% worse than the reported results. I use 8 GPUS with batch size 16. I think the learning hyper parameters and initialization are the same as yours. Could you give me some hint about why the potential problem might be?

Thanks

When I evaluate on a 300 proposals model, the eval info is:

The maxDet is 100 not the max proposals num, Should I change these three threshold or find another eval method?

My eval cmd is

https://github.com/PeizeSun/SparseR-CNN/blob/daf86305eb5e1a1c93d001b457e5d90a8b40e032/projects/SparseR-CNN/sparsercnn/loss.py#L251 Why does the "cost_class" include two items, i.e. pos_cost_class and neg_cost_class? While if self.use_focal == False, cost_class = -out_prob[:, tgt_ids], which includes only one item and corresponds to "pos_cost_class".

I noticed that the paper mentions an iterative training by feeding the previously predicted bbox and feature. Could you help identify which part of the code corresponds to this. And may I ask how deduplication is achieved in this process, since the paper claims to be free of NMS?

Hi, thanks for your great work.

I am a little confused about the tables you provided in the paper.

In Table.1, you mentioned that Sparse RCNN R50-FPN with 100 proposals can reach a 42.8 AP on COCO val, as far as I can tell, this model configuration has no difference compared to the 42.3 AP model which you mentioned in Table.3, Table.5, Table.6, Table.7, Table.8, Table.9, Table.10.

Moreover, the rest metrics of the 42.8 AP model, e.g. AP50, AP75, APs, APm, APl are identical to that of the 42.3 model, both are 61.2, 45.7, 26.7, 44.6 and 57.6.

Cleverly it is impossible, so I am wondering which is the real result.

Hi :) Thank you for sharing code of Sparse RCNN. I really like your novel ideas in this paper.

I would like to ask regarding the instance-interaction module in dynamic head.

• According to the Table 4 in your paper, 37.2 AP is achieved without instance interaction module. However, in the Table 8, using multi-head attention as instance interaction would rather decrease the AP to 35.7 (-1.8). What is the reason for such performance degradation? Would it be due to small number of epochs for training self-attention enough? If yes, then may I ask have you trained with longer training schedule (I would appreciate if you show the training graphs)?

• Also, may I ask how did you implement multi-head attention in dynamic head? I guess you may reduce the dimension of flattened roi featuers (7x7xC -> C) in order to apply attention with object features (since object features dimension is C). Which method did you use in the ablation study?

Furthermore, I would like to ask regarding the inference time usage of SparseRCNN.

• I read your OneNet paper suggesting end-to-end dense detector which is also very impressive to me. Both Sparse RCNN and OneNet (also DETR) use pre-defined number of prediction boxes (100 or 300) to measure detection performance. However, in the visualization results, you used different threshold value for different model (0.2 for Sparse RCNN and 0.4 for OneNet). In other words, hand-crafted component still exists in these models even though NMS is removed. May I ask your opinion regarding this issue and any suggestion?

I have not made any changes to the code. I am trying to train it for a single class called "Dining Table" on my custom dataset. The annotations are in COCO format and they are being loaded perfectly well (I have registered it). However when I try to run the following command:

`python projects/SparseRCNN/train_net.py --num-gpus 1     --config-file projects/SparseRCNN/configs/sparsercnn.res50.100pro.3x.yaml`    

The code breaks with with following error trail

  File "projects/SparseRCNN/train_net.py", line 138, in <module>
    launch(
  File "SparseR-CNN/detectron2/engine/launch.py", line 62, in launch
    main_func(*args)
  File "projects/SparseRCNN/train_net.py", line 130, in main
    return trainer.train()
  File "SparseR-CNN/detectron2/engine/defaults.py", line 419, in train
    super().train(self.start_iter, self.max_iter)
  File "SparseR-CNN/detectron2/engine/train_loop.py", line 134, in train
    self.run_step()
  File "SparseR-CNN/detectron2/engine/defaults.py", line 429, in run_step
    self._trainer.run_step()
  File "SparseR-CNN/detectron2/engine/train_loop.py", line 228, in run_step
    loss_dict = self.model(data)
  File "miniconda3/envs/srcnn_working/lib/python3.8/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "SparseR-CNN/projects/SparseRCNN/sparsercnn/detector.py", line 143, in forward
    loss_dict = self.criterion(output, targets)
  File "miniconda3/envs/srcnn_working/lib/python3.8/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "SparseR-CNN/projects/SparseRCNN/sparsercnn/loss.py", line 161, in forward
    losses.update(self.get_loss(loss, outputs, targets, indices, num_boxes))
  File "SparseR-CNN/projects/SparseRCNN/sparsercnn/loss.py", line 137, in get_loss
    return loss_map[loss](outputs, targets, indices, num_boxes, **kwargs)
  File "SparseR-CNN/projects/SparseRCNN/sparsercnn/loss.py", line 110, in loss_boxes
    src_boxes_ = src_boxes / image_size
RuntimeError: The size of tensor a (390) must match the size of tensor b (397) at non-singleton dimension 0

Dimensions of both src_boxes and image_size are:

image_size torch.Size([397, 4])
src_boxes torch.Size([390, 4])

I am not sure if it is because you have hard coded something somewhere (configs etc.) ? Could you please direct me.

❓ How to do something using detectron2

Describe what you want to do, including:

1. what inputs you will provide, if any:
2. what outputs you are expecting:

❓ What does an API do and how to use it?

Please link to which API or documentation you're asking about from https://detectron2.readthedocs.io/

For meaning of a config, please see https://detectron2.readthedocs.io/modules/config.html#config-references

NOTE:

1. Only general answers are provided. If you want to ask about "why X did not work" for something you did, please use the Unexpected behaviors issue template.

2. About how to implement new models / new dataloader / new training logic, etc., check documentation first.

3. We do not answer machine learning / computer vision questions that are not specific to detectron2, such as how a model works, how to improve your training/make it converge, or what algorithm/methods can be used to achieve X.

我在训练Sparse-rcnn的过程中，在project/Sparse-rcnn/sparsercnn/box_ops.py中对应的地方抛出了boxes的坐标值不合法的错误。请问可能是什么原因导致的呢？ 我打印了boxes的坐标值，发现其中是4个nan，网络结构和数据集都没有修改过。训练使用的是两张2080ti，batch size是8，学习率是0.0025

When I use my own dataset(only 1 class) to evaluation, I got the following error. I'd appreciate it if you could guide me.

Traceback (most recent call last): File "G:/SparseR-CNN-main/projects/SparseRCNN/train_net.py", line 176, in launch( File "G:\SparseR-CNN-main\detectron2\engine\launch.py", line 66, in launch main_func(*args) File "G:/Code+paper/SparseR-CNN-main/projects/SparseRCNN/train_net.py", line 170, in main return trainer.train() File "G:\SparseR-CNN-main\detectron2\engine\defaults.py", line 421, in train super().train(self.start_iter, self.max_iter) File "G:\SparseR-CNN-main\detectron2\engine\train_loop.py", line 144, in train self.after_train() File "G:\SparseR-CNN-main\detectron2\engine\train_loop.py", line 153, in after_train h.after_train() File "G:\SparseR-CNN-main\detectron2\engine\hooks.py", line 361, in after_train self._do_eval() File "G:\SparseR-CNN-main\detectron2\engine\hooks.py", line 331, in _do_eval results = self._func() File "G:\SparseR-CNN-main\detectron2\engine\defaults.py", line 374, in test_and_save_results self._last_eval_results = self.test(self.cfg, self.model) File "G:\SparseR-CNN-main\detectron2\engine\defaults.py", line 542, in test results_i = inference_on_dataset(model, data_loader, evaluator) File "G:\SparseR-CNN-main\detectron2\evaluation\evaluator.py", line 176, in inference_on_dataset results = evaluator.evaluate() File "G:\SparseR-CNN-main\detectron2\evaluation\coco_evaluation.py", line 173, in evaluate self._eval_predictions(predictions, img_ids=img_ids) File "G:\SparseR-CNN-main\detectron2\evaluation\coco_evaluation.py", line 212, in _eval_predictions result["category_id"] = reverse_id_mapping[category_id] KeyError: 1

refactoring code with var unpack which is more pythonic, concise, readable and efficient; how do think this change which has practical value？

Thanks for your contribution!

If you're sending a large PR (e.g., >50 lines), please open an issue first about the feature / bug, and indicate how you want to contribute.

We do not always accept features. See https://detectron2.readthedocs.io/notes/contributing.html#pull-requests about how we handle PRs.

Before submitting a PR, please run dev/linter.sh to lint the code.

[01/28 02:49:28 d2.data.datasets.coco]: Loading datasets/coco/annotations/instances_train2017.json takes 4.76 seconds. Traceback (most recent call last): File "projects/SparseRCNN/train_net.py", line 159, in args=(args,), File "/home/wyw/SparseR-CNN/detectron2/engine/launch.py", line 62, in launch main_func(*args) File "projects/SparseRCNN/train_net.py", line 145, in main trainer = Trainer(cfg) File "/home/wyw/SparseR-CNN/detectron2/engine/defaults.py", line 284, in init data_loader = self.build_train_loader(cfg) File "projects/SparseRCNN/train_net.py", line 52, in build_train_loader return build_detection_train_loader(cfg, mapper=mapper) File "/home/wyw/SparseR-CNN/detectron2/config/config.py", line 201, in wrapped explicit_args = _get_args_from_config(from_config, *args, **kwargs) File "/home/wyw/SparseR-CNN/detectron2/config/config.py", line 236, in _get_args_from_config ret = from_config_func(*args, **kwargs) File "/home/wyw/SparseR-CNN/detectron2/data/build.py", line 309, in _train_loader_from_config proposal_files=cfg.DATASETS.PROPOSAL_FILES_TRAIN if cfg.MODEL.LOAD_PROPOSALS else None, File "/home/wyw/SparseR-CNN/detectron2/data/build.py", line 222, in get_detection_dataset_dicts dataset_dicts = [DatasetCatalog.get(dataset_name) for dataset_name in dataset_names] File "/home/wyw/SparseR-CNN/detectron2/data/build.py", line 222, in dataset_dicts = [DatasetCatalog.get(dataset_name) for dataset_name in dataset_names] File "/home/wyw/SparseR-CNN/detectron2/data/catalog.py", line 58, in get return f() File "/home/wyw/SparseR-CNN/detectron2/data/datasets/coco.py", line 469, in DatasetCatalog.register(name, lambda: load_coco_json(json_file, image_root, name)) File "/home/wyw/SparseR-CNN/detectron2/data/datasets/coco.py", line 71, in load_coco_json meta.thing_classes = thing_classes File "/home/wyw/SparseR-CNN/detectron2/data/catalog.py", line 150, in setattr "to a different value!\n{} != {}".format(key, self.name, oldval, val) AssertionError: Attribute 'thing_classes' in the metadata of 'coco_2017_train' cannot be set to a different value!

我把从coco数据集换成了自己的数据集，但是出现了错误，我应该修改你的代码中的哪几个部分的代码。 Traceback (most recent call last): File "projects/SparseRCNN/train_net.py", line 159, in args=(args,), File "/home/libing/anaconda3/envs/detectron2/lib/python3.7/site-packages/detectron2/engine/launch.py", line 62, in launch main_func(*args) File "projects/SparseRCNN/train_net.py", line 145, in main trainer = Trainer(cfg) File "/home/libing/anaconda3/envs/detectron2/lib/python3.7/site-packages/detectron2/engine/defaults.py", line 312, in init data_loader = self.build_train_loader(cfg) File "projects/SparseRCNN/train_net.py", line 52, in build_train_loader return build_detection_train_loader(cfg, mapper=mapper) File "/home/libing/anaconda3/envs/detectron2/lib/python3.7/site-packages/detectron2/config/config.py", line 201, in wrapped explicit_args = _get_args_from_config(from_config, *args, **kwargs) File "/home/libing/anaconda3/envs/detectron2/lib/python3.7/site-packages/detectron2/config/config.py", line 238, in _get_args_from_config ret = from_config_func(*args, **kwargs) File "/home/libing/anaconda3/envs/detectron2/lib/python3.7/site-packages/detectron2/data/build.py", line 314, in _train_loader_from_config proposal_files=cfg.DATASETS.PROPOSAL_FILES_TRAIN if cfg.MODEL.LOAD_PROPOSALS else None, File "/home/libing/anaconda3/envs/detectron2/lib/python3.7/site-packages/detectron2/data/build.py", line 227, in get_detection_dataset_dicts dataset_dicts = [DatasetCatalog.get(dataset_name) for dataset_name in names] File "/home/libing/anaconda3/envs/detectron2/lib/python3.7/site-packages/detectron2/data/build.py", line 227, in dataset_dicts = [DatasetCatalog.get(dataset_name) for dataset_name in names] File "/home/libing/anaconda3/envs/detectron2/lib/python3.7/site-packages/detectron2/data/catalog.py", line 58, in get return f() File "/home/libing/anaconda3/envs/detectron2/lib/python3.7/site-packages/detectron2/data/datasets/coco.py", line 489, in DatasetCatalog.register(name, lambda: load_coco_json(json_file, image_root, name)) File "/home/libing/anaconda3/envs/detectron2/lib/python3.7/site-packages/detectron2/data/datasets/coco.py", line 80, in load_coco_json meta.thing_classes = thing_classes File "/home/libing/anaconda3/envs/detectron2/lib/python3.7/site-packages/detectron2/data/catalog.py", line 150, in setattr "to a different value!\n{} != {}".format(key, self.name, oldval, val) AssertionError: Attribute 'thing_classes' in the metadata of 'coco_2017_train' cannot be set to a different value! ['person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'traffic light', 'fire hydrant', 'stop sign', 'parking meter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sports ball', 'kite', 'baseball bat', 'baseball glove', 'skateboard', 'surfboard', 'tennis racket', 'bottle', 'wine glass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hot dog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'potted plant', 'bed', 'dining table', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cell phone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddy bear', 'hair drier', 'toothbrush'] != ['ship'] 我把SSDD的数据集转换成了coco的格式，然后跑你的代码出现了这个问题，请问怎么解决？

I want to install Sparse R-CNN, but I was troubled by the 3th code, which is

git clone https://github.com/PeizeSun/SparseR-CNN.git
cd SparseR-CNN
python setup.py build develop

And i got this

running build
running build_py
running build_ext
----windows operation system
C:\Miniconda3\envs\py3.6\lib\site-packages\torch\utils\cpp_extension.py:279: UserWarning: Error checking compiler version for cl: [WinError 2] 系统找不到指定的文件。
  warnings.warn('Error checking compiler version for {}: {}'.format(compiler, error))
building 'detectron2._C' extension
----windows operation system
Emitting ninja build file D:\xm_Project\PythonProject\SparseR-CNN-main\build\temp.win-amd64-3.6\Release\build.ninja...
Compiling objects...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
1.10.2
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MA
NIFESTUAC:NO /LIBPATH:C:\Miniconda3\envs\py3.6\lib\site-packages\torch\lib /LIBPATH:C:\Miniconda3\envs\py3.6\libs /LIBPATH:C:\Miniconda3\
envs\py3.6\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\LIB\amd64" "/LIBPATH:C:\Program Files (x86)\Win
dows Kits\10\lib\10.0.10240.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\8.1\lib\winv6.3\um\x64" c10.lib torch.lib torch_cpu
.lib torch_python.lib /EXPORT:PyInit__C D:\xm_Project\PythonProject\SparseR-CNN-main\build\temp.win-amd64-3.6\Release\xm_Project\PythonPr
oject\SparseR-CNN-main\detectron2\layers\csrc\vision.obj D:\xm_Project\PythonProject\SparseR-CNN-main\build\temp.win-amd64-3.6\Release\xm
_Project\PythonProject\SparseR-CNN-main\detectron2\layers\csrc\box_iou_rotated\box_iou_rotated_cpu.obj D:\xm_Project\PythonProject\Sparse
R-CNN-main\build\temp.win-amd64-3.6\Release\xm_Project\PythonProject\SparseR-CNN-main\detectron2\layers\csrc\cocoeval\cocoeval.obj D:\xm_
Project\PythonProject\SparseR-CNN-main\build\temp.win-amd64-3.6\Release\xm_Project\PythonProject\SparseR-CNN-main\detectron2\layers\csrc\
nms_rotated\nms_rotated_cpu.obj D:\xm_Project\PythonProject\SparseR-CNN-main\build\temp.win-amd64-3.6\Release\xm_Project\PythonProject\Sp
arseR-CNN-main\detectron2\layers\csrc\ROIAlign\ROIAlign_cpu.obj D:\xm_Project\PythonProject\SparseR-CNN-main\build\temp.win-amd64-3.6\Rel
ease\xm_Project\PythonProject\SparseR-CNN-main\detectron2\layers\csrc\ROIAlignRotated\ROIAlignRotated_cpu.obj /OUT:build\lib.win-amd64-3.
6\detectron2\_C.cp36-win_amd64.pyd /IMPLIB:D:\xm_Project\PythonProject\SparseR-CNN-main\build\temp.win-amd64-3.6\Release\xm_Project\Pytho
nProject\SparseR-CNN-main\detectron2\layers\csrc\_C.cp36-win_amd64.lib
LINK : fatal error LNK1181: cannot open input file 'D:\xm_Project\PythonProject\SparseR-CNN-main\build\temp.win-amd64-3.6\Release\xm_Proj
ect\PythonProject\SparseR-CNN-main\detectron2\layers\csrc\vision.obj'
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\BIN\\x86_amd64\\link.exe' failed with exit status 1181

I have Visual C++ 2015 Build Tools、PyTorch1.7.0、python3.6.13 installed on windows 10. This is part of my "cpp_extension.py"

I tried many ways, but i still didn't solve the problem.Although this question may be a little silly, can you help me? Thank you.