RefineMask (CVPR 2021)

Hello, I'm able to run refine mask successfully using mmcv-full 1.3.12 and mmdet - 2.6.0 on windows. It runs fine for training and giving the below error during evaluation after 12 epochs. If possible then can you help me out on this and let me know what mistake I'm doing particularly? Thank you in advance.

File "tools/test.py", line 241, in main() File "tools/test.py", line 206, in main outputs = single_gpu_test(model, data_loader, args.show, args.show_dir, File "C:\Users\anaconda3\envs\mmcv\lib\site-packages\mmdet\apis\test.py", line 28, in single_gpu_test result = model(return_loss=False, rescale=True, **data) File "C:\Users\anaconda3\envs\mmcv\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl return forward_call(*input, **kwargs) File "C:\Users\anaconda3\envs\mmcv\lib\site-packages\mmcv\parallel\data_parallel.py", line 42, in forward return super().forward(*inputs, **kwargs) File "C:\Users\anaconda3\envs\mmcv\lib\site-packages\torch\nn\parallel\data_parallel.py", line 166, in forward return self.module(*inputs[0], **kwargs[0]) File "C:\Users\anaconda3\envs\mmcv\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl return forward_call(*input, **kwargs) File "C:\Users\anaconda3\envs\mmcv\lib\site-packages\mmcv\runner\fp16_utils.py", line 98, in new_func return old_func(*args, **kwargs) File "C:\Users\anaconda3\envs\mmcv\lib\site-packages\mmdet\models\detectors\base.py", line 174, in forward return self.forward_test(img, img_metas, **kwargs) File "C:\Users\anaconda3\envs\mmcv\lib\site-packages\mmdet\models\detectors\base.py", line 147, in forward_test return self.simple_test(imgs[0], img_metas[0], **kwargs) File "C:\Users\anaconda3\envs\mmcv\lib\site-packages\mmdet\models\detectors\two_stage.py", line 182, in simple_test return self.roi_head.simple_test( File "C:\Users\anaconda3\envs\mmcv\lib\site-packages\mmdet\models\roi_heads\standard_roi_head.py", line 264, in simple_test segm_results = self.simple_test_mask( File "C:\Users\anaconda3\envs\mmcv\lib\site-packages\mmdet\models\roi_heads\refine_roi_head.py", line 92, in simple_test_mask if det_bboxes.shape[0] == 0: AttributeError: 'list' object has no attribute 'shape'

Hello,

I'm trying to run RefineMask repository. I have a question regarding this. Is it only run with torch 1.5.0 and mmcv-full 1.5.0 versions? Because I have higher version of torch (1.8.1+cu102) and mmcv-full (1.3.12) and when I run it then it gives an error mmcv 1.3.12 is used but incompatible. I try to install mmcv-full 1.5.0 but it throws an error. Can you guide me how should I proceed further? Thank you in advance.

I use 8GPU training the model, the lr is set to 0.02 following your config, but the loss is nan; when I set the lr to 0.0025 in another issue you answered, the loss is normal. can you give me some help, thanks

Traceback (most recent call last): File "tools/train.py", line 188, in main() File "tools/train.py", line 162, in main model.init_weights() File "/home/ubuntu/.conda/envs/open-mmlab/lib/python3.8/site-packages/mmcv/runner/base_module.py", line 55, in init_weights m.init_weights() TypeError: init_weights() missing 1 required positional argument: 'pretrained'

nice work ! I have a question about SimpleRefineMaskHead in refine_mask_head.py There is not mentioned in the article and README. May I ask how its performance is different from the normal model? What changes have been made to the simple model Looking forward to your reply！

hello! thanks for your beautiful work. I have a question that how to apply refine mask in cascade mask rcnn / HTC? Cause there are three stage in HTC, should I apply refine mask in the last stage or all the three stage? thanks for your reply~

Hi， @zhanggang001 ，I‘m still a beginning level learner and I have a question that don't know how to fix it. Without changing the RefineMask config（except DataRoot and num_classes），the loss suddenly boom in epoch1 or epoch2

2021-05-25 09:52:39,591 - mmdet - INFO - Epoch [1][550/1445] lr: 2.000e-02, eta: 3:27:36, time: 0.354, data_time: 0.002, memory: 5616, loss_rpn_cls: 0.1940, loss_rpn_bbox: 0.1124, loss_cls: 0.6903, acc: 89.3125, loss_bbox: 0.7711, loss_instance: 1.7145, loss_semantic: 1.1350, loss: 4.6172, grad_norm: 49.7330 2021-05-25 09:52:58,284 - mmdet - INFO - Epoch [1][600/1445] lr: 2.000e-02, eta: 3:27:43, time: 0.374, data_time: 0.003, memory: 5616, loss_rpn_cls: 0.1819, loss_rpn_bbox: 0.1232, loss_cls: 0.7582, acc: 88.1094, loss_bbox: 0.8542, loss_instance: 1.6824, loss_semantic: 0.7272, loss: 4.3272, grad_norm: 6.2210 2021-05-25 09:53:13,881 - mmdet - INFO - Epoch [1][650/1445] lr: 2.000e-02, eta: 3:25:04, time: 0.312, data_time: 0.003, memory: 5616, loss_rpn_cls: 10.1637, loss_rpn_bbox: 6.9990, loss_cls: 34.9405, acc: 91.4102, loss_bbox: 13.7452, loss_instance: 234.7180, loss_semantic: 7664.2850, loss: 7964.8515, grad_norm: 340497.7997 2021-05-25 09:53:27,481 - mmdet - INFO - Epoch [1][700/1445] lr: 2.000e-02, eta: 3:21:08, time: 0.272, data_time: 0.003, memory: 5616, loss_rpn_cls: 35289.9099, loss_rpn_bbox: 44193.1067, loss_cls: 875153.7309, acc: 94.6094, loss_bbox: 1115768.8280, loss_instance: 22061.8204, loss_semantic: 499100.5092, loss: 2591567.9840, grad_norm: 112257859.0582 2021-05-25 09:53:41,763 - mmdet - INFO - Epoch [1][750/1445] lr: 2.000e-02, eta: 3:18:13, time: 0.286, data_time: 0.003, memory: 5616, loss_rpn_cls: 20893164722860.7344, loss_rpn_bbox: 7885642268278.1553, loss_cls: 1349368483407166.5000, acc: 86.1094, loss_bbox: 627283726521955.8750, loss_instance: 4757506602120.5703, loss_semantic: 3351921805304.0806, loss: 2013540377899481.0000, grad_norm: 117931859697361120.0000 2021-05-25 09:53:56,211 - mmdet - INFO - Epoch [1][800/1445] lr: 2.000e-02, eta: 3:15:45, time: 0.289, data_time: 0.003, memory: 5616, loss_rpn_cls: 52192068187424022801154048.0000, loss_rpn_bbox: 61156605795124256318685184.0000, loss_cls: 12385031694969291828216463360.0000, acc: 27.7930, loss_bbox: 2810178410764419893789982720.0000, loss_instance: 320446566729177349750784.0000, loss_semantic: 7193256245805316776656896.0000, loss: 15316072270766823574903717888.0000, grad_norm: inf

Thanks for your watching, I'm looking forward for your answering :)

Thanks for the nice work. I have a question for you. Your default is 8 cards, and you set the learning rate to 0.02. If I conduct the experiment from multiple cards to single card, does the learning rate need to be adjusted? I have tried to adjust the learning rate to 0.0025 and 0.005 for experiments, but the results are not satisfactory. May I ask if you have any suggestions on the adjustment of learning rate when switching from multiple cards to single cards? Looking forward to your reply.

Hello,

I would like to train refinemask with resnet18 and I'm trying it but I'm getting the below error. Please help me to fix this issue. Thank you in advance!

File "tools/train.py", line 165, in main() File "tools/train.py", line 161, in main meta=meta) File "/home/Anaconda3/envs/openmmlab/lib/python3.7/site-packages/mmdet/apis/train.py", line 150, in train_detector runner.run(data_loaders, cfg.workflow, cfg.total_epochs) File "/home/Anaconda3/envs/openmmlab/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 125, in run epoch_runner(data_loaders[i], **kwargs) File "/home/Anaconda3/envs/openmmlab/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 50, in train self.run_iter(data_batch, train_mode=True) File "/home/Anaconda3/envs/openmmlab/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 30, in run_iter **kwargs) File "/home/Anaconda3/envs/openmmlab/lib/python3.7/site-packages/mmcv/parallel/data_parallel.py", line 67, in train_step return self.module.train_step(*inputs[0], **kwargs[0]) File "/home/Anaconda3/envs/openmmlab/lib/python3.7/site-packages/mmdet/models/detectors/base.py", line 246, in train_step losses = self(**data) File "/home/Anaconda3/envs/openmmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl result = self.forward(*input, **kwargs) File "/home/Anaconda3/envs/openmmlab/lib/python3.7/site-packages/mmcv/runner/fp16_utils.py", line 84, in new_func return old_func(*args, **kwargs) File "/home/Anaconda3/envs/openmmlab/lib/python3.7/site-packages/mmdet/models/detectors/base.py", line 180, in forward return self.forward_train(img, img_metas, **kwargs) File "/home/Anaconda3/envs/openmmlab/lib/python3.7/site-packages/mmdet/models/detectors/two_stage.py", line 142, in forward_train x = self.extract_feat(img) File "/home/Anaconda3/envs/openmmlab/lib/python3.7/site-packages/mmdet/models/detectors/two_stage.py", line 84, in extract_feat x = self.neck(x) File "/home/Anaconda3/envs/openmmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl result = self.forward(*input, **kwargs) File "/home/Anaconda3/envs/openmmlab/lib/python3.7/site-packages/mmcv/runner/fp16_utils.py", line 84, in new_func return old_func(*args, **kwargs) File "/home/Anaconda3/envs/openmmlab/lib/python3.7/site-packages/mmdet/models/necks/fpn.py", line 172, in forward for i, lateral_conv in enumerate(self.lateral_convs) File "/home/Anaconda3/envs/openmmlab/lib/python3.7/site-packages/mmdet/models/necks/fpn.py", line 172, in for i, lateral_conv in enumerate(self.lateral_convs) File "/home/Anaconda3/envs/openmmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl result = self.forward(*input, **kwargs) File "/home/Anaconda3/envs/openmmlab/lib/python3.7/site-packages/mmcv/cnn/bricks/conv_module.py", line 185, in forward x = self.conv(x) File "/home/ec2-user/Anaconda3/envs/openmmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl result = self.forward(*input, **kwargs) File "/home/Anaconda3/envs/openmmlab/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 419, in forward return self._conv_forward(input, self.weight) File "/home/Anaconda3/envs/openmmlab/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 416, in _conv_forward self.padding, self.dilation, self.groups) RuntimeError: Given groups=1, weight of size [256, 256, 1, 1], expected input[1, 64, 200, 272] to have 256 channels, but got 64 channels instead

I see the samples_per_gpu of Cityscapes is change to 2, but the learning rate is still set to 0.01. I understand the modification of samples_per_gpu of COCO dataset, for the batchsize is 16 (8 GPU * 2 samples_per_gpu) and corresponding to learning rate of 0.02. But I wonder when the samples_per_gpu of Cityscapes dataset is 2, the learning rate should also change?

First of all, thank you very much for sharing your work. I tested on coco dataset, but the test result showed that the whole dataset was empty. May I ask how to solve this problem? thanks.

[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 5000/5000, 18.3 task/s, elapsed: 273s, ETA: 0s writing results to work_dirs/r50-refinemask-1x/result.pkl

Evaluating bbox... Loading and preparing results... The testing results of the whole dataset is empty. load gt json load pred json Traceback (most recent call last): File "/data/user-njf86/ding/RefineMask-main/tools/test.py", line 153, in main() File "/data/user-njf86/ding/RefineMask-main/tools/test.py", line 149, in main dataset.evaluate(outputs, args.eval, classwise=True, **kwargs) File "/data/user-njf86/ding/RefineMask-main/mmdet/datasets/coco.py", line 556, in evaluate metric='segm', File "/data/user-njf86/ding/RefineMask-main/mmdet/datasets/coco.py", line 655, in eval_cocofied_lvis_result lvis_dt = LVISResults(lvis_gt, result_file) File "/home/user-njf/anaconda3/envs/mmdet_test/lib/python3.6/site-packages/lvis/results.py", line 44, in init if 'bbox' in result_anns[0]: IndexError: list index out of range Traceback (most recent call last): File "/home/user-njf/anaconda3/envs/mmdet_test/lib/python3.6/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/home/user-njf/anaconda3/envs/mmdet_test/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/user-njf/anaconda3/envs/mmdet_test/lib/python3.6/site-packages/torch/distributed/launch.py", line 260, in main() File "/home/user-njf/anaconda3/envs/mmdet_test/lib/python3.6/site-packages/torch/distributed/launch.py", line 256, in main cmd=cmd) subprocess.CalledProcessError: Command '['/home/user-njf/anaconda3/envs/mmdet_test/bin/python', '-u', '/data/user-njf86/ding/RefineMask-main/tools/test.py', '--local_rank=0', './configs/refinemask/coco/r50-refinemask-1x.py', 'work_dirs/r50-refinemask-1x/latest.pth', '--launcher', 'none', '--eval', 'bbox', 'segm', '--out', 'work_dirs/r50-refinemask-1x/result.pkl', '--tmpdir', 'work_dirs/r50-refinemask-1x/tmpdir']' returned non-zero exit status 1.

I would like to know how to get the flops of the model When using get_flops.py in tools to get flops, there is an error here

  File "/mmdet/models/detectors/two_stage.py", line 88, in forward_dummy
    roi_outs = self.roi_head.forward_dummy(x, proposals)
TypeError: forward_dummy() missing 1 required positional argument: 'roi_labels'

This is def forward_dummy in two_stage.py:
    def forward_dummy(self, img):
        """Used for computing network flops.

        See `mmdetection/tools/analysis_tools/get_flops.py`
        """
        outs = ()
        # backbone
        x = self.extract_feat(img)
        # rpn
        if self.with_rpn:
            rpn_outs = self.rpn_head(x)
            outs = outs + (rpn_outs, )
        proposals = torch.randn(1000, 4).to(img.device)
        # roi_head
        # roi_labels = torch.zeros(1000)
        roi_outs = self.roi_head.forward_dummy(x, proposals)  # roi_labels
        outs = outs + (roi_outs, )
        return outs

refinemask adds roi_labels in def _mask_forward, refine_roi_head.py during training I tried defining roi_labels with all 0

        roi_labels = torch.zeros(1000)
        roi_outs = self.roi_head.forward_dummy(x, proposals, roi_labels)

but have error

  File "/mmdet/mmdet/models/roi_heads/mask_heads/refine_mask_head.py", line 561, in forward
    instance_logits = self.stage_instance_logits[idx](instance_feats)[torch.arange(len(rois)), roi_labels][:, None]
IndexError: tensors used as indices must be long, byte or bool tensors

when I change the type to bool

  File "/mmdet/mmdet/models/roi_heads/mask_heads/refine_mask_head.py", line 561, in forward
    instance_logits = self.stage_instance_logits[idx](instance_feats)[torch.arange(len(rois)), roi_labels][:, None]
IndexError: The shape of the mask [1000] at index 0 does not match the shape of the indexed tensor [100, 1, 14, 14] at index 1

How to implement the calculation, hope you can help me! I don't understand the dimension and type of the roi_labels

# this is forward_dummy in mmdet/models/roi_heads/standard_roi_head.py
    def forward_dummy(self, x, proposals):
        """Dummy forward function."""
        # bbox head
        outs = ()
        rois = bbox2roi([proposals])
        if self.with_bbox:
            bbox_results = self._bbox_forward(x, rois)
            outs = outs + (bbox_results['cls_score'],
                           bbox_results['bbox_pred'])
        # mask head
        if self.with_mask:
            mask_rois = rois[:100]
            mask_results = self._mask_forward(x, mask_rois)
            outs = outs + (mask_results['mask_pred'], )
        return outs