Hello, I was trying to run gcnet form the MMDetection repository. I wished to train GCNet on my custom dataset in which each image is 800x800 and all the annotations are in proper COCO format. But however, my annotations are for for boxes alone and nothing else. I gave the respective paths and then ran the following command:

./dist_train.sh ../configs/gcnet/mask_rcnn_r101_fpn_r4_gcb_c3-c5_1x_coco.py 2

When I ran this, I had the training begun. Here's a small part of my log file when I was trying to reproduce the error:

'''

loading annotations into memory... loading annotations into memory... Done (t=0.78s) creating index... index created! Done (t=0.78s) creating index... index created! loading annotations into memory... Done (t=0.00s) creating index... index created! 2020-05-23 03:17:35,421 - mmdet - INFO - Start running, host: user@c6e7e60caee9, work_dir: /mnt/user 2.log /mmdetection/tools/work_dirs/mask_rcnn_r101_fpn_r4_gcb_c3-c5_1x_coco 2020-05-23 03:17:35,421 - mmdet - INFO - workflow: [('train', 1)], max: 12 epochs loading annotations into memory... Done (t=0.00s) creating index... index created! 2020-05-23 03:18:11,654 - mmdet - INFO - Epoch [1][50/1730] lr: 0.00198, eta: 4:09:57, time: 0.724, data_time: 0.267, memory: 4932, loss_rpn_cls: 0.5057, loss_rpn_bbox: 0.2989, loss_cls: 1.0971, acc: 85.3574, loss_bbox: 0.0454, loss_mask: 0.4885, loss: 2.4356 2020-05-23 03:18:39,866 - mmdet - INFO - Epoch [1][100/1730] lr: 0.00398, eta: 3:41:52, time: 0.565, data_time: 0.101, memory: 4932, loss_rpn_cls: 0.2872, loss_rpn_bbox: 0.2404, loss_cls: 0.4054, acc: 93.0947, loss_bbox: 0.1764, loss_mask: 0.3700, loss: 1.4794 2020-05-23 03:19:08,571 - mmdet - INFO - Epoch [1][150/1730] lr: 0.00597, eta: 3:33:17, time: 0.574, data_time: 0.101, memory: 5062, loss_rpn_cls: 0.1821, loss_rpn_bbox: 0.2431, loss_cls: 0.4640, acc: 90.5537, loss_bbox: 0.2871, loss_mask: 0.3372, loss: 1.5135 2020-05-23 03:19:37,701 - mmdet - INFO - Epoch [1][200/1730] lr: 0.00797, eta: 3:29:28, time: 0.582, data_time: 0.104, memory: 5271, loss_rpn_cls: 0.1201, loss_rpn_bbox: 0.2288, loss_cls: 0.4637, acc: 87.8535, loss_bbox: 0.3915, loss_mask: 0.3074, loss: 1.5114 2020-05-23 03:20:07,551 - mmdet - INFO - Epoch [1][250/1730] lr: 0.00997, eta: 3:27:59, time: 0.597, data_time: 0.105, memory: 5422, loss_rpn_cls: 0.1004, loss_rpn_bbox: 0.2055, loss_cls: 0.3916, acc: 87.1963, loss_bbox: 0.5066, loss_mask: 0.2894, loss: 1.4935 2020-05-23 03:20:37,901 - mmdet - INFO - Epoch [1][300/1730] lr: 0.01197, eta: 3:27:24, time: 0.607, data_time: 0.107, memory: 5422, loss_rpn_cls: 0.0506, loss_rpn_bbox: 0.1710, loss_cls: 0.3173, acc: 88.8096, loss_bbox: 0.5628, loss_mask: 0.2750, loss: 1.3767 2020-05-23 03:21:08,620 - mmdet - INFO - Epoch [1][350/1730] lr: 0.01397, eta: 3:27:11, time: 0.614, data_time: 0.108, memory: 5422, loss_rpn_cls: 0.0498, loss_rpn_bbox: 0.1539, loss_cls: 0.2873, acc: 89.3711, loss_bbox: 0.5508, loss_mask: 0.2680, loss: 1.3098 2020-05-23 03:21:39,345 - mmdet - INFO - Epoch [1][400/1730] lr: 0.01596, eta: 3:26:55, time: 0.615, data_time: 0.104, memory: 5422, loss_rpn_cls: 0.0801, loss_rpn_bbox: 0.1680, loss_cls: 0.2922, acc: 89.8096, loss_bbox: 0.5162, loss_mask: 0.2573, loss: 1.3137 2020-05-23 03:22:10,246 - mmdet - INFO - Epoch [1][450/1730] lr: 0.01796, eta: 3:26:43, time: 0.618, data_time: 0.107, memory: 5422, loss_rpn_cls: 0.0481, loss_rpn_bbox: 0.1469, loss_cls: 0.2658, acc: 90.3389, loss_bbox: 0.5173, loss_mask: 0.2487, loss: 1.2268 2020-05-23 03:22:41,038 - mmdet - INFO - Epoch [1][500/1730] lr: 0.01996, eta: 3:26:23, time: 0.616, data_time: 0.106, memory: 5422, loss_rpn_cls: 0.0358, loss_rpn_bbox: 0.1340, loss_cls: 0.2562, acc: 90.1816, loss_bbox: 0.5243, loss_mask: 0.2481, loss: 1.1984 2020-05-23 03:23:11,924 - mmdet - INFO - Epoch [1][550/1730] lr: 0.02000, eta: 3:26:04, time: 0.618, data_time: 0.107, memory: 5422, loss_rpn_cls: 0.0413, loss_rpn_bbox: 0.1406, loss_cls: 0.2616, acc: 90.0938, loss_bbox: 0.5202, loss_mask: 0.2372, loss: 1.2007 2020-05-23 03:23:42,898 - mmdet - INFO - Epoch [1][600/1730] lr: 0.02000, eta: 3:25:46, time: 0.619, data_time: 0.109, memory: 5422, loss_rpn_cls: 0.0717, loss_rpn_bbox: 0.1463, loss_cls: 0.2397, acc: 90.8945, loss_bbox: 0.4870, loss_mask: 0.2516, loss: 1.1963 2020-05-23 03:24:14,103 - mmdet - INFO - Epoch [1][650/1730] lr: 0.02000, eta: 3:25:34, time: 0.624, data_time: 0.108, memory: 5422, loss_rpn_cls: 0.0434, loss_rpn_bbox: 0.1369, loss_cls: 0.2699, acc: 89.6836, loss_bbox: 0.5065, loss_mask: 0.2456, loss: 1.2023

''' The above log file is when I was trying to reproduce the error. But when I got the error originally, I got the AP=AR=-1 for all values after first epoch. If you observe the bbox loss is not really changing also the learning rate has quickly come upto 0.02. Can someone please explain to me what's the issue here ?? I am training on two GPU's. Also I wish to include validation at the end of each epoch as well.

ENVIRONMENT: Python 3.6.9 CUDA 10.1 Using vent with ubuntu 18.04. Thanks!!

First, your work bring a new view for the NLNet and create GC Block with much less computation than NonLocal Block, which is useful for my work. But I have read a Paper named Dual Attention Network for Scene Segmentation , the spatial attention this paper used is the similar as NonLocal Block, the network architecture is as below： But the visualization of attention maps is different from what you gave in the paper: What is causing the effect of this article(Dual Attention Network for Scene Segmentation) to be different from yours?

hi @xvjiarui i move the code into maskrcnn-benchmark and run the config of mask_rcnn_r16_ct_c3-c5_r50_sbn_fpn_1x with the settings: 16 images / 8 GPUs, lr=0.02, and using ap.SyncBatchNorm. it encouters the NaN in the first few interations, it seems to use more GPU than mask_rcnn_r50_fpn_1x. when i set lr to be 0.0025, the training can run successfully. so can u give me some tips how to set the lr when using the ap.SyncBatchNorm?

Considering the limitations of hardware performance, I tried to train a lightweight model. So I use resnet18. But resnet18 use basicblock instead of bottleneck, but it seems that you don't inplement it for basicblock. If I want to use gc block in basicblock , how can I designed it? Thank you !

if I only have single GPU, how much performence without SyncBacthNorm will dorp ?, Simply to say, training on single GPU could achieve similar performance within +-1% error or not ?
should I adjust the learning rate to 1/8 of its orignal lr and set more epoch_num to get close result with orignal implements.
thanks!!!

Hi, I've noticed that ImageNet training takes two steps process in which ResNet without GC is trained first and finetuned further using GC block.

You describe that this was to speed up the experiment. Similarly, Kinetics used pretrained ResNet on ImageNet to inflate the Slow-Only model.

Section 4.1 from SGENet paper (https://arxiv.org/pdf/1905.09646.pdf) also notes that it was difficult to train GCNet from beginning. Have you experimented GCNet by training from scratch? Is there a reason why you chose not to train from scratch with GC module attached?

Hello, the performances of layernorm might not be optimized on some inference platforms on edge devices. Have you tried other replacements for layernorm? Replacing it with BN might not be a good idea, so how about simply remove the layernorm layer? Thanks!

After setup the code, I tried to trian the model and got this error: anaconda3/lib/python3.7/site-packages/mmdet-0.6.0+a132aab-py3.7.egg/mmdet/ops/dcn/deform_conv_cuda.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __cudaPopCallConfiguration

How to fix it?

Environment

• PyTorch version 1.1.0
• CUDA version 10.0

Hi, I am trying to use "X-101-FPN | DCN Cascade Mask | GC(c3-c5, r4)", but I am getting errors. I think that's due to using a wrong config file. Would you let me know which config file corresponds to that model?

Thanks!

hi, I use gcnet with the setting as 'resnet50-fpn+c3~c5 r16', but runtime increases about 15ms, could you tell me the reason?

import torch from torch import nn import torch.nn.functional as F

def kaiming_init(module, a=0, mode='fan_out', nonlinearity='relu', bias=0, distribution='normal'): assert distribution in ['uniform', 'normal'] if distribution == 'uniform': nn.init.kaiming_uniform_( module.weight, a=a, mode=mode, nonlinearity=nonlinearity) else: nn.init.kaiming_normal_( module.weight, a=a, mode=mode, nonlinearity=nonlinearity) if hasattr(module, 'bias') and module.bias is not None: nn.init.constant_(module.bias, bias)

def constant_init(module, val, bias=0): nn.init.constant_(module.weight, val) if hasattr(module, 'bias') and module.bias is not None: nn.init.constant_(module.bias, bias)

def last_zero_init(m): if isinstance(m, nn.Sequential): constant_init(m[-1], val=0) m[-1].inited = True else: constant_init(m, val=0) m.inited = True

class ContextBlock2d(nn.Module):

def __init__(self, inplanes, ratio = 1./16.):
    super(ContextBlock2d, self).__init__()

    self.inplanes = inplanes
    self.planes = int(self.inplanes * ratio)
   
    self.channel_add_conv = nn.Sequential(
        nn.Conv2d(self.inplanes, self.planes, kernel_size=1),
        nn.ReLU(inplace=True),
        nn.Conv2d(self.planes, self.inplanes, kernel_size=1)
    )
    
    self.reset_parameters()

def reset_parameters(self):
    last_zero_init(self.channel_add_conv)

def spatial_pool(self, x):
    batch, channel, height, width = x.size()
    # [N, C, 1, 1]
    context = F.avg_pool2d(x, (height, width))

    return context

def forward(self, x):
    # [N, C, 1, 1]
    context = self.spatial_pool(x)

    # [N, C, 1, 1]
    channel_add_term = self.channel_add_conv(context)
    out = x + channel_add_term

    return out

Thanks for sharing your excellent work! I haven't run the code but I am curious about the implementation of GCNet. Is the Wv2 of conv layer before addition is initialized to zero to not affect the initial behaviour of the original backbone? This has been implemented in the Non-local net by set the scale of BN as zero.

Hello,can I change the model in faster-rcnn resnet instead of mask rcnn only by change this demo? //---------------------------------------------------------------------------------------------------------------------

model settings model = dict( type='FasterRCNN', pretrained='torchvision://resnet50', backbone=dict( type='ResNet', depth=50, num_stages=4, out_indices=(0, 1, 2, 3), frozen_stages=1, gcb=dict(ratio=1. / 4., ), stage_with_gcb=(False, True, True, True), style='pytorch'), neck=dict( type='FPN', in_channels=[256, 512, 1024, 2048], out_channels=256, num_outs=5), rpn_head=dict( type='RPNHead', in_channels=256, feat_channels=256, anchor_scales=[8], anchor_ratios=[0.5, 1.0, 2.0], anchor_strides=[4, 8, 16, 32, 64], target_means=[.0, .0, .0, .0], target_stds=[1.0, 1.0, 1.0, 1.0], loss_cls=dict( type='CrossEntropyLoss', use_sigmoid=True, loss_weight=1.0), loss_bbox=dict(type='SmoothL1Loss', beta=1.0 / 9.0, loss_weight=1.0)), bbox_roi_extractor=dict( type='SingleRoIExtractor', roi_layer=dict(type='RoIAlign', out_size=7, sample_num=2), out_channels=256, featmap_strides=[4, 8, 16, 32]),

Hi, Thanks for your great work! In your paper, attention maps of particular query points are shown. Could you share this visualization code? More specifically, did you implement an interactive interface to do the visualization?

Many thanks again.

Supposed I want to get benchmark using GCNet but only object detection on my custom dataset, do I need to have mask for trainging or just bounding box is enough?

Hello, I'm not sure if GCNet in mmdetection use ImageNet pretrained model (specifically GCNet + ResNet pretrained model) cause I saw in the GCNet/config, it only uses standard "torchvision://resnet50" in official mmedetection and "modelzoo://resnet50" in this repo as pretrained model, but in the original paper, the auther firstly trained on ImageNet and transfor to COCO detection task.

Would you explain does mmdetection use (ResNet pretrained model) or (ResNet+GC pretrained model)? if you only use ResNet pretrained model, why does COCO have such a large improvement?