In your config file for train 65, I can see a class order like this: ZeroShotMaskRCNN -> BackgroundAwareRPNHead -> SharedFCSemanticBBoxHead - > SemanticFCNMaskHead

They are classes located in mmdet/models, where are the hook functions that connect among those classes?

Describe the bug when i run the commend "python setup.py develop",there is a ERROR: Error compiling objects for extension I have check the environment and the cuda is availiable. my environment is same to what requirement.txt describe.

I ask to Google and find I shoud to replace "AT_CHECK" to "TORCH_CHECK" and I find "AT_CHECK" in some path: mmdet/opt/ /src/ .cpp

Reproduction python setup.py develop

I am confusing about your work.

1. how is the difference between bbox_head_semantic and convfc_bbox_semantic_head?
2. how is the difference between ba_anchor_head and anchor_semantic_head?
3. how are the difference among zero_shot_faster_rcnn, zero_shot_mask_rcnn and zero_shot_two_stage ?

Hi, There is a problem that i face frequently, which is RuntimeError: CUDA error: no kernel image is available for execution on the device. I dont know why it often occurred, though I was using the exact environment versions like yours (cuda, cuddnn, mmcv, mmdet, etc). Do you know why?

Traceback (most recent call last):
  File "./tools/train.py", line 118, in <module>
    main()
  File "./tools/train.py", line 114, in main
    logger=logger)
  File "/home/merge/Harry/instance_seg/ZSI/mmdet/apis/train.py", line 58, in train_detector
    _dist_train(model, dataset, cfg, validate=validate)
  File "/home/merge/Harry/instance_seg/ZSI/mmdet/apis/train.py", line 192, in _dist_train
    runner.run(data_loaders, cfg.workflow, cfg.total_epochs)
  File "/home/merge/anaconda3/envs/zsi/lib/python3.7/site-packages/mmcv/runner/runner.py", line 358, in run
    epoch_runner(data_loaders[i], **kwargs)
  File "/home/merge/anaconda3/envs/zsi/lib/python3.7/site-packages/mmcv/runner/runner.py", line 264, in train
    self.model, data_batch, train_mode=True, **kwargs)
  File "/home/merge/Harry/instance_seg/ZSI/mmdet/apis/train.py", line 38, in batch_processor
    losses = model(**data)
  File "/home/merge/anaconda3/envs/zsi/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/merge/anaconda3/envs/zsi/lib/python3.7/site-packages/mmcv/parallel/distributed.py", line 50, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/home/merge/anaconda3/envs/zsi/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/merge/Harry/instance_seg/ZSI/mmdet/core/fp16/decorators.py", line 49, in new_func
    return old_func(*args, **kwargs)
  File "/home/merge/Harry/instance_seg/ZSI/mmdet/models/detectors/base.py", line 117, in forward
    return self.forward_train(img, img_meta, **kwargs)
  File "/home/merge/Harry/instance_seg/ZSI/mmdet/models/detectors/zero_shot_two_stage.py", line 179, in forward_train
    x = self.extract_feat(img)
  File "/home/merge/Harry/instance_seg/ZSI/mmdet/models/detectors/zero_shot_two_stage.py", line 105, in extract_feat
    x = self.backbone(img)
  File "/home/merge/anaconda3/envs/zsi/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/merge/Harry/instance_seg/ZSI/mmdet/models/backbones/resnet.py", line 525, in forward
    x = self.relu(x)
  File "/home/merge/anaconda3/envs/zsi/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/merge/anaconda3/envs/zsi/lib/python3.7/site-packages/torch/nn/modules/activation.py", line 99, in forward
    return F.relu(input, inplace=self.inplace)
  File "/home/merge/anaconda3/envs/zsi/lib/python3.7/site-packages/torch/nn/functional.py", line 941, in relu
    result = torch.relu_(input)
RuntimeError: CUDA error: no kernel image is available for execution on the device

I tested cuda on python, it worked fine. That means there was an issue with ZSI, not because my setup environment.

Python 3.7.10 (default, Jun  4 2021, 14:48:32) 
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.rand(10).to("cuda")

tensor([0.8508, 0.1020, 0.9818, 0.8558, 0.5624, 0.0784, 0.0054, 0.5591, 0.3068,
        0.6484], device='cuda:0')
>>> 
>>> torch.cuda.device_count()
4
>>> torch.cuda.is_available()
True
>>> 

  File "/home/merge/Harry/instance_seg/ZSI/mmdet/ops/dcn/deform_conv.py", line 9, in <module>

ImportError: from . import deform_conv_cuda

While starting training, the model could not import from deform_conv_cuda. You are using mmdetection library and there's an issue which is similar to mine but it didn't work https://github.com/open-mmlab/mmdetection/issues/601

I'm trying to divide the dataset into 72 seen classes and 8 unseen classes. Then trained on the newly divided dataset, but it went wrong when testing and did not detect any unseen classes. Could you please offer any advice? This is my config file.

# model settings
model = dict(
    type='ZeroShotFasterRCNN',
    pretrained='torchvision://resnet101',
    backbone=dict(
        type='ResNet',
        depth=101,
        num_stages=4,
        out_indices=(0, 1, 2, 3),
        frozen_stages=1,
        style='pytorch'),
    neck=dict(
        type='FPN',
        in_channels=[256, 512, 1024, 2048],
        out_channels=256,
        num_outs=5),
    rpn_head=dict(
        type='BackgroundAwareRPNHead',
        in_channels=256,
        semantic_dims=300,
        feat_channels=256,
        anchor_scales=[8],
        anchor_ratios=[0.5, 1.0, 2.0],
        anchor_strides=[4, 8, 16, 32, 64],
        target_means=[.0, .0, .0, .0],
        target_stds=[1.0, 1.0, 1.0, 1.0],
        voc_path=None,
        vec_path='data/coco/word_w2v_withbg_72_8.txt',
        sync_bg=True,
        loss_cls=dict(
            type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0),
        loss_bbox=dict(type='SmoothL1Loss', beta=1.0 / 9.0, loss_weight=1.0)),
    bbox_roi_extractor=dict(
        type='SingleRoIExtractor',
        roi_layer=dict(type='RoIAlign', out_size=7, sample_num=2),
        out_channels=256,
        featmap_strides=[4, 8, 16, 32]),
    bbox_head=dict(
        type='SharedFCSemanticBBoxHead',
        num_fcs=2,
        in_channels=256,
        fc_out_channels=1024,
        roi_feat_size=7,
        num_classes=73,
        semantic_dims=300,
        seen_class=True,
        reg_with_semantic=False,
        share_semantic=False,
        with_decoder=True,
        sync_bg=True,
        voc_path='data/coco/vocabulary_w2v.txt',
        vec_path='data/coco/word_w2v_withbg_72_8.txt',
        target_means=[0., 0., 0., 0.],
        target_stds=[0.1, 0.1, 0.2, 0.2],
        reg_class_agnostic=False,
        loss_semantic=dict(
            type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0),
        loss_bbox=dict(type='SmoothL1Loss', beta=1.0, loss_weight=1.0)),
    bbox_with_decoder=True,
    bbox_sync_bg=True)
# model training and testing settings
train_cfg = dict(
    rpn=dict(
        assigner=dict(
            type='MaxIoUAssigner',
            pos_iou_thr=0.7,
            neg_iou_thr=0.3,
            min_pos_iou=0.3,
            ignore_iof_thr=-1),
        sampler=dict(
            type='RandomSampler',
            num=256,
            pos_fraction=0.5,
            neg_pos_ub=-1,
            add_gt_as_proposals=False),
        allowed_border=0,
        pos_weight=-1,
        debug=False),
    rpn_proposal=dict(
        nms_across_levels=False,
        nms_pre=2000,
        nms_post=2000,
        max_num=2000,
        nms_thr=0.7,
        min_bbox_size=0),
    rcnn=dict(
        assigner=dict(
            type='MaxIoUAssigner',
            pos_iou_thr=0.5,
            neg_iou_thr=0.5,
            min_pos_iou=0.5,
            ignore_iof_thr=-1),
        sampler=dict(
            type='RandomSampler',
            num=512,
            pos_fraction=0.25,
            neg_pos_ub=-1,
            add_gt_as_proposals=True),
        pos_weight=-1,
        debug=False))
test_cfg = dict(
    rpn=dict(
        nms_across_levels=False,
        nms_pre=1000,
        nms_post=1000,
        max_num=1000,
        nms_thr=0.7,
        min_bbox_size=0),
    rcnn=dict(
        score_thr=0.05,
        nms=dict(type='nms', iou_thr=0.5),
        max_per_img=100))
# dataset settings
dataset_type = 'CocoDataset_72_8'
data_root = 'data/coco/'
img_norm_cfg = dict(
    mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)
train_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(type='LoadAnnotations', with_bbox=True),
    dict(type='Resize', img_scale=(1333, 800), keep_ratio=True),
    dict(type='RandomFlip', flip_ratio=0.5),
    dict(type='Normalize', **img_norm_cfg),
    dict(type='Pad', size_divisor=32),
    dict(type='DefaultFormatBundle'),
    dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels']),
]
test_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(
        type='MultiScaleFlipAug',
        img_scale=(1333, 800),
        flip=False,
        transforms=[
            dict(type='Resize', keep_ratio=True),
            dict(type='RandomFlip'),
            dict(type='Normalize', **img_norm_cfg),
            dict(type='Pad', size_divisor=32),
            dict(type='ImageToTensor', keys=['img']),
            dict(type='Collect', keys=['img']),
        ])
]
data = dict(
    imgs_per_gpu=2,
    workers_per_gpu=0,
    train=dict(
        type=dataset_type,
        ann_file=data_root + 'annotations/instances_train2014_seen_72_8.json',
        img_prefix=data_root + 'train2014/',
        pipeline=train_pipeline),
    val=dict(
        type=dataset_type,
        ann_file=data_root + 'annotations/instances_val2014_seen_72_8.json',
        img_prefix=data_root + 'val2014/',
        pipeline=test_pipeline),
    test=dict(
        type=dataset_type,
        ann_file=data_root + 'annotations/instances_val2014_gzsi_72_8.json',
        img_prefix=data_root + 'val2014/',
        pipeline=test_pipeline))
# optimizer
# optimizer = dict(type='SGD', lr=0.0075, momentum=0.9, weight_decay=0.0001)
optimizer = dict(type='SGD', lr=0.0025, momentum=0.9, weight_decay=0.0001)
optimizer_config = dict(grad_clip=dict(max_norm=35, norm_type=2))
# learning policy
lr_config = dict(
    policy='step',
    warmup='linear',
    warmup_iters=7000,
    warmup_ratio=1.0 / 3,
    step=[8, 11])
checkpoint_config = dict(interval=12)
# yapf:disable
log_config = dict(
    interval=50,
    hooks=[
        dict(type='TextLoggerHook'),
        # dict(type='TensorboardLoggerHook')
    ])
# yapf:enable
evaluation = dict(interval=1)
# runtime settings
total_epochs = 12
dist_params = dict(backend='nccl')
log_level = 'INFO'
work_dir = './work_dirs/zsi/faster/72_8/'
load_from = None
resume_from = None
workflow = [('train', 1)]

Please help me how to change inference.py to see images in colab?

I get the following error:

_qt.qpa.xcb: could not connect to display qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "/usr/local/lib/python3.7/site-packages/cv2/qt/plugins" even though it was found. This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: xcb._

Below train config setting is correct? It looks ignoring seen classes even during training. I wonder why this data class is called during training.

https://github.com/zhengye1995/Zero-shot-Instance-Segmentation/blob/6b25b318d7821b5f7a48f7d5e6dbd0bc63474e14/configs/zsi/65_15/train/zero-shot-mask-rcnn-BARPN-bbox_mask_sync_bg_65_15_decoder_notanh.py#L141

https://github.com/zhengye1995/Zero-shot-Instance-Segmentation/blob/6b25b318d7821b5f7a48f7d5e6dbd0bc63474e14/configs/zsi/48_17/train/zero-shot-mask-rcnn-BARPN-bbox_mask_sync_bg_decoder.py#L142

I think that CocoDataset_48_17 and CocoDataset_65_15 data class looks appropriate.

Thank you a lot!

I am reproducing this work using the latest mmdet version. Inside the file barpn_head.py, I need to import nms from ops (from mmdet.ops import nms). However, in the new architecture of mmdet, there is no folder named ops inside mmdet, therefore, I copied the whole nnm folder from the old mmdet version. And I got this error: ImportError: mmdet/core/nms/nms_cpu.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN3c106detail23torchInternalAssertFailEPKcS2_jS2_S2_ #10. That is because these two lines inside `nns_wrapper.py":

from . import nms_cpu, nms_cuda
from .soft_nms_cpu import soft_nms_cpu

It is failed to load these modules from the following files:

1. nms_cpu.cpython-37m-x86_64-linux-gnu.so
2. nms_cuda.cpython-37m-x86_64-linux-gnu.so
3. soft_nms_cpu.cpython-37m-x86_64-linux-gnu.so

Do you know how to get away with this problem? Note that in the new mmdet version, it does not have ops folder after compiling from mmcv

Could you please let me know how did you create datasets? I mean:

1. what are vocabulary_w2v.txt, word_w2v_withbg_48_17.txt, word_w2v_withbg_65_15.txt ? And how did you create them?

2. Also, How did you create files in annotations folders such as word_w2v_withbg_65_15.txt, instances_val2014_gzsi_65_15.txt, and instances_val2014_unseen_65_15.json ? Thank you.

Hi, one of the evaluation metric for zero shot object detection is Recall@100, but how to compute it is not very clear. My understanding is following. First, select top 100 detections from an image. Second, mark a predicted bounding box as positive if it has an IoU greater than a threshold (0.5 for example) and no other higher confidence bounding box has been assigned to the same GT box. Third, compute recall@100 for this image number_of_positive_prediction / 100. Forth, compute recall@100 for all images sum(recall@100 for each image) / number_of_image.

Is it correct ? Thanks a lot!

Hi i'm confuse about why need to count this as the final score ?

seen_scores = torch.mm(scores, self.vec.t())
seen_scores = torch.mm(seen_scores, self.vec)
unseen_scores = torch.mm(scores, self.vec.t())
unseen_scores = torch.mm(unseen_scores, self.vec_unseen)

because when in training the score will be semantic_score that is not same as above does it have special meaning and why the score can work?

As you can see the following results, the segmentation results perform well. However, the classification of objects' labels is very poor (mislabeling). Why is that? Are there any mistakes for class labels in your code? because an airplane is mostly classified as a kite or a boat.