code for CVPR paper Zero-shot Instance Segmentation

Overview

Code for CVPR2021 paper

Zero-shot Instance Segmentation

Code requirements

  • python: python3.7
  • nvidia GPU
  • pytorch1.1.0
  • GCC >=5.4
  • NCCL 2
  • the other python libs in requirement.txt

Install

conda create -n zsi python=3.7 -y
conda activate zsi

conda install pytorch=1.1.0 torchvision=0.3.0 cudatoolkit=10.0 -c pytorch

pip install cython && pip --no-cache-dir install -r requirements.txt
   
python setup.py develop

Dataset prepare

  • Download the train and test annotations files for zsi from annotations, put all json label file to

    data/coco/annotations/
    
  • Download MSCOCO-2014 dataset and unzip the images it to path:

    data/coco/train2014/
    data/coco/val2014/
    
  • Training:

    • 48/17 split:

         chmod +x tools/dist_train.sh
         ./tools/dist_train.sh configs/zsi/train/zero-shot-mask-rcnn-BARPN-bbox_mask_sync_bg_decoder.py 4
      
    • 65/15 split:

      chmod +x tools/dist_train.sh
      ./tools/dist_train.sh configs/zsi/train/zero-shot-mask-rcnn-BARPN-bbox_mask_sync_bg_65_15_decoder_notanh.py 4
      
  • Inference & Evaluate:

    • ZSI task:

      • 48/17 split ZSI task:
        • download 48/17 ZSI model, put it in checkpoints/ZSI_48_17.pth

        • inference:

          chmod +x tools/dist_test.sh
          ./tools/dist_test.sh configs/zsi/48_17/test/zsi/zero-shot-mask-rcnn-BARPN-bbox_mask_sync_bg_decoder.py checkpoints/ZSI_48_17.pth 4 --json_out results/zsi_48_17.json
          
        • our results zsi_48_17.bbox.json and zsi_48_17.segm.json can also downloaded from zsi_48_17_reults.

        • evaluate:

          • for zsd performance
            python tools/zsi_coco_eval.py results/zsi_48_17.bbox.json --ann data/coco/annotations/instances_val2014_unseen_48_17.json
            
          • for zsi performance
            python tools/zsi_coco_eval.py results/zsi_48_17.segm.json --ann data/coco/annotations/instances_val2014_unseen_48_17.json --types segm
            
      • 65/15 split ZSI task:
        • download 65/15 ZSI model, put it in checkpoints/ZSI_65_15.pth

        • inference:

          chmod +x tools/dist_test.sh
          ./toools/dist_test.sh configs/zsi/65_15/test/zsi/zero-shot-mask-rcnn-BARPN-bbox_mask_sync_bg_65_15_decoder_notanh.py checkpoints/ZSI_65_15.pth 4 --json_out results/zsi_65_15.json
          
        • our results zsi_65_15.bbox.json and zsi_65_15.segm.json can also downloaded from zsi_65_15_reults.

        • evaluate:

          • for zsd performance
            python tools/zsi_coco_eval.py results/zsi_65_15.bbox.json --ann data/coco/annotations/instances_val2014_unseen_65_15.json
            
          • for zsi performance
            python tools/zsi_coco_eval.py results/zsi_65_15.segm.json --ann data/coco/annotations/instances_val2014_unseen_65_15.json --types segm
            
    • GZSI task:

      • 48/17 split GZSI task:
        • use the same model file ZSI_48_17.pth in ZSI task
        • inference:
          chmod +x tools/dist_test.sh
          ./tools/dist_test.sh configs/zsi/48_17/test/gzsi/zero-shot-mask-rcnn-BARPN-bbox_mask_sync_bg_decoder_gzsi.py checkpoints/ZSI_48_17.pth 4 --json_out results/gzsi_48_17.json
          
        • our results gzsi_48_17.bbox.json and gzsi_48_17.segm.json can also downloaded from gzsi_48_17_results.
        • evaluate:
          • for gzsd
            python tools/gzsi_coco_eval.py results/gzsi_48_17.bbox.json --ann data/coco/annotations/instances_val2014_gzsi_48_17.json --gzsi --num-seen-classes 48
            
          • for gzsi
            python tools/gzsi_coco_eval.py results/gzsi_48_17.segm.json --ann data/coco/annotations/instances_val2014_gzsi_48_17.json --gzsi --num-seen-classes 48 --types segm
            
      • 65/15 split GZSI task:
        • use the same model file ZSI_48_17.pth in ZSI task
        • inference:
          chmod +x tools/dist_test.sh
          ./tools/dist_test.sh configs/zsi/65_15/test/gzsi/zero-shot-mask-rcnn-BARPN-bbox_mask_sync_bg_65_15_decoder_notanh_gzsi.py checkpoints/ZSI_65_15.pth 4 --json_out results/gzsi_65_15.json
          
        • our results gzsi_65_15.bbox.json and gzsi_65_15.segm.json can also downloaded from gzsi_65_15_results.
        • evaluate:
          • for gzsd
            python tools/gzsi_coco_eval.py results/gzsi_65_15.bbox.json --ann data/coco/annotations/instances_val2014_gzsi_65_15.json --gzsd --num-seen-classes 65
            
          • for gzsi
            python tools/gzsi_coco_eval.py results/gzsi_65_15.segm.json --ann data/coco/annotations/instances_val2014_gzsi_65_15.json --gzsd --num-seen-classes 65 --types segm
            

License

ZSI is released under MIT License.

Citing

If you use ZSI in your research or wish to refer to the baseline results published here, please use the following BibTeX entries:

@InProceedings{zhengye2021zsi,
  author  =  {Ye, Zheng and Jiahong, Wu and Yongqiag, Qin and Faen, Zhang and Li, Cui},
  title   =  {Zero-shot Instance Segmentation},
  booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  month = {June},
  year = {2021}
}
Comments
  • Where are the hooks among classes?

    Where are the hooks among classes?

    In your config file for train 65, I can see a class order like this: ZeroShotMaskRCNN -> BackgroundAwareRPNHead -> SharedFCSemanticBBoxHead - > SemanticFCNMaskHead

    They are classes located in mmdet/models, where are the hook functions that connect among those classes?

    opened by algoteam5 9
  • Error compiling objects for extension

    Error compiling objects for extension

    Describe the bug when i run the commend "python setup.py develop",there is a ERROR: Error compiling objects for extension I have check the environment and the cuda is availiable. my environment is same to what requirement.txt describe.

    I ask to Google and find I shoud to replace "AT_CHECK" to "TORCH_CHECK" and I find "AT_CHECK" in some path: mmdet/opt/ /src/ .cpp

    Reproduction python setup.py develop

    opened by wisest-cui 8
  • how are the difference between bbox_head_semantic and convfc_bbox_semantic_head?

    how are the difference between bbox_head_semantic and convfc_bbox_semantic_head?

    I am confusing about your work.

    1. how is the difference between bbox_head_semantic and convfc_bbox_semantic_head?
    2. how is the difference between ba_anchor_head and anchor_semantic_head?
    3. how are the difference among zero_shot_faster_rcnn, zero_shot_mask_rcnn and zero_shot_two_stage ?
    opened by algoteam5 8
  • RuntimeError: CUDA error: no kernel image is available for execution on the device

    RuntimeError: CUDA error: no kernel image is available for execution on the device

    Hi, There is a problem that i face frequently, which is RuntimeError: CUDA error: no kernel image is available for execution on the device. I dont know why it often occurred, though I was using the exact environment versions like yours (cuda, cuddnn, mmcv, mmdet, etc). Do you know why?

    Traceback (most recent call last):
      File "./tools/train.py", line 118, in <module>
        main()
      File "./tools/train.py", line 114, in main
        logger=logger)
      File "/home/merge/Harry/instance_seg/ZSI/mmdet/apis/train.py", line 58, in train_detector
        _dist_train(model, dataset, cfg, validate=validate)
      File "/home/merge/Harry/instance_seg/ZSI/mmdet/apis/train.py", line 192, in _dist_train
        runner.run(data_loaders, cfg.workflow, cfg.total_epochs)
      File "/home/merge/anaconda3/envs/zsi/lib/python3.7/site-packages/mmcv/runner/runner.py", line 358, in run
        epoch_runner(data_loaders[i], **kwargs)
      File "/home/merge/anaconda3/envs/zsi/lib/python3.7/site-packages/mmcv/runner/runner.py", line 264, in train
        self.model, data_batch, train_mode=True, **kwargs)
      File "/home/merge/Harry/instance_seg/ZSI/mmdet/apis/train.py", line 38, in batch_processor
        losses = model(**data)
      File "/home/merge/anaconda3/envs/zsi/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
        result = self.forward(*input, **kwargs)
      File "/home/merge/anaconda3/envs/zsi/lib/python3.7/site-packages/mmcv/parallel/distributed.py", line 50, in forward
        return self.module(*inputs[0], **kwargs[0])
      File "/home/merge/anaconda3/envs/zsi/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
        result = self.forward(*input, **kwargs)
      File "/home/merge/Harry/instance_seg/ZSI/mmdet/core/fp16/decorators.py", line 49, in new_func
        return old_func(*args, **kwargs)
      File "/home/merge/Harry/instance_seg/ZSI/mmdet/models/detectors/base.py", line 117, in forward
        return self.forward_train(img, img_meta, **kwargs)
      File "/home/merge/Harry/instance_seg/ZSI/mmdet/models/detectors/zero_shot_two_stage.py", line 179, in forward_train
        x = self.extract_feat(img)
      File "/home/merge/Harry/instance_seg/ZSI/mmdet/models/detectors/zero_shot_two_stage.py", line 105, in extract_feat
        x = self.backbone(img)
      File "/home/merge/anaconda3/envs/zsi/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
        result = self.forward(*input, **kwargs)
      File "/home/merge/Harry/instance_seg/ZSI/mmdet/models/backbones/resnet.py", line 525, in forward
        x = self.relu(x)
      File "/home/merge/anaconda3/envs/zsi/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
        result = self.forward(*input, **kwargs)
      File "/home/merge/anaconda3/envs/zsi/lib/python3.7/site-packages/torch/nn/modules/activation.py", line 99, in forward
        return F.relu(input, inplace=self.inplace)
      File "/home/merge/anaconda3/envs/zsi/lib/python3.7/site-packages/torch/nn/functional.py", line 941, in relu
        result = torch.relu_(input)
    RuntimeError: CUDA error: no kernel image is available for execution on the device
    
    

    I tested cuda on python, it worked fine. That means there was an issue with ZSI, not because my setup environment.

    Python 3.7.10 (default, Jun  4 2021, 14:48:32) 
    [GCC 7.5.0] :: Anaconda, Inc. on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import torch
    >>> torch.rand(10).to("cuda")
    
    tensor([0.8508, 0.1020, 0.9818, 0.8558, 0.5624, 0.0784, 0.0054, 0.5591, 0.3068,
            0.6484], device='cuda:0')
    >>> 
    >>> torch.cuda.device_count()
    4
    >>> torch.cuda.is_available()
    True
    >>> 
    
    
    opened by harrytrinh96 7
  • ImportError: from . import deform_conv_cuda

    ImportError: from . import deform_conv_cuda

      File "/home/merge/Harry/instance_seg/ZSI/mmdet/ops/dcn/deform_conv.py", line 9, in <module>
    
    ImportError: from . import deform_conv_cuda
    

    While starting training, the model could not import from deform_conv_cuda. You are using mmdetection library and there's an issue which is similar to mine but it didn't work https://github.com/open-mmlab/mmdetection/issues/601

    opened by TrinhDinhPhuc 7
  • Could not detect any unseen class

    Could not detect any unseen class

    I'm trying to divide the dataset into 72 seen classes and 8 unseen classes. Then trained on the newly divided dataset, but it went wrong when testing and did not detect any unseen classes. Could you please offer any advice? This is my config file.

    # model settings
    model = dict(
        type='ZeroShotFasterRCNN',
        pretrained='torchvision://resnet101',
        backbone=dict(
            type='ResNet',
            depth=101,
            num_stages=4,
            out_indices=(0, 1, 2, 3),
            frozen_stages=1,
            style='pytorch'),
        neck=dict(
            type='FPN',
            in_channels=[256, 512, 1024, 2048],
            out_channels=256,
            num_outs=5),
        rpn_head=dict(
            type='BackgroundAwareRPNHead',
            in_channels=256,
            semantic_dims=300,
            feat_channels=256,
            anchor_scales=[8],
            anchor_ratios=[0.5, 1.0, 2.0],
            anchor_strides=[4, 8, 16, 32, 64],
            target_means=[.0, .0, .0, .0],
            target_stds=[1.0, 1.0, 1.0, 1.0],
            voc_path=None,
            vec_path='data/coco/word_w2v_withbg_72_8.txt',
            sync_bg=True,
            loss_cls=dict(
                type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0),
            loss_bbox=dict(type='SmoothL1Loss', beta=1.0 / 9.0, loss_weight=1.0)),
        bbox_roi_extractor=dict(
            type='SingleRoIExtractor',
            roi_layer=dict(type='RoIAlign', out_size=7, sample_num=2),
            out_channels=256,
            featmap_strides=[4, 8, 16, 32]),
        bbox_head=dict(
            type='SharedFCSemanticBBoxHead',
            num_fcs=2,
            in_channels=256,
            fc_out_channels=1024,
            roi_feat_size=7,
            num_classes=73,
            semantic_dims=300,
            seen_class=True,
            reg_with_semantic=False,
            share_semantic=False,
            with_decoder=True,
            sync_bg=True,
            voc_path='data/coco/vocabulary_w2v.txt',
            vec_path='data/coco/word_w2v_withbg_72_8.txt',
            target_means=[0., 0., 0., 0.],
            target_stds=[0.1, 0.1, 0.2, 0.2],
            reg_class_agnostic=False,
            loss_semantic=dict(
                type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0),
            loss_bbox=dict(type='SmoothL1Loss', beta=1.0, loss_weight=1.0)),
        bbox_with_decoder=True,
        bbox_sync_bg=True)
    # model training and testing settings
    train_cfg = dict(
        rpn=dict(
            assigner=dict(
                type='MaxIoUAssigner',
                pos_iou_thr=0.7,
                neg_iou_thr=0.3,
                min_pos_iou=0.3,
                ignore_iof_thr=-1),
            sampler=dict(
                type='RandomSampler',
                num=256,
                pos_fraction=0.5,
                neg_pos_ub=-1,
                add_gt_as_proposals=False),
            allowed_border=0,
            pos_weight=-1,
            debug=False),
        rpn_proposal=dict(
            nms_across_levels=False,
            nms_pre=2000,
            nms_post=2000,
            max_num=2000,
            nms_thr=0.7,
            min_bbox_size=0),
        rcnn=dict(
            assigner=dict(
                type='MaxIoUAssigner',
                pos_iou_thr=0.5,
                neg_iou_thr=0.5,
                min_pos_iou=0.5,
                ignore_iof_thr=-1),
            sampler=dict(
                type='RandomSampler',
                num=512,
                pos_fraction=0.25,
                neg_pos_ub=-1,
                add_gt_as_proposals=True),
            pos_weight=-1,
            debug=False))
    test_cfg = dict(
        rpn=dict(
            nms_across_levels=False,
            nms_pre=1000,
            nms_post=1000,
            max_num=1000,
            nms_thr=0.7,
            min_bbox_size=0),
        rcnn=dict(
            score_thr=0.05,
            nms=dict(type='nms', iou_thr=0.5),
            max_per_img=100))
    # dataset settings
    dataset_type = 'CocoDataset_72_8'
    data_root = 'data/coco/'
    img_norm_cfg = dict(
        mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)
    train_pipeline = [
        dict(type='LoadImageFromFile'),
        dict(type='LoadAnnotations', with_bbox=True),
        dict(type='Resize', img_scale=(1333, 800), keep_ratio=True),
        dict(type='RandomFlip', flip_ratio=0.5),
        dict(type='Normalize', **img_norm_cfg),
        dict(type='Pad', size_divisor=32),
        dict(type='DefaultFormatBundle'),
        dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels']),
    ]
    test_pipeline = [
        dict(type='LoadImageFromFile'),
        dict(
            type='MultiScaleFlipAug',
            img_scale=(1333, 800),
            flip=False,
            transforms=[
                dict(type='Resize', keep_ratio=True),
                dict(type='RandomFlip'),
                dict(type='Normalize', **img_norm_cfg),
                dict(type='Pad', size_divisor=32),
                dict(type='ImageToTensor', keys=['img']),
                dict(type='Collect', keys=['img']),
            ])
    ]
    data = dict(
        imgs_per_gpu=2,
        workers_per_gpu=0,
        train=dict(
            type=dataset_type,
            ann_file=data_root + 'annotations/instances_train2014_seen_72_8.json',
            img_prefix=data_root + 'train2014/',
            pipeline=train_pipeline),
        val=dict(
            type=dataset_type,
            ann_file=data_root + 'annotations/instances_val2014_seen_72_8.json',
            img_prefix=data_root + 'val2014/',
            pipeline=test_pipeline),
        test=dict(
            type=dataset_type,
            ann_file=data_root + 'annotations/instances_val2014_gzsi_72_8.json',
            img_prefix=data_root + 'val2014/',
            pipeline=test_pipeline))
    # optimizer
    # optimizer = dict(type='SGD', lr=0.0075, momentum=0.9, weight_decay=0.0001)
    optimizer = dict(type='SGD', lr=0.0025, momentum=0.9, weight_decay=0.0001)
    optimizer_config = dict(grad_clip=dict(max_norm=35, norm_type=2))
    # learning policy
    lr_config = dict(
        policy='step',
        warmup='linear',
        warmup_iters=7000,
        warmup_ratio=1.0 / 3,
        step=[8, 11])
    checkpoint_config = dict(interval=12)
    # yapf:disable
    log_config = dict(
        interval=50,
        hooks=[
            dict(type='TextLoggerHook'),
            # dict(type='TensorboardLoggerHook')
        ])
    # yapf:enable
    evaluation = dict(interval=1)
    # runtime settings
    total_epochs = 12
    dist_params = dict(backend='nccl')
    log_level = 'INFO'
    work_dir = './work_dirs/zsi/faster/72_8/'
    load_from = None
    resume_from = None
    workflow = [('train', 1)]
    
    opened by LHdagong 5
  • google colab

    google colab

    Please help me how to change inference.py to see images in colab?

    I get the following error:

    _qt.qpa.xcb: could not connect to display qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "/usr/local/lib/python3.7/site-packages/cv2/qt/plugins" even though it was found. This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

    Available platform plugins are: xcb._

    opened by ZahraHeydari95 5
  • Is train dataset config is correct?

    Is train dataset config is correct?

    Below train config setting is correct? It looks ignoring seen classes even during training. I wonder why this data class is called during training.

    https://github.com/zhengye1995/Zero-shot-Instance-Segmentation/blob/6b25b318d7821b5f7a48f7d5e6dbd0bc63474e14/configs/zsi/65_15/train/zero-shot-mask-rcnn-BARPN-bbox_mask_sync_bg_65_15_decoder_notanh.py#L141

    https://github.com/zhengye1995/Zero-shot-Instance-Segmentation/blob/6b25b318d7821b5f7a48f7d5e6dbd0bc63474e14/configs/zsi/48_17/train/zero-shot-mask-rcnn-BARPN-bbox_mask_sync_bg_decoder.py#L142

    I think that CocoDataset_48_17 and CocoDataset_65_15 data class looks appropriate.

    Thank you a lot!

    opened by parallel-lain 4
  • ImportError: mmdet/core/nms/nms_cpu.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN3c106detail23torchInternalAssertFailEPKcS2_jS2_S2_

    ImportError: mmdet/core/nms/nms_cpu.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN3c106detail23torchInternalAssertFailEPKcS2_jS2_S2_

    I am reproducing this work using the latest mmdet version. Inside the file barpn_head.py, I need to import nms from ops (from mmdet.ops import nms). However, in the new architecture of mmdet, there is no folder named ops inside mmdet, therefore, I copied the whole nnm folder from the old mmdet version. And I got this error: ImportError: mmdet/core/nms/nms_cpu.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN3c106detail23torchInternalAssertFailEPKcS2_jS2_S2_ #10. That is because these two lines inside `nns_wrapper.py":

    from . import nms_cpu, nms_cuda
    from .soft_nms_cpu import soft_nms_cpu
    

    It is failed to load these modules from the following files:

    1. nms_cpu.cpython-37m-x86_64-linux-gnu.so
    2. nms_cuda.cpython-37m-x86_64-linux-gnu.so
    3. soft_nms_cpu.cpython-37m-x86_64-linux-gnu.so

    Do you know how to get away with this problem? Note that in the new mmdet version, it does not have ops folder after compiling from mmcv

    opened by TrinhDinhPhuc 4
  • How did you create your json annotations 65/15, vocabulary, and word-to-vec ?

    How did you create your json annotations 65/15, vocabulary, and word-to-vec ?

    Could you please let me know how did you create datasets? I mean:

    1. what are vocabulary_w2v.txt, word_w2v_withbg_48_17.txt, word_w2v_withbg_65_15.txt ? And how did you create them?

    2. Also, How did you create files in annotations folders such as word_w2v_withbg_65_15.txt, instances_val2014_gzsi_65_15.txt, and instances_val2014_unseen_65_15.json ? Thank you.

    opened by harrytrinh96 4
  • How to compute Recall@100

    How to compute Recall@100

    Hi, one of the evaluation metric for zero shot object detection is Recall@100, but how to compute it is not very clear. My understanding is following. First, select top 100 detections from an image. Second, mark a predicted bounding box as positive if it has an IoU greater than a threshold (0.5 for example) and no other higher confidence bounding box has been assigned to the same GT box. Third, compute recall@100 for this image number_of_positive_prediction / 100. Forth, compute recall@100 for all images sum(recall@100 for each image) / number_of_image.

    Is it correct ? Thanks a lot!

    opened by zhongxiangzju 4
  • Question about get_det_bboxes function

    Question about get_det_bboxes function

    Hi i'm confuse about why need to count this as the final score ?

    seen_scores = torch.mm(scores, self.vec.t())
    seen_scores = torch.mm(seen_scores, self.vec)
    unseen_scores = torch.mm(scores, self.vec.t())
    unseen_scores = torch.mm(unseen_scores, self.vec_unseen)
    

    because when in training the score will be semantic_score that is not same as above does it have special meaning and why the score can work?

    opened by JKZhan 10
  • Why does the classifer perform poor?

    Why does the classifer perform poor?

    As you can see the following results, the segmentation results perform well. However, the classification of objects' labels is very poor (mislabeling). Why is that? Are there any mistakes for class labels in your code? because an airplane is mostly classified as a kite or a boat.

    Screenshot from 2021-08-11 18-53-11 Screenshot from 2021-08-11 18-53-22 Screenshot from 2021-08-11 18-53-41 Screenshot from 2021-08-11 18-53-56

    opened by TrinhDinhPhuc 1
Owner
zhengye
CS Phd
zhengye
Public repository of the 3DV 2021 paper "Generative Zero-Shot Learning for Semantic Segmentation of 3D Point Clouds"

Generative Zero-Shot Learning for Semantic Segmentation of 3D Point Clouds Björn Michele1), Alexandre Boulch1), Gilles Puy1), Maxime Bucher1) and Rena

valeo.ai 15 Dec 22, 2022
The official implementation of the CVPR 2021 paper FAPIS: a Few-shot Anchor-free Part-based Instance Segmenter

FAPIS The official implementation of the CVPR 2021 paper FAPIS: a Few-shot Anchor-free Part-based Instance Segmenter Introduction This repo is primari

Khoi Nguyen 8 Dec 11, 2022
An official implementation of "Exploiting a Joint Embedding Space for Generalized Zero-Shot Semantic Segmentation" (ICCV 2021) in PyTorch.

Exploiting a Joint Embedding Space for Generalized Zero-Shot Semantic Segmentation This is an official implementation of the paper "Exploiting a Joint

CV Lab @ Yonsei University 35 Oct 26, 2022
Zsseg.baseline - Zero-Shot Semantic Segmentation

This repo is for our paper A Simple Baseline for Zero-shot Semantic Segmentation

null 98 Dec 20, 2022
Leveraging Instance-, Image- and Dataset-Level Information for Weakly Supervised Instance Segmentation

Leveraging Instance-, Image- and Dataset-Level Information for Weakly Supervised Instance Segmentation This paper has been accepted and early accessed

Yun Liu 39 Sep 20, 2022
The code for our paper "NSP-BERT: A Prompt-based Zero-Shot Learner Through an Original Pre-training Task —— Next Sentence Prediction"

The code for our paper "NSP-BERT: A Prompt-based Zero-Shot Learner Through an Original Pre-training Task —— Next Sentence Prediction"

Sun Yi 201 Nov 21, 2022
Code repo for EMNLP21 paper "Zero-Shot Information Extraction as a Unified Text-to-Triple Translation"

Zero-Shot Information Extraction as a Unified Text-to-Triple Translation Source code repo for paper Zero-Shot Information Extraction as a Unified Text

cgraywang 88 Dec 31, 2022
Code for the AAAI 2022 paper "Zero-Shot Cross-Lingual Machine Reading Comprehension via Inter-Sentence Dependency Graph".

multilingual-mrc-isdg Code for the AAAI 2022 paper "Zero-Shot Cross-Lingual Machine Reading Comprehension via Inter-Sentence Dependency Graph". This r

Liyan 5 Dec 7, 2022
Zero-shot Synthesis with Group-Supervised Learning (ICLR 2021 paper)

GSL - Zero-shot Synthesis with Group-Supervised Learning Figure: Zero-shot synthesis performance of our method with different dataset (iLab-20M, RaFD,

Andy_Ge 62 Dec 21, 2022
Codes for ACL-IJCNLP 2021 Paper "Zero-shot Fact Verification by Claim Generation"

Zero-shot-Fact-Verification-by-Claim-Generation This repository contains code and models for the paper: Zero-shot Fact Verification by Claim Generatio

Liangming Pan 47 Jan 1, 2023
Official Pytorch Implementation of: "Semantic Diversity Learning for Zero-Shot Multi-label Classification"(2021) paper

Semantic Diversity Learning for Zero-Shot Multi-label Classification Paper Official PyTorch Implementation Avi Ben-Cohen, Nadav Zamir, Emanuel Ben Bar

null 28 Aug 29, 2022
TorchDistiller - a collection of the open source pytorch code for knowledge distillation, especially for the perception tasks, including semantic segmentation, depth estimation, object detection and instance segmentation.

This project is a collection of the open source pytorch code for knowledge distillation, especially for the perception tasks, including semantic segmentation, depth estimation, object detection and instance segmentation.

yifan liu 147 Dec 3, 2022
Code for 'Self-Guided and Cross-Guided Learning for Few-shot segmentation. (CVPR' 2021)'

SCL Introduction Code for 'Self-Guided and Cross-Guided Learning for Few-shot segmentation. (CVPR' 2021)' We evaluated our approach using two baseline

null 34 Oct 8, 2022
Original code for "Zero-Shot Domain Adaptation with a Physics Prior"

Zero-Shot Domain Adaptation with a Physics Prior [arXiv] [sup. material] - ICCV 2021 Oral paper, by Attila Lengyel, Sourav Garg, Michael Milford and J

Attila Lengyel 40 Dec 21, 2022
The code of Zero-shot learning for low-light image enhancement based on dual iteration

Zero-shot-dual-iter-LLE The code of Zero-shot learning for low-light image enhancement based on dual iteration. You can get the real night image tests

null 1 Mar 18, 2022
The source code for Generating Training Data with Language Models: Towards Zero-Shot Language Understanding.

SuperGen The source code for Generating Training Data with Language Models: Towards Zero-Shot Language Understanding. Requirements Before running, you

Yu Meng 38 Dec 12, 2022
Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers [CVPR 2021]

Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers [BCNet, CVPR 2021] This is the official pytorch implementation of BCNet built on

Lei Ke 434 Dec 1, 2022
Robust Instance Segmentation through Reasoning about Multi-Object Occlusion [CVPR 2021]

Robust Instance Segmentation through Reasoning about Multi-Object Occlusion [CVPR 2021] Abstract Analyzing complex scenes with DNN is a challenging ta

Irene Yuan 24 Jun 27, 2022
FreeSOLO for unsupervised instance segmentation, CVPR 2022

FreeSOLO: Learning to Segment Objects without Annotations This project hosts the code for implementing the FreeSOLO algorithm for unsupervised instanc

NVIDIA Research Projects 253 Jan 2, 2023