[CVPR 2021] Exemplar-Based Open-Set Panoptic Segmentation Network (EOPSN)

Jaedong Hwang

Last update: Dec 30, 2022

Related tags

Deep Learning EOPSN

Overview

EOPSN: Exemplar-Based Open-Set Panoptic Segmentation Network (CVPR 2021)

PyTorch implementation for EOPSN.

We propose open-set panoptic segmentation task and propose a new baseline called EOPSN. The code is based on Detectron2

Usage

First, install requirements.

pip install -r requirements.txt

Then, install PyTorch 1.5+ and torchvision 0.6+:

conda install -c pytorch pytorch torchvision

Finally, you need to install Detectron2. To prevent version conflict, I recommand to install via included detectron2 folders. Regarding installation issue caused from detectron2, please refer to here.

cd detectron2
pip install -e ./

Data preparation

Download and extract COCO 2017 train and val images with annotations from http://cocodataset.org. We expect the directory structure to be the following:

datasets/coco
  annotations/  # annotation json files
  train2017/    # train images
  val2017/      # val images

To convert closed-set panoptic segmentation to open-set panoptic segmentation, run:

python prepare_unknown.py

The default setting is K=20, you can change here.

Training

To train EOPSN on a single node with 8 gpus for 30,000 iterations run:

python train_net.py --config configs/EOPSN_K20.yaml --num-gpus 8

Note that it requires pre-trained models (Void-suppression). Please download from Goolge Drive.

To train baseline (train) on a single node with 8 gpus for 45,000 iterations run:

python train_net.py --config configs/baseline_K20.yaml --num-gpus 8

If you want to log using WandB, you can add --wandb flag.

Evaluation

To evaluate EOPSN on COCO val5k with a single GPU run:

python train_net.py --config configs/EOPSN_K20.yaml --num-gpus 8 --resume --eval-only

Citations

@inproceedings{hwang2021exemplar,
    author = {Hwang, Jaedong and Oh, Seoung Wug and Lee, Joon-Young and Han, Bohyung},
    title = {Exemplar-Based Open-Set Panoptic Segmentation Network},
    booktitle = {CVPR},
    year = {2021},
}

License

EOPSN is released under the CC BY-NC-SA 4.0 license. Please see the LICENSE file for more information. The detectron2 part is released under the Apache 2.0 license. Please see the detectron2/LICENSE file for more information.

Contributing

We actively welcome your pull requests!

An unofficial personal implementation of UM-Adapt, specifically to tackle joint estimation of panoptic segmentation and depth prediction for autonomous driving datasets.

Semisupervised Multitask Learning This repository is an unofficial and slightly modified implementation of UM-Adapt[1] using PyTorch. This code primar

11 Nov 25, 2022

PanopticBEV - Bird's-Eye-View Panoptic Segmentation Using Monocular Frontal View Images

Bird's-Eye-View Panoptic Segmentation Using Monocular Frontal View Images This r

63 Dec 16, 2022

[CVPR 2021] Released code for Counterfactual Zero-Shot and Open-Set Visual Recognition

Counterfactual Zero-Shot and Open-Set Visual Recognition This project provides implementations for our CVPR 2021 paper Counterfactual Zero-S

144 Dec 24, 2022

A PyTorch implementation of the baseline method in Panoptic Narrative Grounding (ICCV 2021 Oral)

52 Dec 19, 2022

Pytorch Implementation of "Contrastive Representation Learning for Exemplar-Guided Paraphrase Generation"

CRL_EGPG Pytorch Implementation of Contrastive Representation Learning for Exemplar-Guided Paraphrase Generation We use contrastive loss implemented b

25 Nov 14, 2022

Open-Set Recognition: A Good Closed-Set Classifier is All You Need

Open-Set Recognition: A Good Closed-Set Classifier is All You Need Code for our paper: "Open-Set Recognition: A Good Closed-Set Classifier is All You

194 Jan 3, 2023

Official code of the paper "Expanding Low-Density Latent Regions for Open-Set Object Detection" (CVPR 2022)

OpenDet Expanding Low-Density Latent Regions for Open-Set Object Detection (CVPR2022) Jiaming Han, Yuqiang Ren, Jian Ding, Xingjia Pan, Ke Yan, Gui-So

64 Jan 7, 2023

[ICRA2021] Reconstructing Interactive 3D Scene by Panoptic Mapping and CAD Model Alignment

Interactive Scene Reconstruction Project Page | Paper This repository contains the implementation of our ICRA2021 paper Reconstructing Interactive 3D

97 Dec 28, 2022

Implementation of "Efficient Regional Memory Network for Video Object Segmentation" (Xie et al., CVPR 2021).

RMNet This repository contains the source code for the paper Efficient Regional Memory Network for Video Object Segmentation. Cite this work @inprocee

76 Dec 14, 2022

Comments

Adoption to cityscapes

Hi, thanks for providing the code to your method. I am interested in trying this out on another dataset, namely cityscapes. I already know that I would have to adapt the dataloading to cityscapes, what I am still unsure about is which pretrained model I would need.

In this issue you mention that training needs a model pretrained with void-supression, the weights of which you provide. How would I go about pretraining such a model on cityscapes? Is there code in this repository for pretraining on COCO, which I could adapt to cityscapes?

opened by Hoff97 4
About the pre-trained model.

Hi @jd730, thanks for sharing your wonderful work. In the README, you said the training requires "void-suppression" pre-trained model. However, I find that in the config file, you still use the standard pre-trained model here. If I want to train your EOPSN, which pre-trained model should I use? Thx.

opened by GengDavid 4
Infinity losses during training

When running the training, I noticed that sometimes some losses (especially the box regression loss) can become Infinity. This seems to happen especially once many exemplars have been mined. I was wondering if you encountered this issue as well, and if you know of any fixes.

opened by Hoff97 1
Problems of reproducing the results of EOPSN

Hi,

Thanks for the interesting paper and open-sourced code.

Recently, I ran the EOPSN method on K20 setting folllowing the given guideline (w/o any editing) and I found the results of unknown things are quite different from the reported one.

Unk | PQ | SQ | RQ -- | -- | -- | -- EOPSN reported | 11.3 | 73.8 | 15.3 EOPSN reproduced | 15.6 | 79.2 | 19.6

From the table, it seems that the released code achieves a much better improvement than the reported one. However, when I further inspect the predictions of class-wise unkown things, it seems that EOPSN's unkown recognition is dominated by the "car" class and other unkown classes are rarely detected. Moreover, the reproduced results may not support the visualization results in Fig5 since several unkown classes are shown to be detected, e.g., stop sign, keyboard, banana, and toilet. So, could you please release the EOPSN checkpoint which supports the reported results? Thanks a lot.

BTW, I found that the training of EOPSN requires the pre-trained model of Void-Suppression, but the current released codebase only contains the void-train method. I wonder could you please release the void-suppression code for better reproduction? Thanks again.

FYI, the predictions of class-wise unkown things on Void-Suppression method are as follows and the results are identical to the reported ones in the paper

opened by HeimingX 7

Owner

Jaedong Hwang

graduate student @ Seoul National University, Korea

GitHub

[CVPR 2021] Exemplar-Based Open-Set Panoptic Segmentation Network (EOPSN)

Related tags

Overview

EOPSN: Exemplar-Based Open-Set Panoptic Segmentation Network (CVPR 2021)

Usage

Data preparation

Training

Evaluation

Citations

License

Contributing

You might also like...

An unofficial personal implementation of UM-Adapt, specifically to tackle joint estimation of panoptic segmentation and depth prediction for autonomous driving datasets.

PanopticBEV - Bird's-Eye-View Panoptic Segmentation Using Monocular Frontal View Images

[CVPR 2021] Released code for Counterfactual Zero-Shot and Open-Set Visual Recognition

A PyTorch implementation of the baseline method in Panoptic Narrative Grounding (ICCV 2021 Oral)

Pytorch Implementation of "Contrastive Representation Learning for Exemplar-Guided Paraphrase Generation"

Open-Set Recognition: A Good Closed-Set Classifier is All You Need

Official code of the paper "Expanding Low-Density Latent Regions for Open-Set Object Detection" (CVPR 2022)

[ICRA2021] Reconstructing Interactive 3D Scene by Panoptic Mapping and CAD Model Alignment

Implementation of "Efficient Regional Memory Network for Video Object Segmentation" (Xie et al., CVPR 2021).

Comments

Adoption to cityscapes

About the pre-trained model.

Infinity losses during training

Problems of reproducing the results of EOPSN

Owner

Jaedong Hwang

[arXiv'22] Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentation

Unbalanced Feature Transport for Exemplar-based Image Translation (CVPR 2021)

[CVPR 2021] Forecasting the panoptic segmentation of future video frames

Voxel Set Transformer: A Set-to-Set Approach to 3D Object Detection from Point Clouds (CVPR 2022)

The source code of CVPR 2019 paper "Deep Exemplar-based Video Colorization".

Pixel Consensus Voting for Panoptic Segmentation (CVPR 2020)

Implementation for Panoptic-PolarNet (CVPR 2021)

[ACM MM 2021] Yes, "Attention is All You Need", for Exemplar based Colorization

Automatically download the cwru data set, and then divide it into training data set and test data set

Official code for paper Exemplar Based 3D Portrait Stylization.