FreeSOLO for unsupervised instance segmentation, CVPR 2022

Overview

FreeSOLO: Learning to Segment Objects without Annotations

This project hosts the code for implementing the FreeSOLO algorithm for unsupervised instance segmentation.

FreeSOLO: Learning to Segment Objects without Annotations,
Xinlong Wang, Zhiding Yu, Shalini De Mello, Jan Kautz, Anima Anandkumar, Chunhua Shen, Jose M. Alvarez
In: Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2022
arXiv preprint (arXiv 2202.12181)

pipeline

Visual Results

vis

Installation

Prerequisites

  • Linux or macOS with Python >= 3.6
  • PyTorch >= 1.5 and torchvision that matches the PyTorch installation.
  • scikit-image

Install PyTorch in Conda env

# create conda env
conda create -n detectron2 python=3.6
# activate the enviorment
conda activate detectron2
# install PyTorch >=1.5 with GPU
conda install pytorch torchvision -c pytorch

Build Detectron2 from Source

Follow the INSTALL.md to install Detectron2 (commit id 11528ce has been tested).

Datasets

Follow the datasets/README.md to set up the MS COCO dataset.

Pre-trained model

Download the DenseCL pre-trained model from here. Convert it to detectron2's format and put the converted model under "training_dir/pre-trained/DenseCL" directory.

python tools/convert-pretrain-to-detectron2.py {WEIGHT_FILE}.pth {WEIGHT_FILE}.pkl

Usage

Free Mask

Download the prepared free masks in json format from here. Put it under "datasets/coco/annotations" directory. Or, generate it by yourself:

bash inference_freemask.sh

Training

# train with free masks
bash train.sh

# generate pseudo labels
bash gen_pseudo_labels.sh

# self-train
bash train_pl.sh

Testing

Download the trained model from here.

bash test.sh {MODEL_PATH}

Citations

Please consider citing our paper in your publications if the project helps your research. BibTeX reference is as follow.

@article{wang2022freesolo,
  title={{FreeSOLO}: Learning to Segment Objects without Annotations},
  author={Wang, Xinlong and Yu, Zhiding and De Mello, Shalini and Kautz, Jan and Anandkumar, Anima and Shen, Chunhua and Alvarez, Jose M},
  journal={arXiv preprint arXiv:2202.12181},
  year={2022}
}
Comments
  • Evaluation with FreeSOLO_R101_30k_pl.pth

    Evaluation with FreeSOLO_R101_30k_pl.pth

    Hello, thank you very much for releasing the source code. When I run bash test.sh FreeSOLO_R101_30k_pl.pth, I get the following results. The AP on person is 0.903, but all other categories are indeed 0. Am I missing any key experimental settings?

    [04/22 13:53:59] d2.evaluation.evaluator INFO: Inference done 4991/5000. Dataloading: 0.0046 s/iter. Inference: 0.0928 s/iter. Eval: 0.0549 s/iter. Total: 0.1524 s/iter. ETA=0:00:01
    [04/22 13:54:00] d2.evaluation.evaluator INFO: Total inference time: 0:12:40.989735 (0.152350 s / iter per device, on 1 devices)
    [04/22 13:54:00] d2.evaluation.evaluator INFO: Total inference pure compute time: 0:07:43 (0.092807 s / iter per device, on 1 devices)
    [04/22 13:54:05] d2.evaluation.coco_evaluation INFO: Preparing results for COCO format ...
    [04/22 13:54:05] d2.evaluation.coco_evaluation INFO: Saving results to training_dir/FreeSOLO_pl/inference/coco_instances_results.json
    [04/22 13:54:09] d2.evaluation.coco_evaluation INFO: Evaluating predictions with official COCO API...
    [04/22 13:54:55] d2.evaluation.coco_evaluation INFO: Evaluation results for bbox: 
    |  AP   |  AP50  |  AP75  |  APs  |  APm  |  APl  |
    |:-----:|:------:|:------:|:-----:|:-----:|:-----:|
    | 0.011 | 0.028  | 0.009  | 0.004 | 0.010 | 0.031 |
    [04/22 13:54:55] d2.evaluation.coco_evaluation INFO: Per-category bbox AP: 
    | category      | AP    | category     | AP    | category       | AP    |
    |:--------------|:------|:-------------|:------|:---------------|:------|
    | person        | 0.903 | bicycle      | 0.000 | car            | 0.000 |
    | motorcycle    | 0.000 | airplane     | 0.000 | bus            | 0.000 |
    | train         | 0.000 | truck        | 0.000 | boat           | 0.000 |
    | traffic light | 0.000 | fire hydrant | 0.000 | stop sign      | 0.000 |
    | parking meter | 0.000 | bench        | 0.000 | bird           | 0.000 |
    | cat           | 0.000 | dog          | 0.000 | horse          | 0.000 |
    | sheep         | 0.000 | cow          | 0.000 | elephant       | 0.000 |
    | bear          | 0.000 | zebra        | 0.000 | giraffe        | 0.000 |
    | backpack      | 0.000 | umbrella     | 0.000 | handbag        | 0.000 |
    | tie           | 0.000 | suitcase     | 0.000 | frisbee        | 0.000 |
    | skis          | 0.000 | snowboard    | 0.000 | sports ball    | 0.000 |
    | kite          | 0.000 | baseball bat | 0.000 | baseball glove | 0.000 |
    | skateboard    | 0.000 | surfboard    | 0.000 | tennis racket  | 0.000 |
    | bottle        | 0.000 | wine glass   | 0.000 | cup            | 0.000 |
    | fork          | 0.000 | knife        | 0.000 | spoon          | 0.000 |
    | bowl          | 0.000 | banana       | 0.000 | apple          | 0.000 |
    | sandwich      | 0.000 | orange       | 0.000 | broccoli       | 0.000 |
    | carrot        | 0.000 | hot dog      | 0.000 | pizza          | 0.000 |
    | donut         | 0.000 | cake         | 0.000 | chair          | 0.000 |
    | couch         | 0.000 | potted plant | 0.000 | bed            | 0.000 |
    | dining table  | 0.000 | toilet       | 0.000 | tv             | 0.000 |
    | laptop        | 0.000 | mouse        | 0.000 | remote         | 0.000 |
    | keyboard      | 0.000 | cell phone   | 0.000 | microwave      | 0.000 |
    | oven          | 0.000 | toaster      | 0.000 | sink           | 0.000 |
    | refrigerator  | 0.000 | book         | 0.000 | clock          | 0.000 |
    | vase          | 0.000 | scissors     | 0.000 | teddy bear     | 0.000 |
    | hair drier    | 0.000 | toothbrush   | 0.000 |                |       |
    [04/22 13:56:08] d2.evaluation.coco_evaluation INFO: Evaluation results for segm: 
    |  AP   |  AP50  |  AP75  |  APs  |  APm  |  APl  |
    |:-----:|:------:|:------:|:-----:|:-----:|:-----:|
    | 0.012 | 0.030  | 0.008  | 0.001 | 0.005 | 0.036 |
    [04/22 13:56:08] d2.evaluation.coco_evaluation INFO: Per-category segm AP: 
    | category      | AP    | category     | AP    | category       | AP    |
    |:--------------|:------|:-------------|:------|:---------------|:------|
    | person        | 0.926 | bicycle      | 0.000 | car            | 0.000 |
    | motorcycle    | 0.000 | airplane     | 0.000 | bus            | 0.000 |
    | train         | 0.000 | truck        | 0.000 | boat           | 0.000 |
    | traffic light | 0.000 | fire hydrant | 0.000 | stop sign      | 0.000 |
    | parking meter | 0.000 | bench        | 0.000 | bird           | 0.000 |
    | cat           | 0.000 | dog          | 0.000 | horse          | 0.000 |
    | sheep         | 0.000 | cow          | 0.000 | elephant       | 0.000 |
    | bear          | 0.000 | zebra        | 0.000 | giraffe        | 0.000 |
    | backpack      | 0.000 | umbrella     | 0.000 | handbag        | 0.000 |
    | tie           | 0.000 | suitcase     | 0.000 | frisbee        | 0.000 |
    | skis          | 0.000 | snowboard    | 0.000 | sports ball    | 0.000 |
    | kite          | 0.000 | baseball bat | 0.000 | baseball glove | 0.000 |
    | skateboard    | 0.000 | surfboard    | 0.000 | tennis racket  | 0.000 |
    | bottle        | 0.000 | wine glass   | 0.000 | cup            | 0.000 |
    | fork          | 0.000 | knife        | 0.000 | spoon          | 0.000 |
    | bowl          | 0.000 | banana       | 0.000 | apple          | 0.000 |
    | sandwich      | 0.000 | orange       | 0.000 | broccoli       | 0.000 |
    | carrot        | 0.000 | hot dog      | 0.000 | pizza          | 0.000 |
    | donut         | 0.000 | cake         | 0.000 | chair          | 0.000 |
    | couch         | 0.000 | potted plant | 0.000 | bed            | 0.000 |
    | dining table  | 0.000 | toilet       | 0.000 | tv             | 0.000 |
    | laptop        | 0.000 | mouse        | 0.000 | remote         | 0.000 |
    | keyboard      | 0.000 | cell phone   | 0.000 | microwave      | 0.000 |
    | oven          | 0.000 | toaster      | 0.000 | sink           | 0.000 |
    | refrigerator  | 0.000 | book         | 0.000 | clock          | 0.000 |
    | vase          | 0.000 | scissors     | 0.000 | teddy bear     | 0.000 |
    | hair drier    | 0.000 | toothbrush   | 0.000 |                |       |
    [04/22 13:56:09] d2.engine.defaults INFO: Evaluation results for coco_2017_val in csv format:
    [04/22 13:56:09] d2.evaluation.testing INFO: copypaste: Task: bbox
    [04/22 13:56:09] d2.evaluation.testing INFO: copypaste: AP,AP50,AP75,APs,APm,APl
    [04/22 13:56:09] d2.evaluation.testing INFO: copypaste: 0.0113,0.0283,0.0085,0.0045,0.0101,0.0311
    [04/22 13:56:09] d2.evaluation.testing INFO: copypaste: Task: segm
    [04/22 13:56:09] d2.evaluation.testing INFO: copypaste: AP,AP50,AP75,APs,APm,APl
    [04/22 13:56:09] d2.evaluation.testing INFO: copypaste: 0.0116,0.0296,0.0079,0.0005,0.0055,0.0362
    
    opened by scarleatt 2
  • how to generate the free mask with my own  custom data?

    how to generate the free mask with my own custom data?

    i have a instance segmentation dataset ,but its label mask is too accurate compare free mask,i just want to konw how you generate free mask?and i i use my pretrained model(solo model) to generate the free mask ,is that a bad idea?please let me konw if that's not so unconvient ,thaks a lot......

    opened by Vincent630 1
  • Unable to reproduce results

    Unable to reproduce results

    Hi,

    Thank you for your work. I encountered several issues when using the code.

    • I tried to reproduce the results of Freemask by running the first step of the algorithm on the COCO dataset. However, I'm unable to get the same results as in the provided json: for example, the embedding vector is not included in the annotations of the provided json. Am I missing something here?

    • I ran the code with the provided json on the train2017+unlabeled2017 split, but only get 0.1% mask AP after the first step of FreeSOLO. In particular, ran the tools/eval_cocoapi.py script for the class-agnostic evaluation. I noticed that only the pairwise loss is able to go down.

    • Evaluating the provided final model (in this repo) does not produce 12.2% AP50 for detection (as claimed in the paper) but only obtains 9.6%. Do I need to post-process the results before evaluating?

    Were people able to reproduce this? Thanks.

    opened by MarcVisions 0
  • no

    no "training_dir/instances_train2017_densecl_r101.json" file

    Hi, Thanx for your great work,. when run "$ bash inference_freemask.sh", error existed as :no such file. Pls tell me where can i get this json file?

    opened by cynthia-you 0
  • How can I visualize free mask results?

    How can I visualize free mask results?

    To ensure that the free masks are almost accurate, I add following codes in inference_freemaks.py . However, the results saved in the disks is strange: the foreground part is always at the edge of an image.

    after masks = masks.cpu().numpy() ''' masks = masks.cpu().numpy() ii=0 for mask in masks: masks_save = masks[ii] * 255 cv2.imwrite('./results/'+img_name+'_'+str(ii)+'.jpg', masks_save) ii=ii+1 '''

    opened by STAR-811 2
  • The training error

    The training error

    Hello, I meet an error when training freesolo: RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation...

    opened by wavinflaghxm 3
Owner
NVIDIA Research Projects
NVIDIA Research Projects
pytorch implementation of "Contrastive Multiview Coding", "Momentum Contrast for Unsupervised Visual Representation Learning", and "Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination"

Unofficial implementation: MoCo: Momentum Contrast for Unsupervised Visual Representation Learning (Paper) InsDis: Unsupervised Feature Learning via N

Zhiqiang Shen 16 Nov 4, 2020
[CVPR 2022] CoTTA Code for our CVPR 2022 paper Continual Test-Time Domain Adaptation

CoTTA Code for our CVPR 2022 paper Continual Test-Time Domain Adaptation Prerequisite Please create and activate the following conda envrionment. To r

Qin Wang 73 Sep 22, 2022
Leveraging Instance-, Image- and Dataset-Level Information for Weakly Supervised Instance Segmentation

Leveraging Instance-, Image- and Dataset-Level Information for Weakly Supervised Instance Segmentation This paper has been accepted and early accessed

Yun Liu 39 Sep 20, 2022
Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers [CVPR 2021]

Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers [BCNet, CVPR 2021] This is the official pytorch implementation of BCNet built on

Lei Ke 419 Sep 30, 2022
code for CVPR paper Zero-shot Instance Segmentation

Code for CVPR2021 paper Zero-shot Instance Segmentation Code requirements python: python3.7 nvidia GPU pytorch1.1.0 GCC >=5.4 NCCL 2 the other python

zhengye 84 Aug 30, 2022
Robust Instance Segmentation through Reasoning about Multi-Object Occlusion [CVPR 2021]

Robust Instance Segmentation through Reasoning about Multi-Object Occlusion [CVPR 2021] Abstract Analyzing complex scenes with DNN is a challenging ta

Irene Yuan 24 Jun 27, 2022
[ArXiv 2021] Data-Efficient Instance Generation from Instance Discrimination

InsGen - Data-Efficient Instance Generation from Instance Discrimination Data-Efficient Instance Generation from Instance Discrimination Ceyuan Yang,

GenForce: May Generative Force Be with You 91 Oct 1, 2022
TorchDistiller - a collection of the open source pytorch code for knowledge distillation, especially for the perception tasks, including semantic segmentation, depth estimation, object detection and instance segmentation.

This project is a collection of the open source pytorch code for knowledge distillation, especially for the perception tasks, including semantic segmentation, depth estimation, object detection and instance segmentation.

yifan liu 136 Sep 14, 2022
Imposter-detector-2022 - HackED 2022 Team 3IQ - 2022 Imposter Detector

HackED 2022 Team 3IQ - 2022 Imposter Detector By Aneeljyot Alagh, Curtis Kan, Jo

Joshua Ji 3 Aug 20, 2022
The 7th edition of NTIRE: New Trends in Image Restoration and Enhancement workshop will be held on June 2022 in conjunction with CVPR 2022.

NTIRE 2022 - Image Inpainting Challenge Important dates 2022.02.01: Release of train data (input and output images) and validation data (only input) 2

Andrés Romero 30 Sep 17, 2022
This is the official pytorch implementation for the paper: Instance Similarity Learning for Unsupervised Feature Representation.

ISL This is the official pytorch implementation for the paper: Instance Similarity Learning for Unsupervised Feature Representation, which is accepted

null 19 May 4, 2022
IAST: Instance Adaptive Self-training for Unsupervised Domain Adaptation (ECCV 2020)

This repo is the official implementation of our paper "Instance Adaptive Self-training for Unsupervised Domain Adaptation". The purpose of this repo is to better communicate with you and respond to your questions. This repo is almost the same with Another-Version, and you can also refer to that version.

CVSM Group -  email: czhu@bupt.edu.cn 85 Jul 30, 2022
Segmentation in Style: Unsupervised Semantic Image Segmentation with Stylegan and CLIP

Segmentation in Style: Unsupervised Semantic Image Segmentation with Stylegan and CLIP Abstract: We introduce a method that allows to automatically se

Daniil Pakhomov 125 Aug 31, 2022
[CVPR 2022] Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels

Using Unreliable Pseudo Labels Official PyTorch implementation of Semi-Supervised Semantic Segmentation Using Unreliable Pseudo Labels, CVPR 2022. Ple

Haochen Wang 206 Sep 24, 2022
Hyperbolic Image Segmentation, CVPR 2022

Hyperbolic Image Segmentation, CVPR 2022 This is the implementation of paper Hyperbolic Image Segmentation (CVPR 2022). Repository structure assets :

Mina Ghadimi Atigh 41 Sep 16, 2022
Contrastive learning of Class-agnostic Activation Map for Weakly Supervised Object Localization and Semantic Segmentation (CVPR 2022)

CCAM (Unsupervised) Code repository for our paper "CCAM: Contrastive learning of Class-agnostic Activation Map for Weakly Supervised Object Localizati

Computer Vision Insitute, SZU 81 Sep 21, 2022
Scribble-Supervised LiDAR Semantic Segmentation, CVPR 2022 (ORAL)

Scribble-Supervised LiDAR Semantic Segmentation Dataset and code release for the paper Scribble-Supervised LiDAR Semantic Segmentation, CVPR 2022 (ORA

null 80 Sep 29, 2022
Stratified Transformer for 3D Point Cloud Segmentation (CVPR 2022)

Stratified Transformer for 3D Point Cloud Segmentation Xin Lai*, Jianhui Liu*, Li Jiang, Liwei Wang, Hengshuang Zhao, Shu Liu, Xiaojuan Qi, Jiaya Jia

DV Lab 154 Sep 22, 2022
Code for Multiple Instance Active Learning for Object Detection, CVPR 2021

Language: 简体中文 | English Introduction This is the code for Multiple Instance Active Learning for Object Detection, CVPR 2021. Installation A Linux pla

Tianning Yuan 256 Sep 21, 2022