FreeSOLO for unsupervised instance segmentation, CVPR 2022

NVIDIA Research Projects

Last update: Jan 2, 2023

Related tags

Overview

FreeSOLO: Learning to Segment Objects without Annotations

This project hosts the code for implementing the FreeSOLO algorithm for unsupervised instance segmentation.

FreeSOLO: Learning to Segment Objects without Annotations,
Xinlong Wang, Zhiding Yu, Shalini De Mello, Jan Kautz, Anima Anandkumar, Chunhua Shen, Jose M. Alvarez
In: Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2022
arXiv preprint (arXiv 2202.12181)

Visual Results

Installation

Prerequisites

Linux or macOS with Python >= 3.6
PyTorch >= 1.5 and torchvision that matches the PyTorch installation.
scikit-image

Install PyTorch in Conda env

# create conda env
conda create -n detectron2 python=3.6
# activate the enviorment
conda activate detectron2
# install PyTorch >=1.5 with GPU
conda install pytorch torchvision -c pytorch

Build Detectron2 from Source

Follow the INSTALL.md to install Detectron2 (commit id 11528ce has been tested).

Datasets

Follow the datasets/README.md to set up the MS COCO dataset.

Pre-trained model

Download the DenseCL pre-trained model from here. Convert it to detectron2's format and put the converted model under "training_dir/pre-trained/DenseCL" directory.

python tools/convert-pretrain-to-detectron2.py {WEIGHT_FILE}.pth {WEIGHT_FILE}.pkl

Usage

Free Mask

Download the prepared free masks in json format from here. Put it under "datasets/coco/annotations" directory. Or, generate it by yourself:

bash inference_freemask.sh

Training

# train with free masks
bash train.sh

# generate pseudo labels
bash gen_pseudo_labels.sh

# self-train
bash train_pl.sh

Testing

Download the trained model from here.

bash test.sh {MODEL_PATH}

Citations

Please consider citing our paper in your publications if the project helps your research. BibTeX reference is as follow.

@article{wang2022freesolo,
  title={{FreeSOLO}: Learning to Segment Objects without Annotations},
  author={Wang, Xinlong and Yu, Zhiding and De Mello, Shalini and Kautz, Jan and Anandkumar, Anima and Shen, Chunhua and Alvarez, Jose M},
  journal={arXiv preprint arXiv:2202.12181},
  year={2022}
}

Comments

The training error

Hello, I meet an error when training freesolo: RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation...

opened by wavinflaghxm 4

Evaluation with FreeSOLO_R101_30k_pl.pth

Hello, thank you very much for releasing the source code. When I run bash test.sh FreeSOLO_R101_30k_pl.pth, I get the following results. The AP on person is 0.903, but all other categories are indeed 0. Am I missing any key experimental settings?

[04/22 13:53:59] d2.evaluation.evaluator INFO: Inference done 4991/5000. Dataloading: 0.0046 s/iter. Inference: 0.0928 s/iter. Eval: 0.0549 s/iter. Total: 0.1524 s/iter. ETA=0:00:01
[04/22 13:54:00] d2.evaluation.evaluator INFO: Total inference time: 0:12:40.989735 (0.152350 s / iter per device, on 1 devices)
[04/22 13:54:00] d2.evaluation.evaluator INFO: Total inference pure compute time: 0:07:43 (0.092807 s / iter per device, on 1 devices)
[04/22 13:54:05] d2.evaluation.coco_evaluation INFO: Preparing results for COCO format ...
[04/22 13:54:05] d2.evaluation.coco_evaluation INFO: Saving results to training_dir/FreeSOLO_pl/inference/coco_instances_results.json
[04/22 13:54:09] d2.evaluation.coco_evaluation INFO: Evaluating predictions with official COCO API...
[04/22 13:54:55] d2.evaluation.coco_evaluation INFO: Evaluation results for bbox: 
|  AP   |  AP50  |  AP75  |  APs  |  APm  |  APl  |
|:-----:|:------:|:------:|:-----:|:-----:|:-----:|
| 0.011 | 0.028  | 0.009  | 0.004 | 0.010 | 0.031 |
[04/22 13:54:55] d2.evaluation.coco_evaluation INFO: Per-category bbox AP: 
| category      | AP    | category     | AP    | category       | AP    |
|:--------------|:------|:-------------|:------|:---------------|:------|
| person        | 0.903 | bicycle      | 0.000 | car            | 0.000 |
| motorcycle    | 0.000 | airplane     | 0.000 | bus            | 0.000 |
| train         | 0.000 | truck        | 0.000 | boat           | 0.000 |
| traffic light | 0.000 | fire hydrant | 0.000 | stop sign      | 0.000 |
| parking meter | 0.000 | bench        | 0.000 | bird           | 0.000 |
| cat           | 0.000 | dog          | 0.000 | horse          | 0.000 |
| sheep         | 0.000 | cow          | 0.000 | elephant       | 0.000 |
| bear          | 0.000 | zebra        | 0.000 | giraffe        | 0.000 |
| backpack      | 0.000 | umbrella     | 0.000 | handbag        | 0.000 |
| tie           | 0.000 | suitcase     | 0.000 | frisbee        | 0.000 |
| skis          | 0.000 | snowboard    | 0.000 | sports ball    | 0.000 |
| kite          | 0.000 | baseball bat | 0.000 | baseball glove | 0.000 |
| skateboard    | 0.000 | surfboard    | 0.000 | tennis racket  | 0.000 |
| bottle        | 0.000 | wine glass   | 0.000 | cup            | 0.000 |
| fork          | 0.000 | knife        | 0.000 | spoon          | 0.000 |
| bowl          | 0.000 | banana       | 0.000 | apple          | 0.000 |
| sandwich      | 0.000 | orange       | 0.000 | broccoli       | 0.000 |
| carrot        | 0.000 | hot dog      | 0.000 | pizza          | 0.000 |
| donut         | 0.000 | cake         | 0.000 | chair          | 0.000 |
| couch         | 0.000 | potted plant | 0.000 | bed            | 0.000 |
| dining table  | 0.000 | toilet       | 0.000 | tv             | 0.000 |
| laptop        | 0.000 | mouse        | 0.000 | remote         | 0.000 |
| keyboard      | 0.000 | cell phone   | 0.000 | microwave      | 0.000 |
| oven          | 0.000 | toaster      | 0.000 | sink           | 0.000 |
| refrigerator  | 0.000 | book         | 0.000 | clock          | 0.000 |
| vase          | 0.000 | scissors     | 0.000 | teddy bear     | 0.000 |
| hair drier    | 0.000 | toothbrush   | 0.000 |                |       |
[04/22 13:56:08] d2.evaluation.coco_evaluation INFO: Evaluation results for segm: 
|  AP   |  AP50  |  AP75  |  APs  |  APm  |  APl  |
|:-----:|:------:|:------:|:-----:|:-----:|:-----:|
| 0.012 | 0.030  | 0.008  | 0.001 | 0.005 | 0.036 |
[04/22 13:56:08] d2.evaluation.coco_evaluation INFO: Per-category segm AP: 
| category      | AP    | category     | AP    | category       | AP    |
|:--------------|:------|:-------------|:------|:---------------|:------|
| person        | 0.926 | bicycle      | 0.000 | car            | 0.000 |
| motorcycle    | 0.000 | airplane     | 0.000 | bus            | 0.000 |
| train         | 0.000 | truck        | 0.000 | boat           | 0.000 |
| traffic light | 0.000 | fire hydrant | 0.000 | stop sign      | 0.000 |
| parking meter | 0.000 | bench        | 0.000 | bird           | 0.000 |
| cat           | 0.000 | dog          | 0.000 | horse          | 0.000 |
| sheep         | 0.000 | cow          | 0.000 | elephant       | 0.000 |
| bear          | 0.000 | zebra        | 0.000 | giraffe        | 0.000 |
| backpack      | 0.000 | umbrella     | 0.000 | handbag        | 0.000 |
| tie           | 0.000 | suitcase     | 0.000 | frisbee        | 0.000 |
| skis          | 0.000 | snowboard    | 0.000 | sports ball    | 0.000 |
| kite          | 0.000 | baseball bat | 0.000 | baseball glove | 0.000 |
| skateboard    | 0.000 | surfboard    | 0.000 | tennis racket  | 0.000 |
| bottle        | 0.000 | wine glass   | 0.000 | cup            | 0.000 |
| fork          | 0.000 | knife        | 0.000 | spoon          | 0.000 |
| bowl          | 0.000 | banana       | 0.000 | apple          | 0.000 |
| sandwich      | 0.000 | orange       | 0.000 | broccoli       | 0.000 |
| carrot        | 0.000 | hot dog      | 0.000 | pizza          | 0.000 |
| donut         | 0.000 | cake         | 0.000 | chair          | 0.000 |
| couch         | 0.000 | potted plant | 0.000 | bed            | 0.000 |
| dining table  | 0.000 | toilet       | 0.000 | tv             | 0.000 |
| laptop        | 0.000 | mouse        | 0.000 | remote         | 0.000 |
| keyboard      | 0.000 | cell phone   | 0.000 | microwave      | 0.000 |
| oven          | 0.000 | toaster      | 0.000 | sink           | 0.000 |
| refrigerator  | 0.000 | book         | 0.000 | clock          | 0.000 |
| vase          | 0.000 | scissors     | 0.000 | teddy bear     | 0.000 |
| hair drier    | 0.000 | toothbrush   | 0.000 |                |       |
[04/22 13:56:09] d2.engine.defaults INFO: Evaluation results for coco_2017_val in csv format:
[04/22 13:56:09] d2.evaluation.testing INFO: copypaste: Task: bbox
[04/22 13:56:09] d2.evaluation.testing INFO: copypaste: AP,AP50,AP75,APs,APm,APl
[04/22 13:56:09] d2.evaluation.testing INFO: copypaste: 0.0113,0.0283,0.0085,0.0045,0.0101,0.0311
[04/22 13:56:09] d2.evaluation.testing INFO: copypaste: Task: segm
[04/22 13:56:09] d2.evaluation.testing INFO: copypaste: AP,AP50,AP75,APs,APm,APl
[04/22 13:56:09] d2.evaluation.testing INFO: copypaste: 0.0116,0.0296,0.0079,0.0005,0.0055,0.0362

opened by scarleatt 2

how to generate the free mask with my own custom data?

i have a instance segmentation dataset ,but its label mask is too accurate compare free mask,i just want to konw how you generate free mask?and i i use my pretrained model(solo model) to generate the free mask ,is that a bad idea?please let me konw if that's not so unconvient ,thaks a lot......

opened by Vincent630 1
Not able to produce the results

Hi,

When I run bash test.sh {MODEL_PATH} and passing the weights of the downloaded trained model, I get the following error:

Loading and preparing results... Traceback (most recent call last): File "tools/eval_cocoapi.py", line 21, in cocoDt=cocoGt.loadRes(resFile) File "python3.7/site-packages/pycocotools/coco.py", line 319, in loadRes with open(resFile) as f: FileNotFoundError: [Errno 2] No such file or directory: 'training_dir/FreeSOLO_pl/inference/coco_instances_results.json'

opened by h-aboutalebi 0
How can i get the result image?

When i run bash test.sh FreeSOLO_R101_30k_pl.pth.I got the result as this | category | AP | category | AP | category | AP | |:--------------|:------|:-------------|:------|:---------------|:------| | person | 0.665 | bicycle | 0.000 | car | 0.000 | | motorcycle | 0.000 | airplane | nan | bus | 0.000 | | train | 0.000 | truck | 0.000 | boat | 0.000 | | traffic light | 0.000 | fire hydrant | 0.000 | stop sign | 0.000 | | parking meter | nan | bench | 0.000 | bird | 0.000 | | cat | 0.000 | dog | 0.000 | horse | 0.000 | | sheep | nan | cow | 0.000 | elephant | 0.000 | | bear | 0.000 | zebra | 0.000 | giraffe | 0.000 | | backpack | 0.000 | umbrella | 0.000 | handbag | 0.000 | | tie | 0.000 | suitcase | nan | frisbee | 0.000 | | skis | 0.000 | snowboard | 0.000 | sports ball | 0.000 | | kite | 0.000 | baseball bat | 0.000 | baseball glove | 0.000 | | skateboard | 0.000 | surfboard | 0.000 | tennis racket | 0.000 | | bottle | 0.000 | wine glass | 0.000 | cup | 0.000 | | fork | 0.000 | knife | nan | spoon | 0.000 | | bowl | 0.000 | banana | 0.000 | apple | 0.000 | | sandwich | 0.000 | orange | nan | broccoli | 0.000 | | carrot | 0.000 | hot dog | nan | pizza | 0.000 | | donut | 0.000 | cake | nan | chair | 0.000 | | couch | 0.000 | potted plant | 0.000 | bed | 0.000 | | dining table | 0.000 | toilet | 0.000 | tv | 0.000 | | laptop | 0.000 | mouse | 0.000 | remote | 0.000 | | keyboard | 0.000 | cell phone | 0.000 | microwave | 0.000 | | oven | 0.000 | toaster | nan | sink | 0.000 | | refrigerator | 0.000 | book | 0.000 | clock | 0.000 | | vase | 0.000 | scissors | nan | teddy bear | 0.000 | | hair drier | nan | toothbrush | nan | | |

And it cames from python tools/eval_cocoapi.py. Did i missing sth? by the way,how can I get visualization images?

opened by Drm-hello 0
Unable to reproduce results
Hi,

Thank you for your work. I encountered several issues when using the code.

I tried to reproduce the results of Freemask by running the first step of the algorithm on the COCO dataset. However, I'm unable to get the same results as in the provided json: for example, the embedding vector is not included in the annotations of the provided json. Am I missing something here?

I ran the code with the provided json on the train2017+unlabeled2017 split, but only get 0.1% mask AP after the first step of FreeSOLO. In particular, ran the tools/eval_cocoapi.py script for the class-agnostic evaluation. I noticed that only the pairwise loss is able to go down.

Evaluating the provided final model (in this repo) does not produce 12.2% AP50 for detection (as claimed in the paper) but only obtains 9.6%. Do I need to post-process the results before evaluating?

Were people able to reproduce this? Thanks.
opened by MarcVisions 1
no "training_dir/instances_train2017_densecl_r101.json" file

Hi, Thanx for your great work,. when run "$ bash inference_freemask.sh", error existed as :no such file. Pls tell me where can i get this json file?

opened by cynthia-you 1
How can I visualize free mask results?

To ensure that the free masks are almost accurate, I add following codes in inference_freemaks.py . However, the results saved in the disks is strange: the foreground part is always at the edge of an image.

after masks = masks.cpu().numpy() ''' masks = masks.cpu().numpy() ii=0 for mask in masks: masks_save = masks[ii] * 255 cv2.imwrite('./results/'+img_name+'_'+str(ii)+'.jpg', masks_save) ii=ii+1 '''

opened by STAR-811 2

Owner

NVIDIA Research Projects

GitHub

[CVPR 2022] CoTTA Code for our CVPR 2022 paper Continual Test-Time Domain Adaptation

CoTTA Code for our CVPR 2022 paper Continual Test-Time Domain Adaptation Prerequisite Please create and activate the following conda envrionment. To r

87 Jan 8, 2023

pytorch implementation of "Contrastive Multiview Coding", "Momentum Contrast for Unsupervised Visual Representation Learning", and "Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination"

Unofficial implementation: MoCo: Momentum Contrast for Unsupervised Visual Representation Learning (Paper) InsDis: Unsupervised Feature Learning via N

16 Nov 4, 2020

Leveraging Instance-, Image- and Dataset-Level Information for Weakly Supervised Instance Segmentation

Leveraging Instance-, Image- and Dataset-Level Information for Weakly Supervised Instance Segmentation This paper has been accepted and early accessed

39 Sep 20, 2022

Imposter-detector-2022 - HackED 2022 Team 3IQ - 2022 Imposter Detector

HackED 2022 Team 3IQ - 2022 Imposter Detector By Aneeljyot Alagh, Curtis Kan, Jo

3 Aug 20, 2022

Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers [CVPR 2021]

Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers [BCNet, CVPR 2021] This is the official pytorch implementation of BCNet built on

434 Dec 1, 2022

code for CVPR paper Zero-shot Instance Segmentation

Code for CVPR2021 paper Zero-shot Instance Segmentation Code requirements python: python3.7 nvidia GPU pytorch1.1.0 GCC >=5.4 NCCL 2 the other python

86 Dec 13, 2022

Robust Instance Segmentation through Reasoning about Multi-Object Occlusion [CVPR 2021]

Robust Instance Segmentation through Reasoning about Multi-Object Occlusion [CVPR 2021] Abstract Analyzing complex scenes with DNN is a challenging ta

24 Jun 27, 2022

The 7th edition of NTIRE: New Trends in Image Restoration and Enhancement workshop will be held on June 2022 in conjunction with CVPR 2022.

NTIRE 2022 - Image Inpainting Challenge Important dates 2022.02.01: Release of train data (input and output images) and validation data (only input) 2

37 Nov 27, 2022

TorchDistiller - a collection of the open source pytorch code for knowledge distillation, especially for the perception tasks, including semantic segmentation, depth estimation, object detection and instance segmentation.

This project is a collection of the open source pytorch code for knowledge distillation, especially for the perception tasks, including semantic segmentation, depth estimation, object detection and instance segmentation.

147 Dec 3, 2022

[ArXiv 2021] Data-Efficient Instance Generation from Instance Discrimination

InsGen - Data-Efficient Instance Generation from Instance Discrimination Data-Efficient Instance Generation from Instance Discrimination Ceyuan Yang,

GenForce: May Generative Force Be with You

93 Dec 25, 2022

Segmentation in Style: Unsupervised Semantic Image Segmentation with Stylegan and CLIP

Segmentation in Style: Unsupervised Semantic Image Segmentation with Stylegan and CLIP Abstract: We introduce a method that allows to automatically se

134 Dec 19, 2022

[CVPR 2022] Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels

Using Unreliable Pseudo Labels Official PyTorch implementation of Semi-Supervised Semantic Segmentation Using Unreliable Pseudo Labels, CVPR 2022. Ple

268 Dec 24, 2022

Hyperbolic Image Segmentation, CVPR 2022

Hyperbolic Image Segmentation, CVPR 2022 This is the implementation of paper Hyperbolic Image Segmentation (CVPR 2022). Repository structure assets :

46 Dec 29, 2022

Contrastive learning of Class-agnostic Activation Map for Weakly Supervised Object Localization and Semantic Segmentation (CVPR 2022)

CCAM (Unsupervised) Code repository for our paper "CCAM: Contrastive learning of Class-agnostic Activation Map for Weakly Supervised Object Localizati

113 Dec 27, 2022

Scribble-Supervised LiDAR Semantic Segmentation, CVPR 2022 (ORAL)

Scribble-Supervised LiDAR Semantic Segmentation Dataset and code release for the paper Scribble-Supervised LiDAR Semantic Segmentation, CVPR 2022 (ORA

102 Dec 25, 2022

Stratified Transformer for 3D Point Cloud Segmentation (CVPR 2022)

Stratified Transformer for 3D Point Cloud Segmentation Xin Lai*, Jianhui Liu*, Li Jiang, Liwei Wang, Hengshuang Zhao, Shu Liu, Xiaojuan Qi, Jiaya Jia

195 Jan 1, 2023

This is the official pytorch implementation for the paper: Instance Similarity Learning for Unsupervised Feature Representation.

ISL This is the official pytorch implementation for the paper: Instance Similarity Learning for Unsupervised Feature Representation, which is accepted

19 May 4, 2022

IAST: Instance Adaptive Self-training for Unsupervised Domain Adaptation (ECCV 2020)

This repo is the official implementation of our paper "Instance Adaptive Self-training for Unsupervised Domain Adaptation". The purpose of this repo is to better communicate with you and respond to your questions. This repo is almost the same with Another-Version, and you can also refer to that version.

84 Dec 12, 2022

Code for Multiple Instance Active Learning for Object Detection, CVPR 2021

Language: 简体中文 | English Introduction This is the code for Multiple Instance Active Learning for Object Detection, CVPR 2021. Installation A Linux pla

269 Dec 21, 2022