Learning Open-World Object Proposals without Learning to Classify

Overview

Learning Open-World Object Proposals without Learning to Classify

Pytorch implementation for "Learning Open-World Object Proposals without Learning to Classify" (arXiv 2021)

Dahun Kim, Tsung-Yi Lin, Anelia Angelova, In So Kweon, and Weicheng Kuo.

@article{kim2021oln,
  title={Learning Open-World Object Proposals without Learning to Classify},
  author={Kim, Dahun and Lin, Tsung-Yi and Angelova, Anelia and Kweon, In So and Kuo, Weicheng},
  journal={arXiv preprint arXiv:2108.06753},
  year={2021}
}

Introduction

Humans can recognize novel objects in this image despite having never seen them before. “Is it possible to learn open-world (novel) object proposals?” In this paper we propose Object Localization Network (OLN) that learns localization cues instead of foreground vs background classification. Only trained on COCO, OLN is able to propose many novel objects (top) missed by Mask R-CNN (bottom) on an out-of-sample frame in an ego-centric video.


Cross-category generalization on COCO

We train OLN on COCO VOC categories, and test on non-VOC categories. Note our AR@k evaluation does not count those proposals on the 'seen' classes into the budget (k), to avoid evaluating recall on see-class objects.

Method AUC AR@10 AR@30 AR@100 AR@300 AR@1000 Download
OLN-Box 24.8 18.0 26.4 33.4 39.0 45.0 model

Disclaimer

This repo is tested under Python 3.7, PyTorch 1.7.0, Cuda 11.0, and mmcv==1.2.5.

Installation

This repo is built based on mmdetection.

You can use following commands to create conda env with related dependencies.

conda create -n oln python=3.7 -y
conda activate oln
conda install pytorch=1.7.0 torchvision cudatoolkit=11.0 -c pytorch -y
pip install mmcv-full
pip install -r requirements.txt
pip install -v -e . 

Please also refer to get_started.md for more details of installation.

Prepare datasets

COCO dataset is available from official websites. It is recommended to download and extract the dataset somewhere outside the project directory and symlink the dataset root to $OLN/data as below.

object_localization_network
├── mmdet
├── tools
├── configs
├── data
│   ├── coco
│   │   ├── annotations
│   │   ├── train2017
│   │   ├── val2017
│   │   ├── test2017

Testing

Our trained models are available for download here. Place it under trained_weights/latest.pth and run the following commands to test OLN on COCO dataset.

# Multi-GPU distributed testing
bash tools/dist_test_bbox.sh configs/oln_box/oln_box.py \
trained_weights/latest.pth ${NUM_GPUS}
# OR
python tools/test.py configs/oln_box/oln_box.py work_dirs/oln_box/latest.pth --eval bbox

Training

# Multi-GPU distributed training
bash tools/dist_train.sh configs/oln_box/oln_box.py ${NUM_GPUS}

Contact

If you have any questions regarding the repo, please contact Dahun Kim ([email protected]) or create an issue.

Comments
  • About Cross Category Evaluation

    About Cross Category Evaluation

    Hi there, I have a question regarding cross-category evaluation. According to the code, the dataset filter out non-VOC annotations, but this does not filter out images. If the images contain other VOC classes, will they be included in the training set. image

    opened by curiosity654 2
  • Could you please release the training loss curve?

    Could you please release the training loss curve?

    Thank you for the amazing work, however, when I train the model using default configs, the validation results are extremely low. Could you provide the training loss curve for comparison? Thanks in advance. image image

    opened by curiosity654 1
  • Incorrect Dependency Installation Order

    Incorrect Dependency Installation Order

    Following the installation steps, while running pip install mmcv-full==1.2.7 leads to this error:

    Collecting mmcv-full==1.2.7
      Using cached mmcv-full-1.2.7.tar.gz (231 kB)
        ERROR: Command errored out with exit status 1:
         command: 'D:\Software\Anaconda\python.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\intel\\AppData\\Local\\Temp\\pip-install-rmjilsef\\mmcv-full\\setup.py'"'"'; __file__='"'"'C:\\Users\\intel\\AppData\\Local\\Temp\\pip-install-rmjilsef\\mmcv-full\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Users\intel\AppData\Local\Temp\pip-install-rmjilsef\mmcv-full\pip-egg-info'
             cwd: C:\Users\intel\AppData\Local\Temp\pip-install-rmjilsef\mmcv-full\
        Complete output (5 lines):
        Traceback (most recent call last):
          File "<string>", line 1, in <module>
          File "C:\Users\intel\AppData\Local\Temp\pip-install-rmjilsef\mmcv-full\setup.py", line 121, in <module>
            major, minor, *rest = cv2.__version__.split('.')
        AttributeError: module 'cv2' has no attribute '__version__'
        ----------------------------------------
    ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
    

    This seems to be avoided by running the commands in the new order:

    conda create -n oln python=3.7 -y
    conda activate oln
    conda install pytorch=1.7.0 torchvision cudatoolkit=11.0 -c pytorch -y
    pip install -r requirements.txt
    pip install mmcv-full==1.2.7
    pip install -v -e . 
    

    As pip install -r requirements.txt seems to satsify the error AttributeError: module 'cv2' has no attribute '__version__' encountered in pip install mmcv-full==1.2.7

    opened by harshsingh-24 0
  • whether the code about iou sample is released

    whether the code about iou sample is released

    image but the code of OlnRPNHead shows that the iou sample is not used, as the function get_target return to the objectness_targets_list and objectness_weights_list without sample image

    opened by eternaldolphin 0
  • Hello, I have a few questions to ask.

    Hello, I have a few questions to ask.

    Hello, I have a few questions to ask.

    1. How are the effects shown in Figure 5 and Figure 6 generated in the paper?
    2. If you want to generalize the trained model to other data sets, how should you modify the configuration file or code?

    Thank you for your reply.

    opened by mary-0830 0
  • OLN-RPN

    OLN-RPN

    Could you provide the config of OLN-RPN? I remove the part of rcnn in "configs/oln_box/oln_box.py", but the results are worse than the results in the paper. 1656222721984

    opened by gaoyingjay 0
  • Number of class in training

    Number of class in training

    Hi, thanks for the inspiring work. When I retrained the model with the config /oln_box, i received this: "AssertionError: The num_classes (1) in Shared2FCBBoxScoreHead of MMDataParallel does not matches the length of CLASSES 80) in CocoSplitDataset " So, should I set the num_classes = 80 instead of 1 when training with COCO? Or there is something else I should do. Thanks!

    opened by mfanparc 0
Owner
Dahun Kim
Korea Advanced Institute of Science and Technology
Dahun Kim
Unsupervised Semantic Segmentation by Contrasting Object Mask Proposals.

Unsupervised Semantic Segmentation by Contrasting Object Mask Proposals This repo contains the Pytorch implementation of our paper: Unsupervised Seman

Wouter Van Gansbeke 335 Dec 28, 2022
Sparse R-CNN: End-to-End Object Detection with Learnable Proposals, CVPR2021

End-to-End Object Detection with Learnable Proposal, CVPR2021

Peize Sun 1.2k Dec 27, 2022
The implemention of Video Depth Estimation by Fusing Flow-to-Depth Proposals

Flow-to-depth (FDNet) video-depth-estimation This is the implementation of paper Video Depth Estimation by Fusing Flow-to-Depth Proposals Jiaxin Xie,

null 32 Jun 14, 2022
AEI: Actors-Environment Interaction with Adaptive Attention for Temporal Action Proposals Generation

AEI: Actors-Environment Interaction with Adaptive Attention for Temporal Action Proposals Generation A pytorch-version implementation codes of paper:

null 11 Dec 13, 2022
RE-OWOD - Revisiting open world object detection

Revisting Open World Object Detection Installation See INSTALL.md. Dataset Our n

null 7 Jan 5, 2022
A deep learning network built with TensorFlow and Keras to classify gender and estimate age.

Convolutional Neural Network (CNN). This repository contains a source code of a deep learning network built with TensorFlow and Keras to classify gend

Pawel Dziemiach 1 Dec 18, 2021
A deep learning network built with TensorFlow and Keras to classify gender and estimate age.

Convolutional Neural Network (CNN). This repository contains a source code of a deep learning network built with TensorFlow and Keras to classify gend

Pawel Dziemiach 1 Dec 19, 2021
Tools to create pixel-wise object masks, bounding box labels (2D and 3D) and 3D object model (PLY triangle mesh) for object sequences filmed with an RGB-D camera.

Tools to create pixel-wise object masks, bounding box labels (2D and 3D) and 3D object model (PLY triangle mesh) for object sequences filmed with an RGB-D camera. This project prepares training and testing data for various deep learning projects such as 6D object pose estimation projects singleshotpose, as well as object detection and instance segmentation projects.

null 305 Dec 16, 2022
Classify bird species based on their songs using SIamese Networks and 1D dilated convolutions.

The goal is to classify different birds species based on their songs/calls. Spectrograms have been extracted from the audio samples and used as features for classification.

Aditya Dutt 9 Dec 27, 2022
ML model to classify between cats and dogs

Cats-and-dogs-classifier This is my first ML model which can classify between cats and dogs. Here the accuracy is around 75%, however , the accuracy c

Sharath V 4 Aug 20, 2021
a dnn ai project to classify which food people are eating on audio recordings

Deep Learning - EAT Challenge About This project is part of an AI challenge of the DeepLearning course 2021 at the University of Augsburg. The objecti

Marco Tröster 1 Oct 24, 2021
Classify the disease status of a plant given an image of a passion fruit

Passion Fruit Disease Detection I tried to create an accurate machine learning models capable of localizing and identifying multiple Passion Fruits in

null 3 Nov 9, 2021
An implementation of a discriminant function over a normal distribution to help classify datasets.

CS4044D Machine Learning Assignment 1 By Dev Sony, B180297CS The question, report and source code can be found here. Github Repo Solution 1 Based on t

Dev Sony 6 Nov 9, 2021
A Home Assistant custom component for Lobe. Lobe is an AI tool that can classify images.

Lobe This is a Home Assistant custom component for Lobe. Lobe is an AI tool that can classify images. This component lets you easily use an exported m

Kendell R 4 Feb 28, 2022
Rest API Written In Python To Classify NSFW Images.

Rest API Written In Python To Classify NSFW Images.

Wahyusaputra 2 Dec 23, 2021
To Design and Implement Logistic Regression to Classify Between Benign and Malignant Cancer Types

To Design and Implement Logistic Regression to Classify Between Benign and Malignant Cancer Types, from a Database Taken From Dr. Wolberg reports his Clinic Cases.

Astitva Veer Garg 1 Jul 31, 2022
Use graph-based analysis to re-classify stocks and to improve Markowitz portfolio optimization

Dynamic Stock Industrial Classification Use graph-based analysis to re-classify stocks and experiment different re-classification methodologies to imp

Sheng Yang 10 Dec 5, 2022
This is a model to classify Vietnamese sign language using Motion history image (MHI) algorithm and CNN.

Vietnamese sign lagnuage recognition using MHI and CNN This is a model to classify Vietnamese sign language using Motion history image (MHI) algorithm

Phat Pham 3 Feb 24, 2022
Employs neural networks to classify images into four categories: ship, automobile, dog or frog

Neural Net Image Classifier Employs neural networks to classify images into four categories: ship, automobile, dog or frog Viterbi_1.py uses a classic

Riley Baker 1 Jan 18, 2022