VOS: Learning What You Don’t Know by Virtual Outlier Synthesis

Related tags

Deep Learning vos
Overview

VOS

This is the source code accompanying the paper VOS: Learning What You Don’t Know by Virtual Outlier Synthesis by Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li

The codebase is heavily based on ProbDet and Detectron2.

Dataset Preparation

PASCAL VOC

Download the processed VOC 2007 and 2012 dataset from here.

The VOC dataset folder should have the following structure:

 └── VOC_DATASET_ROOT
     |
     ├── JPEGImages
     ├── voc0712_train_all.json
     └── val_coco_format.json

COCO

Download COCO2017 dataset from the official website.

Download the OOD dataset (json file) when the in-distribution dataset is Pascal VOC from here.

Download the OOD dataset (json file) when the in-distribution dataset is BDD-100k from here.

Put the two processed OOD json files to ./anntoations

The COCO dataset folder should have the following structure:

 └── COCO_DATASET_ROOT
     |
     ├── annotations
        ├── xxx (the original json files)
        ├── instances_val2017_ood_wrt_bdd_rm_overlap.json
        └── instances_val2017_ood_rm_overlap.json
     ├── train2017
     └── val2017

BDD-100k

Donwload the BDD-100k images from the official website.

Download the processed BDD-100k json files from here and here.

The BDD dataset folder should have the following structure:

 └── BDD_DATASET_ROOT
     |
     ├── images
     ├── val_bdd_converted.json
     └── train_bdd_converted.json

OpenImages

Download our OpenImages validation splits here. We created a tarball that contains the out-of-distribution data splits used in our paper for hyperparameter tuning. Do not modify or rename the internal folders as those paths are hard coded in the dataset reader. The OpenImages dataset is created in a similar way following this paper.

The OpenImages dataset folder should have the following structure:

 └── OEPNIMAGES_DATASET_ROOT
     |
     ├── coco_classes
     └── ood_classes_rm_overlap

Before training, modify the dataset address in the ./detection/core/datasets/setup_datasets.py according to your local dataset address.

Visualization of the OOD datasets

The OOD images with respect to different in-distribution datasets can be downloaded from ID-VOC-OOD-COCO, ID-VOC-OOD-openimages, ID-BDD-OOD-COCO, ID-BDD-OOD-openimages.

Training

Firstly, enter the detection folder by running

cd detection

Vanilla Faster-RCNN with VOC as the in-distribution dataset


python train_net.py
--num-gpus 8
--config-file VOC-Detection/faster-rcnn/vanilla.yaml 
--random-seed 0 
--resume

Vanilla Faster-RCNN with BDD as the in-distribution dataset

python train_net.py 
--num-gpus 8 
--config-file BDD-Detection/faster-rcnn/vanilla.yaml 
--random-seed 0 
--resume

VOS on ResNet

python train_net_gmm.py 
--num-gpus 8 
--config-file VOC-Detection/faster-rcnn/vos.yaml 
--random-seed 0 
--resume

VOS on RegNet

Before training using the RegNet as the backbone, download the pretrained RegNet backbone from here.

python train_net_gmm.py 
--num-gpus 8 
--config-file VOC-Detection/faster-rcnn/regnetx.yaml 
--random-seed 0 
--resume

Before training on VOS, change "VOS.STARTING_ITER" and "VOS.SAMPLE_NUMBER" in the config file to the desired numbers in paper.

Evaluation

Evaluation with the in-distribution dataset to be VOC

Firstly run on the in-distribution dataset:

python apply_net.py 
--test-dataset voc_custom_val 
--config-file VOC-Detection/faster-rcnn/vos.yaml 
--inference-config Inference/standard_nms.yaml 
--random-seed 0 
--image-corruption-level 0 
--visualize 0

Then run on the OOD dataset:

python apply_net.py
--test-dataset coco_ood_val 
--config-file VOC-Detection/faster-rcnn/vos.yaml 
--inference-config Inference/standard_nms.yaml 
--random-seed 0 
--image-corruption-level 0 
--visualize 0

Obtain the metrics using:

python voc_coco_plot.py 
--name vos 
--thres xxx 
--energy 1 
--seed 0

Here the threshold is determined according to ProbDet. It will be displayed in the screen as you finish evaluating on the in-distribution dataset.

Evaluation with the in-distribution dataset to be BDD

Firstly run on the in-distribution dataset:

python apply_net.py 
--test-dataset bdd_custom_val 
--config-file VOC-Detection/faster-rcnn/vos.yaml 
--inference-config Inference/standard_nms.yaml 
--random-seed 0 
--image-corruption-level 0 
--visualize 0

Then run on the OOD dataset:

python apply_net.py 
--test-dataset coco_ood_val_bdd 
--config-file VOC-Detection/faster-rcnn/vos.yaml 
--inference-config Inference/standard_nms.yaml 
--random-seed 0 
--image-corruption-level 0 
--visualize 0

Obtain the metrics using:

python bdd_coco_plot.py
--name vos 
--thres xxx 
--energy 1 
--seed 0

Pretrained models

The pretrained models for Pascal-VOC can be downloaded from vanilla and VOS-ResNet and VOS-RegNet.

The pretrained models for BDD-100k can be downloaded from vanilla and VOS-ResNet and VOS-RegNet.

VOS on Classification models

Train on WideResNet

cd classification/CIFAR/ & 
python train_virtual.py 
--start_epoch 40 
--sample_number 1000 
--sample_from 10000 
--select 1 
--loss_weight 0.1 

where "start_epoch" denotes the starting epoch of the uncertainty regularization branch.

"sample_number" denotes the size of the in-distribution queue.

"sample_from" and "select" are used to approximate the likelihood threshold during virtual outlier synthesis.

"loss_weight" denotes the weight of the regularization loss.

Please see Section 3 and Section 4.1 in the paper for details.

Train on DenseNet

cd classification/CIFAR/ &
python train_virtual_dense.py 
--start_epoch 40 
--sample_number 1000 
--sample_from 10000 
--select 1 
--loss_weight 0.1 

Evaluation on different classifiers

cd classification/CIFAR/ & 
python test.py 
--model_name xx 
--method_name xx 
--score energy 
--num_to_avg 10

where "model_name" denotes the model architectures. ("res" denotes the WideResNet and "dense" denotes the DenseNet.)

"method_name" denotes the checkpoint name you are loading.

Pretrained models

We provide the pretrained models using WideResNet and DenseNet with the in-distribution dataset to be CIFAR-10.

Citation

If you found any part of this code is useful in your research, please consider citing our paper:

 @article{du2022vos,
      title={VOS: Learning What You Don’t Know by Virtual Outlier Synthesis}, 
      author={Du, Xuefeng and Wang, Zhaoning and Cai, Mu and Li, Yixuan},
      journal={Proceedings of the International Conference on Learning Representations},
      year={2022}
}
Comments
  • Test About OpenImages

    Test About OpenImages

    Dear Authors,

    I have trained the model based on BDD. And I have evaluated the model on the COCO successfully. Now I use the same model to evaluate openimages. There is an error. QQ截图20220301103543

    opened by AmingWu 13
  • Can this method of vos be used in the yolo model?

    Can this method of vos be used in the yolo model?

    I want to use this method in yolo, but during the implementation process, I encountered some problems. Has anyone used this method in yolo? Or have any suggestions?

    opened by konioy 6
  • Not getting good results

    Not getting good results

    Greetings, I am attempting to involve your proposed model in my undertaking. I have utilized the accompanying cfg to prepare my model.

    from vos.detection.default_trainer_gmm import DefaultTrainer from detectron2.config import CfgNode, get_cfg from vos.detection.core.setup import add_probabilistic_config

    cfg = get_cfg()#CfgNode() add_probabilistic_config(cfg)

    cfg_file_name = "vos/detection/configs/VOC-Detection/faster-rcnn/vos.yaml" cfg.merge_from_file(cfg_file_name) cfg.SOLVER.BASE_LR = 0.0001 cfg.SOLVER.IMS_PER_BATCH=6 cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 64 cfg.MODEL.ROI_HEADS.POSITIVE_FRACTION = 0.25 cfg.MODEL.RPN.BATCH_SIZE_PER_IMAGE = 128 cfg.SOLVER.MAX_ITER = 20000 cfg.SOLVER.STEPS = (8000,12000) cfg.VOS.STARTING_ITER = 10000 cfg.SOLVER.CHECKPOINT_PERIOD = 8000 cfg.MODEL.ROI_HEADS.NUM_CLASSES = len(MetadataCatalog.get('my_data').thing_classes)

    Some way or another, I am not obtain great outcomes. Would you be able to diagram why it isn't functioning admirably

    Dataset info image

    Training Graphs:

    image image

    Please let me know if you need more info, to help me.

    opened by imvansh25 6
  • Covariance matrix not PositiveDefinite for larger feature sizes

    Covariance matrix not PositiveDefinite for larger feature sizes

    Hi,

    I tried applying the classification example with a different network. To be specific, I tried the Xception network which has a feature size of 2048 (the WideResNet model from your code has a feature size of 128). This introduces an error, raised by PyTorch:

    ValueError: Expected parameter covariance_matrix (Tensor of shape (2048, 2048)) of distribution MultivariateNormal(loc: torch.Size([2048]), covariance_matrix: torch.Size([2048, 2048])) to satisfy the constraint PositiveDefinite(), but found invalid values:... Interesting enough, this does not happen with the efficientnet-b5 model (which also has a feature size of 2048). Do you maybe know what is going on?

    opened by jorism1993 6
  • Question abuout evaluation

    Question abuout evaluation

    Before evaluation FPR95 and AUROC, the predictions are filtered by "a threshold", why after this filter, the left predictions can be seen as "in-distribution object?", and what's meaning about this optimal_score_threshold. thank you~

        # Compute optimal micro F1 score threshold. We compute the f1 score for
        # every class and score threshold. We then compute the score threshold that
        # maximizes the F-1 score of every class. The final score threshold is the average
        # over all classes.
        precisions = results_api.eval['precision'].mean(0)[:, :, 0, 2]
        recalls = np.expand_dims(results_api.params.recThrs, 1)
        f1_scores = 2 * (precisions * recalls) / (precisions + recalls)
        optimal_f1_score = f1_scores.argmax(0)
        scores = results_api.eval['scores'].mean(0)[:, :, 0, 2]
        optimal_score_threshold = [scores[optimal_f1_score_i, i]
                                   for i, optimal_f1_score_i in enumerate(optimal_f1_score)]
        optimal_score_threshold = np.array(optimal_score_threshold)
        optimal_score_threshold = optimal_score_threshold[optimal_score_threshold != 0]
        optimal_score_threshold = optimal_score_threshold.mean()
    
    opened by AceCoooool 5
  • Missing CIFAR VOS test option?

    Missing CIFAR VOS test option?

    I was wondering if there is a missing python script for testing VOS trained CIFAR models on the OOD datasets to recreate Table 4 from the paper? In classification/CIFAR/test.py I am not seeing any of the VOS code from train_virtual.py referenced for any of the evaluation options.

    Hopefully I'm not overlooking the code and missing it. Thanks.

    opened by warner-benjamin 5
  • Preparing classification datasets

    Preparing classification datasets

    Dear Author,

    I'm trying to run your code and I have a trouble with preparing classification datasets. (places365, iSUN, LSUN-C, LSUN-Resize) I downloaded these datasets but there are only images in the folder. There are no class labels. Would you tell me where I can download classification datasets with class labels? Thank you.

    opened by SUPER-TADORY 5
  • How to get results of the  Mahalanobis method?

    How to get results of the Mahalanobis method?

    Hi,

    Thanks for your public code. The OOD detection benchmarks in your paper are great. I want to know how to get results of the Mahalanobis method based on your public code?

    Thanks.

    opened by songyh15 4
  • Test Result

    Test Result

    Dear Authors,

    this is the evaluation results based on VOC dataset. The results are not consistent with the given results in the paper.

    QQ截图20220227151827

    I first run python apply_net.py --dataset-dir /data1/user/vos/VOC_0712_converted --test-dataset voc_custom_val --config-file VOC-Detection/faster-rcnn/vos.yaml --inference-config Inference/standard_nms.yaml --random-seed 0 --image-corruption-level 0 --visualize 0.

    Then I run python apply_net.py --dataset-dir /data1/user/COCO2017/datasets/coco --test-dataset coco_ood_val --config-file VOC-Detection/faster-rcnn/vos.yaml --inference-config Inference/standard_nms.yaml --random-seed 0 --image-corruption-level 0 --visualize 0.

    Finally, I run CUDA_VISIBLE_DEVICES=4 python voc_coco_plot.py --name vos --thres xxx --energy 1 --seed 0.

    Could you give me some advice? Thank you.

    opened by AmingWu 4
  • Calculating the OOD score using VOS in the classification task

    Calculating the OOD score using VOS in the classification task

    Hello,

    How can I calculate the OOD score on the classification models? I see the "get_ood_scores" function in "vos-main/classification/CIFAR/test.py". Does this function calculate the OOD score for VOS? (or this function only relates to the baselines considered in the paper?).

    opened by ah-ansari 3
  • How to calculate uncertainty score  at inference stage?

    How to calculate uncertainty score at inference stage?

    thanks for your excelent work. I have read your paper and notice that model will calculate uncertainty score according Eq8 at eval stage, but I can not find it in your code. Forgive me that my work env cannot run your code, so that I study your implement only in github.

    Its seems that you just use normal faster-rcnn as eval stage,(link) and I can not find where use the all_predicted_prob_vectors

    opened by captainfffsama 3
Owner
CS Research Group led by Prof. Sharon Li
null
AI Virtual Calculator: This is a simple virtual calculator based on Artificial intelligence.

AI Virtual Calculator: This is a simple virtual calculator that works with gestures using OpenCV. We will use our hand in the air to click on the calc

Md. Rakibul Islam 1 Jan 13, 2022
(JMLR'19) A Python Toolbox for Scalable Outlier Detection (Anomaly Detection)

Python Outlier Detection (PyOD) Deployment & Documentation & Stats Build Status & Coverage & Maintainability & License PyOD is a comprehensive and sca

Yue Zhao 6.4k Nov 22, 2022
SSD: A Unified Framework for Self-Supervised Outlier Detection [ICLR 2021]

SSD: A Unified Framework for Self-Supervised Outlier Detection [ICLR 2021] Pdf: https://openreview.net/forum?id=v5gjXpmR8J Code for our ICLR 2021 pape

Princeton INSPIRE Research Group 111 Nov 21, 2022
Streaming Anomaly Detection Framework in Python (Outlier Detection for Streaming Data)

Python Streaming Anomaly Detection (PySAD) PySAD is an open-source python framework for anomaly detection on streaming multivariate data. Documentatio

Selim Firat Yilmaz 177 Oct 21, 2022
A gesture recognition system powered by OpenPose, k-nearest neighbours, and local outlier factor.

OpenHands OpenHands is a gesture recognition system powered by OpenPose, k-nearest neighbours, and local outlier factor. Currently the system can iden

Paul Treanor 12 Jan 10, 2022
Outlier Exposure with Confidence Control for Out-of-Distribution Detection

OOD-detection-using-OECC This repository contains the essential code for the paper Outlier Exposure with Confidence Control for Out-of-Distribution De

Nazim Shaikh 64 Nov 2, 2022
Certifiable Outlier-Robust Geometric Perception

Certifiable Outlier-Robust Geometric Perception About This repository holds the implementation for certifiably solving outlier-robust geometric percep

null 79 Nov 17, 2022
Deep Anomaly Detection with Outlier Exposure (ICLR 2019)

Outlier Exposure This repository contains the essential code for the paper Deep Anomaly Detection with Outlier Exposure (ICLR 2019). Requires Python 3

Dan Hendrycks 461 Nov 22, 2022
(Py)TOD: Tensor-based Outlier Detection, A General GPU-Accelerated Framework

(Py)TOD: Tensor-based Outlier Detection, A General GPU-Accelerated Framework Background: Outlier detection (OD) is a key data mining task for identify

Yue Zhao 118 Nov 15, 2022
Official Implementation of "LUNAR: Unifying Local Outlier Detection Methods via Graph Neural Networks"

LUNAR Official Implementation of "LUNAR: Unifying Local Outlier Detection Methods via Graph Neural Networks" Adam Goodge, Bryan Hooi, Ng See Kiong and

Adam Goodge 22 Nov 22, 2022
A Python Library for Graph Outlier Detection (Anomaly Detection)

PyGOD is a Python library for graph outlier detection (anomaly detection). This exciting yet challenging field has many key applications, e.g., detect

PyGOD Team 719 Nov 19, 2022
PyTorch code for EMNLP 2021 paper: Don't be Contradicted with Anything! CI-ToD: Towards Benchmarking Consistency for Task-oriented Dialogue System

Don’t be Contradicted with Anything!CI-ToD: Towards Benchmarking Consistency for Task-oriented Dialogue System This repository contains the PyTorch im

Libo Qin 25 Sep 6, 2022
PyTorch code for EMNLP 2021 paper: Don't be Contradicted with Anything! CI-ToD: Towards Benchmarking Consistency for Task-oriented Dialogue System

PyTorch code for EMNLP 2021 paper: Don't be Contradicted with Anything! CI-ToD: Towards Benchmarking Consistency for Task-oriented Dialogue System

Libo Qin 12 Sep 26, 2021
Virtual Dance Reality Stage: a feature that offers you to share a stage with another user virtually

Portrait Segmentation using Tensorflow This script removes the background from an input image. You can read more about segmentation here Setup The scr

null 289 Nov 15, 2022
python debugger and anti-vm that checks if you're in a virtual machine or if someones trying to debug your file

Anti-Debug was made by Love ❌ code ✅ ?? ・What it checks for ・ Kills tools that can be used to debug your file ・ Exits if ran in vm (supports different

Rdimo 31 Aug 9, 2022
PyTorch Implementation for AAAI'21 "Do Response Selection Models Really Know What's Next? Utterance Manipulation Strategies for Multi-turn Response Selection"

UMS for Multi-turn Response Selection Implements the model described in the following paper Do Response Selection Models Really Know What's Next? Utte

Taesun Whang 47 Nov 22, 2022
This program uses trial auth token of Azure Cognitive Services to do speech synthesis for you.

??️ aspeak A simple text-to-speech client using azure TTS API(trial). ?? TL;DR: This program uses trial auth token of Azure Cognitive Services to do s

Levi Zim 341 Nov 25, 2022
Jarvis Project is a basic virtual assistant that uses TensorFlow for learning.

Jarvis_proyect Jarvis Project is a basic virtual assistant that uses TensorFlow for learning. Latest version 0.1 Features: Good morning protocol Tell

Anze Kovac 3 Aug 31, 2022