Estimating and Exploiting the Aleatoric Uncertainty in Surface Normal Estimation

Related tags

Deep Learning surface_normal_uncertainty
Overview

Estimating and Exploiting the Aleatoric Uncertainty in Surface Normal Estimation

Official implementation of the paper

Estimating and Exploiting the Aleatoric Uncertainty in Surface Normal Estimation
ICCV 2021 [oral]
Gwangbin Bae, Ignas Budvytis, and Roberto Cipolla
[arXiv]

The proposed method estimates the per-pixel surface normal probability distribution, from which the expected angular error can be inferred to quantify the aleatoric uncertainty. We also introduce a novel decoder framework where pixel-wise MLPs are trained on a subset of pixels selected based on the uncertainty. Such uncertainty-guided sampling prevents the bias in training towards large planar surfaces, thereby improving the level of the detail in the prediction.

Getting Started

We recommend using a virtual environment.

python3.6 -m venv --system-site-packages ./venv
source ./venv/bin/activate

Install the necessary dependencies by

python3.6 -m pip install -r requirements.txt

Download the pre-trained model weights and sample images.

python download.py && cd examples && unzip examples.zip && cd ..

Running the above will download

  • ./checkpoints/nyu.pt (model trained on NYUv2)
  • ./checkpoints/scannet.pt (model trained on ScanNet)
  • ./examples/*.png (sample images)

Run Demo

To test on your own images, please add them under ./examples/. The images should be in .png or .jpg.

Test using the network trained on NYUv2. We used the ground truth and data split provided by GeoNet.

Please note that the ground truth for NYUv2 is only defined for the center crop of image. The prediction is therefore not accurate outside the center. When testing on your own images, we recommend using the network trained on ScanNet.

python test.py --pretrained nyu --architecture GN

Test using the network trained on ScanNet. We used the ground truth and data split provided by FrameNet.

python test.py --pretrained scannet --architecture BN

Running the above will save the predicted surface normal and uncertainty under ./examples/results/. If successful, you will obtain images like below.

The predictions in the figure above are obtained by the network trained only on ScanNet. The network generalizes well to objects unseen during training (e.g., humans, cars, animals). The last row shows interesting examples where the input image only contains edges.

Citation

If you find our work useful in your research please consider citing our paper:

@InProceedings{Bae2021,
    title   = {Estimating and Exploiting the Aleatoric Uncertainty in Surface Normal Estimation}
    author  = {Gwangbin Bae and Ignas Budvytis and Roberto Cipolla},
    booktitle = {International Conference on Computer Vision (ICCV)},
    year = {2021}                         
}
Comments
  • _pickle.UnpicklingError: invalid load key, '<'.

    _pickle.UnpicklingError: invalid load key, '<'.

    Hello,

    When I run python test.py --pretrained scannet --architecture BN on Win10, it shows this:

    loading checkpoint... ./checkpoints/scannet.pt Loading base model ()...Using cache found in C:\Users\LZS/.cache\torch\hub\rwightman_gen-efficientnet-pytorch_master Done. Removing last two layers (global_pool & classifier). Traceback (most recent call last): File "test.py", line 97, in model = utils.load_checkpoint(checkpoint, model) File "F:\surface_normal_uncertainty\utils\utils.py", line 57, in load_checkpoint ckpt = torch.load(fpath, map_location=lambda storage, loc: storage)['model'] File "F:\Anaconda3\lib\site-packages\torch\serialization.py", line 608, in load return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args) File "F:\Anaconda3\lib\site-packages\torch\serialization.py", line 777, in _legacy_load magic_number = pickle_module.load(f, **pickle_load_args) _pickle.UnpicklingError: invalid load key, '<'.

    I have tried directly downloading the model file, tf_efficientnet_b5_ap-9e82fae8.pth, from https://zzun.app/repo/rwightman-pytorch-image-models-python-deep-learning#releases, and use it to replace the original one.

    I checked their file size and they are all over 100M.

    Could you please give me some idea?

    opened by ZhenshanLiang 5
  • Training on Scannet

    Training on Scannet

    Hello, I didn't find the data loader for Scannet, could I modify the file dataloader_nyu.py to load scannet data? Are there any specific modifications required for scannet? Thanks so much.

    opened by w080707 2
  • Training process

    Training process

    Hi Gwangbin,

    Thanks for sharing your great work! I am curious about the training process, is the network trained by end-to-end training?

    Best wishes, Runsong

    opened by Runsong123 2
  • data alignment

    data alignment

    Screenshot from 2022-01-17 16-37-11

    Thank you for your excellent work！Because there is no code for data processing, could you please introduce how to align the normal parameters output by the network with the GT in the FrameNet dataset (mentioned in the paper).

    opened by wangyusenofficial 1
  • Adding to kornia?

    Adding to kornia?

    Hi,

    This is a great work! Would you consider adding your model to kornia? https://github.com/kornia/kornia I believe, that it could boost the usage and citations for you and make life a bit simpler for your users.

    -- Best, Dmytro

    opened by ducha-aiki 0
Owner
Bae, Gwangbin
PhD student in Computer Vision @ University of Cambridge
Bae, Gwangbin
GitHub
git git《Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking》(CVPR 2021) GitHub:git2] 《Masksembles for Uncertainty Estimation》(CVPR 2021) GitHub:git3]

Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking Ning Wang, Wengang Zhou, Jie Wang, and Houqiang Li Accepted by CVPR

NingWang 236 Dec 22, 2022
Code Release for ICCV 2021 (oral), "AdaFit: Rethinking Learning-based Normal Estimation on Point Clouds"

AdaFit: Rethinking Learning-based Normal Estimation on Point Clouds (ICCV 2021 oral) **Project Page | Arxiv ** Runsong Zhu¹, Yuan Liu², Zhen Dong¹, Te

null 40 Dec 30, 2022
This repository contains codes of ICCV2021 paper: SO-Pose: Exploiting Self-Occlusion for Direct 6D Pose Estimation

SO-Pose This repository contains codes of ICCV2021 paper: SO-Pose: Exploiting Self-Occlusion for Direct 6D Pose Estimation This paper is basically an

shangbuhuan 52 Nov 25, 2022
This is an official implementation for "Exploiting Temporal Contexts with Strided Transformer for 3D Human Pose Estimation".

Exploiting Temporal Contexts with Strided Transformer for 3D Human Pose Estimation This repo is the official implementation of Exploiting Temporal Con

Vegetabird 241 Jan 7, 2023
Re-implementation of the Noise Contrastive Estimation algorithm for pyTorch, following "Noise-contrastive estimation: A new estimation principle for unnormalized statistical models." (Gutmann and Hyvarinen, AISTATS 2010)

Noise Contrastive Estimation for pyTorch Overview This repository contains a re-implementation of the Noise Contrastive Estimation algorithm, implemen

Denis Emelin 42 Nov 24, 2022
This is a Keras implementation of a CNN for estimating age, gender and mask from a camera.

face-detector-age-gender This is a Keras implementation of a CNN for estimating age, gender and mask from a camera. Before run face detector app, expr

Devdreamsolution 2 Dec 4, 2021
Symmetry and Uncertainty-Aware Object SLAM for 6DoF Object Pose Estimation

SUO-SLAM This repository hosts the code for our CVPR 2022 paper "Symmetry and Uncertainty-Aware Object SLAM for 6DoF Object Pose Estimation". ArXiv li

Robot Perception & Navigation Group (RPNG) 97 Jan 3, 2023
A scikit-learn-compatible module for estimating prediction intervals.

|Anaconda|_ MAPIE - Model Agnostic Prediction Interval Estimator MAPIE allows you to easily estimate prediction intervals using your favourite sklearn

SimAI 584 Dec 27, 2022
Official codes: Self-Supervised Learning by Estimating Twin Class Distribution

TWIST: Self-Supervised Learning by Estimating Twin Class Distributions Codes and pretrained models for TWIST: @article{wang2021self, title={Self-Sup

Bytedance Inc. 85 Dec 15, 2022
Improving Object Detection by Estimating Bounding Box Quality Accurately

Improving Object Detection by Estimating Bounding Box Quality Accurately Abstrac

null 2 Apr 14, 2022
LQM - Improving Object Detection by Estimating Bounding Box Quality Accurately

Improving Object Detection by Estimating Bounding Box Quality Accurately Abstract Object detection aims to locate and classify object instances in ima

IM Lab., POSTECH 0 Sep 28, 2022
Aerial Single-View Depth Completion with Image-Guided Uncertainty Estimation (RA-L/ICRA 2020)

Aerial Depth Completion This work is described in the letter "Aerial Single-View Depth Completion with Image-Guided Uncertainty Estimation", by Lucas

ETHZ V4RL 70 Dec 22, 2022
A Weakly Supervised Amodal Segmenter with Boundary Uncertainty Estimation

Paper Khoi Nguyen, Sinisa Todorovic "A Weakly Supervised Amodal Segmenter with Boundary Uncertainty Estimation", accepted to ICCV 2021 Our code is mai

Khoi Nguyen 5 Aug 14, 2022
Face uncertainty quantification or estimation using PyTorch.

Face-uncertainty-pytorch This is a demo code of face uncertainty quantification or estimation using PyTorch. The uncertainty of face recognition is af

Kaen 3 Sep 16, 2022
Official code of "R2RNet: Low-light Image Enhancement via Real-low to Real-normal Network."

R2RNet Official code of "R2RNet: Low-light Image Enhancement via Real-low to Real-normal Network." Jiang Hai, Zhu Xuan, Ren Yang, Yutong Hao, Fengzhu

null 77 Dec 24, 2022
Normal Learning in Videos with Attention Prototype Network

Codes_APN Official codes of CVPR21 paper: Normal Learning in Videos with Attention Prototype Network (https://arxiv.org/abs/2108.11055) Overview of ou

null 11 Dec 13, 2022
An implementation of a discriminant function over a normal distribution to help classify datasets.

CS4044D Machine Learning Assignment 1 By Dev Sony, B180297CS The question, report and source code can be found here. Github Repo Solution 1 Based on t

Dev Sony 6 Nov 9, 2021
A simple rest api that classifies pneumonia infection weather it is Normal, Pneumonia Virus or Pneumonia Bacteria from a chest-x-ray image.

This is a simple rest api that classifies pneumonia infection weather it is Normal, Pneumonia Virus or Pneumonia Bacteria from a chest-x-ray image.

crispengari 3 Jan 8, 2022
Code and dataset for ACL2018 paper "Exploiting Document Knowledge for Aspect-level Sentiment Classification"

Aspect-level Sentiment Classification Code and dataset for ACL2018 [paper] ‘‘Exploiting Document Knowledge for Aspect-level Sentiment Classification’’

Ruidan He 146 Nov 29, 2022