Posterior predictive distributions quantify uncertainties ignored by point estimates.

Overview

The Neural Testbed

Neural Testbed Logo

Introduction

Posterior predictive distributions quantify uncertainties ignored by point estimates. The neural_testbed provides tools for the systematic evaluation of agents that generate such predictions. Crucially, these tools assess not only the quality of marginal predictions per input, but also joint predictions given many inputs. Joint distributions are often critical for useful uncertainty quantification, but they have been largely overlooked by the Bayesian deep learning community.

This library automates the evaluation and analysis of learning agents:

  • Synthetic neural-network-based generative model.
  • Evaluate predictions beyond marginal distributions.
  • Reference implementations of benchmark agents (with tuning).

For a more comprehensive overview, see the accompanying paper.

Technical overview

We outline the key high-level interfaces for our code in base.py:

  • EpistemicSampler: Generates a random sample from agent's predictive distribution.
  • TestbedAgent: Given data, prior_knowledge outputs an EpistemicSampler.
  • TestbedProblem: Reveals training_data, prior_knowledge. Can evaluate the quality of an EpistemicSampler.

If you want to evaluate your algorithm on the testbed, you simply need to define your TestbedAgent and then run it on our experiment.py

def run(agent: testbed_base.TestbedAgent,
        problem: testbed_base.TestbedProblem) -> testbed_base.ENNQuality:
  """Run an agent on a given testbed problem."""
  enn_sampler = agent(problem.train_data, problem.prior_knowledge)
  return problem.evaluate_quality(enn_sampler)

The neural_testbed takes care of the evaluation/logging within the TestbedProblem. This means that the experiment will automatically output data in the correct format. This makes it easy to compare results from different codebases/frameworks, so you can focus on agent design.

How do I get started?

If you are new to neural_testbed you can get started in our colab tutorial. This Jupyter notebook is hosted with a free cloud server, so you can start coding right away without installing anything on your machine. After this, you can follow the instructions below to get neural_testbed running on your local machine:

Installation

We have tested neural_testbed on Python 3.7. To install the dependencies:

  1. Optional: We recommend using a Python virtual environment to manage your dependencies, so as not to clobber your system installation:

    python3 -m venv neural_testbed
    source neural_testbed/bin/activate
    pip install --upgrade pip setuptools
  2. Install neural_testbed directly from github:

    git clone https://github.com/deepmind/neural_testbed.git
    cd neural_testbed
    pip install .
  3. Optional: run the tests by executing ./test.sh from the neural_testbed main directory.

Baseline agents

In addition to our testbed code, we release a collection of benchmark agents. These include the full sets of hyperparameter sweeps necessary to reproduce the paper's results, and can serve as a great starting point for new research. You can have a look at these implementations in the agents/factories/ folder.

We recommended you get started with our colab tutorial. After intallation you can also run an agent directly by executing the following command from the main directory of neural_testbed:

python -m neural_testbed.experiments.run --agent_name=mlp

By default, this will save the results for that agent to csv at /tmp/neural_testbed. You can control these options by flags in the run file. In particular, you can run the agent on the whole sweep of tasks in the Neural Testbed by specifying the flag --problem_id=SWEEP.

Citing

If you use neural_testbed in your work, please cite the accompanying paper:

@misc{osband2021evaluating,
      title={Evaluating Predictive Distributions: Does Bayesian Deep Learning Work?},
      author={Ian Osband and Zheng Wen and Seyed Mohammad Asghari and Vikranth Dwaracherla and Botao Hao and Morteza Ibrahimi and Dieterich Lawson and Xiuyuan Lu and Brendan O'Donoghue and Benjamin Van Roy},
      year={2021},
      eprint={2110.04629},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}
You might also like...
Style-based Point Generator with Adversarial Rendering for Point Cloud Completion (CVPR 2021)
Style-based Point Generator with Adversarial Rendering for Point Cloud Completion (CVPR 2021)

Style-based Point Generator with Adversarial Rendering for Point Cloud Completion (CVPR 2021) An efficient PyTorch library for Point Cloud Completion.

Implementation of the
Implementation of the "PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences" paper.

PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences Introduction Point cloud sequences are irregular and unordered in the spatial dimen

Code for
Code for "PV-RAFT: Point-Voxel Correlation Fields for Scene Flow Estimation of Point Clouds", CVPR 2021

PV-RAFT This repository contains the PyTorch implementation for paper "PV-RAFT: Point-Voxel Correlation Fields for Scene Flow Estimation of Point Clou

Implementation of the
Implementation of the "Point 4D Transformer Networks for Spatio-Temporal Modeling in Point Cloud Videos" paper.

Point 4D Transformer Networks for Spatio-Temporal Modeling in Point Cloud Videos Introduction Point cloud videos exhibit irregularities and lack of or

Point Cloud Denoising input segmentation output raw point-cloud valid/clear fog rain de-noised Abstract Lidar sensors are frequently used in environme
Point Cloud Denoising input segmentation output raw point-cloud valid/clear fog rain de-noised Abstract Lidar sensors are frequently used in environme

Point Cloud Denoising input segmentation output raw point-cloud valid/clear fog rain de-noised Abstract Lidar sensors are frequently used in environme

Synthetic LiDAR sequential point cloud dataset with point-wise annotations
Synthetic LiDAR sequential point cloud dataset with point-wise annotations

SynLiDAR dataset: Learning From Synthetic LiDAR Sequential Point Cloud This is official repository of the SynLiDAR dataset. For technical details, ple

[ICCV 2021 Oral] SnowflakeNet: Point Cloud Completion by Snowflake Point Deconvolution with Skip-Transformer
[ICCV 2021 Oral] SnowflakeNet: Point Cloud Completion by Snowflake Point Deconvolution with Skip-Transformer

This repository contains the source code for the paper SnowflakeNet: Point Cloud Completion by Snowflake Point Deconvolution with Skip-Transformer (ICCV 2021 Oral). The project page is here.

Unofficial implementation of Point-Unet: A Context-Aware Point-Based Neural Network for Volumetric Segmentation

Point-Unet This is an unofficial implementation of the MICCAI 2021 paper Point-Unet: A Context-Aware Point-Based Neural Network for Volumetric Segment

Point-NeRF: Point-based Neural Radiance Fields
Point-NeRF: Point-based Neural Radiance Fields

Point-NeRF: Point-based Neural Radiance Fields Project Sites | Paper | Primary c

Comments
  • Very Slow

    Very Slow

    Hi There,

    For me running evaluations is very slow. I use a GPU, but still evaluating the baseline:uniform_class_probs on leaderboard.CLASSIFICATION_2D_TEST takes very long. At least 20 minutes. I never finished the run, not even the first test.

    Any idea?

    opened by SamuelGabriel 3
  • Could KL on the joint distribution serve as an alternative loss?

    Could KL on the joint distribution serve as an alternative loss?

    This question is paper related, not sure where else to ask it and hope for a response so here it goes.

    One observation from the paper is that, apparently, marginal performance is not necessarily correlated with joint performance.

    That begs the question, what if we used KL on the joint distribution as an alternative loss during training? What would change in marginal and joint evaluation? If I'm not mistaken, currently, we mostly evaluate classification agents by batching a number of marginal losses at a time.

    opened by snovaisg 2
  • Dependencies

    Dependencies

    Hi There,

    Thanks for open-sourcing! The dependencies are really hard to reconstruct as it seems that Jax is changing its API quite heavily between minor releases.

    How to reproduce the bug: Follow your installation manual (in my case with Python 3.9). It will install Jax 0.2 which does not work with your code :( I got it to work by changing the Jax version to 0.1.71 on the CPU but for CUDA I didn't have luck yet.

    (Sorry for the tone wrote this after trying to get the dependencies right for half an hour..)

    opened by SamuelGabriel 4
Owner
DeepMind
DeepMind
Implementation of CVPR'2022:Surface Reconstruction from Point Clouds by Learning Predictive Context Priors

Surface Reconstruction from Point Clouds by Learning Predictive Context Priors (CVPR 2022) Personal Web Pages | Paper | Project Page This repository c

null 136 Dec 12, 2022
Code for Quantifying Ignorance in Individual-Level Causal-Effect Estimates under Hidden Confounding

?? quince Code for Quantifying Ignorance in Individual-Level Causal-Effect Estimates under Hidden Confounding ?? Installation $ git clone [email protected]

Andrew Jesson 19 Jun 23, 2022
A real-time motion capture system that estimates poses and global translations using only 6 inertial measurement units

TransPose Code for our SIGGRAPH 2021 paper "TransPose: Real-time 3D Human Translation and Pose Estimation with Six Inertial Sensors". This repository

Xinyu Yi 261 Dec 31, 2022
Pytorch implementation of Generative Models as Distributions of Functions 🌿

Generative Models as Distributions of Functions This repo contains code to reproduce all experiments in Generative Models as Distributions of Function

Emilien Dupont 117 Dec 29, 2022
CVPR '21: In the light of feature distributions: Moment matching for Neural Style Transfer

In the light of feature distributions: Moment matching for Neural Style Transfer (CVPR 2021) This repository provides code to recreate results present

Nikolai Kalischek 49 Oct 13, 2022
HiddenMarkovModel implements hidden Markov models with Gaussian mixtures as distributions on top of TensorFlow

Class HiddenMarkovModel HiddenMarkovModel implements hidden Markov models with Gaussian mixtures as distributions on top of TensorFlow 2.0 Installatio

Susara Thenuwara 2 Nov 3, 2021
Implicit MLE: Backpropagating Through Discrete Exponential Family Distributions

torch-imle Concise and self-contained PyTorch library implementing the I-MLE gradient estimator proposed in our NeurIPS 2021 paper Implicit MLE: Backp

UCL Natural Language Processing 249 Jan 3, 2023
Universal Probability Distributions with Optimal Transport and Convex Optimization

Sylvester normalizing flows for variational inference Pytorch implementation of Sylvester normalizing flows, based on our paper: Sylvester normalizing

Rianne van den Berg 172 Dec 13, 2022
Mapping Conditional Distributions for Domain Adaptation Under Generalized Target Shift

This repository contains the official code of OSTAR in "Mapping Conditional Distributions for Domain Adaptation Under Generalized Target Shift" (ICLR 2022).

Matthieu Kirchmeyer 5 Dec 6, 2022
SurfEmb (CVPR 2022) - SurfEmb: Dense and Continuous Correspondence Distributions

SurfEmb SurfEmb: Dense and Continuous Correspondence Distributions for Object Pose Estimation with Learnt Surface Embeddings Rasmus Laurvig Haugard, A

Rasmus Haugaard 56 Nov 19, 2022