This is the repo for Uncertainty Quantification 360 Toolkit.

Overview

UQ360

Build Status Documentation Status

The Uncertainty Quantification 360 (UQ360) toolkit is an open-source Python package that provides a diverse set of algorithms to quantify uncertainty, as well as capabilities to measure and improve UQ to streamline the development process. We provide a taxonomy and guidance for choosing these capabilities based on the user's needs. Further, UQ360 makes the communication method of UQ an integral part of development choices in an AI lifecycle. Developers can make a user-centered choice by following the psychology-based guidance on communicating UQ estimates, from concise descriptions to detailed visualizations.

The UQ360 interactive experience provides a gentle introduction to the concepts and capabilities by walking through an example use case. The tutorials and example notebooks offer a deeper, data scientist-oriented introduction. The complete API is also available.

We have developed the package with extensibility in mind. This library is still in development. We encourage the contribution of your uncertianty estimation algorithms, metrics and applications. To get started as a contributor, please join the #uq360-users or #uq360-developers channel of the AIF360 Community on Slack by requesting an invitation here.

Supported Uncertainty Evaluation Metrics

The toolbox provides several standard calibration metrics for classification and regression tasks. This includes Expected Calibration Error (Naeini et al., 2015), Brier Score (Brier, 1950), etc for classification models. Regression metrics include Prediction Interval Coverage Probability (PICP) Chatfield, 1993 and Mean Prediction Interval Width (MPIW) (Shrestha and Solomatine, 2006) among others. The toolbox also provides a novel operation-point agnostic approaches for the assessment of prediction uncertainty estimates called the Uncertainty Characteristic Curve (UCC) (Navratil et al., 2021). Several metrics and diagnosis tools such as reliability diagram (DeGroot and Fienberg, 1983) and risk-vs-rejection rate curves (El-Yaniv et al., 2010) are provides which also support analysis by sub-groups in the population to study fairness implications of acting on given uncertainty estimates.

Supported Uncertainty Estimation Algorithms

UQ algorithms can be broadly classified as intrinsic or extrinsic depending on how the uncertainties are obtained from the AI models. Intrinsic methods encompass models that inherently provides an uncertainty estimate along with its predictions. The toolkit includes algorithms such as variational Bayesian neural networks (BNNs) (Blundell et al., 2015), Gaussian processes (Rasmussen and Williams,2006), quantile regression (Koenker and Bassett, 1978) and hetero/homo-scedastic neuralnetworks (Wakefield, 2013) which are models that fall in this category. The toolkit also includes Horseshoe BNNs (Ghosh et al., 2019) that use sparsity promoting priors and can lead to better-calibrated uncertainties, especially in the small data regime. An Infinitesimal Jackknife (IJ) based algorithm (Ghosh et al., 2020)), provided in the toolkit, is a perturbation-based approach that perform uncertainty quantification by estimating model parameters under different perturbations of the original data. Crucially, here the estimation only requires the model to be trained once on the unperturbed dataset. For models that do not have an inherent notion of uncertainty built into them, extrinsic methods are employed to extract uncertainties post-hoc. The toolkit provides meta-models (Chen et al., 2019)that can be been used to successfully generate reliable confidence measures (in classification), prediction intervals (in regression), and to predict performance metrics such as accuracy on unseen and unlabeled data. For pre-trained models that captures uncertainties to some degree, the toolbox provides extrinsic algorithms that can improve the uncertainty estimation quality. This includes isotonic regression (Zadrozny and Elkan, 2001), Platt-scaling (Platt, 1999), auxiliary interval predictors (Thiagarajan et al., 2020), and UCC-Recalibration (Navratil et al., 2021).

Setup

Supported Configurations:

OS Python version
macOS 3.7
Ubuntu 3.7
Windows 3.7

(Optional) Create a virtual environment

A virtual environment manager is strongly recommended to ensure dependencies may be installed safely. If you have trouble installing the toolkit, try this first.

Conda

Conda is recommended for all configurations though Virtualenv is generally interchangeable for our purposes. Miniconda is sufficient (see the difference between Anaconda and Miniconda if you are curious) and can be installed from here if you do not already have it.

Then, to create a new Python 3.7 environment, run:

conda create --name uq360 python=3.7
conda activate uq360

The shell should now look like (uq360) $. To deactivate the environment, run:

(uq360)$ conda deactivate

The prompt will return back to $ or (base)$.

Note: Older versions of conda may use source activate uq360 and source deactivate (activate uq360 and deactivate on Windows).

Installation

Clone the latest version of this repository:

(uq360)$ git clone https://github.ibm.com/UQ360/UQ360

If you'd like to run the examples and tutorial notebooks, download the datasets now and place them in their respective folders as described in uq360/datasets/data/README.md.

Then, navigate to the root directory of the project which contains setup.py file and run:

(uq360)$ pip install -e .

PIP Installation of Uncertainty Quantification 360

If you would like to quickly start using the UQ360 toolkit without cloning this repository, then you can install the uq360 pypi package as follows.

(your environment)$ pip install uq360

If you follow this approach, you may need to download the notebooks in the examples folder separately.

Using UQ360

The examples directory contains a diverse collection of jupyter notebooks that use UQ360 in various ways. Both examples and tutorial notebooks illustrate working code using the toolkit. Tutorials provide additional discussion that walks the user through the various steps of the notebook. See the details about tutorials and examples here.

Citing UQ360

A technical description of UQ360 is available in this paper. Below is the bibtex entry for this paper.

@misc{uq360-june-2021,
      title={Uncertainty Quantification 360: A Holistic Toolkit for Quantifying 
      and Communicating the Uncertainty of AI}, 
      author={Soumya Ghosh and Q. Vera Liao and Karthikeyan Natesan Ramamurthy 
      and Jiri Navratil and Prasanna Sattigeri 
      and Kush R. Varshney and Yunfeng Zhang},
      year={2021},
      eprint={2106.01410},
      archivePrefix={arXiv},
      primaryClass={cs.AI}
}

Acknowledgements

UQ360 is built with the help of several open source packages. All of these are listed in setup.py and some of these include:

License Information

Please view both the LICENSE file present in the root directory for license information.

Comments
  • Latent metric anomaly detection

    Latent metric anomaly detection

    This is a remake of #24 with signed-off commits

    This was work in most part performed by @fcraighero while at IBM Research Zurich. It implements three algorithms for epistemic uncertainty estimation/anomaly detection based on proxies for data density in the latent space of neural networks.

    It provides the following new posthocUQ methods:

    • uq360.algorithms.layer_scoring.knn.KNN
    • uq360.algorithms.layer_scoring.aklpe.AKLPE
    • uq360.algorithms.layer_scoring.mahalanobis.Mahalanobis

    The first two methods are based on k-Nearest-Neighbor searches, for which we provide three solutions

    • uq360.utils.transformers.nearest_neighbors.exact.ExactNearestNeighbors based on sklearn
    • uq360.utils.transformers.nearest_neighbors.pynndescent.PyNNDNearestNeighbors based on pynndescent
    • uq360.utils.transformers.nearest_neighbors.faiss.FAISSNearestNeighbors based on faiss

    Still TODO: implementing tests (code has been checked of course though). Opening this PR to allow discussions.

    NB: Because I merged the repo IBM/DANTE into UQ360 to preserve history, adding the signoff required some gymnastics. I'm therefore using a second branch

    opened by ndeutschmann 7
  • Adding Test Cases for Calibration Classifier

    Adding Test Cases for Calibration Classifier

    The following PR added test cases for classification_calibration.py The test cases cover the following use cases

    1. The classifier runs in all fit_modes and method types
    2. Adding that there is improvement in Brier score after calibration and the predicted class is unchanged
    opened by adityajain93 4
  • Implementation for multiple black box / meta model based predictors (structured data, text)

    Implementation for multiple black box / meta model based predictors (structured data, text)

    In this PR, four predictors are moved from the performance predictors toolkit into UQ360.

    confidence based passthrough short text structured data We are also moving a bunch of calibrators, transformers, feature implementation, utils along with the predictors.

    contributors: @anupamamurthi @benjamin-elder @marnold6

    opened by anupamamurthi 4
  • Latent metric anomaly detection

    Latent metric anomaly detection

    This was work in most part performed by @fcraighero while at IBM Research Zurich. It implements three algorithms for epistemic uncertainty estimation/anomaly detection based on proxies for data density in the latent space of neural networks.

    It provides the following new posthocUQ methods:

    • uq360.algorithms.layer_scoring.knn.KNN
    • uq360.algorithms.layer_scoring.aklpe.AKLPE
    • uq360.algorithms.layer_scoring.mahalanobis.Mahalanobis

    The first two methods are based on k-Nearest-Neighbor searches, for which we provide three solutions

    • uq360.utils.transformers.nearest_neighbors.exact.ExactNearestNeighbors based on sklearn
    • uq360.utils.transformers.nearest_neighbors.pynndescent.PyNNDNearestNeighbors based on pynndescent
    • uq360.utils.transformers.nearest_neighbors.faiss.FAISSNearestNeighbors based on faiss

    Still TODO: implementing tests (code has been checked of course though). Opening this PR to allow discussions.

    opened by ndeutschmann 2
  • Multi-Class Brier Score implementation has issues

    Multi-Class Brier Score implementation has issues

    The implementation of the multi-class Brier score makes all the targets 1 instead of creating a one-hot vector for each sample. I changed the code to for i in range(len(y_true)):   y_target[i, y_true[i]] = 1.0

    instead of y_target[:, y_true] = 1.0

    opened by jwen307 1
  • fixes #14 multi-class brier score bug

    fixes #14 multi-class brier score bug

    • Fixes #14 multi-class brier score bug and a few other minor bugs.
    • Added a few more tests for calibration utilities.
    • Restricting sklearn version to <1.0.
    • Incrementing version to 0.2.
    opened by pronics2004 0
  • Adding active learning and ensemble of heteroscedastic models

    Adding active learning and ensemble of heteroscedastic models

    Adding algorithms: actively_learned_model ensemble_heteroscdastic_regression

    and their corresponding tutorials. A custom dataset is provided for the active learning tutorial with generated data for a PDE.

    opened by rpestourie 0
  • Clarify documentation for entropy based uncertainty decomposition

    Clarify documentation for entropy based uncertainty decomposition

    In https://github.com/IBM/UQ360/blob/cf02a8aa4242ec8403ec834576d8aa284cb964a2/uq360/metrics/classification_metrics.py#L7

    we need to clarify that y_prob_samples has multiple arrays of size n_samples x n_classes, each a sample from the posterior y | W, x

    opened by nrkarthikeyan 0
  • Make TensorFlow and PyTorch optional requirements

    Make TensorFlow and PyTorch optional requirements

    Both, PyTorch and TensorFlow, are large packages and not required for all parts of UQ360. Even more so, if someone just intends to use the metrics provided by UQ360, the requirements could introduce some conflicts, e.g., if the user requires TensorFlow>=2.6.

    Could we make both requirements optional, e.g., via extras_require{} in the setup.py? The downside would be, as far as I see it, that the default installation pip install uq360 would only install the minimal version and the full version would only be available via, e.g., pip install uq360[full].

    In case you'd find this feature useful, I'd be happy to create a merge request!

    opened by devdrian 1
  • Working GPU Example for demo_bnn_classification.ipynb

    Working GPU Example for demo_bnn_classification.ipynb

    Hallo guys!

    I just wanted to try out your example as mentioned in header. However I get the error

    "Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!"

    when running the fit method of the BnnClassification. Have you tried this example on a GPU machine? When I set device = torch.device("cpu"), it runs smoothly.

    I have taken a closer look in your implementions and found several places, where tensors are created without having the device in mind.

    Best regards

    JFK

    opened by janfelixklein 1
  • Support dataloader in BNNRegression

    Support dataloader in BNNRegression

    Update BNNRegression to support iterating over dataloader during fit similar to BNNClassification.

    https://github.com/IBM/UQ360/blob/main/uq360/algorithms/variational_bayesian_neural_networks/bnn.py#L54

    https://github.com/IBM/UQ360/blob/main/uq360/algorithms/variational_bayesian_neural_networks/bnn.py#L216

    opened by pronics2004 0
  • Assert inputs are the right type

    Assert inputs are the right type

    Assert the inputs are the correct type. There is no way for the end-user to know the right input type only by reading the documentation; therefore, code should speak by itself, asserting the correct type.

    opened by FeSens 1
Owner
International Business Machines
International Business Machines
A python toolbox for predictive uncertainty quantification, calibration, metrics, and visualization

Website, Tutorials, and Docs    Uncertainty Toolbox A python toolbox for predictive uncertainty quantification, calibration, metrics, and visualizatio

Uncertainty Toolbox 1.4k Dec 28, 2022
Face uncertainty quantification or estimation using PyTorch.

Face-uncertainty-pytorch This is a demo code of face uncertainty quantification or estimation using PyTorch. The uncertainty of face recognition is af

Kaen 3 Sep 16, 2022
TensorFlow implementation for Bayesian Modeling and Uncertainty Quantification for Learning to Optimize: What, Why, and How

Bayesian Modeling and Uncertainty Quantification for Learning to Optimize: What, Why, and How TensorFlow implementation for Bayesian Modeling and Unce

Shen Lab at Texas A&M University 8 Sep 2, 2022
The repo of the preprinting paper "Labels Are Not Perfect: Inferring Spatial Uncertainty in Object Detection"

Inferring Spatial Uncertainty in Object Detection A teaser version of the code for the paper Labels Are Not Perfect: Inferring Spatial Uncertainty in

ZINING WANG 21 Mar 3, 2022
Source code for "OmniPhotos: Casual 360° VR Photography"

OmniPhotos: Casual 360° VR Photography Project Page | Video | Paper | Demo | Data This repository contains the source code for creating and viewing Om

Christian Richardt 144 Dec 30, 2022
Office source code of paper UniFuse: Unidirectional Fusion for 360$^\circ$ Panorama Depth Estimation

UniFuse (RAL+ICRA2021) Office source code of paper UniFuse: Unidirectional Fusion for 360$^\circ$ Panorama Depth Estimation, arXiv, Demo Preparation I

Alibaba 47 Dec 26, 2022
Microsoft Cognitive Toolkit (CNTK), an open source deep-learning toolkit

CNTK Chat Windows build status Linux build status The Microsoft Cognitive Toolkit (https://cntk.ai) is a unified deep learning toolkit that describes

Microsoft 17.3k Dec 29, 2022
Microsoft Cognitive Toolkit (CNTK), an open source deep-learning toolkit

CNTK Chat Windows build status Linux build status The Microsoft Cognitive Toolkit (https://cntk.ai) is a unified deep learning toolkit that describes

Microsoft 17k Feb 11, 2021
git git《Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking》(CVPR 2021) GitHub:git2] 《Masksembles for Uncertainty Estimation》(CVPR 2021) GitHub:git3]

Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking Ning Wang, Wengang Zhou, Jie Wang, and Houqiang Li Accepted by CVPR

NingWang 236 Dec 22, 2022
Uncertainty-aware Semantic Segmentation of LiDAR Point Clouds for Autonomous Driving

SalsaNext: Fast, Uncertainty-aware Semantic Segmentation of LiDAR Point Clouds for Autonomous Driving Abstract In this paper, we introduce SalsaNext f

null 308 Jan 4, 2023
Aerial Single-View Depth Completion with Image-Guided Uncertainty Estimation (RA-L/ICRA 2020)

Aerial Depth Completion This work is described in the letter "Aerial Single-View Depth Completion with Image-Guided Uncertainty Estimation", by Lucas

ETHZ V4RL 70 Dec 22, 2022
Code for Deterministic Neural Networks with Appropriate Inductive Biases Capture Epistemic and Aleatoric Uncertainty

Deep Deterministic Uncertainty This repository contains the code for Deterministic Neural Networks with Appropriate Inductive Biases Capture Epistemic

Jishnu Mukhoti 69 Nov 28, 2022
noisy labels; missing labels; semi-supervised learning; entropy; uncertainty; robustness and generalisation.

ProSelfLC: CVPR 2021 ProSelfLC: Progressive Self Label Correction for Training Robust Deep Neural Networks For any specific discussion or potential fu

amos_xwang 57 Dec 4, 2022
Official PyTorch implementation of UACANet: Uncertainty Aware Context Attention for Polyp Segmentation

UACANet: Uncertainty Aware Context Attention for Polyp Segmentation Official pytorch implementation of UACANet: Uncertainty Aware Context Attention fo

Taehun Kim 85 Dec 14, 2022
[CVPR'21] MonoRUn: Monocular 3D Object Detection by Reconstruction and Uncertainty Propagation

MonoRUn MonoRUn: Monocular 3D Object Detection by Reconstruction and Uncertainty Propagation. CVPR 2021. [paper] Hansheng Chen, Yuyao Huang, Wei Tian*

 同济大学智能汽车研究所综合感知研究组 ( Comprehensive Perception Research Group under Institute of Intelligent Vehicles, School of Automotive Studies, Tongji University) 96 Dec 10, 2022
A library for uncertainty representation and training in neural networks.

Epistemic Neural Networks A library for uncertainty representation and training in neural networks. Introduction Many applications in deep learning re

DeepMind 211 Dec 12, 2022
the code for paper "Energy-Based Open-World Uncertainty Modeling for Confidence Calibration"

EOW-Softmax This code is for the paper "Energy-Based Open-World Uncertainty Modeling for Confidence Calibration". Accepted by ICCV21. Usage Commnd exa

Yezhen Wang 36 Dec 2, 2022
Sync2Gen Code for ICCV 2021 paper: Scene Synthesis via Uncertainty-Driven Attribute Synchronization

Sync2Gen Code for ICCV 2021 paper: Scene Synthesis via Uncertainty-Driven Attribute Synchronization 0. Environment Environment: python 3.6 and cuda 10

Haitao Yang 62 Dec 30, 2022
Estimating and Exploiting the Aleatoric Uncertainty in Surface Normal Estimation

Estimating and Exploiting the Aleatoric Uncertainty in Surface Normal Estimation

Bae, Gwangbin 95 Jan 4, 2023