Deep Probabilistic Programming Course @ DIKU

Overview

Syllabus

Part I - Introduction to Deep Probabilistic Programming

Week Topic Exercise Links
1 Introduction to Bayesian Inference Read Pattern Recognition and Machine Learning (PRML), Sections 1.1-1.3, 1.5-1.6 & 2-2.3.4 (inclusive ranges), Intro to Bayesian updating paper, and Pyro paper.

Form up groups and ask a question for each chapter/paper you have read.
Pattern Recognition and Machine Learning

Bayesian Updating Paper

Pyro Paper
2 Variational Inference Read the Variational Inference paper and Pyro tutorials on Stochastic Variational Inference (SVI). Ask three questions about them.

Use Pyro’s Variational Inference support to implement the kidney cancer model. See worksheet and Bayesian Data Analysis 3rd Edition (BDA3) Section 2.7.
Variational Inference Paper

Worksheet

Bayesian Data Analysis

Pyro SVI tutorial: Part I and Part II

Pyro Website
3 Hamiltonian Monte Carlo Read paper on Hamiltonian Monte Carlo and blog post on gradient-based Markov Chain Monte Carlo (MCMC). Look at the source code for Mini-MC.

Ask a question each for HMC, the Mini-MC implementation and discrete variable marginalization.

Implement Bayesian Change-point model in Pyro using NUTS.
Hamiltonian Monte Carlo Paper

Gradient-based MCMC

Mini-MC implementation

Change-point model

Pyro NUTS Example
4 Hidden Markov Models and Discrete Variables. Read Paper on Hidden Markov Models and ask three questions about it.

Read Pyro tutorials on Discrete Variables and Gaussian Mixture Models.

Read Pyro Hidden Markov Model code example and describe in your own words what the different models do.

Add amino acid prediction output to the TorusDBN HMM and show that the posterior predictive distribution of the amino acids matches the one found in data.
Hidden Markov Models

Pyro Discrete Variables Tutorial

Pyro Gaussian Mixture Model Tutorial

Pyro Hidden Markov Model Example

TorusDBN

Optional: Epidemological Inference via HMC
5 Bayesian Regression Models Read PRML Chapter 3 on Linear Models.

Ask 3 questions about the chapter.

Read the Pyro tutorials on Bayesian Regression.

Solve the weather check exercise in the worksheet.
Pyro Bayesian Regression: Part I, Part II

Worksheet
6 Variational Auto-Encoders Read Variational Auto Encoders (VAE) foundations Chapters 1 & 2, and Pyro tutorial on VAE. Ask three questions about the paper and tutorial.

Implement Frey Faces model from VAE paper in Pyro. Rely on the existing VAE implementation (see tutorial link).
Variational Auto Encoders Foundations

Pyro Tutorial on VAE
7 Deep Generative Models Read one of these papers: Interpretable Representation VAE, Causal Effect VAE, Deep Markov Model or DRAW (one paper per group).

Try out the relevant Pyro or PyTorch implementation on your choice of relevant dataset which was not used in the paper.

Make a small (10 minute) presentation about the paper and your experiences with the implementation.
Deep Markov Model

Interpretable Representation VAE

Causal Effect VAE

DRAW

Part II - Deep Probabilistic Programming Project

The second part of the course concerns applying the techniques learned in the first part, as a project solving a practical problem. We have several types of projects depending on the interests of the student.

For those interested in boosting their CV and potentially getting a student job, we warmly recommend working with one of our industrial partners on a suitable probabilistic programming project. For those interested in working with a more academic-oriented project, we have different interesting problems in Computer Science and Biology.

Industrial Projects

Company Area Ideas
 Relion Shift-planning AI Shift planning based on worker availability, historical sales data, weather and holidays.

Employee satisfaction quantification based on previously assigned shifts.

Employee shift assignment based on wishes and need
Paperflow Invoice Recognition AI Talk to us
Hypefactors Media and Reputation Tracking AI Talk to us
‹Your Company› ‹Your Area› Interested in collaboration with our group? contact Ahmad Salim to hear more!

Academic Projects

Type Description Notes/Links
Computer Science Implement a predictive scoring model for your favourite sports game, inspired by FiveThirtyEight. FiveThirtyEight Methodology and Models
Computer Science  Implement a ranking system for your favourite video or board game, inspired by Microsoft TrueSkill. Microsoft TrueSkill Model

Can be implemented in Infer.NET using Expectation Propagation
Computer Science Use Inference Compilation in PyProb to implement a CAPTCHA breaker or a Spaceship Generator Inference Compilation and PyProb. You can use the experimental PyProb bindings for Java.

CAPTCHA breaking with Oxford CAPTCHA Generator.

Spaceship Generator
Computer Science Implement asterisk corrector suggested by XKCD XKCD #2337: Asterisk Correction
Computer Science Implement an inference compilation based program-testing tool like QuickCheck or SmallCheck Inference Compilation

QuickCheck

SmallCheck
Computer Science Magic: The Gathering, Automated Deck Construction. Design a model that finds a good deck automatically based on correlations in existing deck design. Ideas like card substitution models could be integrated too. Magic: The Gathering - Meta Site
Computer Science Use probabilistic programming to explore ideas for solving Eternity II (No $2 million prize anymore, but still interesting from a math point of view) Eternity II
Biology Auto-Encoders or Deep Markov Models for Protein Folding Deep Markov Model

Pyro Deep Markov Model
Biology Inference Compilation for Ancestral Reconstruction Inference Compilation and PyProb. Talk to us for details.
Biology Recurrent Causal Effect VAE for modelling mutations in proteins Talk to us for details.

Recommendations

  • Sometimes sampling can be slow on the CPU for complex models, so try using Google Colab and GPUs and see if that provides an improvement.

Acknowledgements

This course has been developed by Thomas Hamelryck and Ahmad Salim Al-Sibahi. Thanks to Ola Rønning for suggesting the Variational Auto Encoders Foundations paper instead of Auto-Encoding Variational Bayes which we originally proposed to read on week 3. Thanks to Richard Michael for testing out the course and provide us with valuable feedback on the content!

You might also like...
Repository of Jupyter notebook tutorials for teaching the Deep Learning Course at the University of Amsterdam (MSc AI), Fall 2020

Repository of Jupyter notebook tutorials for teaching the Deep Learning Course at the University of Amsterdam (MSc AI), Fall 2020

Computer Vision Script to recognize first person motion, developed as final project for the course
Computer Vision Script to recognize first person motion, developed as final project for the course "Machine Learning and Deep Learning"

Overview of The Code BaseColab/MLDL_FPAR.pdf: it contains the full explanation of our work Base Colab: it contains the base colab used to perform all

Fast, flexible and easy to use probabilistic modelling in Python.
Fast, flexible and easy to use probabilistic modelling in Python.

Please consider citing the JMLR-MLOSS Manuscript if you've used pomegranate in your academic work! pomegranate is a package for building probabilistic

Supervised domain-agnostic prediction framework for probabilistic modelling
Supervised domain-agnostic prediction framework for probabilistic modelling

A supervised domain-agnostic framework that allows for probabilistic modelling, namely the prediction of probability distributions for individual data

NU-Wave: A Diffusion Probabilistic Model for Neural Audio Upsampling
NU-Wave: A Diffusion Probabilistic Model for Neural Audio Upsampling

NU-Wave: A Diffusion Probabilistic Model for Neural Audio Upsampling For Official repo of NU-Wave: A Diffusion Probabilistic Model for Neural Audio Up

Symbolic Parallel Adaptive Importance Sampling for Probabilistic Program Analysis in JAX

SYMPAIS: Symbolic Parallel Adaptive Importance Sampling for Probabilistic Program Analysis Overview | Installation | Documentation | Examples | Notebo

Adversarial Attacks on Probabilistic Autoregressive Forecasting Models.

Attack-Probabilistic-Models This is the source code for Adversarial Attacks on Probabilistic Autoregressive Forecasting Models. This repository contai

Probabilistic Gradient Boosting Machines
Probabilistic Gradient Boosting Machines

PGBM Probabilistic Gradient Boosting Machines (PGBM) is a probabilistic gradient boosting framework in Python based on PyTorch/Numba, developed by Air

Official PyTorch implementation for FastDPM, a fast sampling algorithm for diffusion probabilistic models

Official PyTorch implementation for "On Fast Sampling of Diffusion Probabilistic Models". FastDPM generation on CIFAR-10, CelebA, and LSUN datasets. S

Owner
null
DeepProbLog is an extension of ProbLog that integrates Probabilistic Logic Programming with deep learning by introducing the neural predicate.

DeepProbLog DeepProbLog is an extension of ProbLog that integrates Probabilistic Logic Programming with deep learning by introducing the neural predic

KU Leuven Machine Learning Research Group 94 Dec 18, 2022
All course materials for the Zero to Mastery Deep Learning with TensorFlow course.

All course materials for the Zero to Mastery Deep Learning with TensorFlow course.

Daniel Bourke 3.4k Jan 7, 2023
Implementation of the final project of the course DDA6309 Probabilistic Graphical Model

Task-aware Joint CWS and POS (TCwsPos) This is the implementation of the final project of the course DDA6309 Probabilistic Graphical Models, The Chine

Peng 1 Dec 26, 2021
aka "Bayesian Methods for Hackers": An introduction to Bayesian methods + probabilistic programming with a computation/understanding-first, mathematics-second point of view. All in pure Python ;)

Bayesian Methods for Hackers Using Python and PyMC The Bayesian method is the natural approach to inference, yet it is hidden from readers behind chap

Cameron Davidson-Pilon 25.1k Jan 2, 2023
Probabilistic Programming and Statistical Inference in PyTorch

PtStat Probabilistic Programming and Statistical Inference in PyTorch. Introduction This project is being developed during my time at Cogent Labs. The

Stefano Peluchetti 109 Nov 26, 2022
Modular Probabilistic Programming on MXNet

MXFusion | | | | Tutorials | Documentation | Contribution Guide MXFusion is a modular deep probabilistic programming library. With MXFusion Modules yo

Amazon 100 Dec 10, 2022
PClean: A Domain-Specific Probabilistic Programming Language for Bayesian Data Cleaning

PClean: A Domain-Specific Probabilistic Programming Language for Bayesian Data Cleaning Warning: This is a rapidly evolving research prototype.

MIT Probabilistic Computing Project 190 Dec 27, 2022
InferPy: Deep Probabilistic Modeling with Tensorflow Made Easy

InferPy: Deep Probabilistic Modeling Made Easy InferPy is a high-level API for probabilistic modeling written in Python and capable of running on top

PGM-Lab 141 Oct 13, 2022
Registration Loss Learning for Deep Probabilistic Point Set Registration

RLLReg This repository contains a Pytorch implementation of the point set registration method RLLReg. Details about the method can be found in the 3DV

Felix Järemo Lawin 35 Nov 2, 2022
A Python library for Deep Probabilistic Modeling

Abstract DeeProb-kit is a Python library that implements deep probabilistic models such as various kinds of Sum-Product Networks, Normalizing Flows an

DeeProb-org 46 Dec 26, 2022