A probabilistic programming library for Bayesian deep learning, generative models, based on Tensorflow

Overview

Build Status Doc Status License Join the chat at https://gitter.im/thu-ml/zhusuan

ZhuSuan is a Python probabilistic programming library for Bayesian deep learning, which conjoins the complimentary advantages of Bayesian methods and deep learning. ZhuSuan is built upon TensorFlow. Unlike existing deep learning libraries, which are mainly designed for deterministic neural networks and supervised tasks, ZhuSuan provides deep learning style primitives and algorithms for building probabilistic models and applying Bayesian inference. The supported inference algorithms include:

  • Variational Inference (VI) with programmable variational posteriors, various objectives and advanced gradient estimators (SGVB, REINFORCE, VIMCO, etc.).

  • Importance Sampling (IS) for learning and evaluating models, with programmable proposals.

  • Hamiltonian Monte Carlo (HMC) with parallel chains, and optional automatic parameter tuning.

  • Stochastic Gradient Markov Chain Monte Carlo (SGMCMC): SGLD, PSGLD, SGHMC, and SGNHT.

Installation

ZhuSuan is still under development. Before the first stable release (1.0), please clone the repository and run

pip install .

in the main directory. This will install ZhuSuan and its dependencies automatically. ZhuSuan also requires TensorFlow 1.13.0 or later. Because users should choose whether to install the cpu or gpu version of TensorFlow, we do not include it in the dependencies. See Installing TensorFlow.

If you are developing ZhuSuan, you may want to install in an "editable" or "develop" mode. Please refer to the Contributing section below.

Documentation

Examples

We provide examples on traditional hierarchical Bayesian models and recent deep generative models.

To run the provided examples, you may need extra dependencies to be installed. This can be done by

pip install ".[examples]"
  • Gaussian: HMC
  • Toy 2D Intractable Posterior: SGVB
  • Bayesian Neural Networks: SGVB, SGMCMC
  • Variational Autoencoder (VAE): SGVB, IWAE
  • Convolutional VAE: SGVB
  • Semi-supervised VAE (Kingma, 2014): SGVB, Adaptive IS
  • Deep Sigmoid Belief Networks Adaptive IS, VIMCO
  • Logistic Normal Topic Model: HMC
  • Probabilistic Matrix Factorization: HMC
  • Sparse Variational Gaussian Process: SGVB

Citing ZhuSuan

If you find ZhuSuan useful, please cite it in your publications. We provide a BibTeX entry of the ZhuSuan white paper below.

@ARTICLE{zhusuan2017,
    title={Zhu{S}uan: A Library for {B}ayesian Deep Learning},
    author={Shi, Jiaxin and Chen, Jianfei. and Zhu, Jun and Sun, Shengyang
    and Luo, Yucen and Gu, Yihong and Zhou, Yuhao},
    journal={arXiv preprint arXiv:1709.05870},
    year=2017,
}

Contributing

We always welcome contributions to help make ZhuSuan better. If you would like to contribute, please check out the guidelines here.

Comments
  • Abstraction of empirical distribution.

    Abstraction of empirical distribution.

    I know this might be a bit strange, but I think it would be useful at least to have an option of an empirical distribution. This would make for instance specifying the "inference" networks more symmetric and include more clearly what the full graphical model is. This will also make it consistent to pass the x as an observed variable when we query for the log-probabilities rather than when we build model. This additionally makes the way of building the forward and backward models having the exact same signatures. Taking the example:

    @zs.reuse('model')
    def vae(observed, x_dim, z_dim, n_x, n_z_per_x):
        with zs.BayesianNet(observed=observed) as model:
            z_mean = tf.zeros([n_x, z_dim])
            z = zs.Normal('z', z_mean, std=1., group_ndims=1, n_samples=n_z_per_x)
            lx_z = layers.fully_connected(z, 500)
            lx_z = layers.fully_connected(lx_z, 500)
            x_logits = layers.fully_connected(lx_z, x_dim, activation_fn=None)
            x = zs.Bernoulli('x', x_logits, group_ndims=1)
        return model, x_logits
    
    @zs.reuse('variational')
    def q_net(observed, x_dim, z_dim, n_x, n_z_per_x):
        with zs.BayesianNet(observed=observed) as variational:
            x = zs.Empirical('x', (n_x, x_dim), dtype=tf.int32)
            lz_x = layers.fully_connected(tf.to_float(x), 500)
            lz_x = layers.fully_connected(lz_x, 500)
            z_mean = layers.fully_connected(lz_x, z_dim, activation_fn=None)
            z_logstd = layers.fully_connected(lz_x, z_dim, activation_fn=None)
            z = zs.Normal('z', z_mean, logstd=z_logstd, group_ndims=1,
                          n_samples=n_z_per_x)
        return variational
    
    opened by botev 29
  • pairwise Markov random field

    pairwise Markov random field

    dear authors, thanks for this package. I am wondering if Markov random field can be implemented using zhusuan, is so, do you have a tutorial for me to go through? thank you.

    opened by DanqingZ 13
  • Can't compute prior (local_log_prob) of a StochasticTensor inside tf.scan (in LSTM cell)

    Can't compute prior (local_log_prob) of a StochasticTensor inside tf.scan (in LSTM cell)

    Hi,

    I tried to implement the bayesian_rnn from the docs. However, while trying to compute log_joint, I can't compute the log of prior log_pz because w is declared within a LSTM cell, so I get the following error:

    Traceback (most recent call last):
      File "blstm.py", line 111, in <module>
        joint_ll = log_joint({'x': x, 'y_i': y_i, 'y_v': y_v})
      File "blstm.py", line 106, in log_joint
        log_pz, log_px_z = model.local_log_prob(['w', 'y_v'])  # Error
      File "/Users/jilljenn/code/vae/venv/lib/python3.6/site-packages/zhusuan/model/base.py", line 346, in local_log_prob
        ret.append(s_tensor.log_prob(s_tensor.tensor))
      File "/Users/jilljenn/code/vae/venv/lib/python3.6/site-packages/zhusuan/model/base.py", line 140, in log_prob
        return self._distribution.log_prob(given)
      File "/Users/jilljenn/code/vae/venv/lib/python3.6/site-packages/zhusuan/utils.py", line 215, in _func
        return f(*args, **kwargs)
      File "/Users/jilljenn/code/vae/venv/lib/python3.6/site-packages/zhusuan/distributions/base.py", line 303, in log_prob
        log_p = self._log_prob(given)
      File "/Users/jilljenn/code/vae/venv/lib/python3.6/site-packages/zhusuan/distributions/univariate.py", line 180, in _log_prob
        return c - logstd - 0.5 * precision * tf.square(given - mean)
      File "/Users/jilljenn/code/vae/venv/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py", line 979, in binary_op_wrapper
        return func(x, y, name=name)
      File "/Users/jilljenn/code/vae/venv/lib/python3.6/site-packages/tensorflow/python/ops/gen_math_ops.py", line 8009, in sub
        "Sub", x=x, y=y, name=name)
      File "/Users/jilljenn/code/vae/venv/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
        op_def=op_def)
      File "/Users/jilljenn/code/vae/venv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3392, in create_op
        op_def=op_def)
      File "/Users/jilljenn/code/vae/venv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1760, in __init__
        self._control_flow_post_processing()
      File "/Users/jilljenn/code/vae/venv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1769, in _control_flow_post_processing
        control_flow_util.CheckInputFromValidContext(self, input_tensor.op)
      File "/Users/jilljenn/code/vae/venv/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_util.py", line 263, in CheckInputFromValidContext
        raise ValueError(error_msg + " See info log for more details.")
    ValueError: Cannot use 'model/scan/while/MatMul/Normal.sample/Squeeze' as input to 'Normal.log_prob/sub_1' because 'model/scan/while/MatMul/Normal.sample/Squeeze' is in a while loop. See info log for more details.
    

    Up-to-run code: https://github.com/jilljenn/vae/blob/master/blstm.py#L106

    How to make it?

    opened by jilljenn 11
  • Add sparse GP example

    Add sparse GP example

    This PR implements the variational sparse gaussian process, formulated as in, e.g., section 2.1 of this paper, except that I did not analytically marginalize the inducing point values ((6) in the reference), so that the derivation is simpler, and non-conjugate variational distribution can be used in the future.

    On the UCI Boston and the newly added protein datasets, performance is on par with reports in most literature.

    opened by meta-inf 10
  • Why the std of y_mean is so small?

    Why the std of y_mean is so small?

    Hi, Thank u for your great share first! And when i test the examples\bayesian_neural_nets\bayesian_nn.py, I found that the std of y_mean what i want to predict is too small. As the same data ,when i tested on pyro of torch, the std looks good.

    And i read the code, i found that the y_mean is counted by likelihood function, as well as it should be count by q(w)?

    opened by 28huasheng 7
  • A problem on zs.Concrete

    A problem on zs.Concrete

    ` import tensorflow as tf import zhusuan as zs

    logits = tf.zeros([10, 10]) temparature = 1. labels = zs.Concrete('y', logits=logits, temperature=temparature, n_samples=10, group_ndims=1) prob = labels.log_prob(tf.one_hot(range(10), 10)) print (prob.shape) ` seems only produce 1 tensor with shape ()

    opened by rtz19970824 6
  • Fix `assert_same_dtype`

    Fix `assert_same_dtype`

    In tf there are derived dtypes for reference-based tensors. I.e. Variable().dtype == tf.int32_ref != tf.int32. zs.distributions.utils.assert_same_dtype is invoked whenever a distribution with multiple parameters is constructed, but it did not consider this case. Thus the following code would fail:

    a = tf.Variable(..., dtype=tf.float32) # dtype=tf.float32_ref
    b = <some float32 op>
    dist = zs.MultivariateNormalTriL(.., mean=a, cov_tril=b)  # Invoked assert_same_dtype(mean, cov_tril)
    

    The solution is to use DType.base_dtype.

    opened by meta-inf 6
  • Computation of gradient w.r.t. Distribution parameters

    Computation of gradient w.r.t. Distribution parameters

    I have detailedly studied the code under the folders zhusuan/distributions and zhusuan/model. Yet, I am still confused by how the gradients are computed by Tensorflow. Since a number of samples are used to caluated the cost at each optimization Op, I am not sure why this is reversible when taking the gradients w.r.t. Distribution parameters (e.g. mean and log_std in Normal), how this is handled by Tensorflow. It is much appreciated if you can suggest me some materials which I have missed to read.

    opened by MaxInGaussian 6
  • No module named examples

    No module named examples

    When I run the examples, such as python zhusuan/examples/bayesian_neural_nets/python bayesian_nn.py, I will get the error "ImportError: No module named examples". Please help me. Thank you.

    enhancement 
    opened by rzhangpku 6
  • How to use custom Hamiltonian?

    How to use custom Hamiltonian?

    Hi.

    The parameters of my model have no constraint. I implemented the following two functions and want to use HMC algorithm of zhusuan. How to do that? If I can't use the function directly, how to implement custom StochasticTensor?

    def cal_log_likelihood(p1, p2):
        """Calculate log-likelihood of the model
    
        Parameters
        ----------
        p1 : tf.Tensor, shape=(n_chains, n_states)
        p2 : tf.Tensor, shape=(n_chains, n_states, n_states)
    
        Returns
        -------    
        log_likelihood : tf.Tensor, shape=(n_chains,)
        """
        return log_likelihood
    
    def cal_prior(p1, p2):
        """Calculate logarithm of the prior probability
    
        Parameters
        ----------
        p1 : tf.Tensor, shape=(n_chains, n_states)
        p2 : tf.Tensor, shape=(n_chains, n_states, n_states)
    
        Returns
        -------    
        log_prior_probability : tf.Tensor, shape=(n_chains,)
        """
        return log_prior_probability
    

    Thank you.

    opened by rzu512 5
  • Collaboration with TensorLayer

    Collaboration with TensorLayer

    Hello dear friends,

    I have been reading your work with much interest. With @zsdonghao @luomai @lgarithm we maintain the library TensorLayer (TL), https://github.com/tensorlayer/tensorlayer.

    I think that your work can be really complementary to TL APIs.

    Maybe you would be interested in working together and heading toward a common direction.

    We have a chat group on Slack and/or WeChat if you wanna talk.

    Have a nice day,

    All the best,

    Jonathan DEKHTIAR

    ongoing work 
    opened by DEKHTIARJonathan 5
  • CVE-2007-4559 Patch

    CVE-2007-4559 Patch

    Patching CVE-2007-4559

    Hi, we are security researchers from the Advanced Research Center at Trellix. We have began a campaign to patch a widespread bug named CVE-2007-4559. CVE-2007-4559 is a 15 year old bug in the Python tarfile package. By using extract() or extractall() on a tarfile object without sanitizing input, a maliciously crafted .tar file could perform a directory path traversal attack. We found at least one unsantized extractall() in your codebase and are providing a patch for you via pull request. The patch essentially checks to see if all tarfile members will be extracted safely and throws an exception otherwise. We encourage you to use this patch or your own solution to secure against CVE-2007-4559. Further technical information about the vulnerability can be found in this blog.

    If you have further questions you may contact us through this projects lead researcher Kasimir Schulz.

    opened by TrellixVulnTeam 0
  • Examples code is out dated and doesn't work with Tensorflow 2.x

    Examples code is out dated and doesn't work with Tensorflow 2.x

    Most of the imports used in the examples are from Tensorflow 1.x and are not compatible with Tensorflow 2.x...

    Examples of faulty imports include, TensorFlow.contrib (no longer exists in TF 2.0+)....

    opened by usamazf 2
  • AttributeError: module 'tensorflow' has no attribute 'log'

    AttributeError: module 'tensorflow' has no attribute 'log'

    Hi, I used ZhuSuan library to build bayesian lstm cell. I used the code that was in paper of you:ZhuSuan: A Library for Bayesian Deep python. But I got an error:

    AttributeError: module 'tensorflow' has no attribute 'log' Could someone help me to solve this problem? class BayesianLSTMCell(object): def init(self, num_units, forget_bias=1.0): self._forget_bias = forget_bias w_mean = tf.zeros([2 * num_units + 1, 4 * num_units]) self._w = zs.Normal('w', w_mean, std=1., group_ndims=2) def call(self, state, inputs): c, h = state batch_size = tf.shape(inputs)[0] linear_in = tf.concat([inputs, h, tf.ones([batch_size, 1])], axis=1) linear_out = tf.matmul(linear_in, self._w) # i = input_gate, j = new_input, f = forget_gate, o = output_gate i, j, f, o = tf.split(value=linear_out, num_or_size_splits=4, axis=1) new_c = (c * tf.sigmoid(f + self._forget_bias) + tf.sigmoid(i) * tf.tanh(j)) new_h = tf.tanh(new_c) * tf.sigmoid(o) return new_c, new_h def bayesian_rnn(cell, inputs, seq_len): batch_size = tf.shape(inputs)[0] initializer = (tf.zeros([batch_size, 128]), tf.zeros([batch_size, 128])) c_list, h_list = tf.scan(cell, inputs, initializer=initializer) relevant_outputs = tf.gather_nd( h_list, tf.stack([seq_len - 1, tf.range(batch_size)], axis=1)) logits = tf.squeeze(tf.layers.dense(relevant_outputs, 1), -1) return logits seq_len=5 with zs.BayesianNet() as model: cell = BayesianLSTMCell(128, forget_bias=0.) logits = bayesian_rnn(cell, b, seq_len) _ = zs.Bernoulli(Y, logits, dtype=tf.float32

    opened by neginjv 0
  • Fix vae_ssl: variable name issue and n not match

    Fix vae_ssl: variable name issue and n not match

    Two issues in vae_ssl.py and vae_ssl_adaptive_is.py are fixed now:

    1. The variable names x_labeled and x_unlabeled were reused.
    2. The n in modelling of unlabeled part in vae_ssl.py should be n * n_class.
    opened by csy530216 0
  • The examples of ‘semi_supervised_vae’ cannot run successfully

    The examples of ‘semi_supervised_vae’ cannot run successfully

    There seems to be some little bugs in these examples. For example for t in range(iters): labeled_indices = (np.random.randint(0, n_labeled, size=batch_size)) x_labeled_batch = x_labeled[labeled_indices]

    It throws the error that TypeError: Only integers, slices (:), ellipsis (...), tf.newaxis (None) and scalar tf.int32/tf.int64 tensors are valid indices, got array([17, 13, 15, 32, 80, 63, 47, 90, 90, 15, 23, 75, 51, 82, 10, 77, 5, 56, 54, 4, 63, 21, 11, 82, 36, 46, 85, 59, 38, 28, 40, 8, 28, 56, 56, 28, 25, 56, 75, 39, 52, 12, 18, 24, 22, 61, 93, 74, 18, 39, 97, 84, 67, 70, 20, 83, 90, 10, 1, 4, 63, 13, 19, 98, 65, 74, 82, 39, 54, 79, 92, 19, 6, 56, 82, 20, 61, 82, 88, 85, 9, 80, 40, 6, 21, 85, 8, 74, 78, 23, 27, 72, 37, 31, 20, 76, 97, 91, 87, 25]) in in line 154, in main labeled_indices = (np.random.randint(0, n_labeled, size=batch_size))

    And I use labeled_indices = (np.random.randint(0, n_labeled, size=batch_size)).tolist() It throws the error that ValueError: Value out of range: 1267650600228229401496703205375 in line 155, in main x_labeled_batch = x_labeled[labeled_indices]

    opened by SyDudulu 1
  • Eager executation

    Eager executation

    When I try to scale up my model, I find that tf.scan uses too much memory. Probably I have to use eager execution.

    How well does HMC and SGMCMC work with eager execution and tensorflow 2.0?

    ongoing work 
    opened by rzu512 2
Owner
Tsinghua Machine Learning Group
Tsinghua Machine Learning Group
A probabilistic programming language in TensorFlow. Deep generative models, variational inference.

Edward is a Python library for probabilistic modeling, inference, and criticism. It is a testbed for fast experimentation and research with probabilis

Blei Lab 4.7k Jan 9, 2023
A Python package for Bayesian forecasting with object-oriented design and probabilistic models under the hood.

Disclaimer This project is stable and being incubated for long-term support. It may contain new experimental code, for which APIs are subject to chang

Uber Open Source 1.6k Dec 29, 2022
Deep universal probabilistic programming with Python and PyTorch

Getting Started | Documentation | Community | Contributing Pyro is a flexible, scalable deep probabilistic programming library built on PyTorch. Notab

null 7.7k Dec 30, 2022
Functional tensors for probabilistic programming

Funsor Funsor is a tensor-like library for functions and distributions. See Functional tensors for probabilistic programming for a system description.

null 208 Dec 29, 2022
Probabilistic reasoning and statistical analysis in TensorFlow

TensorFlow Probability TensorFlow Probability is a library for probabilistic reasoning and statistical analysis in TensorFlow. As part of the TensorFl

null 3.8k Jan 5, 2023
Using approximate bayesian posteriors in deep nets for active learning

Bayesian Active Learning (BaaL) BaaL is an active learning library developed at ElementAI. This repository contains techniques and reusable components

ElementAI 687 Dec 25, 2022
Python Library for learning (Structure and Parameter) and inference (Statistical and Causal) in Bayesian Networks.

pgmpy pgmpy is a python library for working with Probabilistic Graphical Models. Documentation and list of algorithms supported is at our official sit

pgmpy 2.2k Dec 25, 2022
Fast, flexible and easy to use probabilistic modelling in Python.

Please consider citing the JMLR-MLOSS Manuscript if you've used pomegranate in your academic work! pomegranate is a package for building probabilistic

Jacob Schreiber 3k Jan 2, 2023
Python library for creating data pipelines with chain functional programming

PyFunctional Features PyFunctional makes creating data pipelines easy by using chained functional operators. Here are a few examples of what it can do

Pedro Rodriguez 2.1k Jan 5, 2023
BAyesian Model-Building Interface (Bambi) in Python.

Bambi BAyesian Model-Building Interface in Python Overview Bambi is a high-level Bayesian model-building interface written in Python. It's built on to

null 861 Dec 29, 2022
pyhsmm MITpyhsmm - Bayesian inference in HSMMs and HMMs. MIT

Bayesian inference in HSMMs and HMMs This is a Python library for approximate unsupervised inference in Bayesian Hidden Markov Models (HMMs) and expli

Matthew Johnson 527 Dec 4, 2022
Statistical Rethinking: A Bayesian Course Using CmdStanPy and Plotnine

Statistical Rethinking: A Bayesian Course Using CmdStanPy and Plotnine Intro This repo contains the python/stan version of the Statistical Rethinking

Andrés Suárez 3 Nov 8, 2022
vartests is a Python library to perform some statistic tests to evaluate Value at Risk (VaR) Models

vartests is a Python library to perform some statistic tests to evaluate Value at Risk (VaR) Models, such as: T-test: verify if mean of distribution i

RAFAEL RODRIGUES 5 Jan 3, 2023
Gaussian processes in TensorFlow

Website | Documentation (release) | Documentation (develop) | Glossary Table of Contents What does GPflow do? Installation Getting Started with GPflow

GPflow 1.7k Jan 6, 2023
A Pythonic introduction to methods for scaling your data science and machine learning work to larger datasets and larger models, using the tools and APIs you know and love from the PyData stack (such as numpy, pandas, and scikit-learn).

This tutorial's purpose is to introduce Pythonistas to methods for scaling their data science and machine learning work to larger datasets and larger models, using the tools and APIs they know and love from the PyData stack (such as numpy, pandas, and scikit-learn).

Coiled 102 Nov 10, 2022
BigDL - Evaluate the performance of BigDL (Distributed Deep Learning on Apache Spark) in big data analysis problems

Evaluate the performance of BigDL (Distributed Deep Learning on Apache Spark) in big data analysis problems.

Vo Cong Thanh 1 Jan 6, 2022
Spaghetti: an open-source Python library for the analysis of network-based spatial data

pysal/spaghetti SPAtial GrapHs: nETworks, Topology, & Inference Spaghetti is an open-source Python library for the analysis of network-based spatial d

Python Spatial Analysis Library 203 Jan 3, 2023
Hatchet is a Python-based library that allows Pandas dataframes to be indexed by structured tree and graph data.

Hatchet Hatchet is a Python-based library that allows Pandas dataframes to be indexed by structured tree and graph data. It is intended for analyzing

Lawrence Livermore National Laboratory 14 Aug 19, 2022