Bayesian dessert for Lasagne

Related tags

Deep Learning lasagne theano deep-learning neural-network uncertainty bayesian-inference variational-inference gelato
Overview

Gelato

Coverage Status

Bayesian dessert for Lasagne

Recent results in Bayesian statistics for constructing robust neural networks have proved that it is one of the best ways to deal with uncertainty, overfitting but still having good performance. Gelato will help to use bayes for neural networks. Library heavily relies on Theano, Lasagne and PyMC3.

Installation

  • from github (assumes bleeding edge pymc3 installed) 
    # pip install git+git://github.com/pymc-devs/pymc3.git
pip install git+https://github.com/ferrine/gelato.git
  • from source 
    git clone https://github.com/ferrine/gelato
pip install -r gelato/requirements.txt
pip install -e gelato

Usage

I use generic approach for decorating all Lasagne at once. Thus, for using Gelato you need to replace import statements for layers only. For constructing a network you need to be the in pm.Model context environment.

Warning

  • lasagne.layers.noise is not supported
  • lasagne.layers.normalization is not supported (theano problems with default updates)
  • functions from lasagne.layers are hidden in gelato as they use Lasagne classes. Some exceptions are done for lasagne.layers.helpers. I'll try to solve the problem generically in future.

Examples

For comprehensive example of using Gelato you can reference this notebook

Life Hack

Any spec class can be used standalone so feel free to use it everywhere

References

Charles Blundell et al: "Weight Uncertainty in Neural Networks" (arXiv preprint arXiv:1505.05424)

You might also like...
Bayesian optimization in PyTorch

BoTorch is a library for Bayesian Optimization built on PyTorch. BoTorch is currently in beta and under active development! Why BoTorch ? BoTorch Prov

null 2.5k Dec 31, 2022
Safe Bayesian Optimization
Safe Bayesian Optimization

SafeOpt - Safe Bayesian Optimization This code implements an adapted version of the safe, Bayesian optimization algorithm, SafeOpt [1], [2]. It also p

Felix Berkenkamp 111 Dec 11, 2022
Bayesian Optimization using GPflow

Note: This package is for use with GPFlow 1. For Bayesian optimization using GPFlow 2 please see Trieste, a joint effort with Secondmind. GPflowOpt GP

GPflow 257 Dec 26, 2022
Code for
Code for "Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equations"

Infinitely Deep Bayesian Neural Networks with SDEs This library contains JAX and Pytorch implementations of neural ODEs and Bayesian layers for stocha

Winnie Xu 95 Nov 26, 2021
(under submission) Bayesian Integration of a Generative Prior for Image Restoration
(under submission) Bayesian Integration of a Generative Prior for Image Restoration

BIGPrior: Towards Decoupling Learned Prior Hallucination and Data Fidelity in Image Restoration Authors: Majed El Helou, and Sabine Süsstrunk {Note: p

Majed El Helou 22 Dec 17, 2022
PClean: A Domain-Specific Probabilistic Programming Language for Bayesian Data Cleaning

PClean: A Domain-Specific Probabilistic Programming Language for Bayesian Data Cleaning Warning: This is a rapidly evolving research prototype.

MIT Probabilistic Computing Project 190 Dec 27, 2022
Bayesian Image Reconstruction using Deep Generative Models
Bayesian Image Reconstruction using Deep Generative Models

Bayesian Image Reconstruction using Deep Generative Models R. Marinescu, D. Moyer, P. Golland For technical inquiries, please create a Github issue. F

Razvan Valentin Marinescu 51 Nov 23, 2022
Few-shot Relation Extraction via Bayesian Meta-learning on Relation Graphs

Few-shot Relation Extraction via Bayesian Meta-learning on Relation Graphs This is an implemetation of the paper Few-shot Relation Extraction via Baye

MilaGraph 36 Nov 22, 2022
Supporting code for the paper
Supporting code for the paper "Dangers of Bayesian Model Averaging under Covariate Shift"

Dangers of Bayesian Model Averaging under Covariate Shift This repository contains the code to reproduce the experiments in the paper Dangers of Bayes

Pavel Izmailov 25 Sep 21, 2022
Comments
  • Exception in example NB

    Exception in example NB

    I'm up-to-date on pymc3 and gelato.

    ---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/Users/twiecki/anaconda/lib/python3.6/site-packages/theano/gof/op.py in __call__(self, *inputs, **kwargs)
    624                 try:
--> 625                     storage_map[ins] = [self._get_test_value(ins)]
    626                     compute_map[ins] = [True]

/Users/twiecki/anaconda/lib/python3.6/site-packages/theano/gof/op.py in _get_test_value(cls, v)
    580         detailed_err_msg = utils.get_variable_trace_string(v)
--> 581         raise AttributeError('%s has no test value %s' % (v, detailed_err_msg))
    582 

AttributeError: Softmax.0 has no test value  
Backtrace when that variable is created:

  File "/Users/twiecki/anaconda/lib/python3.6/site-packages/ipykernel/zmqshell.py", line 533, in run_cell
    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
  File "/Users/twiecki/anaconda/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2717, in run_cell
    interactivity=interactivity, compiler=compiler, result=result)
  File "/Users/twiecki/anaconda/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2821, in run_ast_nodes
    if self.run_code(code, result):
  File "/Users/twiecki/anaconda/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2881, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-18-7dd01309b711>", line 37, in <module>
    prediction = gelato.layers.get_output(network)
  File "/Users/twiecki/anaconda/lib/python3.6/site-packages/lasagne/layers/helper.py", line 190, in get_output
    all_outputs[layer] = layer.get_output_for(layer_inputs, **kwargs)
  File "/Users/twiecki/anaconda/lib/python3.6/site-packages/lasagne/layers/dense.py", line 124, in get_output_for
    return self.nonlinearity(activation)
  File "/Users/twiecki/anaconda/lib/python3.6/site-packages/lasagne/nonlinearities.py", line 44, in softmax
    return theano.tensor.nnet.softmax(x)


During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
<ipython-input-18-7dd01309b711> in <module>()
     44                    prediction,
     45                    observed=target_var,
---> 46                    total_size=total_size)

/Users/twiecki/working/projects/pymc/pymc3/distributions/distribution.py in __new__(cls, name, *args, **kwargs)
     35                 raise TypeError("observed needs to be data but got: {}".format(type(data)))
     36             total_size = kwargs.pop('total_size', None)
---> 37             dist = cls.dist(*args, **kwargs)
     38             return model.Var(name, dist, data, total_size)
     39         else:

/Users/twiecki/working/projects/pymc/pymc3/distributions/distribution.py in dist(cls, *args, **kwargs)
     46     def dist(cls, *args, **kwargs):
     47         dist = object.__new__(cls)
---> 48         dist.__init__(*args, **kwargs)
     49         return dist
     50 

/Users/twiecki/working/projects/pymc/pymc3/distributions/discrete.py in __init__(self, p, *args, **kwargs)
    429         super(Categorical, self).__init__(*args, **kwargs)
    430         try:
--> 431             self.k = tt.shape(p)[-1].tag.test_value
    432         except AttributeError:
    433             self.k = tt.shape(p)[-1]

/Users/twiecki/anaconda/lib/python3.6/site-packages/theano/gof/op.py in __call__(self, *inputs, **kwargs)
    637                         raise ValueError(
    638                             'Cannot compute test value: input %i (%s) of Op %s missing default value. %s' %
--> 639                             (i, ins, node, detailed_err_msg))
    640                     elif config.compute_test_value == 'ignore':
    641                         # silently skip test

ValueError: Cannot compute test value: input 0 (Softmax.0) of Op Shape(Softmax.0) missing default value.  
Backtrace when that variable is created:

  File "/Users/twiecki/anaconda/lib/python3.6/site-packages/ipykernel/zmqshell.py", line 533, in run_cell
    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
  File "/Users/twiecki/anaconda/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2717, in run_cell
    interactivity=interactivity, compiler=compiler, result=result)
  File "/Users/twiecki/anaconda/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2821, in run_ast_nodes
    if self.run_code(code, result):
  File "/Users/twiecki/anaconda/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2881, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-18-7dd01309b711>", line 37, in <module>
    prediction = gelato.layers.get_output(network)
  File "/Users/twiecki/anaconda/lib/python3.6/site-packages/lasagne/layers/helper.py", line 190, in get_output
    all_outputs[layer] = layer.get_output_for(layer_inputs, **kwargs)
  File "/Users/twiecki/anaconda/lib/python3.6/site-packages/lasagne/layers/dense.py", line 124, in get_output_for
    return self.nonlinearity(activation)
  File "/Users/twiecki/anaconda/lib/python3.6/site-packages/lasagne/nonlinearities.py", line 44, in softmax
    return theano.tensor.nnet.softmax(x)
    opened by twiecki 12
  • Integrate opvi

    Integrate opvi

    I'm currently integrating recent changes in PyMC3 to gelato. There are a lot of changes. Everyone is welcome for discussion.

    Here are the most remarkable features:

    • no more with context when using gelato layers
    from gelato.layers import *
import pymc3 as pm
# get data somehow
inp = InputLayer(shape)
out = DenseLayer(inp, 1, W=NormalSpec(sd=LognormalSpec(sd=.1)))
out = DenseLayer(out, 1, W=NormalSpec(sd=LognormalSpec(sd=.1)))
with out.root:
    pm.Normal('y', mu=get_output(out, {inp:x}),
              observed=y)
    approx = pm.fit(10000)
    • Flexible Specs you can do almost everything. What to do if we want different shapes there is an open question
    from gelato import *
import theano.tensor as tt
import pymc3 as pm
func = as_spec_op(tt.nlinalg.matrix_power)
expr0= func(NormalSpec() * LaplaceSpec(), 2)
expr1 = expr0 / 100 - NormalSpec()
with Model() as model:
    var = expr((10, 10))
    assert var.tag.test_value.shape == (10, 10)
    assert len(model.free_RVs) == 3
    fit(100)
U = NormalSpec()
V = UniformSpec()
V = V / V.norm(2)
W = U*V
with pm.Model() as model:
    result = W((3, 2), name='weight_normalization')
    opened by ferrine 2
  • Fix example

    Fix example

    refere to #7. I've updated example using new pm.Minibatch API. All was running good with the following theanorc:

    [global]
device=cpu
floatX=float32
mode=FAST_RUN
optimizer_including=cudnn

[lib]
cnmem=0.95

[nvcc]
fastmath=True
flags = -I/usr/local/cuda-8.0-cudnnv5.1/include -L/usr/local/cuda-8.0-cudnnv5.1/lib64

[blas]
ldflag = -L/usr/lib/openblas-base -Lusr/local/cuda-8.0-cudnnv5.1/lib64 -lopenblas

[DebugMode]
check_finite=1

[cuda]
root=/usr/local/cuda-8.0-cudnnv5.1/

    pip freeze output

    alabaster==0.7.10
algopy==0.5.3
Babel==2.4.0
bleach==2.0.0
CommonMark==0.5.4
cycler==0.10.0
Cython==0.25.2
decorator==4.0.11
docutils==0.13.1
entrypoints==0.2.2
-e git+https://github.com/ferrine/gelato@d1604e9f277baf80b7dddfdf908d8c39ad70661#egg=gelato
h5py==2.7.0
html5lib==0.999999999
imagesize==0.7.1
ipykernel==4.6.1
ipython==6.0.0
ipython-genutils==0.2.0
ipywidgets==6.0.0
Jinja2==2.9.6
joblib==0.11
jsonschema==2.6.0
jupyter==1.0.0
jupyter-client==5.0.1
jupyter-console==5.1.0
jupyter-core==4.3.0
Keras==2.0.4
Lasagne==0.2.dev1
Mako==1.0.6
MarkupSafe==1.0
matplotlib==2.0.0
mistune==0.7.4
more-itertools==3.1.0
nbconvert==5.1.1
nbformat==4.3.0
nbsphinx==0.2.13
nose==1.3.7
notebook==5.0.0
numdifftools==0.9.20
numpy==1.13.0
pandas==0.20.1
pandocfilters==1.4.1
patsy==0.4.1
pexpect==4.2.1
pickleshare==0.7.4
prompt-toolkit==1.0.14
ptyprocess==0.5.1
Pygments==2.2.0
pygpu==0.6.5
-e git+https://github.com/ferrine/pymc3@2dfaeada66014445ccff562719025d5ac4ce12e2#egg=pymc3
pymongo==3.4.0
pyparsing==2.2.0
python-dateutil==2.6.0
pytz==2017.2
PyYAML==3.12
pyzmq==16.0.2
qtconsole==4.3.0
recommonmark==0.4.0
requests==2.13.0
scikit-learn==0.18.1
scipy==0.19.1
seaborn==0.7.1
simplegeneric==0.8.1
six==1.10.0
sklearn==0.0
snowballstemmer==1.2.1
Sphinx==1.5.5
terminado==0.6
testpath==0.3
Theano==0.10.0.dev1
tornado==4.5.1
tqdm==4.11.2
traitlets==4.3.2
wcwidth==0.1.7
webencodings==0.5.1
widgetsnbextension==2.0.0
xmltodict==0.11.0
    opened by ferrine 0
  • Not compatible with latest version of pymc3

    Not compatible with latest version of pymc3

    When I attempt to import gelato, it fails with the following error message:

    ---> 19 class LayerModelMeta(pm.model.InitContextMeta):
     20     """Magic comes here
     21     """

AttributeError: module 'pymc3.model' has no attribute 'InitContextMeta'

    I believe that InitContextMeta no longer exists in pymc3; it's been merged with ContextMeta.

    I don't know if there are plans to update this repository anytime soon, although it does seem like a useful tool, so it would be great if it worked with the latest pymc3.

    opened by quevivasbien 2
Releases(v0.1.0)
Owner
Maxim Kochurov
Researcher @ NTechLab; MSU/Skoltech; Core Dev @ PyMC3, Geoopt
Maxim Kochurov
GitHub
pyhsmm - library for approximate unsupervised inference in Bayesian Hidden Markov Models (HMMs) and explicit-duration Hidden semi-Markov Models (HSMMs), focusing on the Bayesian Nonparametric extensions, the HDP-HMM and HDP-HSMM, mostly with weak-limit approximations.

Bayesian inference in HSMMs and HMMs This is a Python library for approximate unsupervised inference in Bayesian Hidden Markov Models (HMMs) and expli

Matthew Johnson 527 Dec 4, 2022
Bayesian-Torch is a library of neural network layers and utilities extending the core of PyTorch to enable the user to perform stochastic variational inference in Bayesian deep neural networks

Bayesian-Torch is a library of neural network layers and utilities extending the core of PyTorch to enable the user to perform stochastic variational inference in Bayesian deep neural networks. Bayesian-Torch is designed to be flexible and seamless in extending a deterministic deep neural network architecture to corresponding Bayesian form by simply replacing the deterministic layers with Bayesian layers.

Intel Labs 210 Jan 4, 2023
Hierarchical-Bayesian-Defense - Towards Adversarial Robustness of Bayesian Neural Network through Hierarchical Variational Inference (Openreview)

Towards Adversarial Robustness of Bayesian Neural Network through Hierarchical V

LBK 20 Dec 2, 2022
Spearmint Bayesian optimization codebase

Spearmint Spearmint is a software package to perform Bayesian optimization. The Software is designed to automatically run experiments (thus the code n

Formerly: Harvard Intelligent Probabilistic Systems Group -- Now at Princeton 1.5k Dec 29, 2022
Python Environment for Bayesian Learning

Pebl is a python library and command line application for learning the structure of a Bayesian network given prior knowledge and observations. Pebl in

Abhik Shah 103 Jul 14, 2022
Python Library for learning (Structure and Parameter) and inference (Statistical and Causal) in Bayesian Networks.

pgmpy pgmpy is a python library for working with Probabilistic Graphical Models. Documentation and list of algorithms supported is at our official sit

pgmpy 2.2k Jan 3, 2023
Python package for Bayesian Machine Learning with scikit-learn API

Python package for Bayesian Machine Learning with scikit-learn API Installing & Upgrading package pip install https://github.com/AmazaspShumik/sklearn

Amazasp Shaumyan 482 Jan 4, 2023
Simple, but essential Bayesian optimization package

BayesO: A Bayesian optimization framework in Python Simple, but essential Bayesian optimization package. http://bayeso.org Online documentation Instal

Jungtaek Kim 74 Dec 5, 2022
A bare-bones TensorFlow framework for Bayesian deep learning and Gaussian process approximation

Aboleth A bare-bones TensorFlow framework for Bayesian deep learning and Gaussian process approximation [1] with stochastic gradient variational Bayes

Gradient Institute 127 Dec 12, 2022
Python package facilitating the use of Bayesian Deep Learning methods with Variational Inference for PyTorch

PyVarInf PyVarInf provides facilities to easily train your PyTorch neural network models using variational inference. Bayesian Deep Learning with Vari

null 342 Dec 2, 2022