I know this might be a bit strange, but I think it would be useful at least to have an option of an empirical distribution. This would make for instance specifying the "inference" networks more symmetric and include more clearly what the full graphical model is. This will also make it consistent to pass the x as an observed variable when we query for the log-probabilities rather than when we build model. This additionally makes the way of building the forward and backward models having the exact same signatures. Taking the example:

@zs.reuse('model')
def vae(observed, x_dim, z_dim, n_x, n_z_per_x):
    with zs.BayesianNet(observed=observed) as model:
        z_mean = tf.zeros([n_x, z_dim])
        z = zs.Normal('z', z_mean, std=1., group_ndims=1, n_samples=n_z_per_x)
        lx_z = layers.fully_connected(z, 500)
        lx_z = layers.fully_connected(lx_z, 500)
        x_logits = layers.fully_connected(lx_z, x_dim, activation_fn=None)
        x = zs.Bernoulli('x', x_logits, group_ndims=1)
    return model, x_logits

@zs.reuse('variational')
def q_net(observed, x_dim, z_dim, n_x, n_z_per_x):
    with zs.BayesianNet(observed=observed) as variational:
        x = zs.Empirical('x', (n_x, x_dim), dtype=tf.int32)
        lz_x = layers.fully_connected(tf.to_float(x), 500)
        lz_x = layers.fully_connected(lz_x, 500)
        z_mean = layers.fully_connected(lz_x, z_dim, activation_fn=None)
        z_logstd = layers.fully_connected(lz_x, z_dim, activation_fn=None)
        z = zs.Normal('z', z_mean, logstd=z_logstd, group_ndims=1,
                      n_samples=n_z_per_x)
    return variational

dear authors, thanks for this package. I am wondering if Markov random field can be implemented using zhusuan, is so, do you have a tutorial for me to go through? thank you.

Hi,

I tried to implement the bayesian_rnn from the docs. However, while trying to compute log_joint, I can't compute the log of prior log_pz because w is declared within a LSTM cell, so I get the following error:

Traceback (most recent call last):
  File "blstm.py", line 111, in <module>
    joint_ll = log_joint({'x': x, 'y_i': y_i, 'y_v': y_v})
  File "blstm.py", line 106, in log_joint
    log_pz, log_px_z = model.local_log_prob(['w', 'y_v'])  # Error
  File "/Users/jilljenn/code/vae/venv/lib/python3.6/site-packages/zhusuan/model/base.py", line 346, in local_log_prob
    ret.append(s_tensor.log_prob(s_tensor.tensor))
  File "/Users/jilljenn/code/vae/venv/lib/python3.6/site-packages/zhusuan/model/base.py", line 140, in log_prob
    return self._distribution.log_prob(given)
  File "/Users/jilljenn/code/vae/venv/lib/python3.6/site-packages/zhusuan/utils.py", line 215, in _func
    return f(*args, **kwargs)
  File "/Users/jilljenn/code/vae/venv/lib/python3.6/site-packages/zhusuan/distributions/base.py", line 303, in log_prob
    log_p = self._log_prob(given)
  File "/Users/jilljenn/code/vae/venv/lib/python3.6/site-packages/zhusuan/distributions/univariate.py", line 180, in _log_prob
    return c - logstd - 0.5 * precision * tf.square(given - mean)
  File "/Users/jilljenn/code/vae/venv/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py", line 979, in binary_op_wrapper
    return func(x, y, name=name)
  File "/Users/jilljenn/code/vae/venv/lib/python3.6/site-packages/tensorflow/python/ops/gen_math_ops.py", line 8009, in sub
    "Sub", x=x, y=y, name=name)
  File "/Users/jilljenn/code/vae/venv/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/Users/jilljenn/code/vae/venv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3392, in create_op
    op_def=op_def)
  File "/Users/jilljenn/code/vae/venv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1760, in __init__
    self._control_flow_post_processing()
  File "/Users/jilljenn/code/vae/venv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1769, in _control_flow_post_processing
    control_flow_util.CheckInputFromValidContext(self, input_tensor.op)
  File "/Users/jilljenn/code/vae/venv/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_util.py", line 263, in CheckInputFromValidContext
    raise ValueError(error_msg + " See info log for more details.")
ValueError: Cannot use 'model/scan/while/MatMul/Normal.sample/Squeeze' as input to 'Normal.log_prob/sub_1' because 'model/scan/while/MatMul/Normal.sample/Squeeze' is in a while loop. See info log for more details.

Up-to-run code: https://github.com/jilljenn/vae/blob/master/blstm.py#L106

How to make it?

This PR implements the variational sparse gaussian process, formulated as in, e.g., section 2.1 of this paper, except that I did not analytically marginalize the inducing point values ((6) in the reference), so that the derivation is simpler, and non-conjugate variational distribution can be used in the future.

On the UCI Boston and the newly added protein datasets, performance is on par with reports in most literature.

Hi, Thank u for your great share first! And when i test the examples\bayesian_neural_nets\bayesian_nn.py, I found that the std of y_mean what i want to predict is too small. As the same data ,when i tested on pyro of torch, the std looks good.

And i read the code, i found that the y_mean is counted by likelihood function, as well as it should be count by q(w)?

` import tensorflow as tf import zhusuan as zs

logits = tf.zeros([10, 10]) temparature = 1. labels = zs.Concrete('y', logits=logits, temperature=temparature, n_samples=10, group_ndims=1) prob = labels.log_prob(tf.one_hot(range(10), 10)) print (prob.shape) ` seems only produce 1 tensor with shape ()

In tf there are derived dtypes for reference-based tensors. I.e. Variable().dtype == tf.int32_ref != tf.int32. zs.distributions.utils.assert_same_dtype is invoked whenever a distribution with multiple parameters is constructed, but it did not consider this case. Thus the following code would fail:

a = tf.Variable(..., dtype=tf.float32) # dtype=tf.float32_ref
b = <some float32 op>
dist = zs.MultivariateNormalTriL(.., mean=a, cov_tril=b)  # Invoked assert_same_dtype(mean, cov_tril)

The solution is to use DType.base_dtype.

I have detailedly studied the code under the folders zhusuan/distributions and zhusuan/model. Yet, I am still confused by how the gradients are computed by Tensorflow. Since a number of samples are used to caluated the cost at each optimization Op, I am not sure why this is reversible when taking the gradients w.r.t. Distribution parameters (e.g. mean and log_std in Normal), how this is handled by Tensorflow. It is much appreciated if you can suggest me some materials which I have missed to read.

When I run the examples, such as python zhusuan/examples/bayesian_neural_nets/python bayesian_nn.py, I will get the error "ImportError: No module named examples". Please help me. Thank you.

Hi.

The parameters of my model have no constraint. I implemented the following two functions and want to use HMC algorithm of zhusuan. How to do that? If I can't use the function directly, how to implement custom StochasticTensor?

def cal_log_likelihood(p1, p2):
    """Calculate log-likelihood of the model

    Parameters
    ----------
    p1 : tf.Tensor, shape=(n_chains, n_states)
    p2 : tf.Tensor, shape=(n_chains, n_states, n_states)

    Returns
    -------    
    log_likelihood : tf.Tensor, shape=(n_chains,)
    """
    return log_likelihood

def cal_prior(p1, p2):
    """Calculate logarithm of the prior probability

    Parameters
    ----------
    p1 : tf.Tensor, shape=(n_chains, n_states)
    p2 : tf.Tensor, shape=(n_chains, n_states, n_states)

    Returns
    -------    
    log_prior_probability : tf.Tensor, shape=(n_chains,)
    """
    return log_prior_probability

Thank you.

Hello dear friends,

I have been reading your work with much interest. With @zsdonghao @luomai @lgarithm we maintain the library TensorLayer (TL), https://github.com/tensorlayer/tensorlayer.

I think that your work can be really complementary to TL APIs.

Maybe you would be interested in working together and heading toward a common direction.

We have a chat group on Slack and/or WeChat if you wanna talk.

Have a nice day,

All the best,

Jonathan DEKHTIAR

Patching CVE-2007-4559

Hi, we are security researchers from the Advanced Research Center at Trellix. We have began a campaign to patch a widespread bug named CVE-2007-4559. CVE-2007-4559 is a 15 year old bug in the Python tarfile package. By using extract() or extractall() on a tarfile object without sanitizing input, a maliciously crafted .tar file could perform a directory path traversal attack. We found at least one unsantized extractall() in your codebase and are providing a patch for you via pull request. The patch essentially checks to see if all tarfile members will be extracted safely and throws an exception otherwise. We encourage you to use this patch or your own solution to secure against CVE-2007-4559. Further technical information about the vulnerability can be found in this blog.

If you have further questions you may contact us through this projects lead researcher Kasimir Schulz.

Most of the imports used in the examples are from Tensorflow 1.x and are not compatible with Tensorflow 2.x...

Examples of faulty imports include, TensorFlow.contrib (no longer exists in TF 2.0+)....

Hi, I used ZhuSuan library to build bayesian lstm cell. I used the code that was in paper of you:ZhuSuan: A Library for Bayesian Deep python. But I got an error:

AttributeError: module 'tensorflow' has no attribute 'log' Could someone help me to solve this problem? class BayesianLSTMCell(object): def init(self, num_units, forget_bias=1.0): self._forget_bias = forget_bias w_mean = tf.zeros([2 * num_units + 1, 4 * num_units]) self._w = zs.Normal('w', w_mean, std=1., group_ndims=2) def call(self, state, inputs): c, h = state batch_size = tf.shape(inputs)[0] linear_in = tf.concat([inputs, h, tf.ones([batch_size, 1])], axis=1) linear_out = tf.matmul(linear_in, self._w) # i = input_gate, j = new_input, f = forget_gate, o = output_gate i, j, f, o = tf.split(value=linear_out, num_or_size_splits=4, axis=1) new_c = (c * tf.sigmoid(f + self._forget_bias) + tf.sigmoid(i) * tf.tanh(j)) new_h = tf.tanh(new_c) * tf.sigmoid(o) return new_c, new_h def bayesian_rnn(cell, inputs, seq_len): batch_size = tf.shape(inputs)[0] initializer = (tf.zeros([batch_size, 128]), tf.zeros([batch_size, 128])) c_list, h_list = tf.scan(cell, inputs, initializer=initializer) relevant_outputs = tf.gather_nd( h_list, tf.stack([seq_len - 1, tf.range(batch_size)], axis=1)) logits = tf.squeeze(tf.layers.dense(relevant_outputs, 1), -1) return logits seq_len=5 with zs.BayesianNet() as model: cell = BayesianLSTMCell(128, forget_bias=0.) logits = bayesian_rnn(cell, b, seq_len) _ = zs.Bernoulli(Y, logits, dtype=tf.float32

Two issues in vae_ssl.py and vae_ssl_adaptive_is.py are fixed now:

1. The variable names x_labeled and x_unlabeled were reused.
2. The n in modelling of unlabeled part in vae_ssl.py should be n * n_class.

There seems to be some little bugs in these examples. For example for t in range(iters): labeled_indices = (np.random.randint(0, n_labeled, size=batch_size)) x_labeled_batch = x_labeled[labeled_indices]

It throws the error that TypeError: Only integers, slices (:), ellipsis (...), tf.newaxis (None) and scalar tf.int32/tf.int64 tensors are valid indices, got array([17, 13, 15, 32, 80, 63, 47, 90, 90, 15, 23, 75, 51, 82, 10, 77, 5, 56, 54, 4, 63, 21, 11, 82, 36, 46, 85, 59, 38, 28, 40, 8, 28, 56, 56, 28, 25, 56, 75, 39, 52, 12, 18, 24, 22, 61, 93, 74, 18, 39, 97, 84, 67, 70, 20, 83, 90, 10, 1, 4, 63, 13, 19, 98, 65, 74, 82, 39, 54, 79, 92, 19, 6, 56, 82, 20, 61, 82, 88, 85, 9, 80, 40, 6, 21, 85, 8, 74, 78, 23, 27, 72, 37, 31, 20, 76, 97, 91, 87, 25]) in in line 154, in main labeled_indices = (np.random.randint(0, n_labeled, size=batch_size))

And I use labeled_indices = (np.random.randint(0, n_labeled, size=batch_size)).tolist() It throws the error that ValueError: Value out of range: 1267650600228229401496703205375 in line 155, in main x_labeled_batch = x_labeled[labeled_indices]

When I try to scale up my model, I find that tf.scan uses too much memory. Probably I have to use eager execution.

How well does HMC and SGMCMC work with eager execution and tensorflow 2.0?