Hi. Great Repository! I was having trouble recreating the results from some of the papers which you implemented and I found your examples to be more complete than the original author repositories which is very helpful.

I have a question regarding your calculation of the test set log likelihood of the UCI datasets...

In this file (https://github.com/JavierAntoran/Bayesian-Neural-Networks/blob/master/notebooks/regression/mc_dropout_hetero.ipynb) you use the flow...

- make tensor of (samples, mu) and (samples, log_sigma)
- use means.mean() as mu and (means.var() + mean(sigma) ** 2) ** 0.5 as sigma
- calculate sum of all log probability / data instances

Then as a last step you add log(y_sigma) which is the standard deviation of the y values before they were normalized. Why do you do this and where does the necessity to do this come from?

The original MC Dropout and Concrete Dropout repositories use some form of the logsumexp trick do calculate the log probabilities, but so far I have been unable to recreate the UCI results using their method. I get much closer with your method but I cannot justify the last step to myself.

Reference

Concrete Dropout

MC Dropout

Hi,

I’m trying to approximate the posterior predictive distribution that corresponds to the MC Dropout and Bayes By Backprop neural networks (which I see you say is possible in the section MNIST classification in README). I’m new to Python, so I’m having a little trouble figuring out how exactly you do this/ what part of the code carries this out?

I tried to go about it by playing around with your function get_weight_samples, but noticed that this gives me the same weights each time I train the same network. For example, when training the MC Dropout network, I assumed the output of get_weight_samples would change as in theory different nodes are dropped during training each time. My confusion here makes me think that I maybe have misinterpreted what this function is supposed to be doing.

Any clarification would be greatly appreciated! I’m sorry if this wasn’t the right place to post a question of this nature – new to Github and still learning the ropes.

Hi, in BBB, we sample the outputs several times (5 for example) when testing. This is due to the nature of MC. But how do we use these 5 outputs of one sample? Averaging them? Summing them? Or just take the best one as this sample's output vector? Are there any theory about how to use these sampled results? Thanks!

Hi, in bbp_homoscedastic.ipynb, it seems that you choose a normal Gaussian prior rather than a scale mixture prior. I think that mixture Gaussian prior can better model the real distribution of weight w. Thanks!

Hi, in local reparametrisation trick, when computing output of convolution layer, we use alpha * mu^2 to replace sigma^2. But when computing KL, should we also use alpha * mu^2 to replace the weight's variance? Why or why not? Thanks!

Hi @JavierAntoran @stratisMarkou ,

I am currently using SGLD in another project. I have some example code, where the noise added to the parameters is scaled by the learning rate:

noise = torch.randn(n.size()) * param_noise_sigma * lr.

I belive, this is wrong and so I scaled the noise with the square root of the learning rate:

noise = torch.randn(n.size()) * np.sqrt(lr)

This works well. However, you scale the learning rate by

1/np.sqrt(sigma)

If I do this, the noise gets way too big and the network does not learn. Also, in the initial paper by Welling et al. it seems to me that the first approach is right, though I already found out that the variance of the noise scales with lr**2, which is why the square root is necessary.

I hope you can explain to me, why the ()^-1 is necessary. Thank you in advance!

For the bbp_hetero notebook, should be plot be done as: plt.fill_between(np.linspace(-5, 5, 200)* x_std + x_mean, means - aleatoric, means + aleatoric, color = c[0], alpha = 0.3, label = 'Aleatoric') plt.fill_between(np.linspace(-5, 5, 200)* x_std + x_mean, means - epistemic, means + epistemic, color = c[1], alpha = 0.3, label = 'Epistemic')

At the moment, the boundaries are mixed.

Thanks for the attention so far, Celso

def softmax_CE_preact_hessian(last_layer_acts): side = last_layer_acts.shape[1] I = torch.eye(side).type(torch.ByteTensor) # for i != j H = -ai * aj -- Note that these are activations not pre-activations Hl = - last_layer_acts.unsqueeze(1) * last_layer_acts.unsqueeze(2) # for i == j H = ai * (1 - ai) Hl[:, I] = last_layer_acts * (1 - last_layer_acts) return Hl

This function calculates the hessian for the first layer's activations. Why can the hessian be obtained by this process?

for i != j H = -ai * aj -- Note that these are activations not pre-activations

for i == j H = ai * (1 - ai)

Is this process shown in the paper (https://openreview.net/pdf?id=Skdvd2xAZ)?

Looking forward to your reply! Really thank you!

Hi @JavierAntoran , my name is Hao, I'm searching for a valid method for the Uncertainty Explanations, like to decompose the Uncertainty of the prediction back to the features, until I found your paper "Getting a GLUE: A Method for Explaining Uncertainty Estimates", and find it fantastic, do you have implementations of the "CLUE" in this repo or where can I find it, because I see the reference of this github in the paper? I'm researching on using ICU data to do Sepsis Prediction, really appreciate it if you can help.

Thanks in Advance Hao

Hi! First of all, thank you for providing your code！ I think there is an error in class "spike_slab_2GMM" in the "priors.py" file: normalised_like = self.pi1 + torch.exp(N1_ll - max_loglike) + self.pi2 + torch.exp(N2_ll - max_loglike) After calculation and verification, I think it should be changed to: normalised_like = self.pi1 * torch.exp(N1_ll - max_loglike) + self.pi2 * torch.exp(N2_ll - max_loglike) I'm sorry to disturb you in your busy schedule~

Your codes are helpful for me. However, I found it seems there are some mistakes in the file train_BayesByBackprop_MNIST.py and train_BootrapEnsemble_MNIST.py at line 219 and 210 respectively. Those in the other files are right.

Looking forward to your reply.

Firstly, thank you for all these great notebooks, they've been very helpful in building a better understanding of these methods.

I am wondering where the function log_gaussian_loss originates from? I'm struggling to find reference to it in the literature, though very likely looking in the wrong places.

In MC dropout it seems that one output neuron is for the prediction and another that feeds into this loss function, and I'm struggling to get a simpler version working whereby there's one output neuron and a different loss function. Where does this technique originate from?

Thanks again

Hi JavierAntoran,

Thanks for the wonderful code first, and it is really helpful for me working in the related area. I'd like to consult a question about the MC dropout. In BBB with local reparameterization, the activation values are sampled for each data point instead of directly sampling a weight distribution to reduce the computational complexity. So, in MC dropout, shall we do the similar procedure, e.g. dropout hidden units for each data point in training or testing phase? I notice that your MC dropout model seems uses the same dropout for a mini-batch data and the default batch size is 128. Should I change the batch size to 1 to achieve the goal of dropping hidden units for each data point?

Looking forward to your reply. Thanks a lot

Hi @JavierAntoran @stratisMarkou,

First of all, thanks for making all of this code available - it's been great to look through!

Im currently spending some time trying to work through the Weight Uncertainty in Neural Networks in order to implement Bayes-by-Backprop. I was struggling to understand the difference between your implementation of Bayes-by-Backprop and Bayes-by-Backprop with Local Reparameterization.

I was under the impression that the local reparameterization was the following:

https://github.com/JavierAntoran/Bayesian-Neural-Networks/blob/022b9cedb69be3fecc83d4b0efe4b5a848119c2a/src/Bayes_By_Backprop/model.py#L58-L66

However this same approach is used in both methods.

The main difference I see in the code you've implemented is the calculation of the KL Divergence in closed form in the Local Reparameterization version of the code due to the use of a Gaussian prior / posterior distribution.

I was wondering if my understanding of the local reparameterization method was wrong, or if I had simply misunderstood the code?

Any guidance would be much appreciated!

Minor bug. Instead of: data = pd.read_csv('housing.data', header=0, delimiter="\s+").values Use: data = pd.read_csv('housing.data', header=None, delimiter="\s+").values