I'm trying to replicate the MNIST and CIFAR-10 experiments used in the paper. If I want to modify the code to add Conditional Masked Autoregressive Flow, which part of the neural network model should I modify?

Thanks for your help!!

To reproduce, use:

python main.py --dataset GAS --flow glow

Printed message for first epoch:

Warning: Results for GLOW are not as good as for MAF yet.
Warning: Results for GLOW are not as good as for MAF yet.
Warning: Results for GLOW are not as good as for MAF yet.
Warning: Results for GLOW are not as good as for MAF yet.
Warning: Results for GLOW are not as good as for MAF yet.
Train Epoch: 0 [0/852174 (0%)]	Loss: 6.648179
Train Epoch: 0 [100000/852174 (12%)]	Loss: -2.193499
Train Epoch: 0 [200000/852174 (23%)]	Loss: -2.497202
Train Epoch: 0 [300000/852174 (35%)]	Loss: -1.122860
Train Epoch: 0 [400000/852174 (47%)]	Loss: -2.892694
Train Epoch: 0 [500000/852174 (59%)]	Loss: -2.268665
Train Epoch: 0 [600000/852174 (70%)]	Loss: -3.088916
Train Epoch: 0 [700000/852174 (82%)]	Loss: -3.798649
Train Epoch: 0 [800000/852174 (94%)]	Loss: -2.554942

Validation set: Average loss: nan

Best validation at epoch 0: Average loss: inf

Feel free to incorporate any or none of these fixes/additions. Let me know if you'd like me to break up the PR. Thanks!

1. Implement the generative pass (mode == 'inverse') for MADE module. This iteratively builds the output of the generative model given the input (e.g. the Gaussian latent data). The shift and scale parameters for the first element are independent of the data, and are determined simply by the bias weights.

2. Add the HalfTanh module, which applies tanh to the second half of the dimensions of a given tensor. This is used in MADE, since tanh adds stability to the scale parameters.

3. Add a 2D moon dataset example, including a visualization of generated samples. The command

python main.py --dataset MOONS --flow maf --epochs 50

results in the following plot:

4. Changing sign of scale parameter in 'inverse' mode to correspond to the MAF paper.

I'm running into a problem when I call model.forward(..., mode='inverse') before calling model.forward(..., mode='direct'), in which case batch_mean and batch_var are not defined. What is a proper why to fix this problem? I need to use inverse mode first during the training stage in my application.

The code snippet looks like this. I've omitted some application specific codes.

block = [
fnn.MADE(num_inputs, num_hidden),
fnn.BatchNormFlow(num_inputs),
fnn.Reverse(num_inputs)
]
model = fnn.FlowSequential(*blocks)
model.train()
model.forward(..., mode='inverse')

AttributeError: 'BatchNormFlow' object has no attribute 'batch_mean'

Previously we put a tanh on the end of the scale net for MADE, to prevent scale explosion. But in the original implementation the last layer does not have an activation function, instead they have different activation function in the intermediate layers for particular dataset. For coupling layer, the implementation of MAF repo have the scale and translate net apart, and use different activation function for each of those. This commit is trying to mimic these behaviors.

util is an undefined name in this context while datasets.util is imported on line 5.

flake8 testing of https://github.com/ikostrikov/pytorch-flows on Python 3.7.0

$ flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics

./datasets/power.py:33:9: F821 undefined name 'util'
        util.plot_hist_marginals(data_split.x)
        ^
1     F821 undefined name 'util'
1

https://github.com/ikostrikov/pytorch-flows/blob/34a34a0f25a86479e221f5d9bc4a67857d6621d0/flows.py#L114

This currently matches https://github.com/ikostrikov/pytorch-flows/blob/34a34a0f25a86479e221f5d9bc4a67857d6621d0/flows.py#L130 - but L114 should have no minus sign for the log determinant of the Jacobian.

The forward computation is on https://github.com/ikostrikov/pytorch-flows/blob/34a34a0f25a86479e221f5d9bc4a67857d6621d0/flows.py#L113

u = (inputs - m) * torch.exp(-a)

The inverse of this is

inputs = u * torch.exp(a) + m

So the log determinant jacobian of the inverse is a.sum(-1, keepdim=True) not -a.sum(-1, keepdim=True).

Might be missing something here!

In our application, we've found the need to provide the PDF for the latent variables (in our case, it is useful that the PDF depends on the conditional inputs). We've found a small fix to your code that let's you have this option, by just adding an init method to FlowSequential and changing log_probs accordingly. I'll share the different code here. Here is the added init method

    def __init__(self, *args, log_probs = 'gaussian'):
        super(FlowSequential,self).__init__(*args)
        if log_probs = 'gaussian'
            def __log_probs(x, *_):
                return torch.sum(-0.5 * x ** 2 - 0.5 * math.log(2 * math.pi),
                                                            -1, keepdim=True)
            self.log_probs = __log_probs
        else:
            self.log_probs = log_probs

and here is the modified log_probs.

    def log_probs(self, inputs, cond_inputs = None):
        u, log_jacob = self(inputs, cond_inputs)
        log_probs = self.log_probs(u, cond_inputs)
        return (log_probs + log_jacob).sum(-1, keepdim=True)

Then, to provide your own PDF, just do model = FlowSequential(*modules,logprobs=your_own_PDF)

I found that constructing masks using the original MADE paper https://arxiv.org/abs/1502.03509 (nodes are randomly assigned ids) works better than assigning an id deterministically as in https://github.com/ikostrikov/pytorch-flows/blob/5520ebe4e91f5829db695ffd5f1ba7376d92a446/flows.py#L19

(The reason for this is likely simple, that if the output size is not an exact multiple of the number of hidden units, some input features will be assigned more hidden nodes than others)

Does your code support MADEs with an arbitrary number of masked linear layers? My main question is really if the function get_mask works for hidden layers of arbitrary depth. Thanks!

The readme states:

For the moment, I removed MNIST and CIFAR10 because I have plans to add pixel-based models later.

what do you mean by that? What's missing? I might be able to implement some stuff.