Hi, thanks for this great work! I was recently researching on the effect of augmentation noise.

My intuition is that larger growth rate will result in larger capacity of the model. However, by altering DF_74_10 to DF_74_50, I observe that the convergence becomes extremely slow than DF_74_10. I was wondering if you could share any experience on the selection of growth rate. :-)

Congratulations on very good work! I'm very impressed with what you've been able to accomplish, especially taking the limited compute budget into account!

Question 0. In Table 1 you report the model parameters and training time for three datasets, but not ImageNet64. Do you have the time/parameter numbers for ImageNet64? Are they the same as ImageNet32?

Question 1. FID computation. You write the for image generation you sample from N(0, 0.8). Did you compute FID using this N(0, 0.8) or N(0, I)? I ask because I believe FID is slightly biased in favor of GANs and denoising diffusion models, because they usually trade-off quality of individual images for variability. You could find the sweet point in this trade-off for DenseFlow by computing FID for N(0, alpha) where alpha=0.5,0.6,...,1. I'd be curious to see the best number you'd be able to get.

Question 2. Since NF sample ~num_pixels faster than autoregressive models, I'm curious as to whether we could improve samples at the cost of 10-20x longer sampling time. For example, we could sample a batch of 128 fake images, then perform SGD minimizing LLH wrt the 128 fake images. Even if we do 100 SGD steps we'd be ~num_pixels/100 times faster than autoregressive models.

Question 3. Do you think increased network width worked better on ImageNet32 or ImageNet64? I ask because I like to think of ImageNet64 as 4 channel-wise copies of ImageNet32x32, which effectively increases the network width. I imagine if we go to 128x128 or 256x256 the network depth issue of Normalizing Flows may be smaller. What do you think?

Question 4. Did you find batch_size=32 to optimize better for ImageNet64, or was this mainly for memory savings?

Question 5. Did you try using gradient checkpointing?

Question 6. Did you try training in float16?

Question 7. How large was GPU utilization on ImageNet64 with batch_size=32? I.e., if you wrote nvidia-smi would it tell you 50% or 100%?

Hi, I'm having a hard time reproducing the CIFAR-10 results (haven't tried others yet). I am more consistently getting around 3.1bpd on the eval set and can get around 2.9bpd on the training set. I've even tried training longer and different scheduling than is described in the appendix. Was this the best result that you got or an average? Do you have a variance? Do you have a checkpoint that could be released?

Hi! I'm seeing this error when using the evaluation and train command for ImageNet32

Loaded weights for model at 22/22 epochs Traceback (most recent call last): File "/content/DenseFlow/experiments/image/eval_loglik.py", line 72, in bpd = dataset_elbo_bpd(model, eval_loader, device=device, double=eval_args.double) File "/content/DenseFlow/denseflow/utils/loss.py", line 81, in dataset_elbo_bpd bpd += elbo_bpd(model, x).cpu().item() * len(x) File "/content/DenseFlow/denseflow/utils/loss.py", line 28, in elbo_bpd return loglik_bpd(model, x) File "/content/DenseFlow/denseflow/utils/loss.py", line 12, in loglik_bpd return - model.log_prob(x).sum() / (math.log(2) * x.shape.numel()) File "/content/DenseFlow/denseflow/flows/flow.py", line 33, in log_prob x, ldj = transform(x) File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(*input, **kwargs) File "/content/DenseFlow/experiments/image/model/flow_modules.py", line 127, in forward eta, delta_ll = self.cross_unit_coupling(g_in, self.blocks[i-1]) File "/content/DenseFlow/experiments/image/model/flow_modules.py", line 114, in cross_unit_coupling eta_hat = mu + torch.exp(log_scale) * eta RuntimeError: The size of tensor a (64) must match the size of tensor b (16) at non-singleton dimension 3

and with --k 200 File "/content/DenseFlow/experiments/image/eval_loglik.py", line 75, in bpd = dataset_iwbo_bpd(model, eval_loader, k=eval_args.k, kbs=eval_args.kbs, device=device, double=eval_args.double) File "/content/DenseFlow/denseflow/utils/loss.py", line 106, in dataset_iwbo_bpd bpd += iwbo_bpd(model, x, k=k, kbs=kbs).cpu().item() * len(x) File "/content/DenseFlow/denseflow/utils/loss.py", line 57, in iwbo_bpd if kbs: return - iwbo_batched(model, x, k, kbs).sum() / (x.numel() * math.log(2)) File "/content/DenseFlow/denseflow/utils/loss.py", line 44, in iwbo_batched ll_stack = model.log_prob(x_stack) File "/content/DenseFlow/denseflow/flows/flow.py", line 33, in log_prob x, ldj = transform(x) File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(*input, **kwargs) File "/content/DenseFlow/experiments/image/model/flow_modules.py", line 127, in forward eta, delta_ll = self.cross_unit_coupling(g_in, self.blocks[i-1]) File "/content/DenseFlow/experiments/image/model/flow_modules.py", line 114, in cross_unit_coupling eta_hat = mu + torch.exp(log_scale) * eta RuntimeError: The size of tensor a (3200) must match the size of tensor b (16) at non-singleton dimension 3

I'm able to generate samples but not get the bpd scores.

I was comparing DenseFlow against VDM on ImageNet64x64.

DenseFlow: 3.35 BPD, 130M, 1 V100 ~2 weeks VDM: 3.4 BPD, ?M, 128 TPUv3 for ?weeks?

It looks like DenseFlow gets better BPD with ~100x less compute, at the cost of worse FID.

Question 0. Do you know what the training loss of DenseFlow/VDM? I imagine low training loss leads to good FID (perhaps VDM has train=1.0 bpd and valid=3.4 bpd, while DenseFlow has train=3.3 and valid=3.4).

Question 1. Did you make any test cases for the BPD computation? It just sounds too good to be true that we can get better BPD with 100x less compute.

Question 2. It may be there is a trade-off between BPD and FID. That is, DenseFlow get good BPD but bad FID, while VDM gets good FID and worse BPD. Do you believe this is the case? If so, what do you believe cause this phenomenon?