Hi,

Thank you for the paper and code.

I notice that when training with the annealing strategy, the samples is corrupted twice -- the first one is here, and the second one is here. In my understanding, the second one corresponds to { \sigma_1, ... \sigma_L } described in the paper. May I ask why the first one is needed?

Thank you and look forward to your reply!

Hi, thanks for your great work and the code sharing! I have a little confusion after I trace the code. In the original paper, the objective of eq(2) can be expand as eq(5) under the denoising score matching scenario. In the equation, the score network produce gradient of logarithmic data density , which condition on the sample and noise scale . We can see the snippet in the following (in page 6)

However, we can notice that the score network compute score which condition on data sample and label. here is the code of training. So which one is correct?

By the way, this is not a big issue :-) Since new noise conditioning technique is introduced in latter version (NCSNv2), the score network is only condition on sample . Just want to realize the correctness of old version.

I run the baseline model with the default setting, but it seems not converged (cifar10), loss is huge, around 6600000. Is this normal, or just mine this? If it is normal, why is this? Based on my research, traditional langevin dynamics can easily converge with restricted number of steps, such as 50. I'm quite curious why the author set the number of this sampling process to 1000, with a relatively small learning rate.

Thanks for your excellent work, the anneal model converges well, by the way, i just try to figure out why this method works. Can you please tell us?

Hi thanks for the great work. Because I couldn't find any information about custom dataset, I wonder if this repo works for training and generating using custom dataset? If yes, do I have to convert my images to (for example) Cifar10 format?

Hi, and thank you for your work.

I think there may be a bug in the anneal_runner, due to the log of dsm loss on validation. Specifically, the model is put in eval() in the following line

https://github.com/ermongroup/ncsn/blob/c4654a258e54a7da0c916132cebdd0818fae8fd2/runners/anneal_runner.py#L158

and it is never put in train() again.

Best, D

Hey,

I'm currently tracing out the story of diffusion generative models and right now, I'm studying the denoising score matching objective (DSM). I've noticed that your multi-scale approach relies heavily on it (and the original paper is quite old), so I decided that to ask my question here.

I gone through the theory of DSM and got a good grip on how it works and why it works. However, in practice I observe slow convergence (much slower convergence than with ISM) on toy examples. In particular I believe this might be due type of noise distribution selected. While not restricted, it seems everyone goes with a normal distribution since it provides a simple derivative. The derivate being 1/sigma**2 * (orig-perturbed). In practive, I've observed that the scale term in front causes the derivative to take values on the order of 1e4 for sigma=1e-2 and loss jumps around quite heavily. The smaller sigma, the slower the convergence. The loss never actually decreases, but the resulting gradient field looks comparable to what ISM gives.

Did you observe this in your experiments as well?

This may be a bit pedantic but, I wonder if there is a bug in the way the log probabilities are calculated for the annealed GMM distribution. My understanding is that annealed sampling uses scores from the ‘noise distribution’ which should reflect a Gaussian perturbed input. I’m assuming you guys achieve this by adding the noise distribution to the original GMM, hence allowing you to calculate the true log probabilities from the same GMM now scaled with the noise sigmas.

However, adding two Gaussian random variables should add the variances from both distributions as shown (taken from Wikipedia)

In this case sigma_x would be the original GMM sigma (which is set to 1), and sigma_y would correspond to the noise level. So shouldn't the calculation in the lines below be using something like sigma = np.sqrt(sigma**2 + self.sigma**2) instead of just the sigma passed in to the function? In my own tests, sampling with these updated sigmas gives more faithful results to the original distribution.

https://github.com/ermongroup/ncsn/blob/adb98fb5c9533b79933bfee0b3e883dc33f84e7a/models/gmm.py#L45-L46

Hi,

Thank you for your great work. I downloaded the checkpoints on Google drive and tried to do sampling, but I got this error:

RuntimeError: Error(s) in loading state_dict for DataParallel:
	Missing key(s) in state_dict: "module.normalizer.alpha", "module.normalizer.gamma", "module.normalizer.beta", "module.res1.0.normalize2.alpha", "module.res1.0.normalize2.gamma"
....

I was wondering how you and others load the model wrapped with DataParallel? I can train the model from scratch and fix the saving but wanted to check if there is better workaround.