Hi, I am conducting the training on cifar10 and testing the FID (50k samples with 100 timesteps) after trained certain iterations. However, the FID is instable in my experiments. Specifically, it gets 12.00, 7.67, 16.88, 7.78, 6.65, 18.43, 11.51 when trained 80k, 90k, 100k, 150k, 200k, 250k, 300k iterations, respectively (i.e., an up and down pattern). Does anyone find the same issue?

how can i successfully save each n-timesteps of the diffusion sampling step. For so far i edited lines of code in p_sample_loop_progressive(), with something like

for step, i in enumerate(indices):
 

            t = th.tensor([i] * shape[0], device=device)
            
            with th.no_grad():
                out = self.p_sample(
                    model,
                    img,
                    t,
                    clip_denoised=clip_denoised,
                    denoised_fn=denoised_fn,
                    model_kwargs=model_kwargs,
                )
                yield out
                if step % 1000 == 0:
                    print('step',step)

                    sample = out["sample"]
                    sample = ((sample + 1) * 127.5).clamp(0, 255).to(th.uint8)
                    sample = sample.permute(0, 2, 3, 1)
                    sample = sample.contiguous()

                    image.extend([sample.cpu().numpy() for sample in [sample]])
           
                    arr = np.concatenate(image, axis=0)

                    arr = arr[: shape[0]]
   
                    img = Image.fromarray(arr[0])

                    out_path = os.path.join(logger.get_dir(), f"samples_{label}_{str(step).zfill(4)}.npz")
                    out_image = os.path.join(logger.get_dir(), f"samples_{label}_{str(step).zfill(4)}.tif")
                    img.save(out_image, compression='raw')
                    np.savez(out_path, arr)
                img = out["sample"]

but i always get the fully noisy image.

Many thanks in advance

Hi! What's the reason y'all are using explicit utilities to convert float32 to float16 tensors instead of using torch.amp? I'm thinking of building out my own implementation of this repository, using einops, and was curious if this is something I'd need to take into account instead of just using torch.amp.

When I use the checkpoint from the part "Upsampling 256x256 model (280M parameters, trained for 500K iterations) ", the program will report an error. Do you know why this happens？I used the recommended model parameters.

RuntimeError: Error(s) in loading state_dict for UNetModel: Missing key(s) in state_dict: "input_blocks.4.0.skip_connection.weight", "input_blocks.4.0.skip_connection.bias", "input_blocks.7.1.norm.weight", "input_blocks.7.1.norm.bias", "input_blocks.7.1.qkv.weight", "input_blocks.7.1.qkv.bias", "input_blocks.7.1.proj_out.weight", "input_blocks.7.1.proj_out.bias", "input_blocks.8.1.norm.weight", "input_blocks.8.1.norm.bias", "input_blocks.8.1.qkv.weight", "input_blocks.8.1.qkv.bias", "input_blocks.8.1.proj_out.weight", "input_blocks.8.1.proj_out.bias", "input_blocks.10.0.skip_connection.weight", "input_blocks.10.0.skip_connection.bias", "input_blocks.10.1.norm.weight", "input_blocks.10.1.norm.bias", "input_blocks.10.1.qkv.weight", "input_blocks.10.1.qkv.bias", "input_blocks.10.1.proj_out.weight", "input_blocks.10.1.proj_out.bias", "input_blocks.11.1.norm.weight", "input_blocks.11.1.norm.bias", "input_blocks.11.1.qkv.weight", "input_blocks.11.1.qkv.bias", "input_blocks.11.1.proj_out.weight", "input_blocks.11.1.proj_out.bias". Unexpected key(s) in state_dict: "input_blocks.12.0.op.weight", "input_blocks.12.0.op.bias", "input_blocks.13.0.in_layers.0.weight", "input_blocks.13.0.in_layers.0.bias", "input_blocks.13.0.in_layers.2.weight", "input_blocks.13.0.in_layers.2.bias", "input_blocks.13.0.emb_layers.1.weight", "input_blocks.13.0.emb_layers.1.bias", "input_blocks.13.0.out_layers.0.weight", "input_blocks.13.0.out_layers.0.bias", "input_blocks.13.0.out_layers.3.weight", "input_blocks.13.0.out_layers.3.bias", "input_blocks.13.0.skip_connection.weight", "input_blocks.13.0.skip_connection.bias", "input_blocks.13.1.norm.weight", "input_blocks.13.1.norm.bias", "input_blocks.13.1.qkv.weight", "input_blocks.13.1.qkv.bias", "input_blocks.13.1.proj_out.weight", "input_blocks.13.1.proj_out.bias", "input_blocks.14.0.in_layers.0.weight", "input_blocks.14.0.in_layers.0.bias", "input_blocks.14.0.in_layers.2.weight", "input_blocks.14.0.in_layers.2.bias", "input_blocks.14.0.emb_layers.1.weight", "input_blocks.14.0.emb_layers.1.bias", "input_blocks.14.0.out_layers.0.weight", "input_blocks.14.0.out_layers.0.bias", "input_blocks.14.0.out_layers.3.weight", "input_blocks.14.0.out_layers.3.bias", "input_blocks.14.1.norm.weight", "input_blocks.14.1.norm.bias", "input_blocks.14.1.qkv.weight", "input_blocks.14.1.qkv.bias", "input_blocks.14.1.proj_out.weight", "input_blocks.14.1.proj_out.bias", "input_blocks.15.0.op.weight", "input_blocks.15.0.op.bias", "input_blocks.16.0.in_layers.0.weight", "input_blocks.16.0.in_layers.0.bias", "input_blocks.16.0.in_layers.2.weight", "input_blocks.16.0.in_layers.2.bias", "input_blocks.16.0.emb_layers.1.weight", "input_blocks.16.0.emb_layers.1.bias", "input_blocks.16.0.out_layers.0.weight", "input_blocks.16.0.out_layers.0.bias", "input_blocks.16.0.out_layers.3.weight",

Hi,

Thanks for releasing the code, especially in PyTorch. I was wondering if you are also planning to release the checkpoints for models from Table 3?

Hello,

I can't manage to sample from a model. When I am trying to do so : RuntimeError: Error(s) in loading state_dict for UNetModel: Missing key(s) in state_dict: XXX size mismatch for: XXXX

This happens when I try to load the model00000.pt

Do you have an idea how to solve this issue ?

I noticed that in the dot product attention implementation here (the QKVAttention class) the scale is $1 / \sqrt[\leftroot{-2}\uproot{2}4]{d}$ instead of $1 / \sqrt{d}$. It is here in the codebase: https://github.com/openai/improved-diffusion/blob/783b6740edb79fdb7d063250db2c51cc9545dcd1/improved_diffusion/unet.py#L247

It is a minor issue and probably does not matter in the overall performance of models, but thought it's worth pointing out.

Hi, thanks for your great work!

Recently I have been working on the conditional generation of diffusion models, and I found that it has classifier guidance and classifier-free guidance. For the former, a classifier needs to be pre-trained. But I didn't find this pre-trained classifier in your code. I am a little confused if you are using the classifier-free guidance.

After trying out the system on CPU I'm trying to set it up using the GPU and I get the following error:

TypeError: init() got an unexpected keyword argument 'broadcast_buffers'

Tried it on two different machines and got the same result. Anyone came across this?

I’m working on recreating your results and I’m able to train with a bare MSE loss just fine. However, my implementation of the RESCALED_MSE loss (mse for the means, vlb for the variances) gives a loss orders of magnitude larger than the bare MSE loss. This seems wrong, since you have an explicit 1/1000 scaling factor on the vlb loss, for the purpose of not overwhelming the MSE loss.

What is the expected scale of RESCALED_MSE loss?

I tried to train a DDPM to produce cat images. But it seems like the training process does not work properly. Can anyone tell me what happened? No error is reported.

I am using Wsl2.

Here is the code: train_ddpm.sh #!/bin/bash MODEL_FLAGS="--image_size 64 --num_channels 128 --num_res_blocks 3" DIFFUSION_FLAGS="--diffusion_steps 4000 --noise_schedule cosine" TRAIN_FLAGS="--lr 1e-4 --batch_size 128" python scripts/image_train.py --data_dir cats/ $MODEL_FLAGS $DIFFUSION_FLAGS $TRAIN_FLAGS

console outputs: (Generator) root@LAPTOP-I6AJJ63E:/mnt/d/CatPicGenerateDDPM/improved-diffusion# ./train_ddpm.sh Logging to /tmp/openai-2022-07-20-21-25-58-111937 creating model and diffusion... creating data loader... training... ./train_ddpm.sh: line 5: 455 Killed python scripts/image_train.py --data_dir cats/ $MODEL_FLAGS $DIFFUSION_FLAGS $TRAIN_FLAGS

Thank you very much!

I have noticed that sampling with this model takes an extraordinary amount of time, far slower than any individual training loop. At 4000 diffusion steps, it seems to get through around 100-500 epochs before it can generate a single image. Is this normal behavior, and if not are there any fixes to this? Thank you for all of the help.

I was looking through the code to see how the paper was implemented, but I ran into an issue when looking at the part of the paper measuring the KL loss between two Gaussians:

Specifically, the Loss at time t-1 is the KL loss between the predicted gaussian and real gaussian at time t-1. The predicted gaussian is defined as follows:

And the real gaussian is defined as follows:

The formulation of the loss function makes sense to me, but when I look at the code, it looks like the authors are having the model predict mu_tilde (eq 11) as opposed to mu (eq 13). I'm looking at the following function in the code: https://github.com/openai/improved-diffusion/blob/783b6740edb79fdb7d063250db2c51cc9545dcd1/improved_diffusion/gaussian_diffusion.py#L232

In this function, the mean is calculated from epsilon by first calculating the prediction for x_0, then calculating the mean at time t.

To predict x_0, the following function is used: https://github.com/openai/improved-diffusion/blob/783b6740edb79fdb7d063250db2c51cc9545dcd1/improved_diffusion/gaussian_diffusion.py#L328

But, this function looks to be the formulation for the mean function (eq 13)

I have a couple of questions regarding the implementation:

1. Why is the mean function (eq 11) for the real gaussian distribution (eq 12) being used when retrieving the value of the predicted gaussian distribution (eq 3) when the formulation for the predicted gaussian distribution is formulated as a function of eq 13?
2. Why is x_0/x_start being calculated directly from eq 13, the predicted mean?

Thanks for the help!

Hi !

I was wondering if some people tested the diffusion training process with few images ( 1000 or less) and obtained good results ?

I am working with rare pathology images, and I have only 300 of them. I am seeing some unexpected behavior on the generated samples (lack of diversity, color shifts etc...) As far as I am aware, diffusion models work better with low training data points than GANs ?

Thank you

Hello,

I am using the repo to produce synthetic histopathology images.

However, I am observing a weird color shift during training. As training progress, this shift is getting worst and worst. Basically, the images I am working with are all kind of pinkish, but produced samples are sometimes in this range, but can also be yellow, green, red, etc...

Is this a problem that has been observed before with diffusion models ? If yes, is there a know solution or an explanation for this behavior ?

Thank you so much !

I noticed a scaling factor of 0.5 when sampling from p(xt-1|xt). I tried to find the definition of this 0.5 on the paper but failed. Is there any special factor behind this 0.5? would the performance be different without this factor?

https://github.com/openai/improved-diffusion/blob/783b6740edb79fdb7d063250db2c51cc9545dcd1/improved_diffusion/gaussian_diffusion.py#L386