Hi, it's a great work! But I am a little confused about how to generate a new image? Shall I give the sentence tokens and then use them to predict the image tokens? And where to inject the noise? It will be very appreciate that you can answer these questions, thank you!

I changed this line as sugggested to use the GPU:

# This can be changed to a GPU, e.g. 'cuda:0'.
dev = torch.device('cuda:0')

And I tried to execute the notebook. I got the following error message:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/tmp/ipykernel_11/3257249919.py in <module>
      1 import torch.nn.functional as F
      2 
----> 3 z_logits = enc(x)
      4 z = torch.argmax(z_logits, axis=1)
      5 z = F.one_hot(z, num_classes=enc.vocab_size).permute(0, 3, 1, 2).float()

/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1108         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1109                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110             return forward_call(*input, **kwargs)
   1111         # Do not call functions when jit is used
   1112         full_backward_hooks, non_full_backward_hooks = [], []

/opt/conda/lib/python3.8/site-packages/dall_e/encoder.py in forward(self, x)
     91                         raise ValueError('input must have dtype torch.float32')
     92 
---> 93                 return self.blocks(x)

/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1108         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1109                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110             return forward_call(*input, **kwargs)
   1111         # Do not call functions when jit is used
   1112         full_backward_hooks, non_full_backward_hooks = [], []

/opt/conda/lib/python3.8/site-packages/torch/nn/modules/container.py in forward(self, input)
    139     def forward(self, input):
    140         for module in self:
--> 141             input = module(input)
    142         return input
    143 

/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1108         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1109                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110             return forward_call(*input, **kwargs)
   1111         # Do not call functions when jit is used
   1112         full_backward_hooks, non_full_backward_hooks = [], []

/opt/conda/lib/python3.8/site-packages/dall_e/utils.py in forward(self, x)
     41                         w, b = self.w, self.b
     42 
---> 43                 return F.conv2d(x, w, b, padding=(self.kw - 1) // 2)
     44 
     45 def map_pixels(x: torch.Tensor) -> torch.Tensor:

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument weight in method wrapper___slow_conv2d_forward)

I am having trouble getting the dVAE to train properly if I include the KL loss term with a uniform prior over the number of visual tokens. Does anyone here has had similar experiences or problems? The paper mentions an increasing schedule for the kl weight factor but I cant get it to work properly and results are always better if I set the KL loss to zero altogether.

Maybe someone can help?

Thanks for releasing the paper as well as the codes!

Could you give some hints on the hyperparameters of the bottleneck that might affect the performance?

1. The downsampling ratio. In the original VQ-VAE paper, they only use 4 times downsampling (comparing to 8 in DALL-E) and it seems their generated images lack a global structure (I assumed also because they didn't use a powerful prior model). Is that using a higher downsampling rate, the global structure is better preserved? Or easier for the prior model to learn?

2. The codebook size is set to 2^13, have you tried using smaller codebook size? Presumably, as the codebook size shrinks, the VAE can hardly reconstruct the image. What does the reconstructed image with a small codebook look like? Is the texture still preserved but the global structure distorted or something else?

I also have an additional question to the inference stage of the model. Is the image tokens sampled from the prior transformer auto-regressively or using other searching technique? Also, how to control the number of the generated image tokens to be exactly 32 * 32?

Although this codebase is for the vae part. would appreciate if you could help in understanding few components of the transformer part also. In the paper and blog released, you mentioned that you use Child et al paper. Can you elaborate on what you use as the block size? 8/16/32 If we use a block size of 16 for example then how do you implement the convolutional kernel, it has gaps of only 1 block but if you have sparse block of size 16 then it doesn't make sense.

Also, when you are training the gpt style model. Even though the loss and perplexity reduce, how do you identify when the perplexity/loss value of the 1.2B parameter model is sufficient? like is a loss of 4/5 good or should it be <1.

Hello i am brand new to the github community and coding, i have zero idea how to install this but im an artist and it would be an excellent resource for non copyrighted images, i know its alot to ask but can someone please tell me how to install this code i made my account for this specifically for this

I just downloaded the repo to my local file system and used jupyter notebook and then opened and played the notebook. I also downloaded the encoder and decoder to the same folder for ease of loading. It says that 'preprocess' is not defined, but it seems to be. Admittedly, a bit rusty. Running Python 3.9 on Mac OSX. Also, I may be way out of line with respect to the purpose, but I was expecting to see code that took natural language input (e.g. "Show me a penguin on snow") and then DALL*E returns the provided image.

Originally posted by @metaphorz in https://github.com/openai/DALL-E/issues/5#issuecomment-787310649

pytorch:1.7.1 torchversion: 0.8.2 when run the code, it seems wrong:

Traceback (most recent call last): File "E:/github/DALL-E-master/test.py", line 46, in display(T.ToPILImage(mode='RGB')(x[0])) File "C:\ProgramData\Anaconda3\lib\site-packages\torchvision\transforms\transforms.py", line 185, in call return F.to_pil_image(pic, self.mode) File "C:\ProgramData\Anaconda3\lib\site-packages\torchvision\transforms\functional.py", line 202, in to_pil_image 'not {}'.format(type(npimg))) TypeError: Input pic must be a torch.Tensor or NumPy ndarray, not <class 'numpy.ndarray'> Original image:

What's the meaning of logit_laplace_eps here since both the input and output are [0,1] tensors. https://github.com/openai/DALL-E/blob/5be4b236bc3ade6943662354117a0e83752cc322/dall_e/utils.py#L51

Hi, thanks for the code!

It's a simple question, I found the output dimension of decoder is set to 2 * self.output_channels, which is 6. I expect the output dimension should be 3 (RGB). Can you kindly explain the reason?

Thank you in advance!

https://github.com/openai/DALL-E/blob/3381ae9a10bafe3cb1c7c9fff554565ad7751e7f/dall_e/decoder.py#L82

screens / installation

https://disk.yandex.ru/i/kCSzh6LodRkWlQ

CAN You explain me what should i do here and how launch the app?

thank you in advance