Hello, what do you think is the minimum recommended GPU specs (memory etc) for good performance, both for training on a new dataset and for testing the pluralistic completion? Thanks!

Hi, thanks for your nice works. There are some details that bothered me. I would appreciate it if you could give me some advice.

1. BERT choice I noticed --BERT option was be used in all Transformer training and inference processes. Which situation we do not need to select this option?
2. attention mask As the paper described, transformer model capture the unmasked information to predict the probability distribution of missing regions. . But in the code CausalSelfAttention , I found model will capture information at all position, and attention filling mask does not be used except input occlusion. How can it guarantee to just pay attention on unmasked information?
3. auto-regressive As far as I understand, the model generate all masked pixels by a end to end mode rather than auto-regressive mode. During the inference, the model generate one pixel each iteration to improve sampling quality. If it works like I said, how do we guarantee the pixel quality of the first masked position in each iteration during the inference process?

Thanks again.

Thank you for proposing this good idea of using Transformer as a priori information! But why not use an end-to-end network for training, is it because the effect is not good?

Hello,

First of all, super cool model, and thanks for being so helpful with past questions. I was just wondering specifically how you generated pixel-wise completion probability maps as in Fig. 9 of your paper (I get how it's done in theory, I just wanted to see code if possible).

Thanks!

Hello, thank you for the great work. I tried to download the pretrained model but I get this error:

--2021-11-23 18:50:20-- https://www.dropbox.com/s/cqjgcj0serkbdxd/ckpts_ICT.zip?dl=1 Resolving www.dropbox.com (www.dropbox.com)... 162.125.3.18, 2620:100:6018:18::a27d:312 Connecting to www.dropbox.com (www.dropbox.com)|162.125.3.18|:443... connected. HTTP request sent, awaiting response... 301 Moved Permanently Location: /s/dl/cqjgcj0serkbdxd/ckpts_ICT.zip [following] --2021-11-23 18:50:20-- https://www.dropbox.com/s/dl/cqjgcj0serkbdxd/ckpts_ICT.zip Reusing existing connection to www.dropbox.com:443. HTTP request sent, awaiting response... 404 Not Found 2021-11-23 18:50:20 ERROR 404: Not Found.

The link doesn't seem to work.

Hi,

I am confused by the sample_mask function in transformer/utils/util.py, it seems that it does not use the argument num_sample but keeps num_sample=1, is it normal ?

Moreover, you use top_k=40 but the paper uses top_k=50. What is the best choice ?

Thanks,

Hi, @raywzy

I am trying to train the model on ImageNet with the following setting: --data_path /ILSVRC2012/train --validation_path /ILSVRC2012/val --mask_path /PConv-Keras/irregular_mask/train_mask --batch_size 32 --train_epoch 100 --nodes 1 --gpus 8 --node_rank 0 --n_layer 35 --n_embd 1024 --n_head 8 --GELU_2 --image_size 32 --use_ImageFolder

But I am getting Nan for train and test loss (screenshot attached).

This is happening for smaller datasets like Paris streetview (#train_images: 14900, #test_images 100) as well (screenshot attached).

Any suggestions on how to fix this issue?

Hello, Congrats for your nice works. I use a 16G GPU, but a single card can only run batch_size 3. I turned on mixed precision training, and the other settings are --n_layer 35 --n_embd 512 --n_head 8, which is the same as your model trained on Places2. So I want to know how do you use 8 GPUs and set the batch_size to 64 to train the transformer model?

I tried to get it working in Google Colab, but it does seem that 25GB RAM is not enough and it seems to crash during the first step. How much RAM is expected?

Hi, thank you for sharing code.

I want to run training code on grayscale images, but i got following error.

# Mask is 12022, # Image is 12022
# Mask is 12022, # Image is 0
Warnning: There is no trained model found. An initialized model will be used.
Warnning: There is no previous optimizer found. An initialized optimizer will be used.
Resume from Epoch 0
Traceback (most recent call last):
  File "main.py", line 139, in <module>
    mp.spawn(main_worker, nprocs=opts.gpus, args=(opts,))
  File "/home/naoki/.pyenv/versions/3.8.6/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 230, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/home/naoki/.pyenv/versions/3.8.6/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 188, in start_processes
    while not context.join():
  File "/home/naoki/.pyenv/versions/3.8.6/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 150, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/home/naoki/.pyenv/versions/3.8.6/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap
    fn(i, *args)
  File "/home/naoki/ICT/Transformer/main.py", line 73, in main_worker
    trainer.train(loaded_ckpt)
  File "/home/naoki/ICT/Transformer/DDP_trainer.py", line 203, in train
    run_epoch('train')
  File "/home/naoki/ICT/Transformer/DDP_trainer.py", line 139, in run_epoch
    logits, loss = model(x, y)
  File "/home/naoki/.pyenv/versions/3.8.6/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/naoki/.pyenv/versions/3.8.6/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 799, in forward
    output = self.module(*inputs[0], **kwargs[0])
  File "/home/naoki/.pyenv/versions/3.8.6/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/naoki/ICT/Transformer/models/model.py", line 254, in forward
    loss = F.cross_entropy(logits.view(-1, logits.size(-1)), targets.view(-1))
  File "/home/naoki/.pyenv/versions/3.8.6/lib/python3.8/site-packages/torch/nn/functional.py", line 2824, in cross_entropy
    return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: Expected object of scalar type Long but got scalar type Float for argument #2 'target' in call to _thnn_nll_loss_forward

I checked the size and data type before cross entropy loss. ''' logits: torch.Size([12, 1024, 512]), torch.float32 targets: torch.Size([12, 1024]), torch.float32 '''

Could you give me how to solve this problem. Thank you in advance.

Otherwise I get this error:

Traceback (most recent call last):
  File "inference.py", line 10, in <module>
    from datas.dataset import ImageDataset
ImportError: cannot import name 'ImageDataset' from 'datas.dataset' (/content/ICT/Transformer/datas/dataset.py)

Has anyone encountered this error，torch.multiprocessing.spawn.ProcessExitedException: process 0 terminated with signal SIGKILL. I am wondering a solution.

Hi, @raywzy

I tried to use both FFHQ and Places2 pretrained weights of upsampler. However, the Upsampler's pretrained weight of Places does not have enough quality. We know that the weight of the generator is trained during 322000 iterations. Do you think that the attached results are correct?

From left to right, 1st stage output | blended result with masked input image | raw output of upsampler | GT image | masked input image | raw output of upsampler within given mask

Trying to evaluate your codes: WSL2 under Windows 10 Nvidia RTX3090

Upon start provided script for inference (regardless of using ckpts) , from the very beginning got a message:

### Something Wrong ###
  0%|  

After that calculation continue. If "--FFHQ" or "--Places2_Nature" specified - inference finished with no error.

However if " ImageNet" specified - inference finished with an error:

raise AssertionError("Invalid device id")
AssertionError: Invalid device id

NVCC report:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Tue_May__3_19:00:59_Pacific_Daylight_Time_2022
Cuda compilation tools, release 11.7, V11.7.64
Build cuda_11.7.r11.7/compiler.31294372_0

Tried to change line 44 in "run.py" from CUDA_VISIBLE_DEVICES=0,1 to CUDA_VISIBLE_DEVICES=0

-- no luck.

Found the reason for message:

### Something Wrong ###
  0%|  

That happened due to the different file image/mask names (they should be identical). However the rest of the issue still exists. Beside of that after finishing inference (regardless specified ckpts) "output" sub folder in the folder "Guided_Upsample" got created but it is empty. "output" sub folder in the folder "Transformer" got created and consists generated image with 32 or 48 pixel.

Thanks for the nice code! I tried to enable "--random_stroke" so the masks will be generated on the fly. But it seems that the "--mask_path" still needs input, otherwise it will point to the default path which is not on my PC.

Thank you for sharing. We would like to use your model as a baseline for comparison. However, there is currently a little problem based on the inference code you provided, as shown in the below, how can I solve this situation?