Hi, I am trying to implement this repo. I've downloaded the ade20k checkpoints and created a conda env following your yaml file.

When I run the testing command python test.py --name oasis_ade20k --dataset_mode ade20k --gpu_ids 0 \ azureuser@ivan-fantasia-default --dataroot test_images --batch_size 1 I get the following error:

/opt/conda/conda-bld/pytorch_1544176307774/work/aten/src/THC/THCTensorScatterGather.cu:176: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [232,0,0], thread: [101,0,0] Assertion `indexValue >= 0 && indexValue < tensor.sizes[dim]` failed.
Traceback (most recent call last):
  File "test.py", line 25, in <module>
    generated = model(None, label, "generate", None)
  File "/anaconda/envs/oasis/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/anaconda/envs/oasis/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 141, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/anaconda/envs/oasis/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/azureuser/IM/OASIS/models/models.py", line 72, in forward
    fake = self.netEMA(label)
  File "/anaconda/envs/oasis/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/azureuser/IM/OASIS/models/generator.py", line 36, in forward
    z = torch.randn(seg.size(0), self.opt.z_dim, dtype=torch.float32, device=dev)
RuntimeError: Creating MTGP constants failed. at /opt/conda/conda-bld/pytorch_1544176307774/work/aten/src/THC/THCTensorRandom.cu:35

I am running on test_image folder which are some ade20k images.

Any suggestion? Thanks ;)

hello dear author @edgarschnfld

I downloaded the pre-trained model and added noise to the labels that are input for the network but the test failed and the model didn't work for the noisy dataset at all. what should I do for testing the model on noise?

hi dear @SebastianSchildt @SushkoVadim I used another network to create label maps from images (this network: https://github.com/CSAILVision/semantic-segmentation-pytorch) then fed your oasis network with them in test mode, but it doesn't work. it says: [RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED] would you please help ?

thank you so much

While doing my experiments I've replaced SPADE's discriminator with OASIS one and tried this with VGG being both disabled and enabled. In any case images seems to be visually better (even though FID is higher on CityScapes) but what is very noticeable (unfortunately I can't share images) is that OASIS discriminator for some reason encourages repetitive patterns in result images, specifically in broad semantic regions like road. Do you have any idea what could cause that given that the only change in my model I did is replacing MultiscaleDiscriminator with OASISDiscriminator?

I'm sorry for a vague question but I just wonder if there any obvious things I can try to fight this. I also use LabelMix loss with Lambda = 10.

Hey!

Could you explain the reasoning behind beta2 being equal to 0.999 instead of 0.9 (like in SPADE)? Didn't find any mention of this neither in the paper nor in code

hello dear all @SebastianSchildt @LGro @johannes-mueller @schorg @c2bo

Does anybody knows what the input and output of this network is exactly? does it take those labels and the output is the normal pictures? Am I right?

thanks in advance

To reproduce your results, one thing I am not sure, could you please help me to make sure?

For cityscapes dataset, from https://github.com/NVlabs/SPADE, and [https://github.com/mseitzer/pytorch-fid](FID computation). Without doubt, your synthesized images are from your codes.

But how about the real images?

1. Resize. resize real images to the same size with your synthesized (256 * 512) in nearest down sampling?
2. what are the real images? only val image froms cityscapes (gtFine/val) or all images (gtFine/val, gtFine/test, gtFine/train)?

Thanks a lot.

Hi. @edgarschnfld @SushkoVadim

I am a student studying semantic image synthesis. Thank you for the great work. I have two questions about the difference between paper and code.

1. EMA As you cite [Yaz et al., 2018], exponential moving average is a good technique for training GAN. However, in your code https://github.com/boschresearch/OASIS/blob/6e728ec5f5b7b69d6744485aa69a355e0164423c/utils/utils.py#L125-L132 I think below code might be added

model.module.netG.state_dict()[key].data.copy_(
    model.module.netEMA.state_dict()[key].data
)

If not, netG is not trained using EMA.

Yaz, Yasin, et al. "The unusual effectiveness of averaging in GAN training." International Conference on Learning Representations. 2018.

2. 3D noise

If I do not misunderstand your paper, the paper says that the noise of OASIS has been sampled from a 3D normal distribution. And this is one of the main differences with SPADE. However, in your code at, https://github.com/boschresearch/OASIS/blob/f049c37ff711792c09e573586640e1fd11d69d58/models/generator.py#L34-L39 Noise is not sampled from the 3D normal distribution. It was also sampled from a 1D normal distribution. Then expand it to 3D, which replicates the same vector spatial way. In my opinion, this code should be replaced by

z = torch.randn(seg.shape, ...)

I think both two parts are pretty crucial for your paper. If there is any reason for these choices or my fault, please let me know.

Thank you.

Hi @SushkoVadim @edgarschnfld,

Thanks for the excellent work. I noted that when updating EMA, you collect the running stats for BatchNorm before FID computation, image or network saving (see https://github.com/boschresearch/OASIS/blob/master/utils/utils.py#L133).

May I ask about the purpose and intuition of this operation? How significant would it affect the model performance? Thank you in advance.

I noticed that in the code you will calculate FID according to test images and save the best model accordingly(correct me if wrong). I am wondering if is the best model used to report numbers in the paper? Thanks

Hey, I've noticed that the weight reweighting that is used in the code doesn't match the papers equation Indeed, here, the formula is BxHxW/(num_pixels_of_class_i*num_of_non_zeros_classes_in_batch) whereas the paper showcase the following equation: BxHxW/(num_pixels_of_class_i). Did i get something wrong? If not can you clarify which one is the correct equation? Thanks

Hello, Thank you for sharing your great work! I noticed that on line 36 of the losses.py file, coefficients = torch.reciprocal(class_occurence) * torch.numel(label) / (num_of_classes * label.shape[1]) it seems that label.shape[1] should be modified to label.shape[0]here so that the solution is in line with the equation in the paper (HxW), not sure if I understand it correctly? Or is there some detail I'm not noticing? Thanks.

how to generate diverse images from a "Label map"? for example: how to generate "OASIS1"、"OASIS2"、"OASIS3"、"OASIS4"、"OASIS5" form "Label map" what is the terminal line?

Hello @edgarschnfld, @SushkoVadim,

thank you for sharing your work! I was wondering if you have explored the importance of the spectral norm in the discriminator in more detail. I also noticed that you applied the spectral norm to every layer except the last convolution, see https://github.com/boschresearch/OASIS/blob/master/models/discriminator.py#L23 Is this a specific design decision?

It would be great if you could share some insights here. Thanks,

Nikolai

Hi, I'm trying to use OASIS discriminator with my own dataset's semantic segmantation. In short, my ground truth onehot segmantation has 3 channels, and it is shaped like: channel 0: background, channel 1: human, and channel 2 for the fake pixel label as paper said. But when calculate cross entropy loss, the target label seems to have been processed into an one channel label like: 1:background, 2:human whenis_real=Trueor 0: fake when is_real=False. The model still works properly, but the predicted semantic segmentation display order is not quite correct when visualizing. Can you please tell me if my process is correct, any suggestions would be very helpful, thanks.