Official PyTorch implementation of RobustNet (CVPR 2021 Oral)

Hi, Thanks for your great work!

I have run the code for MobileNet, however, I get largely lower results than your reported results.

I got the following results:

MobileNet-Base, mIoU = 20.12%

MobileNet-IBN, mIoU = 27.41%

MobileNet-ISW, mIoU = 23.24%

All results are tested on the last model and on the CityScapes.

I also found an error when loading the dataset,

https://github.com/shachoi/RobustNet/blob/55e385a6600482793cf64544217678196dbcd3b8/datasets/gtav.py#L438

This line will causes ``TypeError: 'bool' object is not subscriptable'' for Pthon 3.

And I change:

mask_copy[(mask == np.array(k))[:,:,0] & (mask == np.array(k))[:,:,1] & (mask == np.array(k))[:,:,2]] = v

to

sel_mask = np.logical_and(mask[:, :, 0] == np.array(k)[0], mask[:, :, 1] == np.array(k)[1], mask[:, :, 2] == np.array(k)[2])
mask_copy[sel_mask] = v

for all the dataset loaders.

I am not sure if this causes the low results. Could you help me to address this problem? Thanks!

Thank you for your outstanding work. You only give the file structure of Cityscapes, but you don't provide the file structure of gtav, Synthia, BDD, etc. can you provide them?

Hi, I tried the experiment with ResNet-101 for ISW by training on the GTAV (I changed the script based on the city_isw_r101.sh). However, I got results that are similar to ResNet-50. For example, the mIoU in Cityscape is 36%.

Do you have any results on ResNet-101 for GTAV? And how to train ISW with ResNet-101 for GTAV?

Hi,

Thanks for such a nice work! I was wondering if it is possible to reproduce somehow the results of Fig. 9, where you compare with DA methods. Did you change the backbone to ResNet-101 + DeepLab v2 to compare with those UDA methods (Appendix A.1)? Or did you downscale their backbone to Resnet-50 and changed their decoder to DeepLab v3+?

If you changed your backbone to ResNet-101 + DeepLab v2, is it possible to run an experiment with this setting? What steps should I take?

Since I confirmed that the author of the paper was Korean, I wrote the question in Korean. If I have to write a question in English, if you give me an answer, I will write it.

안녕하세요. 'RobustNet: Improving Domain Generalization in Urban-Scene Segmentation via Instance Selective Whitening' 양질의 논문을 작성해주셔서, 정말 흥미롭게 잘 읽었습니다. 논문 내용과 관련해서 2가지 질문 사항이 있는데, 답변 해주시면 감사하겠습니다.

1. photometric transformation(i.e. color jittering)에 따른 covariance matrix의 elements 확인

원-이미지가 있고 photometric transformation을 통한 변환-이미지 2개가 있을 때, 각각의 covariance matrix of feature map을 계산하고 2개의 covariance matrix의 값을 직접 확인해보고 싶습니다. 논문에서 언급된 Sensitive covariances는 Illumination이 크게 달라지고, Insensitive covariances는 photometric transformation에 영향을 받지 않고 scene structure에 민감한 부분을 직접 확인해보고 싶은데, 혹시 이에 대한 구현이 있을까요?

2. feature-covariance matrix 차이에 대한 질문

개인적인 생각으로는 일반적인 segmentation model with ResNet backbone은 photometric transformation에 따른 feature-covariance matrix의 elements 차이가 나타나지 않고, model의 layer에 Instance-Normalization 혹은 Loss_DWT와 같은 whitening이 적용된 모델만 photometric transformation에 따른 feature-covariance matrix의 elements가 차이를 나타낼 것으로 보이는데, 혹시 해당 내용이 맞을까요? 아니면, 일반적인 모델도 논문에서 언급된 covariance의 차이를 보일까요?

감사합니다 ! I love LG A.I Research.

Thanks for your great work. I have tested your trained Res50 model successfully on cityscapes. But I noticed that there are no test loaders for other datasets. So I tried to use the val loader of Mapillary to do inference, but the result especially the compose images seem not compatible with validation mean_iou(0.5+). So could you update eval.py or tell me how to eval on other datasets, thanks a lot.

Hi,

I came across this Uniform dataset in your implementation. For example the class GTAVUniform in datasets/gtav.py. This uniform dataset appears to be the default dataset for your baselines, but I haven't found any info about it in the Readme or the paper. Could you help me understand what is it for?

Thanks!

Hi,

I have a question about inference on Mapillary dataset. I found that image resolution in Mapillary is not fixed. So, how did you test model on Mapillary dataset? Resizing all images to the same resolution or using slide window? This makes me confused.

Dear author, In your paper, you mentioned that there are 12403 images for a train set in GTAV. However, in your split file 'split_data/gtav_split_train.txt', there are only 12388 image names. Is this a mistake?

Thank you so much for your great work! I am very interested in your work and am trying your code. However, I only have one gpu. After modifying the code and training the model, it is found that the baseline performance has dropped significantly. Do you have the code for a single CPU? I would be very grateful if you could provide it to me.

Thank you very much for this great work! I am trying to train the baseline model based on your code. However, I can not obtain the reported baseline performance.

I use Cityscapes train set to train the model and test the model on BDD-100K. But the baseline performance is only 42.7, which is 2.3 mIoU lower than your baseline (44.96).

Hello, Thanks for wonderful code and explanations.

I am running your code on ResNet50 with Cityscapes dataset. Specifically I run the following command.

CUDA_VISIBLE_DEVICES=0,1 ./scripts/train_r50os16_cty_ibn.sh # Train: Cityscapes, / ResNet50, IBN-Net

But I am getting error., I am attaching the log file screenshot. Could you let me know what can be the problem.

Thank you for your work in the field of domain generalization. There is a question about self dataset on RobustNet. Can I test my own dataset based on the existing code？

Hey! I saw your email and was able to work on it for today, so from what I saw in the docs it seems you have to be in an additional method when you create your camera, I'm not exactly sure what camera you are using or how it should interact with the rest of the code so for now I replaced the issue file "OpenCVPipeline.java" with this file from the OpenCV documentation. I would consult those further to find out the exact issue you're having. I hope this can help!

Hello, I face an Incompatibility issue between open cv and PyTorch can you mention which PyTorch version did you use ? and what command do you use to install opencv ?