Curriculum Domain Adaptation for Semantic Segmentation of Urban Scenes, ICCV 2017

Overview

AdaptationSeg

This is the Python reference implementation of AdaptionSeg proposed in "Curriculum Domain Adaptation for Semantic Segmentation of Urban Scenes".

Curriculum Domain Adaptation for Semantic Segmentation of Urban Scenes
Yang Zhang; Philip David; Boqing Gong;
International Conference on Computer Vision, 2017
A Curriculum Domain Adaptation Approach to the Semantic Segmentation of Urban Scenes
Yang Zhang; Philip David;  Hassan Foroosh; Boqing Gong;
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019

[TPAMI paper] [ICCV paper] [ArXiv Extended paper] [Poster]

[New] Survey of domain adaptation for semantic segmentation

Check out our new survey of domain adaptation for semantic segmentation in our TPAMI paper.

Review

Overview

Qualitative Results

We introduced a set of constraints to domain-adapt an arbitrary segmentation convolutional neural network (CNN) trained on source domain (synthetic images) to target domain (real images) without accessing target domain annotations.

Overview

Prerequisites

  • Linux
  • A CUDA-enabled NVIDIA GPU; Recommend video memory >= 11GB

Getting Started

Installation

The code requires following dependencies:

  • Python 2/3
  • Theano (installation)
  • Keras>=2.0.5 (Lower version might encounter Conv2DTranspose problem with Theano backend) (installation; You might want to install though pip since conda only offers Keras<=2.0.2)
  • Pillow (installation)

Keras backend setup

Make sure your Keras's image_data_format is channels_first. It is recommended to use Theano as the backend. However Tensorflow should also be okay. Note that using Tensorflow will result in lower initial/baseline model performance because the baseline model was trained using Theano.

How do I check/switch them?

Download dataset

1, Download leftImg8bit_trainvaltest.zip and leftImg8bit_trainextra.zip in CityScape dataset here. (Require registration)

2, Download SYNTHIA-RAND-CITYSCAPES in SYNTHIA dataset here.

3, Download our auxiliary pre-inferred target domain properties (Including both superpixel landmark and label distribution described in the paper) & parsed annotation here.

4, Download the submodule cityscapesScripts for evaluation purpose.

5, Unzip and organize them in this way:

./
├── train_val_DA.py
├── ...
├── cityscapesScripts/
│   ├── ...
│   └── cityscapesscripts/
│       ├── ...
│       └── evaluation/...
└── data/
    ├── Image/
    │   ├── CityScape/           # Unzip from two CityScape zips
    │   │   ├── test/
    │   │   ├── train/
    │   │   ├── train_extra/
    │   │   └── val/
    │   └── SYNTHIA/             # Unzip from the SYNTHIA dataset
    │       └── train/
    ├── label_distribution/      # Unzip from our auxiliary dataset
    │   └── ...
    ├── segmentation_annotation/ # Unzip from our auxiliary dataset
    │   └── ...
    ├── SP_labels/               # Unzip from our auxiliary dataset
    │   └── ...
    └── SP_landmark/             # Unzip from our auxiliary dataset
        └── ...

(Hint: If you have already downloaded the datasets but do not want to move them around, you may want to create some symbolic links of exsiting local datasets)

Training

Run train_val_FCN_DA.py either in your favorite Python IDE or the terminal by typing:

python train_val_FCN_DA.py

This would train the model for six epochs and save the best model during the training. You can stop it and continue to the evaluation during training if you feel it takes too long, however, performance would not be guaranteed then.

Evaluation

After running train_val_FCN_DA.py for at least 500 steps, run test_FCN_DA.py either in your favorite Python IDE or the terminal by typing:

python test_FCN_DA.py

This would evaluate both pre-trained SYNTHIA-FCN and adapted FCN over CityScape dataset and print both mean IoU.

Note

The original framework was implemented in Keras 1 with a custom transposed convolution ops. The performance might be slightly different from the ones reported in the paper. Also, some new commits in TF/Theano optimizer implementation after the code release has broken the losses' numerical stability. I have changed code's optimizer to SGD despite the original paper used Adadelta. You are welcome to try Adadelta/Adam however it seems that they will result in a NaN loss right after training starts. If the NaN problem persists, try to remove the label distribution loss from the training.

Citation

Please cite our paper if this code benefits your reseaarch:

@InProceedings{Zhang_2017_ICCV,
author = {Zhang, Yang and David, Philip and Gong, Boqing},
title = {Curriculum Domain Adaptation for Semantic Segmentation of Urban Scenes},
booktitle={The IEEE International Conference on Computer Vision (ICCV)},
volume={2},
number={5},
pages={6},
month = {Oct},
year = {2017}
}

@ARTICLE{Zhang_2019_TPAMI,
author={Zhang, Yang and David, Philip and Foroosh, Hassan and Gong, Boqing},
journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
title={A Curriculum Domain Adaptation Approach to the Semantic Segmentation of Urban Scenes},
year={2019},
volume={},
number={},
pages={1-1},
doi={10.1109/TPAMI.2019.2903401},
ISSN={1939-3539},
month={},}
Comments
  • ImportError: No module named city_meanIU

    ImportError: No module named city_meanIU

    hello YangZhang When I ran train_val_FCN_DA.py code , I got this error. gnss@gnss:/media/gnss/文档/mask/AdaptationSeg-master$ python train_val_FCN_DA.pyUsing Theano backend. Traceback (most recent call last): File "train_val_FCN_DA.py", line 14, in from city_meanIU import city_meanIU ImportError: No module named city_meanIU

    opened by cuimiao187561 5
  • `image_mean` type problem

    `image_mean` type problem

    Hi, thanks for your great work! It's really impressive that you do the segmentation translation from the virtual world to the real world scene. But when I run your code, I have a little problem with your image_mean type. It seems that you loaded images in RGB (https://github.com/YangZhang4065/AdaptationSeg/blob/master/warp_data.py#L10L25) and subtracted image mean value [103.939, 116.779, 123.68] from them (https://github.com/YangZhang4065/AdaptationSeg/blob/master/train_val_FCN_DA.py#L134).

    However, in the VGG network, they used image mean value [103.939, 116.779, 123.68] in BGR, not RGB (https://gist.github.com/ksimonyan/211839e770f7b538e2d8#description). I just wondered that why you do the operation of RGB-images with BGR-image_mean. (Maybe your SYNTHIA images are saved in BGR type before loading into the model?) I would be very grateful if you could resolve my doubt. Thank you so much. :)

    opened by BassyKuo 4
  • Class-wise evaluation IoU

    Class-wise evaluation IoU

    Hi @YangZhang4065 , Thank you so much for your study. I could train and evaluate your code but I couldn't achieve class wise evaluation IoUs with test_FCN_DA.py. How could I get these metrics ? Thank you.

    opened by dagdelenmustafa 2
  • About Superpixel Landmarks

    About Superpixel Landmarks

    Hi Yang,

    The code of your paper is elegant and easy to reproduce. But where can I find the code to train the multi-class SVM that yields the superpixel annotations of the target domain? Thanks!

    opened by ShichenLiu 1
  • Guidance for downloding leftImg8bit_trainvaltest.zip and leftImg8bit_trainextra.zip

    Guidance for downloding leftImg8bit_trainvaltest.zip and leftImg8bit_trainextra.zip

    Hi Yang Zhang,

    How are you doing?

    I am wondering if the link that in the following instruction is supposed to point so other web page? Current it is pointing to the Pillow installation page.

    1, Download leftImg8bit_trainvaltest.zip and leftImg8bit_trainextra.zip in CityScape dataset here. (Require registration)

    I am new to Pillow, so I don't know if it is possible to down load dataset with it.

    Thank you, Heng

    opened by heng2j 1
  • superpixel landmark and label distribution link not working

    superpixel landmark and label distribution link not working

    hi thanks for your work and effort, but i could not hit the link for superpixel landmark and label distribution. could you check and update the link ?

    opened by dagdelenmustafa 0
  • Problem while calling create_vgg16_FCN  in FCN_da.py

    Problem while calling create_vgg16_FCN in FCN_da.py

    Hi @YangZhang4065 , Im using create_vgg16_FCN to create FCN model , but on calling the function i'm facing following error :

    Negative dimension size caused by subtracting 2 from 1 for '{{node max_pooling2d_21/MaxPool}} = MaxPoolT=DT_FLOAT, data_format="NHWC", explicit_paddings=[], ksize=[1, 2, 2, 1], padding="VALID", strides=[1, 2, 2, 1]' with input shapes: [?,1,160,128].

    I have used padding ='same' on maxpooling in create_vgg16_FCN function , then above error does not comes , but while loading the pretrained weights from Synthia_FCA , i facing incompatible dimensions issue .

    Kindly let me know If m missing something .

    opened by mishpri0209 0
  • Logistic regression model

    Logistic regression model

    Hello @YangZhang4065 , Thanks for the nice work. I'm trying to train model to estimate the global label distribution. Can you give more details of the architecture and training scheme? Thanks!

    opened by river-afk 8
  • About color-constancy

    About color-constancy

    Thanks for your great help first, I wonder how to obtain the gamut-based color constancy method in the Paper. I have searched for a long time, but I can not find it. Can you share me the code about color constancy.

    opened by Katexiang 1
  • Get stuck before training

    Get stuck before training

    Hi,

    I tried to run the code, but the training does not proceed even for one iteration. Actually, it does nothing after printing:

    Start loading files
    Start training
    Epoch 1/100
    

    And starts consuming memory. Any idea on that?

    Thanks.

    opened by fatemeh-slh 3
Owner
Yang Zhang
Perception @ Waymo
Yang Zhang
A PyTorch implementation of the paper "Semantic Image Synthesis via Adversarial Learning" in ICCV 2017

Semantic Image Synthesis via Adversarial Learning This is a PyTorch implementation of the paper Semantic Image Synthesis via Adversarial Learning. Req

Seonghyeon Nam 146 Nov 25, 2022
(CVPR2021) DANNet: A One-Stage Domain Adaptation Network for Unsupervised Nighttime Semantic Segmentation

DANNet: A One-Stage Domain Adaptation Network for Unsupervised Nighttime Semantic Segmentation CVPR2021(oral) [arxiv] Requirements python3.7 pytorch==

W-zx-Y 85 Dec 7, 2022
Adversarial Adaptation with Distillation for BERT Unsupervised Domain Adaptation

Knowledge Distillation for BERT Unsupervised Domain Adaptation Official PyTorch implementation | Paper Abstract A pre-trained language model, BERT, ha

Minho Ryu 29 Nov 30, 2022
code for ICCV 2021 paper 'Generalized Source-free Domain Adaptation'

G-SFDA Code (based on pytorch 1.3) for our ICCV 2021 paper 'Generalized Source-free Domain Adaptation'. [project] [paper]. Dataset preparing Download

Shiqi Yang 84 Dec 26, 2022
Submodular Subset Selection for Active Domain Adaptation (ICCV 2021)

S3VAADA: Submodular Subset Selection for Virtual Adversarial Active Domain Adaptation ICCV 2021 Harsh Rangwani, Arihant Jain*, Sumukh K Aithal*, R. Ve

Video Analytics Lab -- IISc 13 Dec 28, 2022
Code for CVPR2021 "Visualizing Adapted Knowledge in Domain Transfer". Visualization for domain adaptation. #explainable-ai

Visualizing Adapted Knowledge in Domain Transfer @inproceedings{hou2021visualizing, title={Visualizing Adapted Knowledge in Domain Transfer}, auth

Yunzhong Hou 80 Dec 25, 2022
[CVPR2021] Domain Consensus Clustering for Universal Domain Adaptation

[CVPR2021] Domain Consensus Clustering for Universal Domain Adaptation [Paper] Prerequisites To install requirements: pip install -r requirements.txt

Guangrui Li 84 Dec 26, 2022
CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation

CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation [arxiv] This is the official repository for CDTrans: Cross-domain Transformer for

null 238 Dec 22, 2022
CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation

[ICCV2021] TransReID: Transformer-based Object Re-Identification [pdf] The official repository for TransReID: Transformer-based Object Re-Identificati

DamoCV 569 Dec 30, 2022
A Pytorch Implementation of [Source data‐free domain adaptation of object detector through domain

A Pytorch Implementation of Source data‐free domain adaptation of object detector through domain‐specific perturbation Please follow Faster R-CNN and

null 1 Dec 25, 2021
Semantic Segmentation for Real Point Cloud Scenes via Bilateral Augmentation and Adaptive Fusion (CVPR 2021)

Semantic Segmentation for Real Point Cloud Scenes via Bilateral Augmentation and Adaptive Fusion (CVPR 2021) This repository is for BAAF-Net introduce

null 90 Dec 29, 2022
Instance Segmentation in 3D Scenes using Semantic Superpoint Tree Networks

SSTNet Instance Segmentation in 3D Scenes using Semantic Superpoint Tree Networks(ICCV2021) by Zhihao Liang, Zhihao Li, Songcen Xu, Mingkui Tan, Kui J

null 83 Nov 29, 2022
Code release for "Transferable Semantic Augmentation for Domain Adaptation" (CVPR 2021)

Transferable Semantic Augmentation for Domain Adaptation Code release for "Transferable Semantic Augmentation for Domain Adaptation" (CVPR 2021) Paper

null 66 Dec 16, 2022
[arXiv'22] Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentation

Panoptic NeRF Project Page | Paper | Dataset Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentation Xiao Fu*, Shangzhan zhang*,

Xiao Fu 111 Dec 16, 2022
[arXiv'22] Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentation

Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentation Xiao Fu1*  Shangzhan Zhang1*  Tianrun Chen1  Yichong Lu1  Lanyun Zhu2  Xi

Xiao Fu 37 May 17, 2022
Implementation supporting the ICCV 2017 paper "GANs for Biological Image Synthesis"

GANs for Biological Image Synthesis This codes implements the ICCV-2017 paper "GANs for Biological Image Synthesis". The paper and its supplementary m

Anton Osokin 95 Nov 25, 2022
Recall Loss for Semantic Segmentation (This repo implements the paper: Recall Loss for Semantic Segmentation)

Recall Loss for Semantic Segmentation (This repo implements the paper: Recall Loss for Semantic Segmentation) Download Synthia dataset The model uses

null 32 Sep 21, 2022
Variational Attention: Propagating Domain-Specific Knowledge for Multi-Domain Learning in Crowd Counting (ICCV, 2021)

DKPNet ICCV 2021 Variational Attention: Propagating Domain-Specific Knowledge for Multi-Domain Learning in Crowd Counting Baseline of DKPNet is availa

null 19 Oct 14, 2022