Taking A Closer Look at Domain Shift: Category-level Adversaries for Semantics Consistent Domain Adaptation

Overview

Taking A Closer Look at Domain Shift: Category-level Adversaries for Semantics Consistent Domain Adaptation (CVPR2019)

This is a pytorch implementation of CLAN.

Oral Presentation Video

Watch the video

Prerequisites

  • Python 3.6
  • GPU Memory >= 11G
  • Pytorch 1.0.0

Getting started

The data folder is structured as follows:

├── data/
│   ├── Cityscapes/     
|   |   ├── gtFine/
|   |   ├── leftImg8bit/
│   ├── GTA5/
|   |   ├── images/
|   |   ├── labels/
│   ├── SYNTHIA/ 
|   |   ├── RAND_CITYSCAPES/
│   └── 			
└── model/
│   ├── DeepLab_resnet_pretrained.pth
...

Train

CUDA_VISIBLE_DEVICES=0 python CLAN_train.py --snapshot-dir ./snapshots/GTA2Cityscapes

Evaluate

CUDA_VISIBLE_DEVICES=0 python CLAN_evaluate.py --restore-from  ./snapshots/GTA2Cityscapes/GTA5_100000.pth --save ./result/GTA2Cityscapes_100000

Our pretrained model is available via Google Drive

Compute IoU

python CLAN_iou.py ./data/Cityscapes/gtFine/val result/GTA2Cityscapes_100000

Tip: The best-performance model might not be the final one in the last epoch. If you want to evaluate every saved models in bulk, please use CLAN_evaluate_bulk.py and CLAN_iou_bulk.py, the result will be saved in an Excel sheet.

CUDA_VISIBLE_DEVICES=0 python CLAN_evaluate_bulk.py
python CLAN_iou_bulk.py

Visualization Results

(a) (b)

(c) (d)

This code is heavily borrowed from the baseline AdaptSegNet

Citation

If you use this code in your research please consider citing

@article{luo2021category,
  title={Category-Level Adversarial Adaptation for Semantic Segmentation using Purified Features},
  author={Luo, Yawei and Liu, Ping and Zheng, Liang and Guan, Tao and Yu, Junqing and Yang, Yi},
  journal={IEEE Transactions on Pattern Analysis \& Machine Intelligence (TPAMI)},
  year={2021},
}

@inproceedings{luo2019Taking,
title={Taking A Closer Look at Domain Shift: Category-level Adversaries for Semantics Consistent Domain Adaptation},
author={Luo, Yawei and Zheng, Liang and Guan, Tao and Yu, Junqing and Yang, Yi},
booktitle={The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2019}
}

Related works

Seg-Uncertainty

Comments
  • About WeightedBCEWithLogitLoss

    About WeightedBCEWithLogitLoss

    Hi, I have a problem in understanding your code of class WeightedBCEWithLogitLoss, showed as follow. It seems the input is logits, but there isn't any sigmoid or softmax operation done on it. And I'm confused of that two lines. Is the loss computation same as pytorch BCEWithLogitLoss???
    image

    opened by JialeTao 5
  • About source only and t-nse

    About source only and t-nse

    source only-Resnet is 36.6 in many papers, but I didn't get 36.6 in my trial. Can you share your training model with me? Another question ,is the t-nse_fearture's shape like [h * w, 2048], and the label's shape like [h * w]? [ps: you saide you use layer4 features as input in another issues.] In my process, i get the layer4 features which shape is [1,2048,81,61], and then i upsample the features to [1, 2048, 1024, 512] which match the labels shape[1024, 512], and then i reshape the features to [1024 * 512, 2048] and labels to [1024 * 512]. Is that right?

    opened by jarvisWang0903 5
  • About t-sne

    About t-sne

    "As a result, the original well-segmented class are well preserved. We then map the high-dimensional features of (b), (c) and (d) to a 2-D space with t-SNE [28] shown in (e), (f) and (g)" This is the description in your paper, I want to know which layer of output this high-dimensional features refers to,thanks~

    opened by jarvisWang0903 5
  • The evaluation results on GTA2cityscapes

    The evaluation results on GTA2cityscapes

    Hi, I trained your code on GTA52cityscapes following your instruction but failed to reproduce the evaluation results shown in your paper. The mIoU is close to the TAN baseline but worse than your method. The below is what I got. Any suggestions? Thanks!===>road: 84.44 ===>sidewalk: 25.92 ===>building: 78.1 ===>wall: 21.57 ===>fence: 24.06 ===>pole: 26.74 ===>light: 30.62 ===>sign: 19.42 ===>vegetation: 83.48 ===>terrain: 31.98 ===>sky: 73.76 ===>person: 58.13 ===>rider: 25.26 ===>car: 70.47 ===>truck: 35.57 ===>bus: 43.26 ===>train: 0.82 ===>motocycle: 27.86 ===>bicycle: 22.75 ===> mIoU: 41.27

    opened by yingzicy 4
  • About data loading

    About data loading

    Hello, thank for sharing the great work. I have a quick question. When I train the CLAN model, I find that randomly mirror the images and randomly scale the crop_size are used fo r preprocessing data, which is a little different from previous works.

    When training, I can think of it as data augmentation. But, why apply the above preprocessing to the test phase?

    Thanks!

    opened by BinhuiXie 3
  • how to use multi gpu

    how to use multi gpu

    i have two gpu 1080ti i want to multi gpu because 1 gpu has out of memory so i use function" nn.dataparallel (model)" but my gpu don't allocate 2gpu and allocate only one gpu (print(torch.cuda.device_count()) is 2 but don't allocate them) do you know how to solve this problem?

    opened by jungho-93 2
  • 运行问题

    运行问题

    我的运行环境是 pytorch 1.1 python3.6 当 最开始执行时 显示错误为: /home/ygl/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/upsampling.py:129: UserWarning: nn.Upsample is deprecated. Use nn.functional.interpolate instead. warnings.warn("nn.{} is deprecated. Use nn.functional.interpolate instead.".format(self.name)) Fatal Python error: Cannot recover from stack overflow. ... Traceback (most recent call last): File "CLAN_train.py", line 440, in main() File "CLAN_train.py", line 334, in main loss_seg.backward() File "/home/ygl/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/tensor.py", line 102, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph) File "/home/ygl/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/autograd/init.py", line 90, in backward allow_unreachable=True) # allow_unreachable flag File "/home/ygl/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 274, in handler _error_if_any_worker_fails()

    opened by ygjwd12345 2
  • About the damping policy

    About the damping policy

    Thanks for your codes! I have doubt about the damping policy you used to replace the early stopping. I also think training 150k as AdaptSeg did is too time-consuming and shortening the total iteration steps is necessary. The damping weight of adversarial loss in GRL is ascending from 0 to 1 with the iteration increasing. While you set the actually opposite policy where the weight descending from 1 to 0 along with the iterating. So could you share some insights to use such a damping policy? Thanks!

    opened by LHXXHB 2
  • Dataloader is always reported as wrong.

    Dataloader is always reported as wrong.

    Hi! I am trying to reproduce your experiments, however, I always broke in line.309 of CLAN_train.py file. _, batch = next(trainloader_iter) Every time my device reports wrong when meeting this sentence.

    Reported: 'RuntimeError: DataLoader worker (pid(s) 15186) exited unexpectedly'

    My environment is: Python 3.6.8, torch 1.1.0. Could you please let me know how can I do to solve the issue since I really apparate this idea reported in this paper.

    opened by lolinkun 2
  • About the batch size

    About the batch size

    Hi, first congratulations on your excellent work. The only thing that I want to ask is whether the batch size is one? Did you try to use a relatively large batch size??

    opened by hjf1997 1
  • CUDA error: out of memory

    CUDA error: out of memory

    Hi, I trained the model on a 1080Ti, but get CUDA out of memory error.

    python  CLAN_train.py --snapshot-dir ./snapshots/GTA2Cityscapes
    ....
    Traceback (most recent call last):
      File "CLAN_train.py", line 426, in <module>
        main()
      File "CLAN_train.py", line 319, in main
        loss_seg = (loss_calc(pred_source1, labels_s, args.gpu) + loss_calc(pred_source2, labels_s, args.gpu))
      File "CLAN_train.py", line 165, in loss_calc
        return criterion(pred, label)
      File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
        result = self.forward(*input, **kwargs)
      File "/media/ubuntu/76c3afb9-9877-4540-ac95-6a70ac97a8fc/PycharmProjects/UDASS/CLAN/utils/loss.py", line 72, in forward
        predict = P[target_mask.view(N, 1, H, W).repeat(1, C, 1, 1)].view(C, -1)
    RuntimeError: CUDA error: out of memory
    

    The sizes of GTA5 images and Cityscapes images are set as default in the code. So 11G memory is not enough?

    opened by willer94 1
  • About Significance-aware Information Bottleneck Code

    About Significance-aware Information Bottleneck Code

    @RoyalVane Dear authors,

    Thanks for sharing this impressive work! I was wondering if you have public code for reproducing CLAN-IB, CLAN-SIB, and CLAN-Full models, as presented in your TPAMI paper, Category-Level Adversarial Adaptation for Semantic Segmentation using Purified Features.

    Thanks so much!

    opened by ArlenCHEN 0
Owner
Yawei Luo
Computer Vision
Yawei Luo
code for `Look Closer to Segment Better: Boundary Patch Refinement for Instance Segmentation`

Look Closer to Segment Better: Boundary Patch Refinement for Instance Segmentation (CVPR 2021) Introduction PBR is a conceptually simple yet effective

H.Chen 143 Jan 5, 2023
A Closer Look at Structured Pruning for Neural Network Compression

A Closer Look at Structured Pruning for Neural Network Compression Code used to reproduce experiments in https://arxiv.org/abs/1810.04622. To prune, w

Bayesian and Neural Systems Group 140 Dec 5, 2022
PyTorch implementation of the R2Plus1D convolution based ResNet architecture described in the paper "A Closer Look at Spatiotemporal Convolutions for Action Recognition"

R2Plus1D-PyTorch PyTorch implementation of the R2Plus1D convolution based ResNet architecture described in the paper "A Closer Look at Spatiotemporal

Irhum Shafkat 342 Dec 16, 2022
Look Closer: Bridging Egocentric and Third-Person Views with Transformers for Robotic Manipulation

Look Closer: Bridging Egocentric and Third-Person Views with Transformers for Robotic Manipulation Official PyTorch implementation for the paper Look

Rishabh Jangir 20 Nov 24, 2022
Mapping Conditional Distributions for Domain Adaptation Under Generalized Target Shift

This repository contains the official code of OSTAR in "Mapping Conditional Distributions for Domain Adaptation Under Generalized Target Shift" (ICLR 2022).

Matthieu Kirchmeyer 5 Dec 6, 2022
This is the offical website for paper ''Category-consistent deep network learning for accurate vehicle logo recognition''

The Pytorch Implementation of Category-consistent deep network learning for accurate vehicle logo recognition This is the offical website for paper ''

Wanglong Lu 28 Oct 29, 2022
Cycle Consistent Adversarial Domain Adaptation (CyCADA)

Cycle Consistent Adversarial Domain Adaptation (CyCADA) A pytorch implementation of CyCADA. If you use this code in your research please consider citi

Hyunwoo Ko 2 Jan 10, 2022
[NeurIPS 2021] Better Safe Than Sorry: Preventing Delusive Adversaries with Adversarial Training

Better Safe Than Sorry: Preventing Delusive Adversaries with Adversarial Training Code for NeurIPS 2021 paper "Better Safe Than Sorry: Preventing Delu

Lue Tao 29 Sep 20, 2022
Adversarial Adaptation with Distillation for BERT Unsupervised Domain Adaptation

Knowledge Distillation for BERT Unsupervised Domain Adaptation Official PyTorch implementation | Paper Abstract A pre-trained language model, BERT, ha

Minho Ryu 29 Nov 30, 2022
Official PyTorch implementation of CAPTRA: CAtegory-level Pose Tracking for Rigid and Articulated Objects from Point Clouds

CAPTRA: CAtegory-level Pose Tracking for Rigid and Articulated Objects from Point Clouds Introduction This is the official PyTorch implementation of o

Yijia Weng 96 Dec 7, 2022
Single-stage Keypoint-based Category-level Object Pose Estimation from an RGB Image

CenterPose Overview This repository is the official implementation of the paper "Single-stage Keypoint-based Category-level Object Pose Estimation fro

NVIDIA Research Projects 188 Dec 27, 2022
[ICRA 2022] CaTGrasp: Learning Category-Level Task-Relevant Grasping in Clutter from Simulation

This is the official implementation of our paper: Bowen Wen, Wenzhao Lian, Kostas Bekris, and Stefan Schaal. "CaTGrasp: Learning Category-Level Task-R

Bowen Wen 199 Jan 4, 2023
You Only Look One-level Feature (YOLOF), CVPR2021, Detectron2

You Only Look One-level Feature (YOLOF), CVPR2021 A simple, fast, and efficient object detector without FPN. This repo provides a neat implementation

qiang chen 273 Jan 3, 2023
Stacked Hourglass Network with a Multi-level Attention Mechanism: Where to Look for Intervertebral Disc Labeling

⚠️ ‎‎‎ A more recent and actively-maintained version of this code is available in ivadomed Stacked Hourglass Network with a Multi-level Attention Mech

Reza Azad 14 Oct 24, 2022
Code for CVPR2021 "Visualizing Adapted Knowledge in Domain Transfer". Visualization for domain adaptation. #explainable-ai

Visualizing Adapted Knowledge in Domain Transfer @inproceedings{hou2021visualizing, title={Visualizing Adapted Knowledge in Domain Transfer}, auth

Yunzhong Hou 80 Dec 25, 2022
[CVPR2021] Domain Consensus Clustering for Universal Domain Adaptation

[CVPR2021] Domain Consensus Clustering for Universal Domain Adaptation [Paper] Prerequisites To install requirements: pip install -r requirements.txt

Guangrui Li 84 Dec 26, 2022
CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation

CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation [arxiv] This is the official repository for CDTrans: Cross-domain Transformer for

null 238 Dec 22, 2022
CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation

[ICCV2021] TransReID: Transformer-based Object Re-Identification [pdf] The official repository for TransReID: Transformer-based Object Re-Identificati

DamoCV 569 Dec 30, 2022
A Pytorch Implementation of [Source data‐free domain adaptation of object detector through domain

A Pytorch Implementation of Source data‐free domain adaptation of object detector through domain‐specific perturbation Please follow Faster R-CNN and

null 1 Dec 25, 2021