CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation

Last update: Dec 22, 2022

Related tags

Deep Learning CDTrans

Overview

CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation [arxiv]

This is the official repository for CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation

Introduction

Unsupervised domain adaptation (UDA) aims to transfer knowledge learned from a labeled source domain to a different unlabeled target domain. Most existing UDA methods focus on learning domain-invariant feature representation, either from the domain level or category level, using convolution neural networks (CNNs)-based frameworks. With the success of Transformer in various tasks, we find that the cross-attention in Transformer is robust to the noisy input pairs for better feature alignment, thus in this paper Transformer is adopted for the challenging UDA task. Specifically, to generate accurate input pairs, we design a two-way center-aware labeling algorithm to produce pseudo labels for target samples. Along with the pseudo labels, a weight-sharing triple-branch transformer framework is proposed to apply self-attention and cross-attention for source/target feature learning and source-target domain alignment, respectively. Such design explicitly enforces the framework to learn discriminative domain-specific and domain-invariant representations simultaneously. The proposed method is dubbed CDTrans (cross-domain transformer), and it provides one of the first attempts to solve UDA tasks with a pure transformer solution. Extensive experiments show that our proposed method achieves the best performance on all public UDA datasets including Office-Home, Office-31, VisDA-2017, and DomainNet.

Results

Table 1 [UDA results on Office-31]

Methods	Avg.	A->D	A->W	D->A	D->W	W->A	W->D
Baseline(DeiT-S)	86.7	87.6	86.9	74.9	97.7	73.5	99.6
Baseline(DeiT-S)	86.7	model		model		model
CDTrans(DeiT-S)	90.4	94.6	93.5	78.4	98.2	78	99.6
CDTrans(DeiT-S)	90.4	model	model	model	model	model	model
Baseline(DeiT-B)	88.8	90.8	90.4	76.8	98.2	76.4	100
Baseline(DeiT-B)	88.8	model		model		model
CDTrans(DeiT-B)	92.6	97	96.7	81.1	99	81.9	100
CDTrans(DeiT-B)	92.6	model	model	model	model	model	model

Table 2 [UDA results on Office-Home]

Methods	Avg.	Ar->Cl	Ar->Pr	Ar->Re	Cl->Ar	Cl->Pr	Cl->Re	Pr->Ar	Pr->Cl	Pr->Re	Re->Ar	Re->Cl	Re->Pr
Baseline(DeiT-S)	69.8	55.6	73	79.4	70.6	72.9	76.3	67.5	51	81	74.5	53.2	82.7
Baseline(DeiT-S)	69.8	model			model			model			model
CDTrans(DeiT-S)	74.7	60.6	79.5	82.4	75.6	81.0	82.3	72.5	56.7	84.4	77.0	59.1	85.5
CDTrans(DeiT-S)	74.7	model	model	model	model	model	model	model	model	model	model	model	model
Baseline(DeiT-B)	74.8	61.8	79.5	84.3	75.4	78.8	81.2	72.8	55.7	84.4	78.3	59.3	86
Baseline(DeiT-B)	74.8	model			model			model			model
CDTrans(DeiT-B)	80.5	68.8	85	86.9	81.5	87.1	87.3	79.6	63.3	88.2	82	66	90.6
CDTrans(DeiT-B)	80.5	model	model	model	model	model	model	model	model	model	model	model	model

Table 3 [UDA results on VisDA-2017]

Methods	Per-class	plane	bcycl	bus	car	horse	knife	mcycl	person	plant	sktbrd	train	truck
Baseline(DeiT-B)	67.3 (model)	98.1	48.1	84.6	65.2	76.3	59.4	94.5	11.8	89.5	52.2	94.5	34.1
CDTrans(DeiT-B)	88.4 (model)	97.7	86.39	86.87	83.33	97.76	97.16	95.93	84.08	97.93	83.47	94.59	55.3

Table 4 [UDA results on DomainNet]

Base-S	clp	info	pnt	qdr	rel	skt	Avg.	CDTrans-S	clp	info	pnt	qdr	rel	skt	Avg.
clp	-	21.2	44.2	15.3	59.9	46.0	37.3	clp	-	25.3	52.5	23.2	68.3	53.2	44.5
clp	model						37.3	clp	model	model	model	model	model	model	44.5
info	36.8	-	39.4	5.4	52.1	32.6	33.3	info	47.6	-	48.3	9.9	62.8	41.1	41.9
info	model						33.3	info	model	model	model	model	model	model	41.9
pnt	47.1	21.7	-	5.7	60.2	39.9	34.9	pnt	55.4	24.5	-	11.7	67.4	48.0	41.4
pnt	model						34.9	pnt	model	model	model	model	model	model	41.4
qdr	25.0	3.3	10.4	-	18.8	14.0	14.3	qdr	36.6	5.3	19.3	-	33.8	22.7	23.5
qdr	model						14.3	qdr	model	model	model	model	model	model	23.5
rel	54.8	23.9	52.6	7.4	-	40.1	35.8	rel	61.5	28.1	56.8	12.8	-	47.2	41.3
rel	model						35.8	rel	model	model	model	model	model	model	41.3
skt	55.6	18.6	42.7	14.9	55.7	-	37.5	skt	64.3	26.1	53.2	23.9	66.2	-	46.7
skt	model						37.5	skt	model	model	model	model	model	model	46.7
Avg.	43.9	17.7	37.9	9.7	49.3	34.5	32.2	Avg.	53.08	21.86	46.02	16.3	59.7	42.44	39.9

Base-B	clp	info	pnt	qdr	rel	skt	Avg.	CDTrans-B	clp	info	pnt	qdr	rel	skt	Avg.
clp	-	24.2	48.9	15.5	63.9	50.7	40.6	clp	-	29.4	57.2	26.0	72.6	58.1	48.7
clp	model						40.6	clp	model	model	model	model	model	model	48.7
info	43.5	-	44.9	6.5	58.8	37.6	38.3	info	57.0	-	54.4	12.8	69.5	48.4	48.4
info	model						38.3	info	model	model	model	model	model	model	48.4
pnt	52.8	23.3	-	6.6	64.6	44.5	38.4	pnt	62.9	27.4	-	15.8	72.1	53.9	46.4
pnt	model						38.4	pnt	model	model	model	model	model	model	46.4
qdr	31.8	6.1	15.6	-	23.4	18.9	19.2	qdr	44.6	8.9	29.0	-	42.6	28.5	30.7
qdr	model						19.2	qdr	model	model	model	model	model	model	30.7
rel	58.9	26.3	56.7	9.1	-	45.0	39.2	rel	66.2	31.0	61.5	16.2	-	52.9	45.6
rel	model						39.2	rel	model	model	model	model	model	model	45.6
skt	60.0	21.1	48.4	16.6	61.7	-	41.6	skt	69.0	29.6	59.0	27.2	72.5	-	51.5
skt	model						41.6	skt	model	model	model	model	model	model	51.5
Avg.	49.4	20.2	42.9	10.9	54.5	39.3	36.2	Avg.	59.9	25.3	52.2	19.6	65.9	48.4	45.2

Requirements

Installation

pip install -r requirements.txt
(Python version is the 3.7 and the GPU is the V100 with cuda 10.1, cudatoolkit 10.1)

Prepare Datasets

Download the UDA datasets Office-31, Office-Home, VisDA-2017, DomainNet

Then unzip them and rename them under the directory like follow: (Note that each dataset floader needs to make sure that it contains the txt file that contain the path and lable of the picture, which is already in data/the_dataset of this project.)

data
├── OfficeHomeDataset
│   │── class_name
│   │   └── images
│   └── *.txt
├── domainnet
│   │── class_name
│   │   └── images
│   └── *.txt
├── office31
│   │── class_name
│   │   └── images
│   └── *.txt
├── visda
│   │── train
│   │   │── class_name
│   │   │   └── images
│   │   └── *.txt 
│   └── validation
│       │── class_name
│       │   └── images
│       └── *.txt

Prepare DeiT-trained Models

For fair comparison in the pre-training data set, we use the DeiT parameter init our model based on ViT. You need to download the ImageNet pretrained transformer model : DeiT-Small, DeiT-Base and move them to the ./data/pretrainModel directory.

Training

We utilize 1 GPU for pre-training and 2 GPUs for UDA, each with 16G of memory.

Scripts.

Command input paradigm

bash scripts/[pretrain/uda]/[office31/officehome/visda/domainnet]/run_*.sh [deit_base/deit_small]

For example

DeiT-Base scripts

# Office-31     Source: Amazon   ->  Target: Dslr, Webcam
bash scripts/pretrain/office31/run_office_amazon.sh deit_base
bash scripts/uda/office31/run_office_amazon.sh deit_base

#Office-Home    Source: Art      ->  Target: Clipart, Product, Real_World
bash scripts/pretrain/officehome/run_officehome_Ar.sh deit_base
bash scripts/uda/officehome/run_officehome_Ar.sh deit_base

# VisDA-2017    Source: train    ->  Target: validation
bash scripts/pretrain/visda/run_visda.sh deit_base
bash scripts/uda/visda/run_visda.sh deit_base

# DomainNet     Source: Clipart  ->  Target: painting, quickdraw, real, sketch, infograph
bash scripts/pretrain/domainnet/run_domainnet_clp.sh deit_base
bash scripts/uda/domainnet/run_domainnet_clp.sh deit_base

DeiT-Small scripts Replace deit_base with deit_small to run DeiT-Small results. An example of training on office-31 is as follows:

# Office-31     Source: Amazon   ->  Target: Dslr, Webcam
bash scripts/pretrain/office31/run_office_amazon.sh deit_small
bash scripts/uda/office31/run_office_amazon.sh deit_small

Evaluation

# For example VisDA-2017
python test.py --config_file 'configs/uda.yml' MODEL.DEVICE_ID "('0')" TEST.WEIGHT "('../logs/uda/vit_base/visda/transformer_best_model.pth')" DATASETS.NAMES 'VisDA' DATASETS.NAMES2 'VisDA' OUTPUT_DIR '../logs/uda/vit_base/visda/' DATASETS.ROOT_TRAIN_DIR './data/visda/train/train_image_list.txt' DATASETS.ROOT_TRAIN_DIR2 './data/visda/train/train_image_list.txt' DATASETS.ROOT_TEST_DIR './data/visda/validation/valid_image_list.txt'

Acknowledgement

Codebase from TransReID

Comments

Problem in DomainNet training setting

Hi, why do you use combine the training set and the testing set in DomainNet for training and testing? For example, when taking "clipart" as the target domain, you use clipart.txt (combining clipart_train.txt and clipart_test.txt) for both training and testing, which means all the testing samples have been seen during training. For another word, why not use clipart_train.txt for training and use clipart_test.txt for testing?

opened by SikaStar 3
Compile error
I have followed the Requirements of README.md and tried to run the code for a whole day. however, there are still several issues unresolved. I searched on the Google but it did not work. I want get some advice please.

pretrain

bash scripts/pretrain/officehome/run_officehome_Ar.sh deit_small But The Art/Pan/00002.jpg is exist in officehome dataset and Art.txt

train

bash scripts/uda/office31/run_office_amazon.sh deit_small

bash scripts/uda/visda/run_visda.sh deit_small
opened by gliehu 3
Question about the Loss term in this code

Hello, I think there exsists a logical error in your code.

In processor_uda.py, the loss Loss1 of the target samples is not consistant with the loss term your proposed in paper.

Here I take out the error line 339

loss1 = loss_fn(score1, feat1, t_pseudo_target, target_cam)

and the line 337

(self_score1, self_feat1, self_prob1), (score2, feat2, prob2), (score1, feat1, prob1), cross_attn = model(img, t_img, target, cam_label=target_cam, view_label=target_view ) # output: source , target , source_target_fusion

In your code, you use the cls Loss of the fusion samples instead of the target samples, then the loss is employed to optimize the network as the target cls Loss

This is not reasonable and not consistant with Fig. (2) in your paper.

Hope you can check the code. Thanks

opened by myukzzz 1
Visualising the attention

Just wondering if you had any code to allow for visualization of the attention maps of size (B, N, num_patches+1, num_patches+1) to create something like Figure 1a in the paper.

Thanks!

opened by finlay96 1
关于文中主要idea在代码中的体现

您好！感谢您的团队提供的论文和代码，对我有很大启发。我是一名初学者，研究代码很长时间但还是有不少困惑，您能否为我指出文中提到的TWO-WAY CENTER-AWARE PSEUDO LABELING方法和CROSS-DOMAIN TRANSFORMER框架的代码是在哪个文件或函数中？期待着您的回复。

opened by Eureka-JTX 0
关于主流UDA数据集的训练方法

您好，感谢贵团队分享的论文及代码，受益良多。 1、首先，对于实验部分中使用的主流数据集Office31、OfficeHome以及不划分的train和test的DomainNet等，训练阶段是否都是使用source样本+target样本，在测试阶段仍然是拿训练过程中使用过的target样本？请问大致是这样的过程吗？

2、如果实验部分的确是上述第1个问题的情况，那么最终得到的模型很有可能已经过拟合target样本。同时，测试阶段用于test的样本都是网络在训练阶段见过的样本，所以我认为测试阶段得到精度无法正确评估模型在target域上的性能（或者说泛化性）。况且，CDTrans使用的是伪标签的训练方法，对带伪标签的target样本做监督学习，应该会更容易产生过拟合target样本的现象。

3、现在大部分论文的方法也都是这么使用主流数据集的吗？若上述训练和测试方式存在问题，那么那些不划分target训练和测试部分的DA方法，最终可能都是训练出来一个过拟合target样本的模型呢？即使是不使用伪标签的那些DA方法，如果将全部target样本同时用于训练和测试，是不是也是不合理的呢？

总的来讲，我认为DA最终得到的模型，应该要拿那些网络没见过的target样本去测试，才能评估这个模型是否真的适应了target的数据分布。希望作者看到的话，能帮忙看看我的问题以及观点是否有问题吗？感激不尽！😁🤞

opened by Jin-huihuang 0
CDTrans与TransReID的关系？

您好！

发现了一件古怪的事情，在本repo的代码中，似乎从来没有出现过CDTrans这一名字，反而是TransReID反复出现，请问我是否可以认为TransReID就是CDTrans的早期名字呢？

以及，请问TransReID中所需要的，并且也是经常出现的camera, view, 这些到底是什么？如何从数据集中获取它们呢？

谢谢！

opened by WenqingZong 0
Error when training on DomainNet

Hello! I have an issue when I trains a DeiT-S with CDTrans on DomainNet. An error was occured, and I analyzed the error by making use of breakpoint(). Based on the result of debugging, I could see that the image pairs don't remain after image filtering process. In paper for CDTrans, the results on DomainNet can be seen, and I think training should also be possible on this dataset.

My question is what should I do for training on DomainNet without error. Should I modify the parameter 'topk'? or Should I modify WITH_PSEUDO_LABEL_FILTER in uda.yml? If not, is there any other way to handle this problem?

Thank you for reading it!

opened by JoonHyeokJ 0
Single Image inference
What changes need to be made in the code as well as in the configuration file in order to carry out inference on single image.

For example in do_inference_uda

probs = model(img, img, cam_label=camids, view_label=target_view, return_logits=True) What should the camid and target_view be set to.

Furthermore at line 530

evaluator.update((probs[1], vid)) here vid is the class label which is being used to compute evaluate accuracy metrics.

For single image evaluation is this line required? Did you run any experiment on your end for single image classification ?

Would appreciate it if you could give some insights on this.
opened by sparshgarg23 0

Owner

GitHub

Adversarial Adaptation with Distillation for BERT Unsupervised Domain Adaptation

Knowledge Distillation for BERT Unsupervised Domain Adaptation Official PyTorch implementation | Paper Abstract A pre-trained language model, BERT, ha

29 Nov 30, 2022

Unified unsupervised and semi-supervised domain adaptation network for cross-scenario face anti-spoofing, Pattern Recognition

USDAN The implementation of Unified unsupervised and semi-supervised domain adaptation network for cross-scenario face anti-spoofing, which is accepte

11 Nov 3, 2022

Code of TVT: Transferable Vision Transformer for Unsupervised Domain Adaptation

TVT Code of TVT: Transferable Vision Transformer for Unsupervised Domain Adaptation Datasets: Digit: MNIST, SVHN, USPS Object: Office, Office-Home, Vi

37 Dec 15, 2022

code for our paper "Source Data-absent Unsupervised Domain Adaptation through Hypothesis Transfer and Labeling Transfer"

SHOT++ Code for our TPAMI submission "Source Data-absent Unsupervised Domain Adaptation through Hypothesis Transfer and Labeling Transfer" that is ext

75 Dec 16, 2022

The official codes of "Semi-supervised Models are Strong Unsupervised Domain Adaptation Learners".

SSL models are Strong UDA learners Introduction This is the official code of paper "Semi-supervised Models are Strong Unsupervised Domain Adaptation L

26 Dec 26, 2022

A PyTorch implementation for Unsupervised Domain Adaptation by Backpropagation(DANN), support Office-31 and Office-Home dataset

DANN A PyTorch implementation for Unsupervised Domain Adaptation by Backpropagation Prerequisites Linux or OSX NVIDIA GPU + CUDA (may CuDNN) and corre

8 Apr 16, 2022

(CVPR2021) DANNet: A One-Stage Domain Adaptation Network for Unsupervised Nighttime Semantic Segmentation

DANNet: A One-Stage Domain Adaptation Network for Unsupervised Nighttime Semantic Segmentation CVPR2021(oral) [arxiv] Requirements python3.7 pytorch==

85 Dec 7, 2022

IAST: Instance Adaptive Self-training for Unsupervised Domain Adaptation (ECCV 2020)

This repo is the official implementation of our paper "Instance Adaptive Self-training for Unsupervised Domain Adaptation". The purpose of this repo is to better communicate with you and respond to your questions. This repo is almost the same with Another-Version, and you can also refer to that version.

84 Dec 12, 2022

Unsupervised Domain Adaptation for Nighttime Aerial Tracking (CVPR2022)

Unsupervised Domain Adaptation for Nighttime Aerial Tracking (CVPR2022) Junjie Ye, Changhong Fu, Guangze Zheng, Danda Pani Paudel, and Guang Chen. Uns

Intelligent Vision for Robotics in Complex Environment

91 Dec 30, 2022

Code to reproduce the experiments in the paper "Transformer Based Multi-Source Domain Adaptation" (EMNLP 2020)

Transformer Based Multi-Source Domain Adaptation Dustin Wright and Isabelle Augenstein To appear in EMNLP 2020. Read the preprint: https://arxiv.org/a

36 Dec 5, 2022

Official pytorch implement for “Transformer-Based Source-Free Domain Adaptation”

Official implementation for TransDA Official pytorch implement for “Transformer-Based Source-Free Domain Adaptation”. Overview: Result: Prerequisites:

54 Dec 22, 2022

Code for CVPR2021 "Visualizing Adapted Knowledge in Domain Transfer". Visualization for domain adaptation. #explainable-ai

Visualizing Adapted Knowledge in Domain Transfer @inproceedings{hou2021visualizing, title={Visualizing Adapted Knowledge in Domain Transfer}, auth

80 Dec 25, 2022