Adversarial Graph Representation Adaptation for Cross-Domain Facial Expression Recognition (AGRA, ACM 2020, Oral)

Overview

Cross Domain Facial Expression Recognition Benchmark

Implementation of papers:

Pipeline

Environment

Ubuntu 16.04 LTS, Python 3.5, PyTorch 1.3

Note: We also provide docker image for this project, click here. (Tag: py3-pytorch1.3-agra)

Datasets

To apply for the AFE, please complete the AFE Database User Agreement and submit it to [email protected] or [email protected].

Note:

  1. The AFE Database Agreement needs to be signed by the faculty member at a university or college and sent it by email.
  2. In order to comply with relevant regulations, you need to apply for the image data of the following data sets by yourself, including CK+, JAFFE, SFEW 2.0, FER2013, ExpW, RAF.

Pre-Train Model

You can download pre-train models in Baidu Drive (password: tzrf) and OneDrive.

Note: To replace backbone of each methods, you should modify and run getPreTrainedModel_ResNet.py (or getPreTrainedModel_MobileNet.py) in the folder where you want to use the method.

Usage

Before run these script files, you should download datasets and pre-train model, and run getPreTrainedModel_ResNet.py (or getPreTrainedModel_MobileNet.py).

Run ICID

cd ICID
bash Train.sh

Run DFA

cd DFA
bash Train.sh

Run LPL

cd LPL
bash Train.sh

Run DETN

cd DETN
bash TrainOnSourceDomain.sh     # Train Model On Source Domain
bash TransferToTargetDomain.sh  # Then, Transfer Model to Target Domain

Run FTDNN

cd FTDNN
bash Train.sh

Run ECAN

cd ECAN
bash TrainOnSourceDomain.sh     # Train Model On Source Domain
bash TransferToTargetDomain.sh  # Then, Transfer Model to Target Domain

Run CADA

cd CADA
bash TrainOnSourceDomain.sh     # Train Model On Source Domain
bash TransferToTargetDomain.sh  # Then, Transfer Model to Target Domain

Run SAFN

cd SAFN
bash TrainWithSAFN.sh

Run SWD

cd SWD
bash Train.sh

Run AGRA

cd AGRA
bash TrainOnSourceDomain.sh     # Train Model On Source Domain
bash TransferToTargetDomain.sh  # Then, Transfer Model to Target Domain

Result

Souce Domain: RAF

Methods Backbone CK+ JAFFE SFEW2.0 FER2013 ExpW Mean
ICID ResNet-50 74.42 50.70 48.85 53.70 69.54 59.44
DFA ResNet-50 64.26 44.44 43.07 45.79 56.86 50.88
LPL ResNet-50 74.42 53.05 48.85 55.89 66.90 59.82
DETN ResNet-50 78.22 55.89 49.40 52.29 47.58 56.68
FTDNN ResNet-50 79.07 52.11 47.48 55.98 67.72 60.47
ECAN ResNet-50 79.77 57.28 52.29 56.46 47.37 58.63
CADA ResNet-50 72.09 52.11 53.44 57.61 63.15 59.68
SAFN ResNet-50 75.97 61.03 52.98 55.64 64.91 62.11
SWD ResNet-50 75.19 54.93 52.06 55.84 68.35 61.27
Ours ResNet-50 85.27 61.50 56.43 58.95 68.50 66.13

Methods Backbone CK+ JAFFE SFEW2.0 FER2013 ExpW Mean
ICID ResNet-18 67.44 48.83 47.02 53.00 68.52 56.96
DFA ResNet-18 54.26 42.25 38.30 47.88 47.42 46.02
LPL ResNet-18 72.87 53.99 49.31 53.61 68.35 59.63
DETN ResNet-18 64.19 52.11 42.25 42.01 43.92 48.90
FTDNN ResNet-18 76.74 50.23 49.54 53.28 68.08 59.57
ECAN ResNet-18 66.51 52.11 48.21 50.76 48.73 53.26
CADA ResNet-18 73.64 55.40 52.29 54.71 63.74 59.96
SAFN ResNet-18 68.99 49.30 50.46 53.31 68.32 58.08
SWD ResNet-18 72.09 53.52 49.31 53.70 65.85 58.89
Ours ResNet-18 77.52 61.03 52.75 54.94 69.70 63.19

Methods Backbone CK+ JAFFE SFEW2.0 FER2013 ExpW Mean
ICID MobileNet V2 57.36 37.56 38.30 44.47 60.64 47.67
DFA MobileNet V2 41.86 35.21 29.36 42.36 43.66 38.49
LPL MobileNet V2 59.69 40.38 40.14 50.13 62.26 50.52
DETN MobileNet V2 53.49 40.38 35.09 45.88 45.26 44.02
FTDNN MobileNet V2 71.32 46.01 45.41 49.96 62.87 55.11
ECAN MobileNet V2 53.49 43.08 35.09 45.77 45.09 44.50
CADA MobileNet V2 62.79 53.05 43.12 49.34 59.40 53.54
SAFN MobileNet V2 66.67 45.07 40.14 49.90 61.40 52.64
SWD MobileNet V2 68.22 55.40 43.58 50.30 60.04 55.51
Ours MobileNet V2 72.87 55.40 45.64 51.05 63.94 57.78

Souce Domain: AFE

Methods Backbone CK+ JAFFE SFEW2.0 FER2013 ExpW Mean
ICID ResNet-50 56.59 57.28 44.27 46.92 52.91 51.59
DFA ResNet-50 51.86 52.70 38.03 41.93 60.12 48.93
LPL ResNet-50 73.64 61.03 49.77 49.54 55.26 57.85
DETN ResNet-50 56.27 52.11 44.72 42.17 59.80 51.01
FTDNN ResNet-50 61.24 57.75 47.25 46.36 52.89 53.10
ECAN ResNet-50 58.14 56.91 46.33 46.30 61.44 53.82
CADA ResNet-50 72.09 49.77 50.92 50.32 61.70 56.96
SAFN ResNet-50 73.64 64.79 49.08 48.89 55.69 58.42
SWD ResNet-50 72.09 61.50 48.85 48.83 56.22 57.50
Ours ResNet-50 78.57 65.43 51.18 51.31 62.71 61.84

Methods Backbone CK+ JAFFE SFEW2.0 FER2013 ExpW Mean
ICID ResNet-18 54.26 51.17 47.48 46.44 54.85 50.84
DFA ResNet-18 35.66 45.82 34.63 36.88 62.53 43.10
LPL ResNet-18 67.44 62.91 48.39 49.82 54.51 56.61
DETN ResNet-18 44.19 47.23 45.46 45.39 58.41 48.14
FTDNN ResNet-18 58.91 59.15 47.02 48.58 55.29 53.79
ECAN ResNet-18 44.19 60.56 43.26 46.15 62.52 51.34
CADA ResNet-18 72.09 53.99 48.39 48.61 58.50 56.32
SAFN ResNet-18 68.22 61.50 50.46 50.07 55.17 57.08
SWD ResNet-18 77.52 59.15 50.69 51.84 56.56 59.15
Ours ResNet-18 79.84 61.03 51.15 51.95 65.03 61.80

Methods Backbone CK+ JAFFE SFEW2.0 FER2013 ExpW Mean
ICID MobileNet V2 55.04 42.72 34.86 39.94 44.34 43.38
DFA MobileNet V2 44.19 27.70 31.88 35.95 61.55 40.25
LPL MobileNet V2 69.77 50.23 43.35 45.57 51.63 52.11
DETN MobileNet V2 57.36 54.46 32.80 44.11 64.36 50.62
FTDNN MobileNet V2 65.12 46.01 46.10 46.69 53.02 51.39
ECAN MobileNet V2 71.32 56.40 37.61 45.34 64.00 54.93
CADA MobileNet V2 70.54 45.07 40.14 46.72 54.93 51.48
SAFN MobileNet V2 62.79 53.99 42.66 46.61 52.65 51.74
SWD MobileNet V2 64.34 53.52 44.72 50.24 55.85 53.73
Ours MobileNet V2 75.19 54.46 47.25 47.88 61.10 57.18

Mean of All Methods

Souce Domain: RAF

Backbone CK+ JAFFE SFEW2.0 FER2013 ExpW Mean
ResNet-50 75.87 54.30 54.49 54.82 62.09 59.51
ResNet-18 69.43 51.88 47.94 51.72 61.26 56.45
MobileNet V2 60.78 45.15 39.59 47.92 56.46 49.98

Souce Domain: AFE

Backbone CK+ JAFFE SFEW2.0 FER2013 ExpW Mean
ResNet-50 65.41 57.93 47.04 47.26 57.87 55.10
ResNet-18 60.23 56.25 46.95 47.57 58.34 53.87
MobileNet V2 63.57 48.46 40.14 44.91 56.34 50.68

Citation

@article{chen2020cross,
  title={Cross-Domain Facial Expression Recognition: A Unified Evaluation Benchmark and Adversarial Graph Learning},
  author={Chen, Tianshui and Pu, Tao and Wu, Hefeng and Xie, Yuan and Liu, Lingbo and Lin, Liang},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year={2021},
  pages={1-1},
  doi={10.1109/TPAMI.2021.3131222}
}

@inproceedings{xie2020adversarial,
  title={Adversarial Graph Representation Adaptation for Cross-Domain Facial Expression Recognition},
  author={Xie, Yuan and Chen, Tianshui and Pu, Tao and Wu, Hefeng and Lin, Liang},
  booktitle={Proceedings of the 28th ACM international conference on Multimedia},
  year={2020}
}

Contributors

For any questions, feel free to open an issue or contact us:

Comments
  • 复现问题

    复现问题

    你好,我是华南师范大学的一名学生 对于您论文的复现我想请问您几个问题: 复现时按照您的代码和数据集处理方式,从RAF-DB 到CK+ 数据集的迁移 best Accuracy 只有 78.29,无法达到85效果 您给出的docker 的image putao3/images:py3-pytorch1.3-agra 里面是python2的 ,而且没有装pytorch

    opened by loveego 13
  • ExpW and FER test data

    ExpW and FER test data

    Dear authors,

    ExpW does not have a validation split but it appears that you've used a validation set for evaluation. Could you please provide the val lists you used for FER2013 and ExpW?

    Thanks, Manogna

    opened by manogna-s 7
  • Reproducing results on AGRA

    Reproducing results on AGRA

    Good morning,

    I'm trying to reproduce the results of AGRA, but I cannot match the numbers. I believe the issue is due to hyperparameter selection and data processing. Can you confirm that the hyperparameters in the TrainOnSourceDomain.sh and TransferToTargetDomain.sh files are correct?

    Also, can you help me understand the data processing step you performed? I'm using the RAF-DB dataset as source and JAFFE and FER2013 as targets. From what I read in #17, I believe you split the datasets into three equal-sized parts when no train-test split is available. If you have the test split, you divide it into two equal-sized test and validation sets.

    Therefore, you use 1/3 of JAFFE and 1/2 of the test set of FER2013. For the RAF-DB, on the other hand, you directly use the train split provided by the authors.

    Can you confirm these processes? Many thanks

    opened by altndrr 5
  • Not able to find image files in Datasets

    Not able to find image files in Datasets

    Hi Authors, Thank you so much for making the code open source. However I am facing an issue with the RAF/CK+ dataset. I downloaded the dataset from the OneDrive Link as mentioned in the repository. I am not able to find any images in the RAF dataset and CK+ dataset. Kindly tell me where can I find the images corresponding to the dataset. I look forward to your response. Thanks again for your help, Megh

    opened by meghbhalerao 3
  • (打扰)关于快速得到表情识别结果的小问题:关于预训练模型

    (打扰)关于快速得到表情识别结果的小问题:关于预训练模型

    您好,感谢您在学术上的贡献和开源代码 我想快速获得一个可以识别表情的项目,在阅读您的代码过程中发现您提供了预训练模型 所以想请问是否在完成Train on source doman后保存的模型才会拥有基本表情识别的能力? 还是加载您上传的预训练模型直接修改"TrainOnSourceDomain.py"进行test也会拥有一定的表情识别能力(只是准确率不够高)? 时间不太允许完成source doman 的训练再进行transfer,故提出这个比较蠢的疑问,十分感谢!

    opened by laoliu97 2
  • Not able to reproduce results in RAF to CK+

    Not able to reproduce results in RAF to CK+

    Dear Authors,

    Thank you so much for making the code open source.

    However, I am not able to reproduce the results for RAF to CK+, which is given in Table 1 - 72.09 % - the number corresponding to the CADA row.

    This is my config file -

    Log_Name='ResNet50_CropNet_withoutAFN_transferToTargetDomain_RAFtoCK+'
    Resume_Model='ResNet50_CropNet_withoutAFN_trainOnSourceDomain_RAFtoCK+.pkl'
    ##Resume_Model=None
    OutputPath='.'
    GPU_ID=1
    Backbone='ResNet50'
    useAFN='False'
    methodOfAFN='SAFN'
    radius=25
    deltaRadius=1
    weight_L2norm=0.05
    useDAN='True'
    methodOfDAN='CDAN'
    faceScale=112
    sourceDataset='RAF'
    targetDataset='CK+'
    train_batch_size=32
    test_batch_size=32
    useMultiDatasets='False'
    epochs=100
    lr=0.0001
    lr_ad=0.001
    momentum=0.9
    weight_decay=0.0001
    isTest='False'
    showFeature='False'
    class_num=7
    useIntraGCN='True'
    useInterGCN='True'
    useLocalFeature='True'
    useRandomMatrix='False'
    useAllOneMatrix='False'
    useCov='False'
    useCluster='False'
        
    OMP_NUM_THREADS=16 MKL_NUM_THREADS=16 CUDA_VISIBLE_DEVICES=${GPU_ID} python3 TransferToTargetDomain.py \
    --Log_Name ${Log_Name} \
    --OutputPath ${OutputPath} \
    --Backbone ${Backbone} \
    --Resume_Model ${Resume_Model} \
    --GPU_ID ${GPU_ID} \
    --useAFN ${useAFN} \
    --methodOfAFN ${methodOfAFN} \
    --radius ${radius} \
    --deltaRadius ${deltaRadius} \
    --weight_L2norm ${weight_L2norm} \
    --useDAN ${useDAN} \
    --methodOfDAN ${methodOfDAN} \
    --faceScale ${faceScale} \
    --sourceDataset ${sourceDataset} \
    --targetDataset ${targetDataset} \
    --train_batch_size ${train_batch_size} \
    --test_batch_size ${test_batch_size} \
    --useMultiDatasets ${useMultiDatasets} \
    --epochs ${epochs} \
    --lr ${lr} \
    --lr_ad ${lr_ad} \
    --momentum ${momentum} \
    --weight_decay ${weight_decay} \
    --isTest ${isTest} \
    --showFeature ${showFeature} \
    --class_num ${class_num} \
    --useIntraGCN ${useIntraGCN} \
    --useInterGCN ${useInterGCN} \
    --useLocalFeature ${useLocalFeature} \
    --useRandomMatrix ${useRandomMatrix} \
    --useAllOneMatrix ${useAllOneMatrix} \
    --useCov ${useCov} \
    --useCluster ${useCluster}
    

    Kindly let me know if I am missing something. Please help at the earliest, Thank you again, Megh

    opened by meghbhalerao 2
  • 数据集

    数据集

    学者您好,在您提供的CD-FER-Benchmark开源代码中,有关数据集我有一处不明白,在Utils.py里面283行有这样一个路径'/RAF/basic/Annotation/Landmarks_5/,但是在公开数据集里都没有提供Landmarks_5,请问您是怎么生成的呢?或者你怎么制作的呢?可以把您的这几个数据集的Landmarks_5发给我一份吗?或者指导一下我怎么生成这个文件,希望您有空的时候帮助解答一下我的问题,万分感谢!!!

    opened by qitingtingvi 1
  • label stored

    label stored

    Hello, thank you very much for your work. I would like to see how the data in the Label,box and landmark files in your code are stored, ok? All I need is an example

    opened by likanglinlove 1
  • 现有ICID代码,似乎只用了Source 训练集?

    现有ICID代码,似乎只用了Source 训练集?

    Source 选择AFED 数据集, Target使用 FER2013, backbone使用了ResNet18, 然而现有ICID的代码,似乎只用了Source训练集训练模型, Target 测试集准确率也只有38左右。而使用Source 和 Target 训练集一起训练模型时,Target测试集准确率达到了52.66。

    opened by smwanghhh 1
Owner
null
Relative Uncertainty Learning for Facial Expression Recognition

Relative Uncertainty Learning for Facial Expression Recognition The official implementation of the following paper at NeurIPS2021: Title: Relative Unc

null 35 Dec 28, 2022
Software for Multimodalty 2D+3D Facial Expression Recognition (FER) UI

EmotionUI Software for Multimodalty 2D+3D Facial Expression Recognition (FER) UI. demo screenshot (with RealSense) required packages Python >= 3.6 num

Yang Jiao 2 Dec 23, 2021
CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation

CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation [arxiv] This is the official repository for CDTrans: Cross-domain Transformer for

null 238 Dec 22, 2022
CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation

[ICCV2021] TransReID: Transformer-based Object Re-Identification [pdf] The official repository for TransReID: Transformer-based Object Re-Identificati

DamoCV 569 Dec 30, 2022
Web service for facial landmark detection, head pose estimation, facial action unit recognition, and eye-gaze estimation based on OpenFace 2.0

OpenGaze: Web Service for OpenFace Facial Behaviour Analysis Toolkit Overview OpenFace is a fantastic tool intended for computer vision and machine le

Sayom Shakib 4 Nov 3, 2022
OpenFace – a state-of-the art tool intended for facial landmark detection, head pose estimation, facial action unit recognition, and eye-gaze estimation.

OpenFace 2.2.0: a facial behavior analysis toolkit Over the past few years, there has been an increased interest in automatic facial behavior analysis

Tadas Baltrusaitis 5.8k Dec 31, 2022
Unified unsupervised and semi-supervised domain adaptation network for cross-scenario face anti-spoofing, Pattern Recognition

USDAN The implementation of Unified unsupervised and semi-supervised domain adaptation network for cross-scenario face anti-spoofing, which is accepte

null 11 Nov 3, 2022
Py-FEAT: Python Facial Expression Analysis Toolbox

Py-FEAT is a suite for facial expressions (FEX) research written in Python. This package includes tools to detect faces, extract emotional facial expressions (e.g., happiness, sadness, anger), facial muscle movements (e.g., action units), and facial landmarks, from videos and images of faces, as well as methods to preprocess, analyze, and visualize FEX data.

Computational Social Affective Neuroscience Laboratory 147 Jan 6, 2023
Facial detection, landmark tracking and expression transfer library for Windows, Linux and Mac

Welcome to the CSIRO Face Analysis SDK. Documentation for the SDK can be found in doc/documentation.html. All code in this SDK is provided according t

Luiz Carlos Vieira 7 Jul 16, 2020
Facial Expression Detection In The Realtime

The human's facial expressions is very important to detect thier emotions and sentiment. It can be very efficient to use to make our computers make interviews. Furthermore, we have robots now can detect the human's emotions and based on thats take an action .etc. So, It will be better to provide a tool or model for this.

Adel El-Nabarawy 4 Mar 1, 2022
Facial expression detector

A tensorflow convolutional neural network model to detect facial expressions.

Carlos Tardón Rubio 5 Apr 20, 2022
Automatically measure the facial Width-To-Height ratio and get facial analysis results provided by Microsoft Azure

fwhr-calc-website This project is to automatically measure the facial Width-To-Height ratio and get facial analysis results provided by Microsoft Azur

SoohyunPark 1 Feb 7, 2022
Code to reproduce the experiments in the paper "Transformer Based Multi-Source Domain Adaptation" (EMNLP 2020)

Transformer Based Multi-Source Domain Adaptation Dustin Wright and Isabelle Augenstein To appear in EMNLP 2020. Read the preprint: https://arxiv.org/a

CopeNLU 36 Dec 5, 2022
Repository for the paper "Online Domain Adaptation for Occupancy Mapping", RSS 2020

RSS 2020 - Online Domain Adaptation for Occupancy Mapping Repository for the paper "Online Domain Adaptation for Occupancy Mapping", Robotics: Science

Anthony 26 Sep 22, 2022
IAST: Instance Adaptive Self-training for Unsupervised Domain Adaptation (ECCV 2020)

This repo is the official implementation of our paper "Instance Adaptive Self-training for Unsupervised Domain Adaptation". The purpose of this repo is to better communicate with you and respond to your questions. This repo is almost the same with Another-Version, and you can also refer to that version.

CVSM Group -  email: czhu@bupt.edu.cn 84 Dec 12, 2022
RDA: Robust Domain Adaptation via Fourier Adversarial Attacking

RDA: Robust Domain Adaptation via Fourier Adversarial Attacking Updates 08/2021: check out our domain adaptation for video segmentation paper Domain A

null 17 Nov 30, 2022
FedMM: Saddle Point Optimization for Federated Adversarial Domain Adaptation

This repository contains the code accompanying the paper " FedMM: Saddle Point Optimization for Federated Adversarial Domain Adaptation" Paper link: R

null 20 Jun 29, 2022
Cycle Consistent Adversarial Domain Adaptation (CyCADA)

Cycle Consistent Adversarial Domain Adaptation (CyCADA) A pytorch implementation of CyCADA. If you use this code in your research please consider citi

Hyunwoo Ko 2 Jan 10, 2022
Code for CVPR2021 "Visualizing Adapted Knowledge in Domain Transfer". Visualization for domain adaptation. #explainable-ai

Visualizing Adapted Knowledge in Domain Transfer @inproceedings{hou2021visualizing, title={Visualizing Adapted Knowledge in Domain Transfer}, auth

Yunzhong Hou 80 Dec 25, 2022