[ECCV 2020] XingGAN for Person Image Generation

Related tags

Deep Learning pytorch generation image-generation non-local crossing appearance-features deepfashion feature-fusion shape-features eccv2020 eccv-2020 selectiongan
Overview

License CC BY-NC-SA 4.0 Python 3.6 Packagist Last Commit Maintenance Contributing Ask Me Anything !

Contents

XingGAN or CrossingGAN

| Project | Paper |
XingGAN for Person Image Generation
Hao Tang12, Song Bai2, Li Zhang2, Philip H.S. Torr2, Nicu Sebe13.
1University of Trento, Italy, 2University of Oxford, UK, 3Huawei Research Ireland, Ireland.
In ECCV 2020.
The repository offers the official implementation of our paper in PyTorch.

In the meantime, check out our related ACM MM 2019 paper Cycle In Cycle Generative Adversarial Networks for Keypoint-Guided Image Generation, BMVC 2020 oral paper Bipartite Graph Reasoning GANs for Person Image Generation, and ICCV 2021 paper Intrinsic-Extrinsic Preserved GANs for Unsupervised 3D Pose Transfer.

Framework

Comparison Results


License

Creative Commons License
Copyright (C) 2020 University of Trento, Italy.

All rights reserved. Licensed under the CC BY-NC-SA 4.0 (Attribution-NonCommercial-ShareAlike 4.0 International)

The code is released for academic research use only. For commercial use, please contact [email protected].

Installation

Clone this repo.

git clone https://github.com/Ha0Tang/XingGAN
cd XingGAN/

This code requires PyTorch 1.0.0 and python 3.6.9+. Please install the following dependencies:

  • pytorch 1.0.0
  • torchvision
  • numpy
  • scipy
  • scikit-image
  • pillow
  • pandas
  • tqdm
  • dominate

To reproduce the results reported in the paper, you need to run experiments on NVIDIA DGX1 with 4 32GB V100 GPUs for DeepFashion, and 1 32GB V100 GPU for Market-1501.

Dataset Preparation

Please follow SelectionGAN to directly download both Market-1501 and DeepFashion datasets.

This repository uses the same dataset format as SelectionGAN and BiGraphGAN. so you can use the same data for all these methods.

Generating Images Using Pretrained Model

Market-1501

sh scripts/download_xinggan_model.sh market

Then,

  1. Change several parameters in test_market.sh.
  2. Run sh test_market.sh for testing.

DeepFashion

sh scripts/download_xinggan_model.sh deepfashion

Then,

  1. Change several parameters in test_deepfashion.sh.
  2. Run sh test_deepfashion.sh for testing.

Train and Test New Models

Market-1501

  1. Change several parameters in train_market.sh.
  2. Run sh train_market.sh for training.
  3. Change several parameters in test_market.sh.
  4. Run sh test_market.sh for testing.

DeepFashion

  1. Change several parameters in train_deepfashion.sh.
  2. Run sh train_deepfashion.sh for training.
  3. Change several parameters in test_deepfashion.sh.
  4. Run sh test_deepfashion.sh for testing.

Evaluation

We adopt SSIM, mask-SSIM, IS, mask-IS, and PCKh for evaluation of Market-1501. SSIM, IS, PCKh for DeepFashion.

  1. SSIM, mask-SSIM, IS, mask-IS: install python3.5, tensorflow 1.4.1, and scikit-image==0.14.2. Then run, python tool/getMetrics_market.py or python tool/getMetrics_fashion.py.

  2. PCKh: install python2, and pip install tensorflow==1.4.0, then set export KERAS_BACKEND=tensorflow. After that, run python tool/crop_market.py or python tool/crop_fashion.py. Next, download pose estimator and put it under the root folder, and run python compute_coordinates.py. Lastly, run python tool/calPCKH_market.py or python tool/calPCKH_fashion.py.

Please refer to Pose-Transfer for more details.

Acknowledgments

This source code is inspired by both Pose-Transfer and SelectionGAN.

Related Projects

BiGraphGAN | GestureGAN | C2GAN | SelectionGAN | Guided-I2I-Translation-Papers

Citation

If you use this code for your research, please consider giving a star ⭐ and citing our paper 🦖 :

XingGAN

@inproceedings{tang2020xinggan,
  title={XingGAN for Person Image Generation},
  author={Tang, Hao and Bai, Song and Zhang, Li and Torr, Philip HS and Sebe, Nicu},
  booktitle={ECCV},
  year={2020}
}

If you use the original BiGraphGAN, GestureGAN, C2GAN, and SelectionGAN model, please consider giving stars ⭐ and citing the following papers 🦖 :

BiGraphGAN

@inproceedings{tang2020bipartite,
  title={Bipartite Graph Reasoning GANs for Person Image Generation},
  author={Tang, Hao and Bai, Song and Torr, Philip HS and Sebe, Nicu},
  booktitle={BMVC},
  year={2020}
}

GestureGAN

@article{tang2019unified,
  title={Unified Generative Adversarial Networks for Controllable Image-to-Image Translation},
  author={Tang, Hao and Liu, Hong and Sebe, Nicu},
  journal={IEEE Transactions on Image Processing (TIP)},
  year={2020}
}

@inproceedings{tang2018gesturegan,
  title={GestureGAN for Hand Gesture-to-Gesture Translation in the Wild},
  author={Tang, Hao and Wang, Wei and Xu, Dan and Yan, Yan and Sebe, Nicu},
  booktitle={ACM MM},
  year={2018}
}

C2GAN

@article{tang2021total,
  title={Total Generate: Cycle in Cycle Generative Adversarial Networks for Generating Human Faces, Hands, Bodies, and Natural Scenes},
  author={Tang, Hao and Sebe, Nicu},
  journal={IEEE Transactions on Multimedia (TMM)},
  year={2021}
}

@inproceedings{tang2019cycleincycle,
  title={Cycle In Cycle Generative Adversarial Networks for Keypoint-Guided Image Generation},
  author={Tang, Hao and Xu, Dan and Liu, Gaowen and Wang, Wei and Sebe, Nicu and Yan, Yan},
  booktitle={ACM MM},
  year={2019}
}

SelectionGAN

@inproceedings{tang2019multi,
  title={Multi-channel attention selection gan with cascaded semantic guidance for cross-view image translation},
  author={Tang, Hao and Xu, Dan and Sebe, Nicu and Wang, Yanzhi and Corso, Jason J and Yan, Yan},
  booktitle={CVPR},
  year={2019}
}

@article{tang2020multi,
  title={Multi-channel attention selection gans for guided image-to-image translation},
  author={Tang, Hao and Xu, Dan and Yan, Yan and Corso, Jason J and Torr, Philip HS and Sebe, Nicu},
  journal={arXiv preprint arXiv:2002.01048},
  year={2020}
}

Contributions

If you have any questions/comments/bug reports, feel free to open a github issue or pull a request or e-mail to the author Hao Tang ([email protected]).

Collaborations

I'm always interested in meeting new people and hearing about potential collaborations. If you'd like to work together or get in contact with me, please email [email protected]. Some of our projects are listed here.

Progress is impossible without change, and those who cannot change their minds cannot change anything.

Comments
  • Start point

    Start point

    Hi Hao

    I'm trying to download the images but keep getting the following error:

    Forbidden

    You don't have permission to access /~hao.tang/uploads/models/XingGAN/ on this server. Apache/2.4. .... Server at disi.unitn.it Port 80

    May you please help where might be the problem?

    opened by Mathilda88 5
  • reproducing results using pretrained Deep Fashion Model -- quality seems not as good as expected

    reproducing results using pretrained Deep Fashion Model -- quality seems not as good as expected

    Dear paper authors

    I am working on a NeurIPS paper for PoseMorphing, so I wanted a good comparison with your state-of-the-art method.

    I tried to run your pretrained deepfashion model, and it worked. However, the results seem worse than I expected, can it be that the latest pytorch 1.7 has broken some detail in your version?

    I write the command I used to run your model, and 2 example results. Maybe you can tell me if they look as expected, or something has gone wrong?

    python test.py --dataroot deepfashion/ --name deepfashion_XingGAN --model XingGAN --phase test --dataset_mode keypoint --norm instance --batchSize 1 --resize_or_crop no --gpu_ids 0 --BP_input_nc 18 --no_flip --which_model_netG Xing --checkpoints_dir ./checkpoints --pairLst /deepfahion/fasion-resize-pairs-test.csv --which_epoch latest --results_dir ./results --display_id 0

    fashionMENTees_Tanksid0000730104_7additional jpg___fashionMENTees_Tanksid0000730104_1front jpg_vis fashionWOMENBlouses_Shirtsid0000337203_3back jpg___fashionWOMENBlouses_Shirtsid0000337203_2side jpg_vis

    fashionMENTees_Tanksid0000730104_7additional jpg___fashionMENTees_Tanksid0000730104_1front jpg_vis

    fashionWOMENBlouses_Shirtsid0000337203_3back jpg___fashionWOMENBlouses_Shirtsid0000337203_2side jpg_vis

    thanks a lot for your help to make research reproducible

    opened by nikjetchev 3
  • How to visualize the pose map?

    How to visualize the pose map?

    Thanks for your novel work. It helps me a lot! And I wonder how to visualize the pose map as the figure shown in your paper? I can't find the visualization process in this repo.

    截屏2021-01-06 下午5 09 49
    opened by pumpkinnan97 1
  • Some question about the SSIM metric

    Some question about the SSIM metric

    Thanks for your novel work. When I calculated the SSIM metric, it's a little lower than the value that in the paper. Here are the results that I obtained. Market1501 dataset image

    Some visual results that obtained with the pretrained model. /0265_c6s1_056351_01.jpg___0265_c3s1_062642_05.jpg_vis.jpg image 1322_c3s3_035678_02.jpg___1322_c4s5_061435_01.jpg_vis.jpg image

    I don't know where I got it wrong. Thanks for your time.

    opened by zympsyche 1
  • Training Problems

    Training Problems

    @Ha0Tang Thanks for your novel work. I have trained the marker dataset follow your guidance. But I have a question that With the increase of training iteration, the loss of each part also increases except D_PP and D_PB。what are the two parts mean? I also wanted to ask how many epochs were used in the pre-training models you provided image image

    opened by yinyiyu 1
  • Face swapping(future work)

    Face swapping(future work)

    Hey @Ha0Tang Thanks for sharing such awesome work! I was wondering if we utilize your algorithm and facial landmarks to do face swapping and generate talking head models?

    opened by amil-rp-work 1
  • Accuracy without Co-attention Fusion Module

    Accuracy without Co-attention Fusion Module

    Thanks for this great work, you have mentioned the accuracy with both SA and AS blocks but under the absence of the co-attention fusion module in the paper and I wonder how did you get the result in this case? Did you have a direct FC layer at the end of the attention modules? How can we replicate that result?

    opened by parakh08 0
  • Image size does not match to the pose heat map size

    Image size does not match to the pose heat map size

    Hi, Thanks for your great work. When I load the deepfashion dataset, I found the size of image is 256X256 but the size of the heat map is 256X176. How can I use this data? Do I need to resize the image?

    opened by SenHe 0
  • estimator not work

    estimator not work

    Hi, why I always get [-1, -1] * 18 when I try to run compute_coordinates.py? Such as: 0000_c6s3_047142_03.jpg: [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]: [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1] 0609_c5s2_022005_05.jpg: [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]: [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1] 0304_c5s1_068698_01.jpg: [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]: [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1] -1_c6s4_006527_05.jpg: [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]: [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1] ... For any images I test, the cooridiantes always are [-1, -1] * 18, how can I solve this error? Thanks a lot!

    opened by ldincredible 0
Owner
Hao Tang
To develop a complete mind: Study the science of art; Study the art of science. Learn how to see. Realize that everything connects to everything else.
Hao Tang
GitHub http://disi.unitn.it/~hao.tang/project/XingGAN.htm
[ECCV 2020] Reimplementation of 3DDFAv2, including face mesh, head pose, landmarks, and more.

Stable Head Pose Estimation and Landmark Regression via 3D Dense Face Reconstruction Reimplementation of (ECCV 2020) Towards Fast, Accurate and Stable

Remilia Scarlet 221 Dec 30, 2022
1st Place Solution to ECCV-TAO-2020: Detect and Represent Any Object for Tracking

Instead, two models for appearance modeling are included, together with the open-source BAGS model and the full set of code for inference. With this code, you can achieve around mAP@23 with TAO test set (based on our estimation).

null 79 Oct 8, 2022
Repository for Traffic Accident Benchmark for Causality Recognition (ECCV 2020)

Causality In Traffic Accident (Under Construction) Repository for Traffic Accident Benchmark for Causality Recognition (ECCV 2020) Overview Data Prepa

Tackgeun 21 Nov 20, 2022
git《Learning Pairwise Inter-Plane Relations for Piecewise Planar Reconstruction》(ECCV 2020) GitHub:

Learning Pairwise Inter-Plane Relations for Piecewise Planar Reconstruction Code for the ECCV 2020 paper by Yiming Qian and Yasutaka Furukawa Getting

null 37 Dec 4, 2022
Code for our paper at ECCV 2020: Post-Training Piecewise Linear Quantization for Deep Neural Networks

PWLQ Updates 2020/07/16 - We are working on getting permission from our institution to release our source code. We will release it once we are granted

null 54 Dec 15, 2022
dataset for ECCV 2020 "Motion Capture from Internet Videos"

Motion Capture from Internet Videos Motion Capture from Internet Videos Junting Dong*, Qing Shuai*, Yuanqing Zhang, Xian Liu, Xiaowei Zhou, Hujun Bao

ZJU3DV 98 Dec 7, 2022
Code for the paper: Adversarial Training Against Location-Optimized Adversarial Patches. ECCV-W 2020.

Adversarial Training Against Location-Optimized Adversarial Patches arXiv | Paper | Code | Video | Slides Code for the paper: Sukrut Rao, David Stutz,

Sukrut Rao 32 Dec 13, 2022
《Unsupervised 3D Human Pose Representation with Viewpoint and Pose Disentanglement》(ECCV 2020) GitHub: [fig9]

Unsupervised 3D Human Pose Representation [Paper] The implementation of our paper Unsupervised 3D Human Pose Representation with Viewpoint and Pose Di

null 42 Nov 24, 2022
SNE-RoadSeg in PyTorch, ECCV 2020

SNE-RoadSeg Introduction This is the official PyTorch implementation of SNE-RoadSeg: Incorporating Surface Normal Information into Semantic Segmentati

null 242 Dec 20, 2022
Code for ECCV 2020 paper "Contacts and Human Dynamics from Monocular Video".

Contact and Human Dynamics from Monocular Video This is the official implementation for the ECCV 2020 spotlight paper by Davis Rempe, Leonidas J. Guib

Davis Rempe 207 Jan 5, 2023
[ECCV 2020] Gradient-Induced Co-Saliency Detection

Gradient-Induced Co-Saliency Detection Zhao Zhang*, Wenda Jin*, Jun Xu, Ming-Ming Cheng ⭐ Project Home » The official repo of the ECCV 2020 paper Grad

Zhao Zhang 35 Nov 25, 2022
Code for Towards Streaming Perception (ECCV 2020) :car:

sAP — Code for Towards Streaming Perception ECCV Best Paper Honorable Mention Award Feb 2021: Announcing the Streaming Perception Challenge (CVPR 2021

Martin Li 85 Dec 22, 2022
Code for paper ECCV 2020 paper: Who Left the Dogs Out? 3D Animal Reconstruction with Expectation Maximization in the Loop.

Who Left the Dogs Out? Evaluation and demo code for our ECCV 2020 paper: Who Left the Dogs Out? 3D Animal Reconstruction with Expectation Maximization

Benjamin Biggs 29 Dec 28, 2022
PyTorch implementation of ECCV 2020 paper "Foley Music: Learning to Generate Music from Videos "

Foley Music: Learning to Generate Music from Videos This repo holds the code for the framework presented on ECCV 2020. Foley Music: Learning to Genera

Chuang Gan 30 Nov 3, 2022
Sign Language Translation with Transformers (COLING'2020, ECCV'20 SLRTP Workshop)

transformer-slt This repository gathers data and code supporting the experiments in the paper Better Sign Language Translation with STMC-Transformer.

Kayo Yin 107 Dec 27, 2022
Source code for "Progressive Transformers for End-to-End Sign Language Production" (ECCV 2020)

Progressive Transformers for End-to-End Sign Language Production Source code for "Progressive Transformers for End-to-End Sign Language Production" (B

null 58 Dec 21, 2022
IAST: Instance Adaptive Self-training for Unsupervised Domain Adaptation (ECCV 2020)

This repo is the official implementation of our paper "Instance Adaptive Self-training for Unsupervised Domain Adaptation". The purpose of this repo is to better communicate with you and respond to your questions. This repo is almost the same with Another-Version, and you can also refer to that version.

CVSM Group - email: czhu@bupt.edu.cn 84 Dec 12, 2022
Boundary-preserving Mask R-CNN (ECCV 2020)

BMaskR-CNN This code is developed on Detectron2 Boundary-preserving Mask R-CNN ECCV 2020 Tianheng Cheng, Xinggang Wang, Lichao Huang, Wenyu Liu Video

Hust Visual Learning Team 178 Nov 28, 2022
Self-Supervised Monocular 3D Face Reconstruction by Occlusion-Aware Multi-view Geometry Consistency[ECCV 2020]

Self-Supervised Monocular 3D Face Reconstruction by Occlusion-Aware Multi-view Geometry Consistency(ECCV 2020) This is an official python implementati

null 304 Jan 3, 2023