[ACM MM 2019 Oral] Cycle In Cycle Generative Adversarial Networks for Keypoint-Guided Image Generation

Related tags

Deep Learning image-generation keypoints landmarks gans image-translation facial-landmarks image-to-image-translation
Overview

License CC BY-NC-SA 4.0 Python 3.6 Packagist Last Commit Maintenance Contributing Ask Me Anything !

Contents

Cycle-In-Cycle GANs

| Conference Paper | Extended Paper | Project |
Cycle In Cycle Generative Adversarial Networks for Keypoint-Guided Image Generation
Hao Tang1, Dan Xu2, Gaowen Liu3, Wei Wang4, Nicu Sebe1 and Yan Yan3
1University of Trento, 2University of Oxford, 3Texas State University, 4EPFL
The repository offers the official implementation of our paper in PyTorch.

In the meantime, check out our related BMVC 2020 oral paper Bipartite Graph Reasoning GANs for Person Image Generation, ECCV 2020 paper XingGAN for Person Image Generation, and ICCV 2021 paper Intrinsic-Extrinsic Preserved GANs for Unsupervised 3D Pose Transfer.

C2GAN Framework

Framework

License

Creative Commons License
Copyright (C) 2019 University of Trento, Italy.

All rights reserved. Licensed under the CC BY-NC-SA 4.0 (Attribution-NonCommercial-ShareAlike 4.0 International)

The code is released for academic research use only. For commercial use, please contact [email protected].

Installation

Clone this repo.

git clone https://github.com/Ha0Tang/C2GAN
cd C2GAN/

This code requires PyTorch 0.4.1+ and python 3.6.9+. Please install dependencies by

pip install -r requirements.txt (for pip users)

or

./scripts/conda_deps.sh (for Conda users)

To reproduce the results reported in the paper, you would need an NVIDIA TITAN Xp GPUs.

Dataset Preparation

For your convenience we provide download scripts:

bash ./datasets/download_c2gan_dataset.sh RaFD_image_landmark
  • RaFD_image_landmark: 3.0 GB

or you can use ./scripts/convert_pts_to_figure.m to convert the generated pts files to figures.

Prepare the datasets like in this folder after the download has finished. Please cite their paper if you use the data.

Generating Images Using Pretrained Model

  • You need download a pretrained model (e.g., Radboud) with the following script:
bash ./scripts/download_c2gan_model.sh Radboud
  • The pretrained model is saved at ./checkpoints/{name}_pretrained/latest_net_G.pth.
  • Then generate the result using
python test.py --dataroot ./datasets/Radboud --name Radboud_pretrained --model c2gan --which_model_netG unet_256 --which_direction AtoB --dataset_mode aligned --norm batch --gpu_ids 0 --batch 16;

The results will be saved at ./results/. Use --results_dir {directory_path_to_save_result} to specify the results directory.

  • For your own experiments, you might want to specify --netG, --norm, --no_dropout to match the generator architecture of the trained model.

Train and Test New Models

  • Download a dataset using the previous script (e.g., Radboud).
  • To view training results and loss plots, run python -m visdom.server and click the URL http://localhost:8097.
  • Train a model:
sh ./train_c2gan.sh
  • To see more intermediate results, check out ./checkpoints/Radboud_c2gan/web/index.html.
  • Test the model:
sh ./test_c2gan.sh
  • The test results will be saved to a html file here: ./results/Radboud_c2gan/latest_test/index.html.

Acknowledgments

This source code is inspired by Pix2pix, and GestureGAN.

Related Projects

BiGraphGAN | XingGAN | GestureGAN | SelectionGAN | Guided-I2I-Translation-Papers

Citation

If you use this code for your research, please cite our paper.

C2GAN

@article{tang2021total,
  title={Total Generate: Cycle in Cycle Generative Adversarial Networks for Generating Human Faces, Hands, Bodies, and Natural Scenes},
  author={Tang, Hao and Sebe, Nicu},
  journal={IEEE Transactions on Multimedia (TMM)},
  year={2021}
}

@inproceedings{tang2019cycleincycle,
  title={Cycle In Cycle Generative Adversarial Networks for Keypoint-Guided Image Generation},
  author={Tang, Hao and Xu, Dan and Liu, Gaowen and Wang, Wei and Sebe, Nicu and Yan, Yan},
  booktitle={ACM MM},
  year={2019}
}

If you use the original BiGraphGAN, XingGAN, GestureGAN, and SelectionGAN model, please cite the following papers:

BiGraphGAN

@inproceedings{tang2020bipartite,
  title={Bipartite Graph Reasoning GANs for Person Image Generation},
  author={Tang, Hao and Bai, Song and Torr, Philip HS and Sebe, Nicu},
  booktitle={BMVC},
  year={2020}
}

XingGAN

@inproceedings{tang2020xinggan,
  title={XingGAN for Person Image Generation},
  author={Tang, Hao and Bai, Song and Zhang, Li and Torr, Philip HS and Sebe, Nicu},
  booktitle={ECCV},
  year={2020}
}

GestureGAN

@article{tang2019unified,
  title={Unified Generative Adversarial Networks for Controllable Image-to-Image Translation},
  author={Tang, Hao and Liu, Hong and Sebe, Nicu},
  journal={IEEE Transactions on Image Processing (TIP)},
  year={2020}
}

@inproceedings{tang2018gesturegan,
  title={GestureGAN for Hand Gesture-to-Gesture Translation in the Wild},
  author={Tang, Hao and Wang, Wei and Xu, Dan and Yan, Yan and Sebe, Nicu},
  booktitle={ACM MM},
  year={2018}
}

SelectionGAN

@inproceedings{tang2019multi,
  title={Multi-channel attention selection gan with cascaded semantic guidance for cross-view image translation},
  author={Tang, Hao and Xu, Dan and Sebe, Nicu and Wang, Yanzhi and Corso, Jason J and Yan, Yan},
  booktitle={CVPR},
  year={2019}
}

@article{tang2020multi,
  title={Multi-channel attention selection gans for guided image-to-image translation},
  author={Tang, Hao and Xu, Dan and Yan, Yan and Corso, Jason J and Torr, Philip HS and Sebe, Nicu},
  journal={arXiv preprint arXiv:2002.01048},
  year={2020}
}

Contributions

If you have any questions/comments/bug reports, feel free to open a github issue or pull a request or e-mail to the author Hao Tang ([email protected]).

Collaborations

I'm always interested in meeting new people and hearing about potential collaborations. If you'd like to work together or get in contact with me, please email [email protected]. Some of our projects are listed here.

If you can do what you do best and be happy, you're further along in life than most people.

Comments
  • An issue about data processing

    An issue about data processing

    I got into trouble when I reproduced your paper, can you help me?

    My question is about the processing of the Radbound dataset. I downloaded your script for your Radbound dataset and found that the face images were not equal to the number of landmark pictures.

    I noticed what you mentioned in your paper: 'We first remove those images in who's face is not detected correctly the public OpenFace software, leading to 5,628 training information and leading 1,407 testing image pair'. The training set and the test set add up to 7,035 images, which corresponds to the number of face images which I downloaded (there are 1005 images in each face expression folder in the data set. if all the face images are combined with lankmarks, the total number of image pairs is just 7035).

    But the problem is that you filtered the undetected landmark, so there aren't enough landmarks to pair with the face images. So I'd like to ask how much data you actually used in your experiments. If 7035 images were used, as the paper says, how do you solve the problem of the missing lankmarks?

    opened by sitadivon 8
  • abnormal output during training

    abnormal output during training

    Hi, I met a problem during training the network.

    At train phase, due to opt.display_freq=1, at each step, 8 images are generated, where real_A, real_B, input_C, input_D should correspond to the input of the network. But I observed that they did not form a pair. See the 12th epoch output. In this picture, real_A and real_B are obviously not the same person.

    I have read the code carefully, but I can’t find the reason. Do you know why this happened?

    opened by sitadivon 4
  • Get wrong result during test phase

    Get wrong result during test phase

    Hi, Hao Tang! I met with difficulties again. Could you help me to figure them out?

    I tried to reproduce your experiment but got wrong result. I used the weight you provided to run the network. While the result of SSIM was only 0.82 (0.86 in your paper), PSNR was 23.53 (21.9 in your paper). I checked the generated images in the result folder while they were normal and the input images sucessfully translated into another expressions .

    Here is my specific workflow:

    1. Download dataset and weight
    2. Pair face with landmark, and lead to a row of images [neutral_face_A, other_expression_B, other_landmark_B, neutral_landmark_A]
    3. Split the whole dataset, and get 1407 images for testset. (BTW, I tried different seeds to make testset, but the value of SSIM are all around 0.82)
    4. Run the test shell script with the default args you set before.
    5. Calculate SSIM and PSNR of each pairs of images and take the average.

    Do I need to train the network by myself to reproduce the experiment, or I just did something wrong during the test process?

    opened by sitadivon 4
  • Why not add L1 loss of Keypoint generator?

    Why not add L1 loss of Keypoint generator?

    The whole loss consists of: GAN loss(both Keypoint and Image) Cycle loss(both Keypoint and Image) L1 loss(Only Image)

    I wonder why you not add L1 loss of Keypoint. Did you experiment with that?

    opened by sitadivon 3
  • RuntimeError: uniform_ expects to return a [from, to) range, but found from=1 > to=0.02

    RuntimeError: uniform_ expects to return a [from, to) range, but found from=1 > to=0.02

    Thank you for your great repository!

    I encountered error in networks RuntimeError: uniform_ expects to return a [from, to) range, but found from=1 > to=0.02

    some range transformation needed to solve this issue

    but I guess this could be something related to pytorch version error but same error appears on pytorch 0.4.1 :(

    opened by Ella77 3
  • Undertsanding model behavior for Recovered images during testing

    Undertsanding model behavior for Recovered images during testing

    Hello @Ha0Tang , I tried to reproduce the results of the Keypoint guided person Image generation using your C2GAN code on Market1501 data. I downloaded the data from https://www.kaggle.com/pengcw1/market-1501/data and organized the data into train and test set as discussed in the paper. I use the Resnet9_block model as the Generator network and the hyperparameters are chosen from the paper for training.

    During testing, I get some results which indicate that the model is learning the target pose. I am unable to explain why or how the recovered_A image looks exactly like a copy of the real_A. For instance, consider the below sample: image Note: The L1 loss and MSE loss are computed between the Real_A and recovered_A.

    During testing, The output Fake_B = netG(combined_realA_inputC) though not perfect, is believable But in case of Recovered_A = netG(combined_FakeB_inputD) , it is surprising to me that the generator is able to output the texture on the shirt though only the images FakeB and inputD were provided at input. The model has NOT been trained on this sample as both train and test sets are disjoint sets.
    I spent some time analysing the code but I am unable to explain this behavior. Is this expected from the model? If yes, what was the motivation ? Kindly help me understand this behavior of the model.

    Thanks and best regards, jysa01

    opened by jysa01 1
  • Training for Market1501

    Training for Market1501

    Hi, thanks for the great paper and work. I am trying to reproduce the results for Market1501 dataset. I use the same hyperparameters mentioned in the paper.

    • During training, after 14 epochs, the Keypoint generator stops producing any output for fake and recovered images. Have you experienced this? Any leads on how this can be fixed?
    • The paper states that OpenPose is used for body keypoint output for the first few epochs. Please explain how this has been incorporated in the code. I do not see any loading of a pretrained openpose model in the training code.
    • A little more detail on how the training was conducted for Market 1501 will be helpful to me (which model for netG and netD)

    Thanks, jysa01

    opened by jysa01 0
  • Testing of images and train for high resolution images

    Testing of images and train for high resolution images

    Thanks for sharing the code. The concept is good.

    I want to get pose transfer of source image. For this i want to use MVC dataset (https://github.com/MVC-Datasets/MVC) contains 4 poses of images(streight view,left side view, right side view, back view). I preprocessed these images that, 1.Non-human images have to be deleted using face-detection 2.I created key-points for these images using open-pose

    I want to train these images such that image A:streight view, image B: left view I have a following questions which can solve my problem to train:

    1. Suppose I trained for those two types images (streight, left view), to get a lefft view of test_image, how can I give its keypoints?? (I don't have left view of test_image. I want generate it)

    2. How to train for high resolution images??

    Thanks in Advance. Regards, Sandhya

    opened by sandhyalaxmiK 1
Owner
Hao Tang
To develop a complete mind: Study the science of art; Study the art of science. Learn how to see. Realize that everything connects to everything else.
Hao Tang
GitHub http://disi.unitn.it/~hao.tang/project/C2GAN.html
UDP++ (ECCVW 2020 Oral), (Winner of COCO 2020 Keypoint Challenge).

UDP-Pose This is the pytorch implementation for UDP++, which won the Fisrt place in COCO Keypoint Challenge at ECCV 2020 Workshop. Top-Down Results on

null 20 Jul 29, 2022
Learning Correspondence from the Cycle-consistency of Time (CVPR 2019)

TimeCycle Code for Learning Correspondence from the Cycle-consistency of Time (CVPR 2019, Oral). The code is developed based on the PyTorch framework,

Xiaolong Wang 706 Nov 29, 2022
Pytorch Implementation for NeurIPS (oral) paper: Pixel Level Cycle Association: A New Perspective for Domain Adaptive Semantic Segmentation

Pixel-Level Cycle Association This is the Pytorch implementation of our NeurIPS 2020 Oral paper Pixel-Level Cycle Association: A New Perspective for D

null 87 Oct 19, 2022
[ICLR 2021, Spotlight] Large Scale Image Completion via Co-Modulated Generative Adversarial Networks

Large Scale Image Completion via Co-Modulated Generative Adversarial Networks, ICLR 2021 (Spotlight) Demo | Paper [NEW!] Time to play with our interac

Shengyu Zhao 373 Jan 2, 2023
π-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis

π-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis Project Page | Paper | Data Eric Ryan Chan*, Marco Monteiro*, Pe

null 375 Dec 31, 2022
Image Deblurring using Generative Adversarial Networks

DeblurGAN arXiv Paper Version Pytorch implementation of the paper DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks. Our netwo

Orest Kupyn 2.2k Jan 1, 2023
Pytorch implementation for reproducing StackGAN_v2 results in the paper StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks

StackGAN-v2 StackGAN-v1: Tensorflow implementation StackGAN-v1: Pytorch implementation Inception score evaluation Pytorch implementation for reproduci

Han Zhang 809 Dec 16, 2022
Flickr-Faces-HQ (FFHQ) is a high-quality image dataset of human faces, originally created as a benchmark for generative adversarial networks (GAN)

Flickr-Faces-HQ Dataset (FFHQ) Flickr-Faces-HQ (FFHQ) is a high-quality image dataset of human faces, originally created as a benchmark for generative

NVIDIA Research Projects 2.9k Dec 28, 2022
Semi-supervised Representation Learning for Remote Sensing Image Classification Based on Generative Adversarial Networks

SSRL-for-image-classification Semi-supervised Representation Learning for Remote Sensing Image Classification Based on Generative Adversarial Networks

Feng 2 Nov 19, 2021
Multi-View Consistent Generative Adversarial Networks for 3D-aware Image Synthesis (CVPR2022)

Multi-View Consistent Generative Adversarial Networks for 3D-aware Image Synthesis Multi-View Consistent Generative Adversarial Networks for 3D-aware

Xuanmeng Zhang 78 Dec 10, 2022
Single-stage Keypoint-based Category-level Object Pose Estimation from an RGB Image

CenterPose Overview This repository is the official implementation of the paper "Single-stage Keypoint-based Category-level Object Pose Estimation fro

NVIDIA Research Projects 188 Dec 27, 2022
Cycle Consistent Adversarial Domain Adaptation (CyCADA)

Cycle Consistent Adversarial Domain Adaptation (CyCADA) A pytorch implementation of CyCADA. If you use this code in your research please consider citi

Hyunwoo Ko 2 Jan 10, 2022
Pre-trained model, code, and materials from the paper "Impact of Adversarial Examples on Deep Learning Models for Biomedical Image Segmentation" (MICCAI 2019).

Adaptive Segmentation Mask Attack This repository contains the implementation of the Adaptive Segmentation Mask Attack (ASMA), a targeted adversarial

Utku Ozbulak 53 Jul 4, 2022
Official code for CVPR2022 paper: Depth-Aware Generative Adversarial Network for Talking Head Video Generation

?? Depth-Aware Generative Adversarial Network for Talking Head Video Generation (CVPR 2022) ?? If DaGAN is helpful in your photos/projects, please hel

Fa-Ting Hong 503 Jan 4, 2023
This Repo is the official CUDA implementation of ICCV 2019 Oral paper for CARAFE: Content-Aware ReAssembly of FEatures

Introduction This Repo is the official CUDA implementation of ICCV 2019 Oral paper for CARAFE: Content-Aware ReAssembly of FEatures. @inproceedings{Wa

Jiaqi Wang 42 Jan 7, 2023
A Fast and Accurate One-Stage Approach to Visual Grounding, ICCV 2019 (Oral)

One-Stage Visual Grounding ***** New: Our recent work on One-stage VG is available at ReSC.***** A Fast and Accurate One-Stage Approach to Visual Grou

Zhengyuan Yang 118 Dec 5, 2022
Pytorch implementation for "Large-Scale Long-Tailed Recognition in an Open World" (CVPR 2019 ORAL)

Large-Scale Long-Tailed Recognition in an Open World [Project] [Paper] [Blog] Overview Open Long-Tailed Recognition (OLTR) is the author's re-implemen

Zhongqi Miao 761 Dec 26, 2022
Code for "PVNet: Pixel-wise Voting Network for 6DoF Pose Estimation" CVPR 2019 oral

Good news! We release a clean version of PVNet: clean-pvnet, including how to train the PVNet on the custom dataset. Use PVNet with a detector. The tr

ZJU3DV 722 Dec 27, 2022
Code for 'Self-Guided and Cross-Guided Learning for Few-shot segmentation. (CVPR' 2021)'

SCL Introduction Code for 'Self-Guided and Cross-Guided Learning for Few-shot segmentation. (CVPR' 2021)' We evaluated our approach using two baseline

null 34 Oct 8, 2022