Semi-supervised Adversarial Learning to Generate Photorealistic Face Images of New Identities from 3D Morphable Model

Overview

Semi-supervised Adversarial Learning to Generate Photorealistic Face Images of New Identities from 3D Morphable Model

Baris Gecer 1, Binod Bhattarai 1, Josef Kittler 2, & Tae-Kyun Kim 1
1 Department of Electrical and Electronic Engineering, Imperial College London, UK
2 Centre for Vision, Speech and Signal Processing, University of Surrey, UK

This repository provides a Tensorflow implementation of our study where we propose a novel end-to-end semi-supervised adversarial framework to generate photorealistic face images of new identities with wide ranges of expressions, poses, and illuminations conditioned by a 3D morphable model.



(This documentation is still under construction, please refer to our paper for more details)

Approach

Our approach aims to synthesize photorealistic images conditioned by a given synthetic image by 3DMM. It regularizes cycle consistency by introducing an additional adversarial game between the two generator networks in an unsupervised fashion. Thus the under-constraint cycle loss is supervised to have correct matching between the two domains by the help of a limited number of paired data. We also encourage the generator to preserve face identity by a set-based supervision through a pretrained classification network.

Dependencies

Data

  • Generate synthetic images using any 3DMM model i.e. LSFM or Basel Face Model by running gen_syn_latent.m
  • Align and crop all datasets using MTCNN to 108x108

Usage

Train by the following script

$ python main.py    --log_dir [path2_logdir] --data_dir [path2_datadir] --syn_dataset [synthetic_dataset_name]
                    --dataset [real_dataset_name] --dataset_3dmm [300W-3D & AFLW2000_dirname] --input_scale_size 108

Add --load_path [paused_training_logdir] to continue a training

Generate realistic images after training by the following script

$ python main.py    --log_dir [path2_logdir] --data_dir [path2_datadir] --syn_dataset [synthetic_dataset_name]
                    --dataset [real_dataset_name] --dataset_3dmm [300W-3D & AFLW2000_dirname] --input_scale_size 108
                    --save_syn_dataset [saving_dir] --train_generator False --generate_dataset True --pretrained_gen [path2_logdir + /model.ckpt]

Pretrained Model

You can download the pretrained model

More Results


Citation

if you find this work is useful for your research, please cite our paper:

@inproceedings{gecer2018semi,
  title={Semi-supervised adversarial learning to generate photorealistic face images of new identities from 3D morphable model},
  author={Gecer, Baris and Bhattarai, Binod and Kittler, Josef and Kim, Tae-Kyun},
  booktitle={Proceedings of the European Conference on Computer Vision (ECCV)},
  pages={217--234},
  year={2018}
}


Acknowledgement

This work was supported by the EPSRC Programme Grant ‘FACER2VM’ (EP/N007743/1). Baris Gecer is funded by the Turkish Ministry of National Education. This study is morally motivated to improve face recognition to help prediction of genetic disorders visible on human face in earlier stages.

Code borrows heavily from carpedm20's BEGAN implementation.

Comments
  • Pretrained model

    Pretrained model

    Dear Baris, thank you for your codes! Do you mind helping me a bit to launch it? I am stuck with attempt not to train it, but to use your model. It asks me why '--dataset' is empty... but why shouldn't it be if I am not training the model, but willing only to synthesis some new faces?

    opened by voa18105 8
  • The question about 300W-3D database generate tranining data

    The question about 300W-3D database generate tranining data

    According to the readme.txt in 3DDFA , I can't generated the training data with the MATLAB code. Except the testing picture in the original folder. When add the *.mat from "300W-3D" and *_face.mat from"300W-3D-Face" to the 'input/' folder , just come out the all black picture. Can you give me some idea ? Thank you very much.

    opened by Gincoeng 2
  • windows or linux?

    windows or linux?

    when i see the environment.yml, it seems that it should be run in windows. but in the readme file, the symbol $ indicate it is run in linix, which system should i should the code? Thank you so much!

    opened by jiyuay 2
  • test

    test

    Why dataset_3dmm and dataset are needed for testing? How to generate realistic image from synthetic one without needing to prepare additional dataset?

    opened by dinashi 1
  • codes to run MTCNN for preprocess

    codes to run MTCNN for preprocess

    Thanks for great paper.

    In README, you say

    Align and crop all datasets using MTCNN to 108x108

    but are there any available codes to run MTCNN to align and crop images?

    opened by aiueogawa 1
Owner
Baris Gecer
I am currently PhD. student at Imperial College, London and working on face recognition with generative adversarial learning
Baris Gecer
Python library for tracking human heads with FLAME (a 3D morphable head model)

Video Head Tracker 3D tracking library for human heads based on FLAME (a 3D morphable head model). The tracking algorithm is inspired by face2face. It

null 61 Dec 25, 2022
Official repository for the CVPR 2021 paper "Learning Feature Aggregation for Deep 3D Morphable Models"

Deep3DMM Official repository for the CVPR 2021 paper Learning Feature Aggregation for Deep 3D Morphable Models. Requirements This code is tested on Py

null 38 Dec 27, 2022
Official codebase for running the small, filtered-data GLIDE model from GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models.

GLIDE This is the official codebase for running the small, filtered-data GLIDE model from GLIDE: Towards Photorealistic Image Generation and Editing w

OpenAI 2.9k Jan 4, 2023
An image base contains 490 images for learning (400 cars and 90 boats), and another 21 images for testingAn image base contains 490 images for learning (400 cars and 90 boats), and another 21 images for testing

SVM Données Une base d’images contient 490 images pour l’apprentissage (400 voitures et 90 bateaux), et encore 21 images pour fait des tests. Prétrait

Achraf Rahouti 3 Nov 30, 2021
Morphable Detector for Object Detection on Demand

Morphable Detector for Object Detection on Demand (ICCV 2021) PyTorch implementation of the paper Morphable Detector for Object Detection on Demand. I

null 9 Feb 23, 2022
Project looking into use of autoencoder for semi-supervised learning and comparing data requirements compared to supervised learning.

Project looking into use of autoencoder for semi-supervised learning and comparing data requirements compared to supervised learning.

Tom-R.T.Kvalvaag 2 Dec 17, 2021
Semi-supervised Representation Learning for Remote Sensing Image Classification Based on Generative Adversarial Networks

SSRL-for-image-classification Semi-supervised Representation Learning for Remote Sensing Image Classification Based on Generative Adversarial Networks

Feng 2 Nov 19, 2021
Unified unsupervised and semi-supervised domain adaptation network for cross-scenario face anti-spoofing, Pattern Recognition

USDAN The implementation of Unified unsupervised and semi-supervised domain adaptation network for cross-scenario face anti-spoofing, which is accepte

null 11 Nov 3, 2022
UniMoCo: Unsupervised, Semi-Supervised and Full-Supervised Visual Representation Learning

UniMoCo: Unsupervised, Semi-Supervised and Full-Supervised Visual Representation Learning This is the official PyTorch implementation for UniMoCo pape

dddzg 49 Jan 2, 2023
Pytorch implementation of our method for high-resolution (e.g. 2048x1024) photorealistic video-to-video translation.

vid2vid Project | YouTube(short) | YouTube(full) | arXiv | Paper(full) Pytorch implementation for high-resolution (e.g., 2048x1024) photorealistic vid

NVIDIA Corporation 8.1k Jan 1, 2023
Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding

The Hypersim Dataset For many fundamental scene understanding tasks, it is difficult or impossible to obtain per-pixel ground truth labels from real i

Apple 1.3k Jan 4, 2023
Face Synthetics dataset is a collection of diverse synthetic face images with ground truth labels.

The Face Synthetics dataset Face Synthetics dataset is a collection of diverse synthetic face images with ground truth labels. It was introduced in ou

Microsoft 608 Jan 2, 2023
Hybrid CenterNet - Hybrid-supervised object detection / Weakly semi-supervised object detection

Hybrid-Supervised Object Detection System Object detection system trained by hybrid-supervision/weakly semi-supervision (HSOD/WSSOD): This project is

null 5 Dec 10, 2022
Shape-aware Semi-supervised 3D Semantic Segmentation for Medical Images

SASSnet Code for paper: Shape-aware Semi-supervised 3D Semantic Segmentation for Medical Images(MICCAI 2020) Our code is origin from UA-MT You can fin

klein 125 Jan 3, 2023
Adversarial Color Enhancement: Generating Unrestricted Adversarial Images by Optimizing a Color Filter

ACE Please find the preliminary version published at BMVC 2020 in the folder BMVC_version, and its extended journal version in Journal_version. Datase

null 28 Dec 25, 2022
A tensorflow/keras implementation of StyleGAN to generate images of new Pokemon.

PokeGAN A tensorflow/keras implementation of StyleGAN to generate images of new Pokemon. Dataset The model has been trained on dataset that includes 8

null 19 Jul 26, 2022
SMPL-X: A new joint 3D model of the human body, face and hands together

SMPL-X: A new joint 3D model of the human body, face and hands together [Paper Page] [Paper] [Supp. Mat.] Table of Contents License Description News I

Vassilis Choutas 1k Jan 9, 2023
CoSMA: Convolutional Semi-Regular Mesh Autoencoder. From Paper "Mesh Convolutional Autoencoder for Semi-Regular Meshes of Different Sizes"

Mesh Convolutional Autoencoder for Semi-Regular Meshes of Different Sizes Implementation of CoSMA: Convolutional Semi-Regular Mesh Autoencoder arXiv p

Fraunhofer SCAI 10 Oct 11, 2022
code for paper "Not All Unlabeled Data are Equal: Learning to Weight Data in Semi-supervised Learning" by Zhongzheng Ren*, Raymond A. Yeh*, Alexander G. Schwing.

Not All Unlabeled Data are Equal: Learning to Weight Data in Semi-supervised Learning Overview This code is for paper: Not All Unlabeled Data are Equa

Jason Ren 22 Nov 23, 2022