Semi-supervised Adversarial Learning to Generate Photorealistic Face Images of New Identities from 3D Morphable Model

Baris Gecer

Last update: Dec 29, 2022

Related tags

Deep Learning tensorflow generative-adversarial-network gan synthetic-images face-recognition 3dmm face-generation eccv-2018

Overview

Semi-supervised Adversarial Learning to Generate Photorealistic Face Images of New Identities from 3D Morphable Model

Baris Gecer¹, Binod Bhattarai¹, Josef Kittler², & Tae-Kyun Kim¹
¹Department of Electrical and Electronic Engineering, Imperial College London, UK
²Centre for Vision, Speech and Signal Processing, University of Surrey, UK

This repository provides a Tensorflow implementation of our study where we propose a novel end-to-end semi-supervised adversarial framework to generate photorealistic face images of new identities with wide ranges of expressions, poses, and illuminations conditioned by a 3D morphable model.

(This documentation is still under construction, please refer to our paper for more details)

Approach

Our approach aims to synthesize photorealistic images conditioned by a given synthetic image by 3DMM. It regularizes cycle consistency by introducing an additional adversarial game between the two generator networks in an unsupervised fashion. Thus the under-constraint cycle loss is supervised to have correct matching between the two domains by the help of a limited number of paired data. We also encourage the generator to preserve face identity by a set-based supervision through a pretrained classification network.

Dependencies

Data

Generate synthetic images using any 3DMM model i.e. LSFM or Basel Face Model by running gen_syn_latent.m
Align and crop all datasets using MTCNN to 108x108

Usage

Train by the following script

$ python main.py    --log_dir [path2_logdir] --data_dir [path2_datadir] --syn_dataset [synthetic_dataset_name]
                    --dataset [real_dataset_name] --dataset_3dmm [300W-3D & AFLW2000_dirname] --input_scale_size 108

Add --load_path [paused_training_logdir] to continue a training

Generate realistic images after training by the following script

$ python main.py    --log_dir [path2_logdir] --data_dir [path2_datadir] --syn_dataset [synthetic_dataset_name]
                    --dataset [real_dataset_name] --dataset_3dmm [300W-3D & AFLW2000_dirname] --input_scale_size 108
                    --save_syn_dataset [saving_dir] --train_generator False --generate_dataset True --pretrained_gen [path2_logdir + /model.ckpt]

Pretrained Model

You can download the pretrained model

More Results

Citation

if you find this work is useful for your research, please cite our paper:

@inproceedings{gecer2018semi,
  title={Semi-supervised adversarial learning to generate photorealistic face images of new identities from 3D morphable model},
  author={Gecer, Baris and Bhattarai, Binod and Kittler, Josef and Kim, Tae-Kyun},
  booktitle={Proceedings of the European Conference on Computer Vision (ECCV)},
  pages={217--234},
  year={2018}
}

Acknowledgement

This work was supported by the EPSRC Programme Grant ‘FACER2VM’ (EP/N007743/1). Baris Gecer is funded by the Turkish Ministry of National Education. This study is morally motivated to improve face recognition to help prediction of genetic disorders visible on human face in earlier stages.

Code borrows heavily from carpedm20's BEGAN implementation.

Comments

Pretrained model

Dear Baris, thank you for your codes! Do you mind helping me a bit to launch it? I am stuck with attempt not to train it, but to use your model. It asks me why '--dataset' is empty... but why shouldn't it be if I am not training the model, but willing only to synthesis some new faces?

opened by voa18105 8
The question about 300W-3D database generate tranining data

According to the readme.txt in 3DDFA , I can't generated the training data with the MATLAB code. Except the testing picture in the original folder. When add the *.mat from "300W-3D" and *_face.mat from"300W-3D-Face" to the 'input/' folder , just come out the all black picture. Can you give me some idea ? Thank you very much.

opened by Gincoeng 2
windows or linux?

when i see the environment.yml, it seems that it should be run in windows. but in the readme file, the symbol $ indicate it is run in linix, which system should i should the code? Thank you so much!

opened by jiyuay 2
test

Why dataset_3dmm and dataset are needed for testing? How to generate realistic image from synthetic one without needing to prepare additional dataset?

opened by dinashi 1
codes to run MTCNN for preprocess

Thanks for great paper.

In README, you say

Align and crop all datasets using MTCNN to 108x108

but are there any available codes to run MTCNN to align and crop images?

opened by aiueogawa 1

Semi-supervised Adversarial Learning to Generate Photorealistic Face Images of New Identities from 3D Morphable Model

Related tags

Overview

Semi-supervised Adversarial Learning to Generate Photorealistic Face Images of New Identities from 3D Morphable Model

Approach

Dependencies

Data

Usage

Pretrained Model

More Results

Citation

Acknowledgement

Comments

Pretrained model

The question about 300W-3D database generate tranining data

windows or linux?

test

codes to run MTCNN for preprocess

Owner

Baris Gecer

Python library for tracking human heads with FLAME (a 3D morphable head model)

Official repository for the CVPR 2021 paper "Learning Feature Aggregation for Deep 3D Morphable Models"

Official codebase for running the small, filtered-data GLIDE model from GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models.

An image base contains 490 images for learning (400 cars and 90 boats), and another 21 images for testingAn image base contains 490 images for learning (400 cars and 90 boats), and another 21 images for testing

Morphable Detector for Object Detection on Demand

Project looking into use of autoencoder for semi-supervised learning and comparing data requirements compared to supervised learning.

Semi-supervised Representation Learning for Remote Sensing Image Classification Based on Generative Adversarial Networks

Unified unsupervised and semi-supervised domain adaptation network for cross-scenario face anti-spoofing, Pattern Recognition

UniMoCo: Unsupervised, Semi-Supervised and Full-Supervised Visual Representation Learning

Pytorch implementation of our method for high-resolution (e.g. 2048x1024) photorealistic video-to-video translation.

Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding

Face Synthetics dataset is a collection of diverse synthetic face images with ground truth labels.

Hybrid CenterNet - Hybrid-supervised object detection / Weakly semi-supervised object detection

Shape-aware Semi-supervised 3D Semantic Segmentation for Medical Images

Adversarial Color Enhancement: Generating Unrestricted Adversarial Images by Optimizing a Color Filter

A tensorflow/keras implementation of StyleGAN to generate images of new Pokemon.

SMPL-X: A new joint 3D model of the human body, face and hands together

CoSMA: Convolutional Semi-Regular Mesh Autoencoder. From Paper "Mesh Convolutional Autoencoder for Semi-Regular Meshes of Different Sizes"

code for paper "Not All Unlabeled Data are Equal: Learning to Weight Data in Semi-supervised Learning" by Zhongzheng Ren, Raymond A. Yeh, Alexander G. Schwing.

Semi-supervised Adversarial Learning to Generate Photorealistic Face Images of New Identities from 3D Morphable Model

Related tags

Overview

Semi-supervised Adversarial Learning to Generate Photorealistic Face Images of New Identities from 3D Morphable Model

Approach

Dependencies

Data

Usage

Pretrained Model

More Results

Citation

Acknowledgement

Comments

Pretrained model

The question about 300W-3D database generate tranining data

windows or linux?

test

codes to run MTCNN for preprocess

Owner

Baris Gecer

Python library for tracking human heads with FLAME (a 3D morphable head model)

Official repository for the CVPR 2021 paper "Learning Feature Aggregation for Deep 3D Morphable Models"

Official codebase for running the small, filtered-data GLIDE model from GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models.

An image base contains 490 images for learning (400 cars and 90 boats), and another 21 images for testingAn image base contains 490 images for learning (400 cars and 90 boats), and another 21 images for testing

Morphable Detector for Object Detection on Demand

Project looking into use of autoencoder for semi-supervised learning and comparing data requirements compared to supervised learning.

Semi-supervised Representation Learning for Remote Sensing Image Classification Based on Generative Adversarial Networks

Unified unsupervised and semi-supervised domain adaptation network for cross-scenario face anti-spoofing, Pattern Recognition

UniMoCo: Unsupervised, Semi-Supervised and Full-Supervised Visual Representation Learning

Pytorch implementation of our method for high-resolution (e.g. 2048x1024) photorealistic video-to-video translation.

Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding

Face Synthetics dataset is a collection of diverse synthetic face images with ground truth labels.

Hybrid CenterNet - Hybrid-supervised object detection / Weakly semi-supervised object detection

Shape-aware Semi-supervised 3D Semantic Segmentation for Medical Images

Adversarial Color Enhancement: Generating Unrestricted Adversarial Images by Optimizing a Color Filter

A tensorflow/keras implementation of StyleGAN to generate images of new Pokemon.

SMPL-X: A new joint 3D model of the human body, face and hands together

CoSMA: Convolutional Semi-Regular Mesh Autoencoder. From Paper "Mesh Convolutional Autoencoder for Semi-Regular Meshes of Different Sizes"

code for paper "Not All Unlabeled Data are Equal: Learning to Weight Data in Semi-supervised Learning" by Zhongzheng Ren*, Raymond A. Yeh*, Alexander G. Schwing.

code for paper "Not All Unlabeled Data are Equal: Learning to Weight Data in Semi-supervised Learning" by Zhongzheng Ren, Raymond A. Yeh, Alexander G. Schwing.