DeepFaceEditing: Deep Face Generation and Editing with Disentangled Geometry and Appearance Control

Overview

DeepFaceEditing: Deep Face Generation and Editing with Disentangled Geometry and Appearance Control

One version of our system is implemented using the Jittor, and you need to install Jittor first.

HomePage: http://www.geometrylearning.com/DeepFaceEditing/

Teaser Image

Abstract

Recent facial image synthesis methods have been mainly based on conditional generative models. Sketch-based conditions can effectively describe the geometry of faces, including the contours of facial components, hair structures, as well as salient edges (e.g., wrinkles) on face surfaces but lack effective control of appearance, which is influenced by color, material, lighting condition, etc. To have more control of generated results, one possible approach is to apply existing disentangling works to disentangle face images into geometry and appearance representations. However, existing disentangling methods are not optimized for human face editing, and cannot achieve fine control of facial details such as wrinkles. To address this issue, we propose DeepFaceEditing, a structured disentanglement framework specifically designed for face images to support face generation and editing with disentangled control of geometry and appearance. We adopt a local-to-global approach to incorporate the face domain knowledge: local component images are decomposed into geometry and appearance representations, which are fused consistently using a global fusion module to improve generation quality. We exploit sketches to assist in extracting a better geometry representation, which also supports intuitive geometry editing via sketching. The resulting method can either extract the geometry and appearance representations from face images, or directly extract the geometry representation from face sketches. Such representations allow users to easily edit and synthesize face images, with decoupled control of their geometry and appearance. Both qualitative and quantitative evaluations show the superior detail and appearance control abilities of our method compared to state-of-the-art methods.

Prerequisites

  1. System

 - Ubuntu 16.04 or later

 - NVIDIA GPU + CUDA 10.0

  1. Software

 - Python 3.7

 - Jittor. More details in Jittor

  • Pillow

  • numpy

 - Packages

pip install -r requirements.txt

How to use

Please download the pre-trained model [Google Drive], then put those under 'checkpoints'.

Extract sketches from real images:

python test_model.py --geo ./images/geometry.png --output ./results/sketch_gen.png --gen_sketch

geometry imagegeometry sketch

Reconstruct faces of image A:

images provide geometry:

python test_model.py --geo ./images/geometry.png --appear ./images/geometry.png --output ./results/recon_result.png --geo_type image

Generate faces with geometry of image A and appearance of image B:

images provide geometry:

python test_model.py --geo ./images/geometry.png --appear ./images/appearance.png --output ./results/image_result.png --geo_type image

sketches provide geometry:

python test_model.py --geo ./results/sketch_gen.png --appear ./images/appearance.png --output ./results/sketch_result.png --geo_type sketch

Citation

If you found this code useful please cite our work as:

@article {chenDeepFaceEditing2021,
    author = {Chen, Shu-Yu and Liu, Feng-Lin and Lai, Yu-Kun and Rosin, Paul L. and Li, Chunpeng and Fu, Hongbo and Gao, Lin},
    title = {{DeepFaceEditing}: Deep Generation of Face Images from Sketches},
    journal = {ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH 2021)},
    year = {2021},
    volume = 40,
    pages = {90:1--90:15},
    number = 4
}
Issues
  • Fixed patch selection?

    Fixed patch selection?

    Just notice that this released demo uses fixed bounding boxes for eyes, nose, mouth (https://github.com/IGLICT/DeepFaceEditing-Jittor/blob/master/combine_model.py#L15).

    I wonder how do you define such bounding boxes in more general cases? Do you use the same fixed values as the ones shown in this released demo, or use some keypoint detection to generate such boxes?

    opened by nexuslrf 1
  • Wrong filename

    Wrong filename

    https://github.com/IGLICT/DeepFaceEditing-Jittor/blob/32b7bce2c60679523e27ca73d8e165418b57fb77/combine_model.py#L43

    load the weight of global fuse
    Traceback (most recent call last):
      File "test_model.py", line 55, in <module>
        model.initialize()
      File "/content/DeepFaceEditing-Jittor/combine_model.py", line 43, in initialize
        self.netG.load('./checkpoints/global_fuse_25.pkl')
      File "/usr/local/lib/python3.7/dist-packages/jittor/__init__.py", line 997, in load
        self.load_parameters(load(path))
      File "/usr/local/lib/python3.7/dist-packages/jittor/__init__.py", line 698, in load
        model_dict = safeunpickle(path)
      File "/usr/local/lib/python3.7/dist-packages/jittor/__init__.py", line 73, in safeunpickle
        with open(path, "rb") as f:
    FileNotFoundError: [Errno 2] No such file or directory: './checkpoints/global_fuse_25.pkl'
    

    checkpoints

    opened by woctezuma 1
  • Cannot generate realistic facial images when using hand-drawn sketches.

    Cannot generate realistic facial images when using hand-drawn sketches.

    Hi, I recently tested this model by using some hand-drawn sketches (rather sketches by sketch generators). I found the generated images look much worse than using generated sketches.

    For example when the input sketch is

    sketch_paper_1

    Your model will just output such an immature unrealistic facial image.

    sketch_result_paper_1

    To make sure this is not the problem with my drawing, I also tested the sketches shown on your paper (e.g., Fig. 15). The generated results are still much worse than the ones shown on the paper.

    For example, this sketched head shown in Fig. 15

    Screen Shot 2022-04-20 at 12 02 34 PM

    generates the face below,

    sketch_result_paper_3

    this is completely different from the results in Fig. 15.

    Could you please explain why such differences exist, and how to generate the correct images.

    Thanks in advance.

    opened by nexuslrf 0
  • About result

    About result

    I followed your instructions to run this code, there is no issues, but the result images what I obtained is blank. Any suggestion about it?

    Here is the running process:

    (deepface) [email protected]:~/DeepFaceEditing-Jittor$ python test_model.py --geo ./images/geometry.png --output ./results/sketch_gen.png --gen_sketch [i 1021 21:22:46.292933 44 compiler.py:941] Jittor(1.3.1.9) src: /home/lisicheng/anaconda3/envs/deepface/lib/python3.7/site-packages/jittor [i 1021 21:22:46.294142 44 compiler.py:942] g++ at /usr/local/bin/g++(7.5.0) [i 1021 21:22:46.294178 44 compiler.py:943] cache_path: /home/lisicheng/.cache/jittor/jt1.3.1/g++7.5.0/py3.7.11/Linux-5.11.0-3x6e/11thGenIntelRCxdc/default [i 1021 21:22:46.296077 44 init.py:372] Found nvcc(10.0.130) at /usr/local/cuda-10.0/bin/nvcc. [i 1021 21:22:46.361783 44 init.py:372] Found gdb(9.2) at /usr/bin/gdb. [i 1021 21:22:46.365229 44 init.py:372] Found addr2line(2.34) at /usr/bin/addr2line. [i 1021 21:22:46.442669 44 compiler.py:993] cuda key:cu10.0.130_sm_86 [i 1021 21:22:46.564720 44 init.py:187] Total mem: 31.22GB, using 10 procs for compiling. [i 1021 21:22:46.629541 44 jit_compiler.cc:27] Load cc_path: /usr/local/bin/g++ [i 1021 21:22:46.730638 44 init.cc:61] Found cuda archs: [86,] [w 1021 21:22:46.808636 44 compiler.py:1326] CUDA arch(86)>75 will be backward-compatible [i 1021 21:22:46.812635 44 compile_extern.py:497] mpicc not found, distribution disabled. [i 1021 21:22:46.854007 44 compile_extern.py:29] found /usr/local/cuda-10.0/include/cublas.h [i 1021 21:22:46.856392 44 compile_extern.py:29] found /usr/local/cuda-10.0/lib64/libcublas.so [i 1021 21:23:35.793154 44 compile_extern.py:29] found /usr/local/cuda-10.0/include/cudnn.h [i 1021 21:23:35.798631 44 compile_extern.py:29] found /usr/local/cuda-10.0/lib64/libcudnn.so [i 1021 21:32:56.286736 44 compile_extern.py:29] found /usr/local/cuda-10.0/include/curand.h [i 1021 21:32:56.305076 44 compile_extern.py:29] found /usr/local/cuda-10.0/lib64/libcurand.so [i 1021 21:32:59.990031 44 cuda_flags.cc:32] CUDA enabled.

    opened by simonlisss 0
  • the link of pre-trained model was unavailable

    the link of pre-trained model was unavailable

    Hi, thanks for your excellent work. But now I found that the link for downloading pre-trained model was unavailable. Could you please share it again? Thanks a lot!!!

    opened by lijiae 1
  • Can you share your training details?

    Can you share your training details?

    Congratulations. But I have a little request, can you share your training details? I reproduced it according to your paper, but it didn't work. Thanks.

    opened by Jeffin-x 1
  • Why GeometryEncoder instead GlobalGenerator

    Why GeometryEncoder instead GlobalGenerator

    Hello, nice job! I have a question, why don't use the encoder(including downsample and resnet-blocks) part of GlobalGenerator with ngf=32 as Geometry Encoder directly, but train a GeometryEncoder with ngf=64. Thanks.

    opened by Michel-liu 3
  • CUDA error at ~/.local/lib/python3.8/site-packages/jittor/src/ops/array_op.cc:32  code=222( cudaErrorUnsupportedPtxVersion ) cudaStreamCreateWithFlags(&stream, cudaStreamNonBlocking)

    CUDA error at ~/.local/lib/python3.8/site-packages/jittor/src/ops/array_op.cc:32 code=222( cudaErrorUnsupportedPtxVersion ) cudaStreamCreateWithFlags(&stream, cudaStreamNonBlocking)

    Hi:

    My env:

    • OS: Ubuntu 20.04
    • Cuda: 11.3
    • cudnn 8.2.0
    • gcc/g++: 10.2

    Failed to run test code as:

      DeepFaceEditing-Jittor git:(main) python test_model.py --geo ./results/sketch_gen.png --appear ./images/appearance.png --output ./results/sketch_result.png --geo_type sketch
    [i 0512 11:37:12.017235 64 compiler.py:851] Jittor(1.2.2.71) src: ~/.local/lib/python3.8/site-packages/jittor
    [i 0512 11:37:12.017286 64 compiler.py:852] g++ at /usr/bin/g++
    [i 0512 11:37:12.017319 64 compiler.py:853] cache_path: ~/.cache/jittor/default/g++
    [i 0512 11:37:12.019891 64 __init__.py:257] Found /usr/local/cuda/bin/nvcc(11.3.58) at /usr/local/cuda/bin/nvcc.
    [i 0512 11:37:12.070950 64 __init__.py:257] Found gdb(9.2) at /usr/bin/gdb.
    [i 0512 11:37:12.074234 64 __init__.py:257] Found addr2line(2.34) at /usr/bin/addr2line.
    [i 0512 11:37:12.108394 64 compiler.py:893] pybind_include: -I/usr/include/python3.8 -I~/.local/lib/python3.8/site-packages/pybind11/include
    [i 0512 11:37:12.116724 64 compiler.py:895] extension_suffix: .cpython-38-x86_64-linux-gnu.so
    [i 0512 11:37:12.207914 64 __init__.py:169] Total mem: 31.27GB, using 10 procs for compiling.
    terminate called after throwing an instance of 'std::runtime_error'
      what():  [f 0512 11:37:12.404950 64 helper_cuda.h:126] CUDA error at ~/.local/lib/python3.8/site-packages/jittor/src/ops/array_op.cc:32  code=222( cudaErrorUnsupportedPtxVersion ) cudaStreamCreateWithFlags(&stream, cudaStreamNonBlocking)
    

    Any suggestions?

    opened by jiapei100 1
Owner
Intelligent Graphics Laboratory, Institute of Computing Technology
null
Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance

Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance Project Page | Paper | Data This repository contains an implementatio

Lior Yariv 389 May 19, 2022
A large-scale face dataset for face parsing, recognition, generation and editing.

CelebAMask-HQ [Paper] [Demo] CelebAMask-HQ is a large-scale face image dataset that has 30,000 high-resolution face images selected from the CelebA da

switchnorm 1.6k May 26, 2022
VGGFace2-HQ - A high resolution face dataset for face editing purpose

The first open source high resolution dataset for face swapping!!! A high resolution version of VGGFace2 for academic face editing purpose

Naiyuan Liu 165 May 21, 2022
Disentangled Lifespan Face Synthesis

Disentangled Lifespan Face Synthesis Project Page | Paper Demo on Colab Preparation Please follow this github to prepare the environments and dataset.

何森 45 May 19, 2022
StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation

StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation Demo video: CVPR 2021 Oral: Single Channel Manipulation: Localized or attribu

Zongze Wu 227 May 16, 2022
Implementation of StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation in PyTorch

StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation Implementation of StyleSpace Analysis: Disentangled Controls for StyleGAN Ima

Xuanchi Ren 78 May 16, 2022
DVG-Face: Dual Variational Generation for Heterogeneous Face Recognition, TPAMI 2021

DVG-Face: Dual Variational Generation for HFR This repo is a PyTorch implementation of DVG-Face: Dual Variational Generation for Heterogeneous Face Re

null 36 May 23, 2022
Self-Supervised Monocular 3D Face Reconstruction by Occlusion-Aware Multi-view Geometry Consistency[ECCV 2020]

Self-Supervised Monocular 3D Face Reconstruction by Occlusion-Aware Multi-view Geometry Consistency(ECCV 2020) This is an official python implementati

null 286 May 13, 2022
SLAMP: Stochastic Latent Appearance and Motion Prediction

SLAMP: Stochastic Latent Appearance and Motion Prediction Official implementation of the paper SLAMP: Stochastic Latent Appearance and Motion Predicti

Kaan Akan 27 Mar 23, 2022
Pytorch implementation for A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose

A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose Paper | Website | Data A-NeRF: Articulated Neural Radiance F

Shih-Yang Su 123 May 24, 2022
[CVPR'21] DeepSurfels: Learning Online Appearance Fusion

DeepSurfels: Learning Online Appearance Fusion Paper | Video | Project Page This is the official implementation of the CVPR 2021 submission DeepSurfel

Online Reconstruction 48 May 20, 2022
Unified tracking framework with a single appearance model

Paper: Do different tracking tasks require different appearance model? [ArXiv] (comming soon) [Project Page] (comming soon) UniTrack is a simple and U

ZhongdaoWang 268 May 26, 2022
Canonical Appearance Transformations

CAT-Net: Learning Canonical Appearance Transformations Code to accompany our paper "How to Train a CAT: Learning Canonical Appearance Transformations

STARS Laboratory 53 Mar 29, 2022
A customisable game where you have to quickly click on black tiles in order of appearance while avoiding clicking on white squares.

W.I.P-Aim-Memory-Game A customisable game where you have to quickly click on black tiles in order of appearance while avoiding clicking on white squar

dE_soot 1 Dec 8, 2021
[CVPR 2020] Interpreting the Latent Space of GANs for Semantic Face Editing

InterFaceGAN - Interpreting the Latent Space of GANs for Semantic Face Editing Figure: High-quality facial attributes editing results with InterFaceGA

GenForce: May Generative Force Be with You 1.2k May 27, 2022
InterFaceGAN - Interpreting the Latent Space of GANs for Semantic Face Editing

InterFaceGAN - Interpreting the Latent Space of GANs for Semantic Face Editing Figure: High-quality facial attributes editing results with InterFaceGA

GenForce: May Generative Force Be with You 1.2k May 22, 2022
Realtime Face Anti Spoofing with Face Detector based on Deep Learning using Tensorflow/Keras and OpenCV

Realtime Face Anti-Spoofing Detection ?? Realtime Face Anti Spoofing Detection with Face Detector to detect real and fake faces Please star this repo

Prem Kumar 73 May 18, 2022
(ICCV 2021) Official code of "Dressing in Order: Recurrent Person Image Generation for Pose Transfer, Virtual Try-on and Outfit Editing."

Dressing in Order (DiOr) ?? [Paper] ?? [Webpage] ?? [Running this code] The official implementation of "Dressing in Order: Recurrent Person Image Gene

Aiyu Cui 207 May 20, 2022
Official codebase for running the small, filtered-data GLIDE model from GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models.

GLIDE This is the official codebase for running the small, filtered-data GLIDE model from GLIDE: Towards Photorealistic Image Generation and Editing w

OpenAI 2.4k May 17, 2022