FaceVerse: a Fine-grained and Detail-controllable 3D Face Morphable Model from a Hybrid Dataset (CVPR2022)

Overview

FaceVerse

FaceVerse: a Fine-grained and Detail-controllable 3D Face Morphable Model from a Hybrid Dataset

Lizhen Wang, Zhiyuan Chen, Tao Yu, Chenguang Ma, Liang Li, Yebin Liu CVPR 2022

Tsinghua University & Ant Group

[Dataset] [Project Page]

teaser

Abstract

We present FaceVerse, a fine-grained 3D Neural Face Model, which is built from hybrid East Asian face datasets containing 60K fused RGB-D images and 2K high-fidelity 3D head scan models. A novel coarse-to-fine structure is proposed to take better advantage of our hybrid dataset. In the coarse module, we generate a base parametric model from large-scale RGB-D images, which is able to predict accurate rough 3D face models in different genders, ages, etc. Then in the fine module, a conditional StyleGAN architecture trained with high-fidelity scan models is introduced to enrich elaborate facial geometric and texture details. Note that different from previous methods, our base and detailed modules are both changeable, which enables an innovative application of adjusting both the basic attributes and the facial details of 3D face models. Furthermore, we propose a single-image fitting framework based on differentiable rendering. Rich experiments show that our method outperforms the state-of-the-art methods.

results Fig.1 Single-image fitting results using FaceVerse model.

FaceVerse PCA model and pre-trained checkpoints

Please download the zip file of version 0 or version 1 (recommended) and unzip it in the ./data folder.

FaceVerse version 0 [download]: paper version.

v0

Fig.2 Single-image reconstruction results of version 0 (base model, detail model and expression refined final model).

FaceVerse version 1 [download]:

  • Refine the shape of the base PCA model: symmetrical and more detailed.

  • Remove the points inside the mouth.

  • Refine the expression PCA components.

v1

Fig.3 Single-image reconstruction results of version 1 (base model, detail model and expression refined final model).

FaceVerse version 2 [download] (only the PCA base model for video tracking, please use version 1 for image fitting):

  • Fit the expression components to the 52 blendshapes defined by Apple. Please check 'exp_name_list' in faceverse_simple_v2.npy for the mapping relation.

  • Provide a simplification option (normal with 28632 vertices, simplified with 6335 vertices): you can use the selected points of FaceVerse v2 by:

python tracking_online.py  --version 2 --use_simplification
python tracking_offline.py --input example/videos/test.mp4 --res_folder example/video_results --version 2 --use_simplification
  • Refine the shape of the base PCA model: orthogonalization.

Fig.4 Real-time online tracking results (30 fps) of version 2. The real-time version is accelerated by point-base rendering using cuda (this version has not been released).

Requirements

  • Python 3.9
  • PyTorch 1.11.0
  • torchvision 0.11.1
  • PyTorch3D 0.6.0
  • Cuda 11.3
  • ONNX Runtime
  • OpenCV
  • Numpy
  • tqdm
  • ninja

You need to compile the ops provided by stylegan2-pytorch using ninja:

cd third_libs/stylegan_ops
python3 setup.py install

Single-image fitting

Reconstructing a 3D face from a single image. There are three processes: (a) reconstructed by PCA model; (b) refined by the detailed generator; (c) refined by the expression generator.

An example input with a image folder (sampled from the FFHQ dataset):

python3 fit_images.py --version 1 --input example/images --res_folder example/image_results --save_ply

Note: the detailed refinement is based on differentiable rendering, which is quite time-consuming (over 10 minutes).

Video-based tracking using our PCA base model

offline_tracking

Offline tracking input with a video (our code will crop the face region using the first frame, --use_simplification can be only used for version >= 2):

python tracking_offline.py --input example/videos/test.mp4 --res_folder example/video_results --version 2

Online tracking using your PC camera (our code will crop the face region using the first frame, --use_simplification can be only used for version >= 2):

python tracking_online.py  --version 2

online_tracking

Note: the tracking is based on differentiable rendering and only has 2 fps.

Citation

If you use this dataset for your research, please consider citing:

@InProceedings{wang2022faceverse,
title={FaceVerse: a Fine-grained and Detail-controllable 3D Face Morphable Model from a Hybrid Dataset},
author={Wang, Lizhen and Chen, Zhiyua and Yu, Tao and Ma, Chenguang and Li, Liang and Liu, Yebin},
booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR2022)},
month={June},
year={2022},
}

Contact

Acknowledgement & License

The code is partially borrowed from 3DMM-Fitting-Pytorch, stylegan2-pytorch and OpenSeeFace. And many thanks to the volunteers participated in data collection. Our License

Comments
  • How do you fit the expression components to the 52 blendshapes defined by Apple?

    How do you fit the expression components to the 52 blendshapes defined by Apple?

    Fitstly, many thanks for your excellent work!

    I wonder how you fit the expression components to the 52 blendshapes defined by Apple, can you share the way to do that?

    Thanks!

    opened by hungsing92 8
  • How to get the inference result of apple52 blendshapes from single video file?

    How to get the inference result of apple52 blendshapes from single video file?

    Hi, how to get the inference result of apple52 blendshapes? I saw the code only have a optimization based function, this could be every time-consuming. I though it could a be single model outputs all blendshapes and rotation in a single forward.

    opened by jinfagang 7
  • Could you upload the model files onto Baidu netdisk ?

    Could you upload the model files onto Baidu netdisk ?

    FaceVerse is a great work. And thanks for your open source. But I can't download the model files becaause google drive limited quota. Could you upload the model files onto Baidu netdisk. It's more convenient for Chinese users. That's the error :

    抱歉,您目前无法查看或下载此文件。
    最近查看或下载此文件的用户过多。请稍后再尝试访问此文件。如果您尝试访问的文件特别大或已与很多人共享,那么您最长可能需要等待 24 小时才能查看或下载该文件。如果您在 24 小时后仍然无法访问文件,请与您的网域管理员联系。

    opened by huhai463127310 2
  • Render mesh in static position?

    Render mesh in static position?

    Hi again,

    I am currently trying to use the offline tracking to render the face mesh in a static position and rotation while processing a video. So ideally I want the face to remain in the same location, and simply change expressions, rather than following the face movement and rotation. Do you have any advice on how I could implement this or where in the current code I could start looking and adjusting? I've been going through the different scripts, but some advice would be greatly appreciated. Thanks!

    opened by JamesBrod 2
  • Handling Video Input

    Handling Video Input

    Hi! Really cool work! I was wondering if you're offline tracking does anything special in terms of stabilisation? I've noticed that using the image input demo across frames of a video leads to the mesh being more shakey compared to the specific tracking script and was wondering how you handled it.

    Thanks!

    opened by JamesBrod 2
  • 如何在tracking任务中获得arkit参数

    如何在tracking任务中获得arkit参数

    正如README.md中Fig.4文字上面显示的图片一样,获得: ['browDownLeft','browDownRight', 'browInnerUp', 'browOuterUpLeft', 'browOuterUpRight', 'cheekPuff', 'cheekSquintLeft', 'cheekSquintRight', 'eyeBlinkLeft', 'eyeBlinkRight', 'eyeLookDownLeft', 'eyeLookDownRight', 'eyeLookInLeft', 'eyeLookInRight', 'eyeLookOutLeft', 'eyeLookOutRight', 'eyeLookUpLeft', 'eyeLookUpRight', 'eyeSquintLeft', 'eyeSquintRight', 'eyeWideLeft', 'eyeWideRight', 'jawForward', 'jawLeft', 'jawOpen', 'jawRight', 'mouthClose', 'mouthDimpleLeft', 'mouthDimpleRight', 'mouthFrownLeft', 'mouthFrownRight', 'mouthFunnel', 'MouthLeft', 'mouthLowerDownLeft', 'mouthLowerDownRight', 'mouthPressLeft', 'mouthPressRight', 'mouthPucker', 'MouthRight', 'mouthRollLower', 'mouthRollUpper', 'mouthShrugLower', 'mouthShrugUpper', 'mouthSmileLeft', 'mouthSmileRight', 'mouthStretchLeft', 'mouthStretchRight', 'mouthUpperUpLeft', 'mouthUpperUpRight', 'noseSneerLeft', 'noseSneerRight','tongueOut'] 对应的浮点数数组

    opened by pbhfcycssjlmm 1
  • About the training code

    About the training code

    Hello author, I am interested in your work~. I just send you a e-mail with [[email protected]], could you send me the training code? If it can be shared will be great thanks~

    opened by hylq66 0
  • Some questions about this project

    Some questions about this project

    Dear author, it is a perfect work. When i use this shell, there are some error. How can i solve it? python3 fit_images.py --version 1 --input example/images --res_folder example/image_results --save_ply --align

    image

    opened by hylq66 1
  • Some questions about this project

    Some questions about this project

    nice job!I have a few questions, hope to get your reply q1.Will the training part and the CUDA accelerated rendering version be announced soon? q2.For example, how to calculate 52blendshapes for your V2 version of the design process? q3.Do you have an idea for an end-to-end training and inference solution? Can you provide a case or open source project to learn from?

    opened by positive666 1
  • When migrate to other standard arkit models, the effect is very bad

    When migrate to other standard arkit models, the effect is very bad

    HI, Whether it complies with the standard arkit specification. When I retarget to other standard arkit models, the effect is very bad. Is there a mistake for the expression base order of faceverse_simple_v2.npy? Thanks.

    opened by youngstu 3
Owner
Lizhen Wang
Lizhen Wang
Code and data of the Fine-Grained R2R Dataset proposed in paper Sub-Instruction Aware Vision-and-Language Navigation

Fine-Grained R2R Code and data of the Fine-Grained R2R Dataset proposed in the EMNLP2020 paper Sub-Instruction Aware Vision-and-Language Navigation. C

YicongHong 31 Sep 15, 2022
Python library for tracking human heads with FLAME (a 3D morphable head model)

Video Head Tracker 3D tracking library for human heads based on FLAME (a 3D morphable head model). The tracking algorithm is inspired by face2face. It

null 40 Sep 21, 2022
Hybrid CenterNet - Hybrid-supervised object detection / Weakly semi-supervised object detection

Hybrid-Supervised Object Detection System Object detection system trained by hybrid-supervision/weakly semi-supervision (HSOD/WSSOD): This project is

null 4 Mar 21, 2022
Official repository for the CVPR 2021 paper "Learning Feature Aggregation for Deep 3D Morphable Models"

Deep3DMM Official repository for the CVPR 2021 paper Learning Feature Aggregation for Deep 3D Morphable Models. Requirements This code is tested on Py

null 35 Sep 8, 2022
Morphable Detector for Object Detection on Demand

Morphable Detector for Object Detection on Demand (ICCV 2021) PyTorch implementation of the paper Morphable Detector for Object Detection on Demand. I

null 9 Feb 23, 2022
The coda and data for "Measuring Fine-Grained Domain Relevance of Terms: A Hierarchical Core-Fringe Approach" (ACL '21)

We propose a hierarchical core-fringe learning framework to measure fine-grained domain relevance of terms – the degree that a term is relevant to a broad (e.g., computer science) or narrow (e.g., deep learning) domain.

Jie Huang 13 Mar 14, 2022
The implementation of CVPR2021 paper Temporal Query Networks for Fine-grained Video Understanding, by Chuhan Zhang, Ankush Gupta and Andrew Zisserman.

Temporal Query Networks for Fine-grained Video Understanding ?? This repository contains the implementation of CVPR2021 paper Temporal_Query_Networks

null 49 Aug 15, 2022
[ICCV 2021] Counterfactual Attention Learning for Fine-Grained Visual Categorization and Re-identification

Counterfactual Attention Learning Created by Yongming Rao*, Guangyi Chen*, Jiwen Lu, Jie Zhou This repository contains PyTorch implementation for ICCV

Yongming Rao 76 Sep 9, 2022
This is the official PyTorch implementation of the paper "TransFG: A Transformer Architecture for Fine-grained Recognition" (Ju He, Jie-Neng Chen, Shuai Liu, Adam Kortylewski, Cheng Yang, Yutong Bai, Changhu Wang, Alan Yuille).

TransFG: A Transformer Architecture for Fine-grained Recognition Official PyTorch code for the paper: TransFG: A Transformer Architecture for Fine-gra

Ju He 284 Sep 26, 2022
WHENet: Real-time Fine-Grained Estimation for Wide Range Head Pose

WHENet: Real-time Fine-Grained Estimation for Wide Range Head Pose Yijun Zhou and James Gregson - BMVC2020 Abstract: We present an end-to-end head-pos

null 346 Sep 28, 2022
PyTorch implementation for Stochastic Fine-grained Labeling of Multi-state Sign Glosses for Continuous Sign Language Recognition.

Stochastic CSLR This is the PyTorch implementation for the ECCV 2020 paper: Stochastic Fine-grained Labeling of Multi-state Sign Glosses for Continuou

Zhe Niu 23 Sep 24, 2022
Code release for The Devil is in the Channels: Mutual-Channel Loss for Fine-Grained Image Classification (TIP 2020)

The Devil is in the Channels: Mutual-Channel Loss for Fine-Grained Image Classification Code release for The Devil is in the Channels: Mutual-Channel

PRIS-CV: Computer Vision Group 228 Oct 1, 2022
Code for Talk-to-Edit (ICCV2021). Paper: Talk-to-Edit: Fine-Grained Facial Editing via Dialog.

Talk-to-Edit (ICCV2021) This repository contains the implementation of the following paper: Talk-to-Edit: Fine-Grained Facial Editing via Dialog Yumin

Yuming Jiang 203 Sep 16, 2022
official Pytorch implementation of ICCV 2021 paper FuseFormer: Fusing Fine-Grained Information in Transformers for Video Inpainting.

FuseFormer: Fusing Fine-Grained Information in Transformers for Video Inpainting By Rui Liu, Hanming Deng, Yangyi Huang, Xiaoyu Shi, Lewei Lu, Wenxiu

null 72 Aug 22, 2022
PyTorch implementation of Weak-shot Fine-grained Classification via Similarity Transfer

SimTrans-Weak-Shot-Classification This repository contains the official PyTorch implementation of the following paper: Weak-shot Fine-grained Classifi

BCMI 59 Jul 11, 2022
Official PyTorch implementation of N-ImageNet: Towards Robust, Fine-Grained Object Recognition with Event Cameras (ICCV 2021)

N-ImageNet: Towards Robust, Fine-Grained Object Recognition with Event Cameras Official PyTorch implementation of N-ImageNet: Towards Robust, Fine-Gra

null 23 Sep 19, 2022
SnapMix: Semantically Proportional Mixing for Augmenting Fine-grained Data (AAAI 2021)

SnapMix: Semantically Proportional Mixing for Augmenting Fine-grained Data (AAAI 2021) PyTorch implementation of SnapMix | paper Method Overview Cite

DavidHuang 121 Aug 26, 2022
Fine-grained Control of Image Caption Generation with Abstract Scene Graphs

Faster R-CNN pretrained on VisualGenome This repository modifies maskrcnn-benchmark for object detection and attribute prediction on VisualGenome data

Shizhe Chen 7 Apr 20, 2021
TransFGU: A Top-down Approach to Fine-Grained Unsupervised Semantic Segmentation

TransFGU: A Top-down Approach to Fine-Grained Unsupervised Semantic Segmentation Zhaoyun Yin, Pichao Wang, Fan Wang, Xianzhe Xu, Hanling Zhang, Hao Li

DamoCV 14 Sep 19, 2022