The code of paper 'Learning to Aggregate and Personalize 3D Face from In-the-Wild Photo Collection'

Related tags

Deep Learning 3DFaceReconstruction-LAP
Overview

Learning to Aggregate and Personalize 3D Face from In-the-Wild Photo Collection

Pytorch implemetation of paper 'Learning to Aggregate and Personalize 3D Face from In-the-Wild Photo Collection'

Introduction

This repository contains demo of LAP (Learning to Aggregate and Personalize) framework for reconstructing 3D face. Right now we provide an early version of demo for testing on in-the-wild images. The output size is 128 and the model is finetuned on CelebAMask-HQ Dataset.

Requirments

The code is tested on pytorch 1.3.0 with torchvision 0.4.1

pip install torch==1.3.0
pip install torchvision==0.4.1

Neural renderer is needed to render the reconstructed images or videos

pip install neural_renderer_pytorch

It may fail if you have a GCC version below 5. If you do not want to upgrade your GCC, one alternative solution is to use conda's GCC and compile the package from source. For example:

conda install gxx_linux-64=7.3
git clone https://github.com/daniilidis-group/neural_renderer.git
cd neural_renderer
python setup.py install

Facenet is also needed to detect and crop human faces in images.

pip install facenet-pytorch

DEMO

Download the pretrained model, and then run:

python demo.py --input ./images --result ./results --checkpoint_lap ./demo/checkpoint300.pth

Options:

--gpu: enable gpu

--detect_human_face: enable automatic human face detection and cropping using MTCNN provided in facenet-pytorch

--render_video: render 3D animations using neural_renderer (GPU is required)

Note:

The output depth is transformed by several options and functions, including tanh(), depth_rescaler and depth_inv_rescaler for better visualization. You could search along these options to find the original output depth and rescale it to a required range. The defined direction of normal in normal maps may be different to your required setting. If you want to accelarate the inference procedure, you may delete the branches irrelavant to reconstruct depth, and set anti_aliasing=False in each renderer.

License

The code contained in this repository is under MIT License and is free for commercial and non-commercial purposes. The dependencies, in particular, neural-renderer-pytorch, facenet, may have its own license.

Citation

@InProceedings{Zhang_2021_CVPR,
    author    = {Zhang, Zhenyu and Ge, Yanhao and Chen, Renwang and Tai, Ying and Yan, Yan and Yang, Jian and Wang, Chengjie and Li, Jilin and Huang, Feiyue},
    title     = {Learning To Aggregate and Personalize 3D Face From In-the-Wild Photo Collection},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    year      = {2021},
    pages     = {14214-14224}
}
Comments
  • About training in resolution of 256×256

    About training in resolution of 256×256

    Hi! Thank you very much for such a great work!

    I have a question about training in the size of 256×256 mentioned in your paper. Based on unsuper3d framework, I find that a single rendering time (i.e., rendering warped_depth) for 256×256 takes ~0.48 s in one NVIDIA Tesla V100 GPU, meaning that it takes ~38 days to train for 30 epoch. So I want to know if you have adopted a special differentiable renderer or parallel rendering method to speed up.

    Again thank you for your work, looking forward to your reply :>.

    opened by YunjieYu 1
  • Multiple input

    Multiple input

    Hi Zhang! Thanks for sharing your great work! Would you please provide the demo of multi input reconstruction? Also, the reconstructions of Asian faces looked not lifelike. Maybe because of the training dataset. So do you have plan to release the training code?

    opened by li-fang 1
  • Excuse me, I want to ask you a question

    Excuse me, I want to ask you a question

    Hello,JesseZhang92 ！Does your project will disclose the code of the model part of the model, how does the two indicators of MAD, SIDE assessment? I am looking forward to your reply.Thanks!

    opened by huyu-coder 1
  • AttributeError: 'Demo' object has no attribute 'canon_depth'

    AttributeError: 'Demo' object has no attribute 'canon_depth'

    I get the following error when trying to run the model on the sample images provided.

    Loading checkpoint from ./demo/checkpoint300.pth Saving checkpoint to ./demo/checkpoint300_fix2.pth Processing ./images/001.png /usr/local/lib/python3.7/dist-packages/torch/nn/functional.py:2705: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details. warnings.warn("Default grid_sample and affine_grid behavior has changed " Rendering video animations Traceback (most recent call last): File "demo.py", line 347, in result_code = model.run(pil_im) File "demo.py", line 252, in run self.render_animation() File "demo.py", line 256, in render_animation b, h, w = self.canon_depth.shape AttributeError: 'Demo' object has no attribute 'canon_depth'

    It looks like the object of class Demo as defined in demo.py does not have the canon_depth attribute in the current version of the code.

    opened by sudoboi 1
  • Can not open the link

    Can not open the link "pretrained model"

    In the DEMO part, I cannot open the link "pretrained model", thus I can't download the model. Maybe it's because I cannot log in google drive ? Could you please give me some advice?

    opened by coder-gx 0
  • RuntimeError: CUDA error: invalid device function

    RuntimeError: CUDA error: invalid device function

    Hey guys,

    I made a colab notebook for this project. I'm currently getting this error

    Traceback (most recent call last): File "demo.py", line 341, in <module> model = Demo(args) File "demo.py", line 53, in __init__ self.renderer_mr = Renderer(cfgs, im_size=128) File "/content/3DFaceReconstruction-LAP/lap/renderer/renderer_mr.py", line 45, in __init__ self.inv_k_mat = torch.inverse(k_mat).unsqueeze(0) RuntimeError: CUDA error: invalid device function

    How to fix this?

    Here is a link to my notebook , you can make a copy and edit it

    https://colab.research.google.com/drive/18tIkvLIaN-_v3sLW2ykkcLGpqxwRN13z?usp=sharing

    opened by GeorvityLabs 3
Owner
Tencent YouTu Research
Tencent YouTu Research
GitHub
This is the code for the paper "Jinkai Zheng, Xinchen Liu, Wu Liu, Lingxiao He, Chenggang Yan, Tao Mei: Gait Recognition in the Wild with Dense 3D Representations and A Benchmark. (CVPR 2022)"

Gait3D-Benchmark This is the code for the paper "Jinkai Zheng, Xinchen Liu, Wu Liu, Lingxiao He, Chenggang Yan, Tao Mei: Gait Recognition in the Wild

null 82 Jan 4, 2023
Code release of paper "Deep Multi-View Stereo gone wild"

Deep MVS gone wild Pytorch implementation of "Deep MVS gone wild" (Paper | website) This repository provides the code to reproduce the experiments of

François Darmon 53 Dec 24, 2022
Code for ICCV2021 paper SPEC: Seeing People in the Wild with an Estimated Camera

SPEC: Seeing People in the Wild with an Estimated Camera [ICCV 2021] SPEC: Seeing People in the Wild with an Estimated Camera, Muhammed Kocabas, Chun-

Muhammed Kocabas 187 Dec 26, 2022
Realtime Face Anti Spoofing with Face Detector based on Deep Learning using Tensorflow/Keras and OpenCV

Realtime Face Anti-Spoofing Detection ?? Realtime Face Anti Spoofing Detection with Face Detector to detect real and fake faces Please star this repo

Prem Kumar 86 Aug 3, 2022
Source code and notebooks to reproduce experiments and benchmarks on Bias Faces in the Wild (BFW).

Face Recognition: Too Bias, or Not Too Bias? Robinson, Joseph P., Gennady Livitz, Yann Henon, Can Qin, Yun Fu, and Samson Timoner. "Face recognition:

Joseph P. Robinson 41 Dec 12, 2022
Code for HLA-Face: Joint High-Low Adaptation for Low Light Face Detection (CVPR21)

HLA-Face: Joint High-Low Adaptation for Low Light Face Detection The official PyTorch implementation for HLA-Face: Joint High-Low Adaptation for Low L

Wenjing Wang 77 Dec 8, 2022
A pytorch implementation of the CVPR2021 paper "VSPW: A Large-scale Dataset for Video Scene Parsing in the Wild"

VSPW: A Large-scale Dataset for Video Scene Parsing in the Wild A pytorch implementation of the CVPR2021 paper "VSPW: A Large-scale Dataset for Video

null 45 Nov 29, 2022
Pytorch implementation of ICASSP 2022 paper Attention Probe: Vision Transformer Distillation in the Wild

Attention Probe: Vision Transformer Distillation in the Wild Jiahao Wang, Mingdeng Cao, Shuwei Shi, Baoyuan Wu, Yujiu Yang In ICASSP 2022 This code is

IIGROUP 6 Sep 21, 2022
A machine learning benchmark of in-the-wild distribution shifts, with data loaders, evaluators, and default models.

WILDS is a benchmark of in-the-wild distribution shifts spanning diverse data modalities and applications, from tumor identification to wildlife monitoring to poverty mapping.

P-Lambda 437 Dec 30, 2022
[CVPR'21] Learning to Recommend Frame for Interactive Video Object Segmentation in the Wild

IVOS-W Paper Learning to Recommend Frame for Interactive Video Object Segmentation in the Wild Zhaoyun Yin, Jia Zheng, Weixin Luo, Shenhan Qian, Hanli

SVIP Lab 38 Dec 12, 2022
Learning High-Speed Flight in the Wild

Learning High-Speed Flight in the Wild This repo contains the code associated to the paper Learning Agile Flight in the Wild. For more information, pl

Robotics and Perception Group 391 Dec 29, 2022
Official Pytorch implementation of "Learning to Estimate Robust 3D Human Mesh from In-the-Wild Crowded Scenes", CVPR 2022

Learning to Estimate Robust 3D Human Mesh from In-the-Wild Crowded Scenes / 3DCrowdNet News ?? 3DCrowdNet achieves the state-of-the-art accuracy on 3D

Hongsuk Choi 113 Dec 21, 2022
Face Library is an open source package for accurate and real-time face detection and recognition

Face Library Face Library is an open source package for accurate and real-time face detection and recognition. The package is built over OpenCV and us

null 52 Nov 9, 2022
Face and Pose detector that emits MQTT events when a face or human body is detected and not detected.

Face Detect MQTT Face or Pose detector that emits MQTT events when a face or human body is detected and not detected. I built this as an alternative t

Jacob Morris 38 Oct 21, 2022
Official Implementation and Dataset of "PPR10K: A Large-Scale Portrait Photo Retouching Dataset with Human-Region Mask and Group-Level Consistency", CVPR 2021

Portrait Photo Retouching with PPR10K Paper | Supplementary Material PPR10K: A Large-Scale Portrait Photo Retouching Dataset with Human-Region Mask an

null 184 Dec 11, 2022
img2pose: Face Alignment and Detection via 6DoF, Face Pose Estimation

img2pose: Face Alignment and Detection via 6DoF, Face Pose Estimation Figure 1: We estimate the 6DoF rigid transformation of a 3D face (rendered in si

Vítor Albiero 519 Dec 29, 2022
[TIP 2021] SADRNet: Self-Aligned Dual Face Regression Networks for Robust 3D Dense Face Alignment and Reconstruction

SADRNet Paper link: SADRNet: Self-Aligned Dual Face Regression Networks for Robust 3D Dense Face Alignment and Reconstruction Requirements python

Multimedia Computing Group, Nanjing University 99 Dec 30, 2022
A large-scale face dataset for face parsing, recognition, generation and editing.

CelebAMask-HQ [Paper] [Demo] CelebAMask-HQ is a large-scale face image dataset that has 30,000 high-resolution face images selected from the CelebA da

switchnorm 1.7k Dec 26, 2022
An algorithm that handles large-scale aerial photo co-registration, based on SURF, RANSAC and PyTorch autograd.

An algorithm that handles large-scale aerial photo co-registration, based on SURF, RANSAC and PyTorch autograd.

Luna Yue Huang 41 Oct 29, 2022