Website | ArXiv | Get Start | Video
PIRenderer
The source code of the ICCV2021 paper "PIRenderer: Controllable Portrait Image Generation via Semantic Neural Rendering" (ICCV2021)
The proposed PIRenderer can synthesis portrait images by intuitively controlling the face motions with fully disentangled 3DMM parameters. This model can be applied to tasks such as:
-
Intuitive Portrait Image Editing
Intuitive Portrait Image Control
Pose & Expression Alignment
-
Motion Imitation
Same & Corss-identity Reenactment
-
Audio-Driven Facial Reenactment
Audio-Driven Reenactment
News
- 2021.9.20 Code for PyTorch is available!
Colab Demo
Coming soon
Get Start
1). Installation
Requirements
- Python 3
- PyTorch 1.7.1
- CUDA 10.2
Conda Installation
# 1. Create a conda virtual environment.
conda create -n PIRenderer python=3.6
conda activate PIRenderer
conda install -c pytorch pytorch=1.7.1 torchvision cudatoolkit=10.2
# 2. Install other dependencies
pip install -r requirements.txt
2). Dataset
We train our model using the VoxCeleb. You can download the demo dataset for inference or prepare the dataset for training and testing.
Download the demo dataset
The demo dataset contains all 514 test videos. You can download the dataset with the following code:
./scripts/download_demo_dataset.sh
Or you can choose to download the resources with these links:
Google Driven & BaiDu Driven with extraction passwords ”p9ab“
Then unzip and save the files to ./dataset
Prepare the dataset
-
The dataset is preprocessed follow the method used in First-Order. You can follow the instructions in their repo to download and crop videos for training and testing.
-
After obtaining the VoxCeleb videos, we extract 3DMM parameters using Deep3DFaceReconstruction.
The folder are with format as:
${DATASET_ROOT_FOLDER} └───path_to_videos └───train └───xxx.mp4 └───xxx.mp4 ... └───test └───xxx.mp4 └───xxx.mp4 ... └───path_to_3dmm_coeff └───train └───xxx.mat └───xxx.mat ... └───test └───xxx.mat └───xxx.mat ... -
We save the video and 3DMM parameters in a lmdb file. Please run the following code to do this
python scripts/prepare_vox_lmdb.py \ --path path_to_videos \ --coeff_3dmm_path path_to_3dmm_coeff \ --out path_to_output_dir
3). Training and Inference
Inference
The trained weights can be downloaded by running the following code:
./scripts/download_weights.sh
Or you can choose to download the resources with these links: coming soon. Then save the files to ./result/face
Reenactment
Run the the demo for face reenactment:
python -m torch.distributed.launch --nproc_per_node=1 --master_port 12345 inference.py \
--config ./config/face.yaml \
--name face \
--no_resume \
--output_dir ./vox_result/face_reenactment
The output results are saved at ./vox_result/face_reenactment
Intuitive Control
coming soon
Train
Our model can be trained with the following code
python -m torch.distributed.launch --nproc_per_node=4 --master_port 12345 train.py \
--config ./config/face.yaml \
--name face
Citation
If you find this code is helpful, please cite our paper
@misc{ren2021pirenderer,
title={PIRenderer: Controllable Portrait Image Generation via Semantic Neural Rendering},
author={Yurui Ren and Ge Li and Yuanqi Chen and Thomas H. Li and Shan Liu},
year={2021},
eprint={2109.08379},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
Acknowledgement
We build our project base on imaginaire. Some dataset preprocessing methods are derived from video-preprocessing.
89 Dec 21, 2022
69 Oct 13, 2022
163 Dec 22, 2022
221 Jan 7, 2023
109 Dec 29, 2022
187 Dec 26, 2022
82 Dec 27, 2022
202 Dec 30, 2022
18 Sep 2, 2022
112 Nov 28, 2022
36 Nov 4, 2022
107 Dec 20, 2022
52 Nov 25, 2022
83 Nov 27, 2022
103 Dec 14, 2022
159 Dec 30, 2022
148 Dec 16, 2022