(CVPR 2021) Lifting 2D StyleGAN for 3D-Aware Face Generation

Related tags

Deep Learning LiftedGAN
Overview

Lifting 2D StyleGAN for 3D-Aware Face Generation

Official implementation of paper "Lifting 2D StyleGAN for 3D-Aware Face Generation".

Requirements

You can create the conda environment by using conda env create -f environment.yml

Training

Training from pre-trained StyleGAN2

Download our pre-trained StyleGAN and face embedding network from here for training. Unzip them into the pretrained/ folder. Then you can start training by:

python tools/train.py config/ffhq_256.py

Note that you do not need an image dataset here becuase we simply lift the StyleGAN2 using images generated by itself.

Training from custom data

We use a re-cropped version of FFHQ to fit the style of our face embedding network. You can find this dataset here. The cats dataset can be found here. To train a StyleGAN2 from you own dataset, check the content under stylegan2-pytorch folder. After training a StyleGAN2, you can lift it using our training code. Note that our method might not apply to other kind of images, if they are very different from human faces.

Testing

Pre-trained Models:

Google Drive

Sampling random faces

You can generate random samples from a lifted gan by running:

python tools/generate_images.py /path/to/the/checkpoint --output_dir results/

Make sure the checkpoint file and its config.py file are under the same folder.

Generating controlled faces

You can generate GIF animations of rotated faces by running:

python tools/generate_poses.py /path/to/the/checkpoint --output_dir results/ --type yaw

Similarly, you can generate faces with different light directions:

python tools/generate_lighting.py /path/to/the/checkpoint --output_dir results/

Testing FID

We use the code from rosinality's stylegan2-pytorch to compute FID. To compute the FID, you first need to compute the statistics of real images:

python utils/calc_inception.py /path/to/the/dataset/lmdb

You might skip this step if you are using our pre-calculated statistics file (link). Then, to test the FID, you can run:

python tools/test_fid.py /path/to/the/checkpoint --inception /path/to/the/inception/file

Acknowledgment

Part of this code is based on Wu's Unsup3D and Rosinality's StyleGAN2-Pytorch.

Comments
  • 'fixed Renderer not setting device on look_at_rotation'

    'fixed Renderer not setting device on look_at_rotation'

    This can cause a 'runtimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!' when running on cuda because look_at_rotation defaults to cpu

    opened by nekomi2 1
  • How to get good results?

    How to get good results?

    Hi, I downloaded the pretrained zip from the README, and ran

     python generate_poses.py pretrained/stylegan2-ffhq-cropped-300000.pt  --output_dir results/ --type yaw

    image

    which gave a result like this. what should I do to improve the results?

    opened by nekomi2 0
  • FFHQ re-cropped code

    FFHQ re-cropped code

    You use a re-cropped version of FFHQ to fit the style of our face embedding network. You have already provide the dataset, could you please release the preprocess code for re-cropped too?

    opened by gongmm 0
  • how to get obj file？

    how to get obj file？

    thank you for your sharing. I am a beginner and I have some questions. I followed the readme file and just got pictures and dynamic images. I want to get the model obj file what should I do

    Another problem is that when I test fit, the result I get is 76.922. Seems to be much worse? Or is this supposed to subtract it from 100, so the real fit is 24?

    Thank you, looking forward to your reply

    opened by Yangleyang 1
  • Inference using own input

    Inference using own input

    I've gone through the implementation, and started to wonder how to put my own photo input as a input of stylegan2 here.

    As far as I know, I first need to convert a photo to a latent vector, and the plug the latent vector to the model. Is there a specific implementation that allow me to do such conversion?

    or are there other ways to use photos that I've got as input to the model?

    thanks in advance :)

    opened by Youngwoo-git 1
  • RuntimeError: svd_cuda: the updating process of SBDSDC did not converge (error: 7)

    RuntimeError: svd_cuda: the updating process of SBDSDC did not converge (error: 7)

    It is a great work; however, I encounter an issue while performing model training. Sometime, the program shows the following error:

    Traceback (most recent call last): File "train.py", line 88, in main(args) File "train.py", line 65, in main watchlist, summary, global_step = network.train_step() File "/home/li-yun/Desktop/3d-gen/LiftedGAN/models/lifted_gan.py", line 415, in train_step loss_albedo_rank = torch.norm(albedo_residual_flat, p='nuc') # error....... File "/home/li-yun/anaconda3/envs/lifted_gan/lib/python3.6/site-packages/torch/functional.py", line 924, in norm return _VF.nuclear_norm(input, keepdim=keepdim) RuntimeError: svd_cuda: the updating process of SBDSDC did not converge (error: 7)

    How can I fix this issue? Thanks.

    opened by LiYunJamesPhD 0
Owner
Yichun Shi
Yichun Shi
GitHub
[SIGGRAPH'22] StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets

[Project] [PDF] This repository contains code for our SIGGRAPH'22 paper "StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets" by Axel Sauer, Katja

null 742 Jan 4, 2023
Official implementation of "StyleCariGAN: Caricature Generation via StyleGAN Feature Map Modulation" (SIGGRAPH 2021)

StyleCariGAN in PyTorch Official implementation of StyleCariGAN:Caricature Generation via StyleGAN Feature Map Modulation in PyTorch Requirements PyTo

PeterZhouSZ 49 Oct 31, 2022
Official implementation of "StyleCariGAN: Caricature Generation via StyleGAN Feature Map Modulation" (SIGGRAPH 2021)

StyleCariGAN: Caricature Generation via StyleGAN Feature Map Modulation This repository contains the official PyTorch implementation of the following

Wonjong Jang 270 Dec 30, 2022
DVG-Face: Dual Variational Generation for Heterogeneous Face Recognition, TPAMI 2021

DVG-Face: Dual Variational Generation for HFR This repo is a PyTorch implementation of DVG-Face: Dual Variational Generation for Heterogeneous Face Re

null 52 Dec 30, 2022
Non-Official Pytorch implementation of "Face Identity Disentanglement via Latent Space Mapping" https://arxiv.org/abs/2005.07728 Using StyleGAN2 instead of StyleGAN

Face Identity Disentanglement via Latent Space Mapping - Implement in pytorch with StyleGAN 2 Description Pytorch implementation of the paper Face Ide

Daniel Roich 58 Dec 24, 2022
Code for Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation (CVPR 2021)

Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation (CVPR 2021) Hang Zhou, Yasheng Sun, Wayne Wu, Chen Cha

Hang_Zhou 628 Dec 28, 2022
StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation

StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation Demo video: CVPR 2021 Oral: Single Channel Manipulation: Localized or attribu

Zongze Wu 267 Dec 30, 2022
Implementation of StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation in PyTorch

StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation Implementation of StyleSpace Analysis: Disentangled Controls for StyleGAN Ima

Xuanchi Ren 86 Dec 7, 2022
StyleGAN-Human: A Data-Centric Odyssey of Human Generation

StyleGAN-Human: A Data-Centric Odyssey of Human Generation Abstract: Unconditional human image generation is an important task in vision and graphics,

stylegan-human 762 Jan 8, 2023
A large-scale face dataset for face parsing, recognition, generation and editing.

CelebAMask-HQ [Paper] [Demo] CelebAMask-HQ is a large-scale face image dataset that has 30,000 high-resolution face images selected from the CelebA da

switchnorm 1.7k Dec 26, 2022
Disentangled Face Attribute Editing via Instance-Aware Latent Space Search, accepted by IJCAI 2021.

Instance-Aware Latent-Space Search This is a PyTorch implementation of the following paper: Disentangled Face Attribute Editing via Instance-Aware Lat

null 67 Dec 21, 2022
[TIP 2021] SADRNet: Self-Aligned Dual Face Regression Networks for Robust 3D Dense Face Alignment and Reconstruction

SADRNet Paper link: SADRNet: Self-Aligned Dual Face Regression Networks for Robust 3D Dense Face Alignment and Reconstruction Requirements python

Multimedia Computing Group, Nanjing University 99 Dec 30, 2022
git git《Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking》(CVPR 2021) GitHub:git2] 《Masksembles for Uncertainty Estimation》(CVPR 2021) GitHub:git3]

Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking Ning Wang, Wengang Zhou, Jie Wang, and Houqiang Li Accepted by CVPR

NingWang 236 Dec 22, 2022
[CVPR 2021] Modular Interactive Video Object Segmentation: Interaction-to-Mask, Propagation and Difference-Aware Fusion

[CVPR 2021] Modular Interactive Video Object Segmentation: Interaction-to-Mask, Propagation and Difference-Aware Fusion

Rex Cheng 364 Jan 3, 2023
Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers [CVPR 2021]

Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers [BCNet, CVPR 2021] This is the official pytorch implementation of BCNet built on

Lei Ke 434 Dec 1, 2022
Semi-supervised Semantic Segmentation with Directional Context-aware Consistency (CVPR 2021)

Semi-supervised Semantic Segmentation with Directional Context-aware Consistency (CAC) Xin Lai*, Zhuotao Tian*, Li Jiang, Shu Liu, Hengshuang Zhao, Li

Jia Research Lab 137 Dec 14, 2022
Implementation for the paper SMPLicit: Topology-aware Generative Model for Clothed People (CVPR 2021)

SMPLicit: Topology-aware Generative Model for Clothed People [Project] [arXiv] License Software Copyright License for non-commercial scientific resear

Enric Corona 225 Dec 13, 2022
Semi-supervised Semantic Segmentation with Directional Context-aware Consistency (CVPR 2021)

Semi-supervised Semantic Segmentation with Directional Context-aware Consistency (CAC) Xin Lai*, Zhuotao Tian*, Li Jiang, Shu Liu, Hengshuang Zhao, Li

DV Lab 137 Dec 14, 2022
Scale-aware Automatic Augmentation for Object Detection (CVPR 2021)

SA-AutoAug Scale-aware Automatic Augmentation for Object Detection Yukang Chen, Yanwei Li, Tao Kong, Lu Qi, Ruihang Chu, Lei Li, Jiaya Jia [Paper] [Bi

DV Lab 182 Dec 29, 2022