Multi-View Consistent Generative Adversarial Networks for 3D-aware Image Synthesis
Multi-View Consistent Generative Adversarial Networks for 3D-aware Image Synthesis.
Xuanmeng Zhang, Zhedong Zheng, Daiheng Gao, Bang Zhang, Pan Pan, Yi Yang
CVPR 2022.
News:
- [2022-04-30] We release the paper, video, code, and checkpoints.
Abstract
3D-aware image synthesis aims to generate images of objects from multiple views by learning a 3D representation. However, one key challenge remains: existing approaches lack geometry constraints, hence usually fail to generate multi-view consistent images. To address this challenge, we propose Multi-View Consistent Generative Adversarial Networks (MVCGAN) for high-quality 3D-aware image synthesis with geometry constraints. By leveraging the underlying 3D geometry information of generated images, i.e., depth and camera transformation matrix, we explicitly establish stereo correspondence between views to perform multi-view joint optimization. In particular, we enforce the photometric consistency between pairs of views and integrate a stereo mixup mechanism into the training process, encouraging the model to reason about the correct 3D shape. Besides, we design a two-stage training strategy with feature-level multi-view joint optimization to improve the image quality. Extensive experiments on three datasets demonstrate that MVCGAN achieves the state-of-the-art performance for 3D-aware image synthesis.
Please refer to the supplementary video for more visualization results.
Getting Started
Installation
Install dependencies by:
pip install -r requirements.txt
Datasets
Pretrained Checkpoints
| Dataset | Resolution | Download |
|---|---|---|
| CelebAHQ | 512 | Google Drive |
| FFHQ | 512 | Google Drive |
| AFHQ | 512 | Google Drive |
Training
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python main.py --output_dir celebahq_exp --port 12361 --curriculum CelebAHQ
Please modify the configuration file curriculums.py to adjust to your own dataset path.
Rendering
CUDA_VISIBLE_DEVICES=0 python render_multiview_image.py --path ${CHECKPOINT_PATH} --output_dir render_dir --output_size 512 --curriculum FFHQ
Acknowledgment
Our implementation of MVCGAN is partly based on the following codebases. We gratefully thank the authors for their wonderful works: pi-gan, pytorch_GAN_zoo.
Citation
If you find our code or paper useful, please consider citing:
@inproceedings{zhang2022multiview,
title={Multi-View Consistent Generative Adversarial Networks for 3D-aware Image Synthesis},
author={Zhang, Xuanmeng and Zheng, Zhedong and Gao, Daiheng and Zhang, Bang and Pan, Pan and Yang, Yi},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year={2022}
}
7 Apr 4, 2022
5 Oct 12, 2022
21 Nov 23, 2022
129 Dec 11, 2022
26 Dec 28, 2022
2 Jan 10, 2022
3k Jan 8, 2023
![[ICLR 2021, Spotlight] Large Scale Image Completion via Co-Modulated Generative Adversarial Networks](https://github.com/zsyzzsoft/co-mod-gan/raw/master/imgs/demo.gif)
373 Jan 2, 2023
2.2k Jan 1, 2023
375 Dec 31, 2022
138 Dec 28, 2022
254 Dec 27, 2022
93 Dec 17, 2022
809 Dec 16, 2022
349 Dec 29, 2022
147 Dec 31, 2022
11 Feb 8, 2022
239 Dec 26, 2022