GRAF
This repository contains official code for the paper GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis.
You can find detailed usage instructions for training your own models and using pre-trained models below.
If you find our code or paper useful, please consider citing
@inproceedings{Schwarz2020NEURIPS,
title = {GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis},
author = {Schwarz, Katja and Liao, Yiyi and Niemeyer, Michael and Geiger, Andreas},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
year = {2020}
}
Installation
First you have to make sure that you have all dependencies in place. The simplest way to do so, is to use anaconda.
You can create an anaconda environment called graf using
conda env create -f environment.yml
conda activate graf
Next, for nerf-pytorch install torchsearchsorted. Note that this requires torch>=1.4.0 and CUDA >= v10.1. You can install torchsearchsorted via
cd submodules/nerf_pytorch
pip install -r requirements.txt
cd torchsearchsorted
pip install .
cd ../../../
Demo
You can now test our code via:
python eval.py configs/carla.yaml --pretrained --rotation_elevation
This script should create a folder results/carla_128_from_pretrained/eval/ where you can find generated videos varying camera pose for the Cars dataset.
Datasets
If you only want to generate images using our pretrained models you do not need to download the datasets. The datasets are only needed if you want to train a model from scratch.
Cars
To download the Cars dataset from the paper simply run
cd data
./download_carla.sh
cd ..
This creates a folder data/carla/ downloads the images as a zip file and extracts them to data/carla/. While we do not use camera poses in this project we provide them for completeness. Your can download them by running
cd data
./download_carla_poses.sh
cd ..
This downloads the camera intrinsics (single file, equal for all images) and extrinsics corresponding to each image.
Faces
Download celebA. Then replace data/celebA in configs/celebA.yaml with *PATH/TO/CELEBA*/Img/img_align_celebA.
Download celebA_hq. Then replace data/celebA_hq in configs/celebAHQ.yaml with *PATH/TO/CELEBA_HQ*.
Cats
Download the CatDataset. Run
cd data
python preprocess_cats.py PATH/TO/CATS/DATASET
cd ..
to preprocess the data and save it to data/cats. If successful this script should print: Preprocessed 9407 images.
Birds
Download CUB-200-2011 and the corresponding Segmentation Masks. Run
cd data
python preprocess_cub.py PATH/TO/CUB-200-2011 PATH/TO/SEGMENTATION/MASKS
cd ..
to preprocess the data and save it to data/cub. If successful this script should print: Preprocessed 8444 images.
Usage
When you have installed all dependencies, you are ready to run our pre-trained models for 3D-aware image synthesis.
Generate images using a pretrained model
To evaluate a pretrained model, run
python eval.py CONFIG.yaml --pretrained --fid_kid --rotation_elevation --shape_appearance
where you replace CONFIG.yaml with one of the config files in ./configs.
This script should create a folder results/EXPNAME/eval with FID and KID scores in fid_kid.csv, videos for rotation and elevation in the respective folders and an interpolation for shape and appearance, shape_appearance.png.
Note that some pretrained models are available for different image sizes which you can choose by setting data:imsize in the config file to one of the following values:
configs/carla.yaml:
data:imsize 64 or 128 or 256 or 512
configs/celebA.yaml:
data:imsize 64 or 128
configs/celebAHQ.yaml:
data:imsize 256 or 512
Train a model from scratch
To train a 3D-aware generative model from scratch run
python train.py CONFIG.yaml
where you replace CONFIG.yaml with your config file. The easiest way is to use one of the existing config files in the ./configs directory which correspond to the experiments presented in the paper. Note that this will train the model from scratch and will not resume training for a pretrained model.
You can monitor on http://localhost:6006 the training process using tensorboard:
cd OUTPUT_DIR
tensorboard --logdir ./monitoring --port 6006
where you replace OUTPUT_DIR with the respective output directory.
For available training options, please take a look at configs/default.yaml.
Evaluation of a new model
For evaluation of the models run
python eval.py CONFIG.yaml --fid_kid --rotation_elevation --shape_appearance
where you replace CONFIG.yaml with your config file.
Multi-View Consistency Check
You can evaluate the multi-view consistency of the generated images by running a Multi-View-Stereo (MVS) algorithm on the generated images. This evaluation uses COLMAP and make sure that you have COLMAP installed to run
python eval.py CONFIG.yaml --reconstruction
where you replace CONFIG.yaml with your config file. You can also evaluate our pretrained models via:
python eval.py configs/carla.yaml --pretrained --reconstruction
This script should create a folder results/EXPNAME/eval/reconstruction/ where you can find generated multi-view images in images/ and the corresponding 3D reconstructions in models/.
Further Information
GAN training
This repository uses Lars Mescheder's awesome framework for GAN training.
NeRF
We base our code for the Generator on this great Pytorch reimplementation of Neural Radiance Fields.
89 Dec 10, 2022
235 Dec 26, 2022
16 Oct 5, 2022
565 Jan 5, 2023
536 Dec 20, 2022
56 Dec 28, 2022
66 Dec 16, 2022
360 Jan 6, 2023
12 Nov 30, 2022
93 Nov 8, 2022
18 May 27, 2022
95 Jan 8, 2023
53 Dec 24, 2022
524 Jan 8, 2023
41 May 18, 2022
23 Dec 7, 2022
101 Dec 11, 2022
68 Dec 14, 2022
230 Dec 31, 2022