HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields
This is the code for "HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields".
This codebase implements HyperNeRF using JAX, building on JaxNeRF.
Demo
We provide an easy-to-get-started demo using Google Colab!
These Colabs will allow you to train a basic version of our method using Cloud TPUs (or GPUs) on Google Colab.
Note that due to limited compute resources available, these are not the fully featured models and will train quite slowly and the quality will likely not be that great. If you would like to train a fully featured model, please refer to the instructions below on how to train on your own machine.
| Description | Link |
|---|---|
| Process a video into a dataset |  |
| Train HyperNeRF | |
| Render HyperNeRF Videos | |
Setup
The code can be run under any environment with Python 3.8 and above. (It may run with lower versions, but we have not tested it).
We recommend using Miniconda and setting up an environment:
conda create --name hypernerf python=3.8
Next, install the required packages:
pip install -r requirements.txt
Install the appropriate JAX distribution for your environment by following the instructions here. For example:
# For CUDA version 11.1
pip install --upgrade "jax[cuda111]" -f https://storage.googleapis.com/jax-releases/jax_releases.html
Training
After preparing a dataset, you can train a Nerfie by running:
export DATASET_PATH=/path/to/dataset
export EXPERIMENT_PATH=/path/to/save/experiment/to
python train.py \
--base_folder $EXPERIMENT_PATH \
--gin_bindings="data_dir='$DATASET_PATH'" \
--gin_configs configs/test_local.gin
To plot telemetry to Tensorboard and render checkpoints on the fly, also launch an evaluation job by running:
python eval.py \
--base_folder $EXPERIMENT_PATH \
--gin_bindings="data_dir='$DATASET_PATH'" \
--gin_configs configs/test_local.gin
The two jobs should use a mutually exclusive set of GPUs. This division allows the training job to run without having to stop for evaluation.
Configuration
- We use Gin for configuration.
- We provide a couple preset configurations.
- Please refer to
config.pyfor documentation on what each configuration does. - Preset configs:
hypernerf_vrig_ds_2d.gin: The deformable surface configuration for the validation rig (novel-view synthesis) experiments.hypernerf_vrig_ap_2d.gin: The axis-aligned plane configuration for the validation rig (novel-view synthesis) experiments.hypernerf_interp_ds_2d.gin: The deformable surface configuration for the interpolation experiments.hypernerf_interp_ap_2d.gin: The axis-aligned plane configuration for the interpolation experiments.
Dataset
The dataset uses the same format as Nerfies.
Citing
If you find our work useful, please consider citing:
@article{park2021hypernerf
author = {Park, Keunhong and Sinha, Utkarsh and Hedman, Peter and Barron, Jonathan T. and Bouaziz, Sofien and Goldman, Dan B and Martin-Brualla, Ricardo and Seitz, Steven M.},
title = {HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields},
journal = {arXiv preprint arXiv:2106.13228},
year = {2021},
}
Shouldn't it be like the quadratic shape when |x| < scale
73 Nov 6, 2022
18 Oct 6, 2022
94 Jan 6, 2023
72 Jan 6, 2023
3 Jul 14, 2021
56 Nov 15, 2022
408 Jan 1, 2023
1 Oct 26, 2021
7 Apr 5, 2022
4 May 26, 2022
32 Nov 9, 2021
16 Oct 11, 2022
3k Jan 5, 2023
1k Jan 2, 2023
397 Dec 30, 2022
13.9k Dec 26, 2022
121 Dec 30, 2022
138 Dec 12, 2022