DirectVoxGO
DirectVoxGO (Direct Voxel Grid Optimization, see our paper) reconstructs a scene representation from a set of calibrated images capturing the scene.
- NeRF-comparable quality for synthesizing novel views from our scene representation.
- Super-fast convergence: Our
15 mins/scenevs. NeRF's10~20+ hrs/scene. - No cross-scene pre-training required: We optimize each scene from scratch.
- Better rendering speed: Our
<1 secsvs. NeRF's29 secsto synthesize a800x800images.
Below run-times (mm:ss) of our optimization progress are measured on a machine with a single RTX 2080 Ti GPU.
github_teaser.mp4
Update
- 2021.11.23: Support CO3D dataset.
- 2021.11.23: Initial release. Issue page is disabled for now. Feel free to contact
[email protected]if you have any questions.
Installation
git clone [email protected]:sunset1995/DirectVoxGO.git
cd DirectVoxGO
pip install -r requirements.txt
Pytorch installation is machine dependent, please install the correct version for your machine. The tested version is pytorch 1.8.1 with python 3.7.4.
Dependencies (click to expand)
PyTorch,numpy: main computation.scipy,lpips: SSIM and LPIPS evaluation.tqdm: progress bar.mmcv: config system.opencv-python: image processing.imageio,imageio-ffmpeg: images and videos I/O.
Download: datasets, trained models, and rendered test views
Directory structure for the datasets (click to expand; only list used files)
data
├── nerf_synthetic # Link: https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1
│ └── [chair|drums|ficus|hotdog|lego|materials|mic|ship]
│ ├── [train|val|test]
│ │ └── r_*.png
│ └── transforms_[train|val|test].json
│
├── Synthetic_NSVF # Link: https://dl.fbaipublicfiles.com/nsvf/dataset/Synthetic_NSVF.zip
│ └── [Bike|Lifestyle|Palace|Robot|Spaceship|Steamtrain|Toad|Wineholder]
│ ├── intrinsics.txt
│ ├── rgb
│ │ └── [0_train|1_val|2_test]_*.png
│ └── pose
│ └── [0_train|1_val|2_test]_*.txt
│
├── BlendedMVS # Link: https://dl.fbaipublicfiles.com/nsvf/dataset/BlendedMVS.zip
│ └── [Character|Fountain|Jade|Statues]
│ ├── intrinsics.txt
│ ├── rgb
│ │ └── [0|1|2]_*.png
│ └── pose
│ └── [0|1|2]_*.txt
│
├── TanksAndTemple # Link: https://dl.fbaipublicfiles.com/nsvf/dataset/TanksAndTemple.zip
│ └── [Barn|Caterpillar|Family|Ignatius|Truck]
│ ├── intrinsics.txt
│ ├── rgb
│ │ └── [0|1|2]_*.png
│ └── pose
│ └── [0|1|2]_*.txt
│
├── deepvoxels # Link: https://drive.google.com/drive/folders/1ScsRlnzy9Bd_n-xw83SP-0t548v63mPH
│ └── [train|validation|test]
│ └── [armchair|cube|greek|vase]
│ ├── intrinsics.txt
│ ├── rgb/*.png
│ └── pose/*.txt
│
└── co3d # Link: https://github.com/facebookresearch/co3d
└── [donut|teddybear|umbrella|...]
├── frame_annotations.jgz
├── set_lists.json
└── [129_14950_29917|189_20376_35616|...]
├── images
│ └── frame*.jpg
└── masks
└── frame*.png
Synthetic-NeRF, Synthetic-NSVF, BlendedMVS, Tanks&Temples, DeepVoxels datasets
We use the datasets organized by NeRF, NSVF, and DeepVoxels. Download links:
- Synthetic-NeRF dataset (manually extract the
nerf_synthetic.ziptodata/) - Synthetic-NSVF dataset (manually extract the
Synthetic_NSVF.ziptodata/) - BlendedMVS dataset (manually extract the
BlendedMVS.ziptodata/) - Tanks&Temples dataset (manually extract the
TanksAndTemple.ziptodata/) - DeepVoxels dataset (manually extract the
synthetic_scenes.ziptodata/deepvoxels/)
Download all our trained models and rendered test views at this link to our logs.
CO3D dataset
We also support the recent Common Objects In 3D dataset. Our method only performs per-scene reconstruction and no cross-scene generalization.
GO
Train
To train lego scene and evaluate testset PSNR at the end of training, run:
$ python run.py --config configs/nerf/lego.py --render_test
Use --i_print and --i_weights to change the log interval.
Evaluation
To only evaluate the testset PSNR, SSIM, and LPIPS of the trained lego without re-training, run:
$ python run.py --config configs/nerf/lego.py --render_only --render_test \
--eval_ssim --eval_lpips_vgg
Use --eval_lpips_alex to evaluate LPIPS with pre-trained Alex net instead of VGG net.
Reproduction
All config files to reproduce our results:
$ ls configs/*
configs/blendedmvs:
Character.py Fountain.py Jade.py Statues.py
configs/nerf:
chair.py drums.py ficus.py hotdog.py lego.py materials.py mic.py ship.py
configs/nsvf:
Bike.py Lifestyle.py Palace.py Robot.py Spaceship.py Steamtrain.py Toad.py Wineholder.py
configs/tankstemple:
Barn.py Caterpillar.py Family.py Ignatius.py Truck.py
configs/deepvoxels:
armchair.py cube.py greek.py vase.py
Your own config files
Check the comments in configs/default.py for the configuable settings. The default values reproduce our main setup reported in our paper. We use mmcv's config system. To create a new config, please inherit configs/default.py first and then update the fields you want. Below is an example from configs/blendedmvs/Character.py:
_base_ = '../default.py'
expname = 'dvgo_Character'
basedir = './logs/blended_mvs'
data = dict(
datadir='./data/BlendedMVS/Character/',
dataset_type='blendedmvs',
inverse_y=True,
white_bkgd=True,
)
Development and tuning guide
Extention to new dataset
Adjusting the data related config fields to fit your camera coordinate system is recommend before implementing a new one. We provide two visualization tools for debugging.
- Inspect the camera and the allocated BBox.
- Export via
--export_bbox_and_cams_only {filename}.npz:python run.py --config configs/nerf/mic.py --export_bbox_and_cams_only cam_mic.npz
- Visualize the result:
python tools/vis_train.py cam_mic.npz
- Export via
- Inspect the learned geometry after coarse optimization.
- Export via
--export_coarse_only {filename}.npz(assumedcoarse_last.taravailable in the train log):python run.py --config configs/nerf/mic.py --export_coarse_only coarse_mic.npz
- Visualize the result:
python tools/vis_volume.py coarse_mic.npz 0.001 --cam cam_mic.npz
- Export via
| Inspecting the cameras & BBox | Inspecting the learned coarse volume |
|---|---|
 |
 |
Speed and quality tradeoff
We have reported some ablation experiments in our paper supplementary material. Setting N_iters, N_rand, num_voxels, rgbnet_depth, rgbnet_width to larger values or setting stepsize to smaller values typically leads to better quality but need more computation. Only stepsize is tunable in testing phase, while all the other fields should remain the same as training.
Acknowledgement
The code base is origined from an awesome nerf-pytorch implementation, but it becomes very different from the code base now.
![CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.](https://avatars.githubusercontent.com/u/72448878?v=4)
Experiments on my own dataset found that the reconstructed objects are also flat.
51 Sep 22, 2022
6 Jun 27, 2022
141 Dec 30, 2022
3 Nov 30, 2021
26 Dec 12, 2022
168 Nov 29, 2022
16 Dec 15, 2022
55 Dec 3, 2022
35 Nov 16, 2022
194 Jan 3, 2023
41 Jan 6, 2023
149 Jan 8, 2023
76 Jan 2, 2023
313 Dec 27, 2022
114 Oct 16, 2022
243 Jan 7, 2023
182 Dec 30, 2022