D-NeRF: Neural Radiance Fields for Dynamic Scenes

Related tags

Deep Learning D-NeRF
Overview

D-NeRF: Neural Radiance Fields for Dynamic Scenes

[Project] [Paper]

D-NeRF is a method for synthesizing novel views, at an arbitrary point in time, of dynamic scenes with complex non-rigid geometries. We optimize an underlying deformable volumetric function from a sparse set of input monocular views without the need of ground-truth geometry nor multi-view images.

This project is an extension of NeRF enabling it to model dynmaic scenes. The code heavily relays on NeRF-pytorch.

D-NeRF

Installation

git clone https://github.com/albertpumarola/D-NeRF.git
cd D-NeRF
conda create -n dnerf python=3.6
conda activate dnerf
pip install -r requirements.txt
cd torchsearchsorted
pip install .
cd ..

Download Pre-trained Weights

You can download the pre-trained models from drive or dropbox. Unzip the downloaded data to the project root dir in order to test it later. See the following directory structure for an example:

├── logs 
│   ├── mutant
│   ├── standup 
│   ├── ...

Download Dataset

You can download the datasets from drive or dropbox. Unzip the downloaded data to the project root dir in order to train. See the following directory structure for an example:

├── data 
│   ├── mutant
│   ├── standup 
│   ├── ...

Demo

We provide simple jupyter notebooks to explore the model. To use them first download the pre-trained weights and dataset.

Description Jupyter Notebook
Synthesize novel views at an arbitrary point in time. render.ipynb
Reconstruct mesh at an arbitrary point in time. reconstruct.ipynb
Quantitatively evaluate trained model. metrics.ipynb

Test

First download pre-trained weights and dataset. Then,

python run_dnerf.py --config configs/mutant.txt --render_only --render_test

This command will run the mutant experiment. When finished, results are saved to ./logs/mutant/renderonly_test_799999 To quantitatively evaluate model run metrics.ipynb notebook

Train

First download the dataset. Then,

conda activate dnerf
export PYTHONPATH='path/to/D-NeRF'
export CUDA_VISIBLE_DEVICES=0
python run_dnerf.py --config configs/mutant.txt

Citation

If you use this code or ideas from the paper for your research, please cite our paper:

@article{pumarola2020d,
  title={D-NeRF: Neural Radiance Fields for Dynamic Scenes},
  author={Pumarola, Albert and Corona, Enric and Pons-Moll, Gerard and Moreno-Noguer, Francesc},
  journal={arXiv preprint arXiv:2011.13961},
  year={2020}
}
Comments
  • low efficiency

    low efficiency

    Hi, thanks for sharing this interesting work. I don't know whether it's the problem of the implementation, but the training can only reach around 15% util of an RTX-2080ti, and as a result extremely slow compared to the standard nerf-pytorch implementation, is this what we should expect?

    Thanks

    opened by ray8828 1
  • change to enable laptop investigation of disp net

    change to enable laptop investigation of disp net

    adds modified chunk sizes for 8gb laptop work (for balls/mutant configs) adds L2 dispnet regularization options adds dispnet visualization to render notebook

    opened by stanlew7531 1
  • Can you provide a sample of 'transforms_render.json'?

    Can you provide a sample of 'transforms_render.json'?

    Hi @albertpumarola, wonderful work! I'm new in this area. When I read the code, I found there is no 'transforms_render.json' file. Could you please provide a sample for reference?

    opened by syguan96 0
  • how to make your own dataset?

    how to make your own dataset?

    I am very interested in your repo D-NeRF and have some questions for you. Could D-NeRF train our own dataset? Could you share how you made your own dataset? How did you create your own dataset? Could you share the script to make the dataset? Looking forward to your reply and sharing! Thank you very much!

    opened by l-mengwei 0
  • about argument

    about argument "z_vals" in "batchify_rays" function

    In function "batchify_rays" from the code, you put 'z_vals' in "render_rays" with shape (N_rays, ~) while the chunked 'ray_batch' has the shape of (chunk, ~). Therefore, an error occurs with the issue of size mismatch at the line "pts = rays_o[...,None,:] + rays_d[...,None,:] * z_vals[...,:,None]" in 'render_rays'

    opened by 2j1ejyu 0
  • code of building the dataset

    code of building the dataset

    Thanks so much for your work. Is it possible to share the code of building the dataset from Adobe Mixamo? Like how to sample camera transformation matrix and get the transformation file.

    opened by rocketappslab 0
  • Reproduce the results in paper

    Reproduce the results in paper

    Dear author,

    I use the default configurations to train. But the PSNR results are better than those in paper.

    What adjustments to configuration should I make so that I can reproduce the results in CVPR?

    Thanks

    opened by derrick-xwp 0
Owner
Albert Pumarola
Computer Vision Researcher at Facebook Reality Labs
Albert Pumarola
GitHub
(Arxiv 2021) NeRF--: Neural Radiance Fields Without Known Camera Parameters

NeRF--: Neural Radiance Fields Without Known Camera Parameters Project Page | Arxiv | Colab Notebook | Data Zirui Wang¹, Shangzhe Wu², Weidi Xie², Min

Active Vision Laboratory 411 Dec 26, 2022
Unofficial & improved implementation of NeRF--: Neural Radiance Fields Without Known Camera Parameters

[Unofficial code-base] NeRF--: Neural Radiance Fields Without Known Camera Parameters [ Project | Paper | Official code base ] ⬅️ Thanks the original

Jianfei Guo 239 Dec 22, 2022
Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields.

This repository contains the code release for Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields. This implementation is written in JAX, and is a fork of Google's JaxNeRF implementation. Contact Jon Barron if you encounter any issues.

Google 625 Dec 30, 2022
This repository contains a PyTorch implementation of "AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis".

AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis | Project Page | Paper | PyTorch implementation for the paper "AD-NeRF: Audio

null 551 Dec 29, 2022
Code release for DS-NeRF (Depth-supervised Neural Radiance Fields)

Depth-supervised NeRF: Fewer Views and Faster Training for Free Project | Paper | YouTube Pytorch implementation of our method for learning neural rad

null 524 Jan 8, 2023
A PyTorch implementation of NeRF (Neural Radiance Fields) that reproduces the results.

NeRF-pytorch NeRF (Neural Radiance Fields) is a method that achieves state-of-the-art results for synthesizing novel views of complex scenes. Here are

Yen-Chen Lin 3.2k Jan 8, 2023
Code release for NeRF (Neural Radiance Fields)

NeRF: Neural Radiance Fields Project Page | Video | Paper | Data Tensorflow implementation of optimizing a neural representation for a single scene an

null 6.5k Jan 1, 2023
Pytorch implementation for A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose

A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose Paper | Website | Data A-NeRF: Articulated Neural Radiance F

Shih-Yang Su 172 Dec 22, 2022
Build upon neural radiance fields to create a scene-specific implicit 3D semantic representation, Semantic-NeRF

Semantic-NeRF: Semantic Neural Radiance Fields Project Page | Video | Paper | Data In-Place Scene Labelling and Understanding with Implicit Scene Repr

Shuaifeng Zhi 243 Jan 7, 2023
Point-NeRF: Point-based Neural Radiance Fields

Point-NeRF: Point-based Neural Radiance Fields Project Sites | Paper | Primary c

Qiangeng Xu 662 Jan 1, 2023
"SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Image", Dejia Xu, Yifan Jiang, Peihao Wang, Zhiwen Fan, Humphrey Shi, Zhangyang Wang

SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Image [Paper] [Website] Pipeline Code Environment pip install -r requirements

VITA 250 Jan 5, 2023
Stereo Radiance Fields (SRF): Learning View Synthesis for Sparse Views of Novel Scenes

Stereo Radiance Fields (SRF): Learning View Synthesis for Sparse Views of Novel Scenes

null 111 Dec 29, 2022
PyTorch implementation of paper "Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes", CVPR 2021

Neural Scene Flow Fields PyTorch implementation of paper "Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes", CVPR 20

Zhengqi Li 585 Jan 4, 2023
Instant-nerf-pytorch - NeRF trained SUPER FAST in pytorch

instant-nerf-pytorch This is WORK IN PROGRESS, please feel free to contribute vi

null 94 Nov 22, 2022
This repository contains the code for the CVPR 2021 paper "GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields"

GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields Project Page | Paper | Supplementary | Video | Slides | Blog | Talk If

null 1.1k Dec 30, 2022
This is the code for Deformable Neural Radiance Fields, a.k.a. Nerfies.

Deformable Neural Radiance Fields This is the code for Deformable Neural Radiance Fields, a.k.a. Nerfies. Project Page Paper Video This codebase conta

Google 1k Jan 9, 2023
Open source repository for the code accompanying the paper 'Non-Rigid Neural Radiance Fields Reconstruction and Novel View Synthesis of a Deforming Scene from Monocular Video'.

Non-Rigid Neural Radiance Fields This is the official repository for the project "Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synt

Facebook Research 296 Dec 29, 2022
Code for KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs

KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs Check out the paper on arXiv: https://arxiv.org/abs/2103.13744 This repo cont

Christian Reiser 373 Dec 20, 2022
PyTorch implementation for MINE: Continuous-Depth MPI with Neural Radiance Fields

MINE: Continuous-Depth MPI with Neural Radiance Fields Project Page | Video PyTorch implementation for our ICCV 2021 paper. MINE: Towards Continuous D

Zijian Feng 325 Dec 29, 2022