Code release for DS-NeRF (Depth-supervised Neural Radiance Fields)

Overview

Depth-supervised NeRF: Fewer Views and Faster Training for Free

Project | Paper | YouTube

Pytorch implementation of our method for learning neural radiance fields that takes advantage of depth supervised by 3D point clouds. It can be used to train NeRF models given only very few input views.

Depth-supervised NeRF: Fewer Views and Faster Training for Free

arXiv 2107.02791, 2021

Kangle Deng1, Andrew Liu2, Jun-Yan Zhu1, Deva Ramanan1,3,

1CMU, 2Google, 3Argo AI


We propose DS-NeRF (Depth-supervised Neural Radiance Fields), a model for learning neural radiance fields that takes advantage of depth supervised by 3D point clouds.

NeRF trained with 2 views:

DS-NeRF trained with 2 views:


Quick Start

Dependencies

Install requirements:

pip install -r requirements.txt

You will also need COLMAP installed to compute poses if you want to run on your data.

Data

Download data for the example scene: fern_2v

bash download_example_data.sh

To play with other scenes presented in the paper, download the data here.

Pre-trained Models

You can download the pre-trained models here. Place the downloaded directory in ./logs in order to test it later. See the following directory structure for an example:

├── logs 
│   ├── fern_2v    # downloaded logs
│   ├── flower_2v  # downloaded logs

How to Run?

Generate camera poses and sparse depth information using COLMAP (optional)

This step is necessary only when you want to run on your data.

First, place your scene directory somewhere. See the following directory structure for an example:

├── data
│   ├── fern_2v
│   ├── ├── images
│   ├── ├── ├── image001.png
│   ├── ├── ├── image002.png

To generate the poses and sparse point cloud:

python imgs2poses.py <your_scenedir>

Testing

Once you have the experiment directory (downloaded or trained on your own) in ./logs,

  • to render a video:
python run_nerf.py --config configs/fern_dsnerf.txt --render_only

Training

To train a DS-NeRF on the example fern dataset:

python run_nerf.py --config configs/fern_dsnerf.txt

You can create your own experiment configuration to try other datasets.


Citation

If you find this repository useful for your research, please cite the following work.

@article{kangle2021dsnerf,
  title={Depth-supervised NeRF: Fewer Views and Faster Training for Free},
  author={Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan},
  journal={arXiv preprint arXiv:2107.02791},
  year={2021}
}

Credits

This code borrows heavily from nerf-pytorch.

Comments
  • NDC space and depth loss

    NDC space and depth loss

    I am using NDC space and I have depth values collected from a sensor, should I normalize these values between 0 and 1 or should I convert them into NDC space? How should we calculate the depth loss while using NDC space as the depth_col value comes between 0 and 1 in NDC space but the target_depth is bigger than 1.

    Thanks.

    opened by mertkiray 5
  • Question about the regularization added to the loss

    Question about the regularization added to the loss

    opened by saunair 4
  • Cannot reproduce performance on forn_2v

    Cannot reproduce performance on forn_2v

    Hi~ Thank you very much for the open source code and paper

    I tried to train a model with default config,but I got poor performance on forn_2v.I strictly followed the instructions to start the reproduction experiment,However, my experimental results are quite different from the released results; result of mine: 000 002 007 log of last iter: image

    Am i missing something ? Really appreciate for your help.

    opened by AliceMegatron 3
  • Why calculate the weights twice

    Why calculate the weights twice

    Hi, thanks for your great work! I have the question about why to recalculate weights when calculating sigma_loss. While we already have the weights in the forward here.

    opened by Trent-tangtao 2
  • Unable to reproduce the fern_2v result

    Unable to reproduce the fern_2v result

    Hi, thank you for your great work.

    I use

    python run_nerf.py --config configs/fern_2v.txt --render_only" 
    

    command with the pretrained model, it gave me these results

    https://user-images.githubusercontent.com/45889437/202093543-8c02a2f1-8989-48d7-8ff5-ead60cdbdf36.mp4

    https://user-images.githubusercontent.com/45889437/202093571-40752187-6e50-4847-912f-cbdd1ac6dbe9.mp4

    while when I re-trained the model from scratch with this command

    python run_nerf.py --config configs/fern_2v.txt --weighted_loss 
    

    and render it with my trained model, it gave this following results

    https://user-images.githubusercontent.com/45889437/202093604-a700141a-77a3-4e85-a25d-b842d8cf902e.mp4

    https://user-images.githubusercontent.com/45889437/202094729-efc05d95-0c5c-4d06-8b8a-1505e3b295b4.mp4

    Do you have any clue ? Is the sigma-loss necessary in order to reproduce the same quality with the given-pretrained model? Thanks

    opened by qiwen98 2
  • Question about ray distance t and depth D

    Question about ray distance t and depth D

    Hi,

    Thank you for the great work.

    If my understanding is correct, the depth D of a 3D point is its Z value in a camera's local coordinate system. It is different from the ray distance t, which is the distance from the point to the camera center. In fact, D = sin(θ) × t, where θ is the angle between the camera plane and the ray. Therefore, even in an ideal case, the distribution of h(t) and D are not expected to be the same.

    However, in section 3.2, I see you minimizing the KL divergence between these two distributions. Wouldn't that cause some problems? Or did I miss something?

    I am new to nerf, please forgive me if the answer is trivial.

    Thank you

    opened by xiechun-tsukuba 2
  • Zero weights appear while computing SigmaLoss

    Zero weights appear while computing SigmaLoss

    Hi, thanks for your nice work!

    I found an issue while trying to use your KL-divergence loss. In loss.py, almost half of alpha values become zero due to negative raw density values. alpha = raw2alpha(raw[...,3] + noise, dists) # [N_rays, N_samples]

    This produces zero weights for such points, and ends up passing a bunch of zeros into the log function. weights = alpha * torch.cumprod(torch.cat([torch.ones((alpha.shape[0], 1)).to(device), 1.-alpha + 1e-10], -1), -1)[:, :-1] loss = -torch.log(weights) * torch.exp(-(z_vals - depths[:,None]) ** 2 / (2 * err)) * dists

    How did you deal with this problem in your experiments? (eg. perhaps by masking out those points while computing the loss, or by using a softplus activation in raw2alpha instead of relu?)

    Thanks in advance!

    opened by totolacky 2
  • how to generate sparse 3d points given known cameras

    how to generate sparse 3d points given known cameras

    Hi, thank you for the work.

    I have a question about the sparse 3d points given known cameras. I have read this issue and this issue.

    I also carefully read this link of colmap. There are multiple steps, like image.txt, cameras.txt, and point3D.txt. I would like to know how to prepare these to align your work to better reproduction for comparison.

    I think this is the way (with known camera poses) you prepare NeRF real (LLFF) and DTU since we need unified poses for training and test images. The steps are as follows:

    (1) estimate poses with training and test images (because we need a unified coordinate for training and test). (2) With camera poses of training images, we generate sparse 3d points. Note, we cannot access test images to generate sparse 3d points.

    In the code, the paper only provides code for python imgs2poses.py <your_scenedir>

    It would be better if a script like python imgs2poses.py <your_scenedir> <your camera poses> <your camera intrinsics> is provided. I would like to know how to prepare these to align your work to better reproduction for comparison.

    Thank you.

    opened by Wanggcong 2
  • The value range of depth loss

    The value range of depth loss

    Hello, I try to run your code with my own data, but I found the depth loss is pretty large (0.25). I'm not sure if it is because the generated depth is not right in my own data, so I want to know the depth loss's value in your training.

    opened by iYuqinL 2
  • Confusion about the implementation of weight calculation for depth and RGB rendering

    Confusion about the implementation of weight calculation for depth and RGB rendering

    Hi, thanks for your neat and excellent work! However, I am confused about the implementation of the weight for depth and RGB rendering. In the code implementation, it seems that the weight is calculated by the product of T(t) and alpha(t) (https://github.com/dunbar12138/DSNeRF/blob/168a13edd8a08396e3ad775283eeb65145317205/run_nerf_helpers.py#L374) while it is formulated as the product of T(t) and sigma(t) in the paper.

    opened by JerryX1110 2
  • Potential view inconsistency in depth loss

    Potential view inconsistency in depth loss

    Hi,

    I noticed that you compute depth in input views in https://github.com/dunbar12138/DSNeRF/blob/main/load_llff.py#L371 but the depth given by NeRF model is in cannonical view https://github.com/dunbar12138/DSNeRF/blob/main/run_nerf_helpers.py#L377

    Shouldn't both the depths be brought to same view before computing depth loss? Please let me know if I missed something

    opened by NagabhushanSN95 2
  • I wonder how to make the depth file.

    I wonder how to make the depth file.

    Hi~ I'm going to apply it to the new data instead of the dataset that provided your great work, dsnerf. So I'd appreciate it if you could tell me how to generate the 'train_depths.npy' file and how to make the information (depth, coord, error). image

    opened by SEJUNHONG 1
  • How to generate all of the other necessary files for run_nerf.py?

    How to generate all of the other necessary files for run_nerf.py?

    When I run my images through COLMAP, I get a Sparse directory and can run imgs2poses which generates poses_bounds.npy.

    How do I go about converting my test and training images to .npy format and how do I generate bds.npy, test_poses.py, train_poses.py, etc...

    Thanks for any assistance.

    opened by TylerRust-1 1
  • Some problems with the code about implementing the deep supervision part

    Some problems with the code about implementing the deep supervision part

    Hello, I recently tried to add depth KL scatter loss to my own project, and found the core code corresponding to this section was then found to be here https://github.com/dunbar12138/DSNeRF/blob/d9deb70ac1464348aebbe5e32fc98aeca1920f70/loss.py#L87 But in my actual use I found a problem that the weights will often be 0 image So when calculating the loss, if you put the weights directly into the log function, this can easily result in results with inf image But I looked at the code and found that there was nothing to prevent this situation

    So I would like to ask guys if anyone has encountered this situation, and I think a very small number can be added to the weights to prevent this as follows:

    -torch.log(weights+1e-10) * torch.exp(-(z_vals - depths[:,None]) ** 2 / (2 * err)) * dists
    

    Sorry for disturbing you and hope to receive an answer, thans a lot (^▽^) !

    opened by YZsZY 1
  • Could you share args.txt and config.txt?

    Could you share args.txt and config.txt?

    Greetings! I find the config files that you release now are actually neglecting many important training options, for example, whether use weighted_loss or not (refer to this issue #30). Besides, the pretrained logs you provided in download_models.sh are actually missing the args.txt and config.txt. Since these hyper-parameters are vital to reproduce the reported performances in paper, could you please completely upload these files for each scene and each dataset? Or it is really hard for for future works to make experimental comparisons with DSNeRF. Thanks!

    opened by machengcheng2016 1
  • Question about SigmaLoss with DSNeRF paper.

    Question about SigmaLoss with DSNeRF paper.

    Hi, Thank you for sharing your great code. I'm glad to learn about DSNeRF with your kind and detailed explanation

    I have a question about sigmaLoss with paper.

    I guess sigma loss is the ray distribution loss in the paper notation by h(t). From the paper, h(t) = T(t)*σ(t) is defined. But the code in class SigmaLoss from loss.py, I see

            # sigma = F.relu(raw[...,3] + noise)
            alpha = raw2alpha(raw[...,3] + noise, dists)  # [N_rays, N_samples]
            weights = alpha * torch.cumprod(torch.cat([torch.ones((alpha.shape[0], 1)).to(device), 1.-alpha + 1e-10], -1), -1)[:, :-1]
            
            loss = -torch.log(weights) * torch.exp(-(z_vals - depths[:,None]) ** 2 / (2 * err)) * dists
            loss = torch.sum(loss, dim=1)
            
            return loss
    

    loss = -torch.log(weights) * torch.exp(-(z_vals - depths[:,None]) ** 2 / (2 * err)) * dists Here, the loss function is defined by weights and $T_i$. I think that from the paper's h(t) definition, the loss should be defined by loss = torch.log(h) * torch.exp(-(z_vals - depths[:,None]) ** 2 / (2 * err)) * dists where h = sigma * torch.cumprod(torch.cat([torch.ones((alpha.shape[0], 1)).to(device), 1.-alpha + 1e-10], -1), -1)[:, :-1] But, the actual code is not. Can I guess why this code is different from the DSNeRF paper? Additionally, can I ask why the '-' term is added in front of torch.log(weights)?

    Thanks

    opened by jhq1234 4
  • Running out of memory when rendering from trained model

    Running out of memory when rendering from trained model

    Hello,

    Thank you very much for your great implementation! I have successfully trained the model on my own data, but am now running into a GPU memory problem when trying to render a novel view. Is there any way to decrease this memory load while "rendering only"? It would generally be great to get a bit more information on the different variables in the config file.

    Thank you very much!

    opened by jakob-ropers-snkeos 2
(Arxiv 2021) NeRF--: Neural Radiance Fields Without Known Camera Parameters

NeRF--: Neural Radiance Fields Without Known Camera Parameters Project Page | Arxiv | Colab Notebook | Data Zirui Wang¹, Shangzhe Wu², Weidi Xie², Min

Active Vision Laboratory 411 Dec 26, 2022
Unofficial & improved implementation of NeRF--: Neural Radiance Fields Without Known Camera Parameters

[Unofficial code-base] NeRF--: Neural Radiance Fields Without Known Camera Parameters [ Project | Paper | Official code base ] ⬅️ Thanks the original

Jianfei Guo 239 Dec 22, 2022
Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields.

This repository contains the code release for Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields. This implementation is written in JAX, and is a fork of Google's JaxNeRF implementation. Contact Jon Barron if you encounter any issues.

Google 625 Dec 30, 2022
This repository contains a PyTorch implementation of "AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis".

AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis | Project Page | Paper | PyTorch implementation for the paper "AD-NeRF: Audio

null 551 Dec 29, 2022
A PyTorch implementation of NeRF (Neural Radiance Fields) that reproduces the results.

NeRF-pytorch NeRF (Neural Radiance Fields) is a method that achieves state-of-the-art results for synthesizing novel views of complex scenes. Here are

Yen-Chen Lin 3.2k Jan 8, 2023
D-NeRF: Neural Radiance Fields for Dynamic Scenes

D-NeRF: Neural Radiance Fields for Dynamic Scenes [Project] [Paper] D-NeRF is a method for synthesizing novel views, at an arbitrary point in time, of

Albert Pumarola 291 Jan 2, 2023
Pytorch implementation for A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose

A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose Paper | Website | Data A-NeRF: Articulated Neural Radiance F

Shih-Yang Su 172 Dec 22, 2022
A minimal TPU compatible Jax implementation of NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis

NeRF Minimal Jax implementation of NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. Result of Tiny-NeRF RGB Depth

Soumik Rakshit 11 Jul 24, 2022
Build upon neural radiance fields to create a scene-specific implicit 3D semantic representation, Semantic-NeRF

Semantic-NeRF: Semantic Neural Radiance Fields Project Page | Video | Paper | Data In-Place Scene Labelling and Understanding with Implicit Scene Repr

Shuaifeng Zhi 243 Jan 7, 2023
Point-NeRF: Point-based Neural Radiance Fields

Point-NeRF: Point-based Neural Radiance Fields Project Sites | Paper | Primary c

Qiangeng Xu 662 Jan 1, 2023
This repository contains the source code for the paper "DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks",

DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks Project Page | Video | Presentation | Paper | Data L

Facebook Research 281 Dec 22, 2022
PyTorch implementation for MINE: Continuous-Depth MPI with Neural Radiance Fields

MINE: Continuous-Depth MPI with Neural Radiance Fields Project Page | Video PyTorch implementation for our ICCV 2021 paper. MINE: Towards Continuous D

Zijian Feng 325 Dec 29, 2022
Official code release for "GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis"

GRAF This repository contains official code for the paper GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis. You can find detailed usage i

null 349 Dec 29, 2022
Instant-nerf-pytorch - NeRF trained SUPER FAST in pytorch

instant-nerf-pytorch This is WORK IN PROGRESS, please feel free to contribute vi

null 94 Nov 22, 2022
This is the code for Deformable Neural Radiance Fields, a.k.a. Nerfies.

Deformable Neural Radiance Fields This is the code for Deformable Neural Radiance Fields, a.k.a. Nerfies. Project Page Paper Video This codebase conta

Google 1k Jan 9, 2023
Open source repository for the code accompanying the paper 'Non-Rigid Neural Radiance Fields Reconstruction and Novel View Synthesis of a Deforming Scene from Monocular Video'.

Non-Rigid Neural Radiance Fields This is the official repository for the project "Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synt

Facebook Research 296 Dec 29, 2022
Code for KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs

KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs Check out the paper on arXiv: https://arxiv.org/abs/2103.13744 This repo cont

Christian Reiser 373 Dec 20, 2022
This is the code for "HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields".

HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields This is the code for "HyperNeRF: A Higher-Dimensional

Google 702 Jan 2, 2023
BARF: Bundle-Adjusting Neural Radiance Fields 🤮 (ICCV 2021 oral)

BARF ?? : Bundle-Adjusting Neural Radiance Fields Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey IEEE International Conference on Comp

Chen-Hsuan Lin 539 Dec 28, 2022