Stereo Radiance Fields (SRF): Learning View Synthesis for Sparse Views of Novel Scenes

Related tags

Deep Learning srf
Overview

Stereo Radiance Fields

Julian Chibane, Aayush Bansal, Verica Lazova, Gerard Pons-Moll
Stereo Radiance Fields (SRF): Learning View Synthesis for Sparse Views of Novel Scenes
In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021

Teaser

Paper - Supplementaty - Video - Project Website - Arxiv - If you find our project useful, please cite us. Citation (Bibtex)

Install

A linux system with python environment manager conda and a full and system wide installation of the CUDA Toolkit 10.1 is required for the project (the latter only for compilation of the torchsearchsorted library).

The following commands clone the repo on your machine and install an environment, "srf", containing all dependencies.

git clone ADD LINK
cd SRF_git
conda env create -f srf_env.yml

Please close the terminal session at this point, and reopen it at the same location. This is done to ensure conda correctly loads all packages.

conda activate srf
pip install torchsearchsorted/

Data Setup

With the next commands the DTU MVS dataset is downloaded and put in place.

wget http://roboimagedata2.compute.dtu.dk/data/MVS/Rectified.zip -P data/
unzip data/Rectified.zip -d data/
mv data/Rectified/* data/DTU_MVS
rmdir data/Rectified

Quick Start with Pretrained Model

To synthesise novel views use the following command

python generator.py --config configs/finetune_scan23.txt --video --render_factor 8 --generate_specific_samples scan23 --fixed_batch 1 --ft_path checkpoint.tar --gen_pose 0

where --config specifies the path to the experiment configuration and --gen_pose is the frame number from 0-55 (including both).

Training

Coming soon.

Contact

For questions and comments please contact Julian Chibane via mail.

License

Copyright (c) 2021 Julian Chibane, Max-Planck-Gesellschaft

By downloading and using this code you agree to the terms in the LICENSE.

You agree to cite the Stereo Radiance Fields (SRF): Learning View Synthesis for Sparse Views of Novel Scenes paper in documents and papers that report on research using this software or the manuscript.

Show LICENSE (click to expand) Please read carefully the following terms and conditions and any accompanying documentation before you download and/or use this software and associated documentation files (the "Software").

The authors hereby grant you a non-exclusive, non-transferable, free of charge right to copy, modify, merge, publish, distribute, and sublicense the Software for the sole purpose of performing non-commercial scientific research, non-commercial education, or non-commercial artistic projects.

Any other use, in particular any use for commercial purposes, is prohibited. This includes, without limitation, incorporation in a commercial product, use in a commercial service, or production of other artefacts for commercial purposes. For commercial inquiries, please see above contact information.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

You understand and agree that the authors are under no obligation to provide either maintenance services, update services, notices of latent defects, or corrections of defects with regard to the Software. The authors nevertheless reserve the right to update, modify, or discontinue the Software at any time.

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

Comments
  • split.pkl was removed from the GitHub repository

    split.pkl was removed from the GitHub repository

    Based on the latest version, I got this error: FileNotFoundError: [Errno 2] No such file or directory: './data/DTU_MVS//split.pkl'

    I tried to copy the original split_shuffle.pkl to that location and it works. I guess you should not remove that file?

    opened by yizhou-wang 3
  • Failed to compile torchsearchsorted

    Failed to compile torchsearchsorted

    Thanks for this valuable work!

    I failed to run pip install torchsearchsorted/. I have Ubuntu 18.04 + GCC v8.4 + CUDA 10.1 + python 3.6.12 + pytorch 1.6. But on another Ubuntu 16.04 (GCC 5.4) machine, it can be compiled successfully.

    Is that an error due to any package version issues? Could you please help? Thanks!

    Here I show one of the error messages (more detailed error messages can be provided upon request):

        ninja: build stopped: subcommand failed.       
      Traceback (most recent call last):                    
        File "/home/t-yizhouwang/anaconda3/envs/srf/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 1515, in _run_ninja_build 
          env=env)                                                                    
        File "/home/t-yizhouwang/anaconda3/envs/srf/lib/python3.6/subprocess.py", line 438, in run
          output=stdout, stderr=stderr)  
      subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1. 
    
    opened by yizhou-wang 3
  • Memory leak during inference

    Memory leak during inference

    Hi, I am attempting for novel view synthesis by running the pretrained inference with the provided command and code. Even if I reduce the number of ray samples, the memory was exploding (500G memory + a lot of swap on the machine). Does anybody else experience this? or could the author kindly comment on this?

    opened by chris-nbi 2
  • What is the N_rand to reproduce paper result?

    What is the N_rand to reproduce paper result?

    Hi, I am using a Tesla V100, and I can only run the training script with N_rand set to 64, but the default N_rand is 4096. Are you using that big batch size during the training?

    opened by FomalhautB 1
  • training requirement of GPU memory

    training requirement of GPU memory

    Hi,

    Thank you for releasing the code.

    I am trying to train this model on my dataset. I just revised the dataset-setup code. The resolution of my image is 2000x2000. and I set ref_imgs number to 5. The GPU is RTX 2080 with 12GB memory. I try to train this model several times, but I got the following err message:

    image

    So I want to ask the requirement of GPU memory for training this model.

    Best, Hengfei

    opened by hengfei-wang 1
  • Custom datasets and camera extrinsics.

    Custom datasets and camera extrinsics.

    Hello jchibane, I'm wondering whether your code can be applied to other datasets such as ShapeNet and custom blender datasets. How did you get the raw pose_extrinsics for the DTU dataset ? image Thanks.

    opened by Chester-CS 1
  • Pre-trained ckpt files are out of date

    Pre-trained ckpt files are out of date

    When I run inference using the latest code, I got this error: KeyError: 'sim_combinations' at https://github.com/jchibane/srf/blob/ce97b2797578a96818c2b53b3a38d34218d2f55f/model.py#L213

    I guess the pre-trained ckpt files are out of date without this key included? (Comment this line out will work for inference.) Thanks!

    opened by yizhou-wang 1
  • Obtaining 3D mesh as in Fig6

    Obtaining 3D mesh as in Fig6

    Hi, I was hoping to obtain the 3D mesh as shown in Figure 6. I assume you'd follow the steps to threshold the density predictions and run Marching cubes on a dense grid of points. Could you perhaps provide the exact detail? or upload the code to obtain the paper's Figure6 mesh output?

    thanks,

    opened by november07 1
  • Could you share the pretrained model on DTU before finetuning on Scan106 or Scan23?

    Could you share the pretrained model on DTU before finetuning on Scan106 or Scan23?

    I want to try to finetune on new scenes but it is too slow to download all the DTU datasets. So I want to finetune SRF directly based on the pretrained model on DTU. Could you share the pretrained model on DTU before finetuning on Scan106 or Scan23?

    opened by EchoTHChen 0
  • coordinate system

    coordinate system

    Dear author,

    can you give more explainations to following two coordinate system? x_down_y_down_z_cam_dir x_down_y_up_z_neg_cam_dir

    How can I draw to visualize the xyz axises?

    Thanks

    opened by derrick-xwp 1
  •  name 'images' is not defined

    name 'images' is not defined

    File "E:\OTHERS_WORKS\srf\srf\dataloader.py", line 101, in getitem imgs, poses, poses_idx = DTU.load_scan_data(sample, self.load_mode, self.num_reference_views + 1, self.cfg, self.load_specific_reference_poses, self.load_fixed, self.shuffle) File "E:\OTHERS_WORKS\srf\srf\data\load_DTU.py", line 59, in load_scan_data poses_idx = random.sample(images[scan],num_views) NameError: name 'images' is not defined

    opened by derrick-xwp 1
  • near and far values in config

    near and far values in config

    Hello, I was wondering how the near and far values were decided upon for ray sampling. The near value = 500, far =1000 at the moment, how were this decided? and if I were to be trying on a different dataset, on what basis should I choose and near and far values?

    opened by chris-nbi 1
Owner
null
A minimal TPU compatible Jax implementation of NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis

NeRF Minimal Jax implementation of NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. Result of Tiny-NeRF RGB Depth

Soumik Rakshit 11 Jul 24, 2022
Open source repository for the code accompanying the paper 'Non-Rigid Neural Radiance Fields Reconstruction and Novel View Synthesis of a Deforming Scene from Monocular Video'.

Non-Rigid Neural Radiance Fields This is the official repository for the project "Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synt

Facebook Research 296 Dec 29, 2022
[ICCV 2021 Oral] NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereo

NerfingMVS Project Page | Paper | Video | Data NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereo Yi Wei, Shaohui

Yi Wei 369 Dec 24, 2022
PyTorch implementation of paper "Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes", CVPR 2021

Neural Scene Flow Fields PyTorch implementation of paper "Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes", CVPR 20

Zhengqi Li 585 Jan 4, 2023
D-NeRF: Neural Radiance Fields for Dynamic Scenes

D-NeRF: Neural Radiance Fields for Dynamic Scenes [Project] [Paper] D-NeRF is a method for synthesizing novel views, at an arbitrary point in time, of

Albert Pumarola 291 Jan 2, 2023
"SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Image", Dejia Xu, Yifan Jiang, Peihao Wang, Zhiwen Fan, Humphrey Shi, Zhangyang Wang

SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Image [Paper] [Website] Pipeline Code Environment pip install -r requirements

VITA 250 Jan 5, 2023
This repository contains a PyTorch implementation of "AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis".

AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis | Project Page | Paper | PyTorch implementation for the paper "AD-NeRF: Audio

null 551 Dec 29, 2022
Official code release for "GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis"

GRAF This repository contains official code for the paper GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis. You can find detailed usage i

null 349 Dec 29, 2022
[ICCV'21] UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction

UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction Project Page | Paper | Supplementary | Video This reposit

null 331 Dec 28, 2022
Official PyTorch Implementation of paper "Deep 3D Mask Volume for View Synthesis of Dynamic Scenes", ICCV 2021.

Deep 3D Mask Volume for View Synthesis of Dynamic Scenes Official PyTorch Implementation of paper "Deep 3D Mask Volume for View Synthesis of Dynamic S

Ken Lin 17 Oct 12, 2022
[ICCV'21] Neural Radiance Flow for 4D View Synthesis and Video Processing

NeRFlow [ICCV'21] Neural Radiance Flow for 4D View Synthesis and Video Processing Datasets The pouring dataset used for experiments can be download he

null 44 Dec 20, 2022
This is a JAX implementation of Neural Radiance Fields for learning purposes.

learn-nerf This is a JAX implementation of Neural Radiance Fields for learning purposes. I've been curious about NeRF and its follow-up work for a whi

Alex Nichol 62 Dec 20, 2022
Pytorch implementation for A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose

A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose Paper | Website | Data A-NeRF: Articulated Neural Radiance F

Shih-Yang Su 172 Dec 22, 2022
Python scripts form performing stereo depth estimation using the high res stereo model in PyTorch .

PyTorch-High-Res-Stereo-Depth-Estimation Python scripts form performing stereo depth estimation using the high res stereo model in PyTorch. Stereo dep

Ibai Gorordo 26 Nov 24, 2022
RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matching

RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matching This repository contains the source code for our paper: RAFT-Stereo: Multilevel

Princeton Vision & Learning Lab 328 Jan 9, 2023
[ICCV 2021 (oral)] Planar Surface Reconstruction from Sparse Views

Planar Surface Reconstruction From Sparse Views Linyi Jin, Shengyi Qian, Andrew Owens, David F. Fouhey University of Michigan ICCV 2021 (Oral) This re

Linyi Jin 89 Jan 5, 2023
This repository contains the code for the CVPR 2021 paper "GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields"

GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields Project Page | Paper | Supplementary | Video | Slides | Blog | Talk If

null 1.1k Dec 30, 2022
This is the code for Deformable Neural Radiance Fields, a.k.a. Nerfies.

Deformable Neural Radiance Fields This is the code for Deformable Neural Radiance Fields, a.k.a. Nerfies. Project Page Paper Video This codebase conta

Google 1k Jan 9, 2023
(Arxiv 2021) NeRF--: Neural Radiance Fields Without Known Camera Parameters

NeRF--: Neural Radiance Fields Without Known Camera Parameters Project Page | Arxiv | Colab Notebook | Data Zirui Wang¹, Shangzhe Wu², Weidi Xie², Min

Active Vision Laboratory 411 Dec 26, 2022