Official repo for AutoInt: Automatic Integration for Fast Neural Volume Rendering in CVPR 2021

Overview

AutoInt: Automatic Integration for Fast Neural Volume Rendering
CVPR 2021

Project Page | Video | Paper

Open Colab
PyTorch implementation of automatic integration.
AutoInt: Automatic Integration for Fast Neural Volume Rendering
David B. Lindell*, Julien N. P. Martel*, Gordon Wetzstein
Stanford University
*denotes equal contribution
in CVPR 2021

Quickstart

To get started quickly, we provide a collab link above. Otherwise, you can clone this repo and follow the below instructions.

To setup a conda environment, download example training data, begin the training process, and launch Tensorboard:

conda env create -f environment.yml
conda activate autoint 
cd experiment_scripts
python train_1d_integral.py
tensorboard --logdir=../logs --port=6006

This example will fit a grad network to a 1D signal and evaluate the integral. You can monitor the training in your browser at localhost:6006. You can also train a network on the sparse tomography problem presented in the paper with python train_sparse_tomography.py.

Autoint for Neural Rendering

Automatic integration can be used to learn closed form solutions to the volume rendering equation, which is an integral equation accumulates transmittance and emittance along rays to render an image. While conventional neural renderers require hundreds of samples along each ray to evaluate these integrals (and hence hundreds of costly forward passes through a network), AutoInt allows evaluating these integrals far fewer forward passes.

Training

To run AutoInt for neural rendering, first set up the conda environment with

conda env create -f environment.yml
conda activate autoint 

Then, download the datasets to the data folder. We allow training on any of three datasets. The synthetic Blender data from NeRF and the LLFF scenes are hosted here. The DeepVoxels data are hosted here.

Finally, use the provided config files in the experiment_scripts/configs folder to train on these datasets. For example, to train on a NeRF Blender dataset, run the following

python train_autoint_radiance_field.py --config ./configs/config_blender_tiny.ini
tensorboard --logdir=../logs/ --port=6006

This will train a small, low-resolution scene. To train scenes at high-resolution (requires a few days of training time), use the config_blender.ini, config_deepvoxels.ini, or config_llff.ini config files.

Rendering

Rendering from a trained model can be done with the following command.

python train_autoint_radiance_field.py --config /path/to/config/file --render_model ../logs/path/to/log/directory <epoch number> --render_ouput /path/to/output/folder

Here, the --render_model command indicates the log directory where the code saves the models and checkpoints. For example, this would be ../logs/blender_lego for the default Blender dataset. Then, the epoch number can be found by looking at numbers of the the saved checkpoint filenames in ../logs/blender_lego/checkpoints/. Finally, --render_output should specify a folder where the output rendered images will be generated.

Citation

@inproceedings{autoint2021,
  title={AutoInt: Automatic Integration for Fast Neural Volume Rendering},
  author={David B. Lindell and Julien N. P. Martel and Gordon Wetzstein},
  year={2021},
  booktitle={Proc. CVPR},
}
You might also like...
The official repo of the CVPR 2021 paper Group Collaborative Learning for Co-Salient Object Detection .

GCoNet The official repo of the CVPR 2021 paper Group Collaborative Learning for Co-Salient Object Detection . Trained model Download final_gconet.pth

 This repository contains the code for the CVPR 2020 paper
This repository contains the code for the CVPR 2020 paper "Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision"

Differentiable Volumetric Rendering Paper | Supplementary | Spotlight Video | Blog Entry | Presentation | Interactive Slides | Project Page This repos

Code for "Modeling Indirect Illumination for Inverse Rendering", CVPR 2022

Modeling Indirect Illumination for Inverse Rendering Project Page | Paper | Data Preparation Set up the python environment conda create -n invrender p

The official implementation of our CVPR 2021 paper - Hybrid Rotation Averaging: A Fast and Robust Rotation Averaging Approach

Graph Optimizer This repo contains the official implementation of our CVPR 2021 paper - Hybrid Rotation Averaging: A Fast and Robust Rotation Averagin

An integration of several popular automatic augmentation methods, including OHL (Online Hyper-Parameter Learning for Auto-Augmentation Strategy) and AWS (Improving Auto Augment via Augmentation Wise Weight Sharing) by Sensetime Research.

An integration of several popular automatic augmentation methods, including OHL (Online Hyper-Parameter Learning for Auto-Augmentation Strategy) and AWS (Improving Auto Augment via Augmentation Wise Weight Sharing) by Sensetime Research.

git git《Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking》(CVPR 2021) GitHub:git2] 《Masksembles for Uncertainty Estimation》(CVPR 2021) GitHub:git3]
git git《Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking》(CVPR 2021) GitHub:git2] 《Masksembles for Uncertainty Estimation》(CVPR 2021) GitHub:git3]

Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking Ning Wang, Wengang Zhou, Jie Wang, and Houqiang Li Accepted by CVPR

Scale-aware Automatic Augmentation for Object Detection (CVPR 2021)
Scale-aware Automatic Augmentation for Object Detection (CVPR 2021)

SA-AutoAug Scale-aware Automatic Augmentation for Object Detection Yukang Chen, Yanwei Li, Tao Kong, Lu Qi, Ruihang Chu, Lei Li, Jiaya Jia [Paper] [Bi

Deep RGB-D Saliency Detection with Depth-Sensitive Attention and Automatic Multi-Modal Fusion (CVPR'2021, Oral)

DSA^2 F: Deep RGB-D Saliency Detection with Depth-Sensitive Attention and Automatic Multi-Modal Fusion (CVPR'2021, Oral) This repo is the official imp

Automatic self-diagnosis program (python required)Automatic self-diagnosis program (python required)

auto-self-checker 자동으로 자가진단 해주는 프로그램(python 필요) 중요 이 프로그램이 실행될때에는 절대로 마우스포인터를 움직이거나 키보드를 건드리면 안된다(화면인식, 마우스포인터로 직접 클릭) 사용법 프로그램을 구동할 폴더 내의 cmd창에서 pip

Comments
  • Reasons behind writing AutoInt extension for PyTorch

    Reasons behind writing AutoInt extension for PyTorch

    Hi, I am really interested in your ideas, and hurry to implement a toy example to test it.

    I find that you write a comprehensive and complicated extension in autoint to automatically "extract" the gradient network from integral network and implement colorful methods, especially for the draw() one.

    However, I wonder that torch.autograd.functional.jacobian has done exactly the same job for you, if we are only talking about getting the partial derivative of the outputs w.r.t the inputs.

    Attempting to answer this question by myself, I write a simple MLP and try to regress a polynomial function by using jacobian or computing derivative manually to "extract" the gradient network. Surprisingly, I find that jacobian can be 60x times slower than manual computation.

    So, is speed one of the reasons behind you writing an extension?

    opened by RaymondJiangkw 2
  • 2D and 3D case.

    2D and 3D case.

    Hi dear authors and thanks for such fantastic work.

    Could you please guide me on how one might use, or extend the autoint framework for 2d and 3d integration? One use case for the 2d case might be fitting the derivative network to an image.

    Thank you.

    @davelindell @wetzste1

    opened by elientumba2019 1
  • How to measure the GPU memory consumption in Table 1?

    How to measure the GPU memory consumption in Table 1?

    Hi, thanks for your sharing of this excellent project. And I would like to ask how you measure the GPU memory consumption of NeRF and AutoInt as in Table 1?

    Thanks in advance!

    opened by Totoro97 1
Owner
Stanford Computational Imaging Lab
Next-generation computational imaging and display systems.
Stanford Computational Imaging Lab
Gesture-Volume-Control - This Python program can adjust the system's volume by using hand gestures

Gesture-Volume-Control This Python program can adjust the system's volume by usi

VatsalAryanBhatanagar 1 Dec 30, 2021
Hand Gesture Volume Control is AIML based project which uses image processing to control the volume of your Computer.

Hand Gesture Volume Control Modules There are basically three modules Handtracking Program Handtracking Module Volume Control Program Handtracking Pro

VITTAL 1 Jan 12, 2022
一些经典的CTR算法的复现; LR, FM, FFM, AFM, DeepFM,xDeepFM, PNN, DCN, DCNv2, DIFM, AutoInt, FiBiNet,AFN,ONN,DIN, DIEN ... (pytorch, tf2.0)

CTR Algorithm 根据论文, 博客, 知乎等方式学习一些CTR相关的算法 理解原理并自己动手来实现一遍 pytorch & tf2.0 保持一颗学徒的心! Schedule Model pytorch tensorflow2.0 paper LR ✔️ ✔️ \ FM ✔️ ✔️ Fac

luo han 149 Dec 20, 2022
This is the official implementation code repository of Underwater Light Field Retention : Neural Rendering for Underwater Imaging (Accepted by CVPR Workshop2022 NTIRE)

Underwater Light Field Retention : Neural Rendering for Underwater Imaging (UWNR) (Accepted by CVPR Workshop2022 NTIRE) Authors: Tian Ye†, Sixiang Che

jmucsx 17 Dec 14, 2022
Official Repo for ICCV2021 Paper: Learning to Regress Bodies from Images using Differentiable Semantic Rendering

[ICCV2021] Learning to Regress Bodies from Images using Differentiable Semantic Rendering Getting Started DSR has been implemented and tested on Ubunt

Sai Kumar Dwivedi 83 Nov 27, 2022
Official PyTorch Implementation of paper "Deep 3D Mask Volume for View Synthesis of Dynamic Scenes", ICCV 2021.

Deep 3D Mask Volume for View Synthesis of Dynamic Scenes Official PyTorch Implementation of paper "Deep 3D Mask Volume for View Synthesis of Dynamic S

Ken Lin 17 Oct 12, 2022
Fast algorithms to compute an approximation of the minimal volume oriented bounding box of a point cloud in 3D.

ApproxMVBB Status Build UnitTests Homepage Fast algorithms to compute an approximation of the minimal volume oriented bounding box of a point cloud in

Gabriel Nützi 390 Dec 31, 2022
Style-based Point Generator with Adversarial Rendering for Point Cloud Completion (CVPR 2021)

Style-based Point Generator with Adversarial Rendering for Point Cloud Completion (CVPR 2021) An efficient PyTorch library for Point Cloud Completion.

Microsoft 119 Jan 2, 2023
PyTorch implementation of paper "IBRNet: Learning Multi-View Image-Based Rendering", CVPR 2021.

IBRNet: Learning Multi-View Image-Based Rendering PyTorch implementation of paper "IBRNet: Learning Multi-View Image-Based Rendering", CVPR 2021. IBRN

Google Interns 371 Jan 3, 2023
This repo contains the official code of our work SAM-SLR which won the CVPR 2021 Challenge on Large Scale Signer Independent Isolated Sign Language Recognition.

Skeleton Aware Multi-modal Sign Language Recognition By Songyao Jiang, Bin Sun, Lichen Wang, Yue Bai, Kunpeng Li and Yun Fu. Smile Lab @ Northeastern

Isen (Songyao Jiang) 128 Dec 8, 2022