Pytorch implementation of paper: "NeurMiPs: Neural Mixture of Planar Experts for View Synthesis"

James Lin

Last update: Dec 13, 2022

Related tags

Deep Learning neurmips

Overview

NeurMips: Neural Mixture of Planar Experts for View Synthesis

This is the official repo for PyTorch implementation of paper "NeurMips: Neural Mixture of Planar Experts for View Synthesis", CVPR 2022.

Paper | Project page | Video

🌱 Prerequisites

OS: Ubuntu 20.04.4 LTS
GPU: NVIDIA TITAN RTX
Python package manager conda

🌱 Setup

Datasets

Download and put datasets under folder data/ by running:

bash run/dataset.sh

For more details of file structure and camera convention, please refer to Dataset.

Environment

Install all python packages for training and evaluation with conda environment setup file:

conda env create -f environment.yml
conda activate neurmips

CUDA extension installation

Compile the extension directly by running:

cd cuda/
python setup.py develop

Note that if you need to modify this CUDA code, simply compile again after your modification.

Pretrained models (optional)

Download pretrained model weights for evaluation without training from scratch:

bash run/checkpoints.sh

🌱 Usage

We provide hyperparameters for each experiment in config file configs/*.yaml, which is used for training and evaluation. For example, replica-kitchen.yaml corresponds to Replica dataset Kitchen scene, and tat-barn.yaml corresponds to Tanks&Temple dataset Barn scene.

Training

Train the teacher and experts model by running:

bash run/train.sh [config]
# example: bash run/train.sh replica-kitchen

Evaluation

Render testing images and evaluate metrics (i.e. PSNR, SSIM, LPIPS) by running:

bash run/eval.sh [config]
# example: bash run/eval.sh replica-kitchen

The rendered images are put under folder output_images/[config]/experts/color/valid/

CUDA Acceleration

To render testing images with optimized CUDA code by running:

bash run/eval_fast.sh [config]
# example: bash run/eval_fast.sh replica-kitchen

The rendered images are put under folder output_images/[config]/experts_cuda/color/valid/

BibTex

@inproceedings{lin2022neurmips,
  title={NeurMiPs: Neural Mixture of Planar Experts for View Synthesis},
  author = {Lin, Zhi-Hao and Ma, Wei-Chiu and Hsu, Hao-Yu and Wang, Yu-Chiang Frank and Wang, Shenlong},
  year={2022},
  booktitle={CVPR},
}

Comments

Number of epochs does not match the description in the paper
Dear Authors,

Thank you for the great work and the open-sourced code!

I noticed that the number of epochs in the configs does not match the description in the paper.

In the camera version: it is said that "We first train the teacher model for 6K epochs, then we distill the planar experts for 1.5K epochs. Finally, we fine-tune the experts for 2.5K epochs."

However, in the config for replica dataset:

epoch: teacher: 2000 distill: 1000 finetune: 1000

in the config for tat dataset:

epoch: teacher: 2800 distill: 1000 finetune: 3500

, none of them matches the description in the paper.

I was wondering whether it is a typo in the paper or the currently open-sourced configs are not for those PSNR numbers reported in the paper?

Thank you!
opened by licj15 2
Why the box_factor is set as 4 for replica dataset

Dear Authors,

Thanks for the great work and the open sourced code! When I am reading the code, I noticed that box_factor is set as 4 for replica. Although box_factor is not mentioned in the paper, but from its corresponding code, I guess it is use to create a box with is box_factor times larger than the bounds of the points.

However, if my understanding is correct, why not set it as 1.0 which best matches the fact that the walls for indoor scenes are exactly at the bounds of the points?

If my understanding is wrong, could you elaborate more about how you decide box_factor as 4? So that it may help to train on other scenes (e.g., scenes collected by other users from real world).

Thank you!

opened by licj15 2
Organization custom data like replica

Hi, thanks for great work! I wonder how to organize custom dataset like replica dataset. I read issue #2, but I want to customize dataloader like replica. How to train on custom datasets, such as a sequence of pictures taken with a handheld camera? How to prepare depth.npy file?

and

Thank you.

opened by hyunJIN7 1

RuntimeError: radix_sort: failed on 1st step: cudaErrorInvalidDevice: invalid device ordinal

Loading from checkpoint: /neurmips/checkpoints/replica-kitchen-experts.pth
Baked planes alpha as [200 * 200]
Traceback (most recent call last):
  File "experts_test_fast.py", line 220, in main
    mnh_cuda.sort_by_key_float32_int64(active_depth, reorder_indices_by_depth)
RuntimeError: radix_sort: failed on 1st step: cudaErrorInvalidDevice: invalid device ordinal

Hi @zhihao-lin , Great work! Any ideas for this error? Downloaded the pretrained model and run fast eval.

opened by cchen156 1

Train on custom dataset

Hi, nice work! I wonder if there is any chance to train on custom datasets, such as a sequence of pictures taken with a handheld camera? How to prepare them for training if there is? Thank you.

opened by linghai06 1
Any luck running this on a 3000 series GPU?

It appears the version of cuda toolkit and maybe some other stuff set in the environment.yml isn't compatible with the sm_86 compute arch. Have you tried running this on nVidia 3000 series cards and can you advise how one might successfully install / compile / run the code on them?

opened by reynoldscem 2

Pytorch implementation of paper: "NeurMiPs: Neural Mixture of Planar Experts for View Synthesis"

Related tags

Overview

NeurMips: Neural Mixture of Planar Experts for View Synthesis

Paper | Project page | Video

🌱 Prerequisites

🌱 Setup

Datasets

Environment

CUDA extension installation

Pretrained models (optional)

🌱 Usage

Training

Evaluation

CUDA Acceleration

Comments

Number of epochs does not match the description in the paper

Why the box_factor is set as 4 for replica dataset

Organization custom data like replica

RuntimeError: radix_sort: failed on 1st step: cudaErrorInvalidDevice: invalid device ordinal

Train on custom dataset

Any luck running this on a 3000 series GPU?

Owner

James Lin

HashNeRF-pytorch - Pure PyTorch Implementation of NVIDIA paper on Instant Training of Neural Graphics primitives

ALBERT-pytorch-implementation - ALBERT pytorch implementation

The LaTeX and Python code for generating the paper, experiments' results and visualizations reported in each paper is available (whenever possible) in the paper's directory

Official PyTorch implementation for paper Context Matters: Graph-based Self-supervised Representation Learning for Medical Images

A PyTorch re-implementation of the paper 'Exploring Simple Siamese Representation Learning'. Reproduced the 67.8% Top1 Acc on ImageNet.

This is a Pytorch implementation of the paper: Self-Supervised Graph Transformer on Large-Scale Molecular Data.

Official pytorch implementation of paper "Image-to-image Translation via Hierarchical Style Disentanglement".

PyTorch implementation of paper "Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes", CVPR 2021

Official pytorch implementation of paper "Inception Convolution with Efficient Dilation Search" (CVPR 2021 Oral).

Official implementation of our paper "LLA: Loss-aware Label Assignment for Dense Pedestrian Detection" in Pytorch.

An implementation of Geoffrey Hinton's paper "How to represent part-whole hierarchies in a neural network" in Pytorch.

Official implementation of our CVPR2021 paper "OTA: Optimal Transport Assignment for Object Detection" in Pytorch.

This is the official PyTorch implementation of the paper "TransFG: A Transformer Architecture for Fine-grained Recognition" (Ju He, Jie-Neng Chen, Shuai Liu, Adam Kortylewski, Cheng Yang, Yutong Bai, Changhu Wang, Alan Yuille).

[PyTorch] Official implementation of CVPR2021 paper "PointDSC: Robust Point Cloud Registration using Deep Spatial Consistency". https://arxiv.org/abs/2103.05465

PyTorch implementation of the Deep SLDA method from our CVPRW-2020 paper "Lifelong Machine Learning with Deep Streaming Linear Discriminant Analysis"

The official pytorch implementation of our paper "Is Space-Time Attention All You Need for Video Understanding?"

PyTorch 1.5 implementation for paper DECOR-GAN: 3D Shape Detailization by Conditional Refinement.

Pytorch implementation of our paper under review — Lottery Jackpots Exist in Pre-trained Models

Official Pytorch Implementation of: "ImageNet-21K Pretraining for the Masses"(2021) paper