HashNeRF-pytorch - Pure PyTorch Implementation of NVIDIA paper on Instant Training of Neural Graphics primitives

Overview

HashNeRF-pytorch

Instant-NGP recently introduced a Multi-resolution Hash Encoding for neural graphics primitives like NeRFs. The original NVIDIA implementation mainly in C++/CUDA, based on tiny-cuda-nn, can train NeRFs upto 100x faster!

This project is a pure PyTorch implementation of Instant-NGP, built with the purpose of enabling AI Researchers to play around and innovate further upon this method.

This project is built on top of the super-useful NeRF-pytorch implementation.

Convergence speed w.r.t. Vanilla NeRF

HashNeRF-pytorch (left) vs NeRF-pytorch (right):

Chair.Convergence.mp4

After training for just 5k iterations (~10 minutes on a single 1050Ti), you start seeing a crisp chair rendering. :)

Instructions

Download the nerf-synthetic dataset from here: Google Drive.

To train a chair HashNeRF model:

python run_nerf.py --config configs/chair.txt --finest_res 512 --log2_hashmap_size 19 --lrate 0.01 --lrate_decay 10

To train for other objects like ficus/hotdog, replace configs/chair.txt with configs/{object}.txt:

hotdog_ficus

Extras

The code-base has additional support for:

  • Total Variation Loss for smoother embeddings (use --tv-loss-weight to enable)
  • Sparsity-inducing loss on the ray weights (use --sparse-loss-weight to enable)

TODO:

  • Voxel pruning during training and/or inference
  • Accelerated ray tracing, early ray termination

Citation

Kudos to Thomas Müller and the NVIDIA team for this amazing work, that will greatly help accelerate Neural Graphics research:

@article{mueller2022instant,
    title = {Instant Neural Graphics Primitives with a Multiresolution Hash Encoding},
    author = {Thomas M\"uller and Alex Evans and Christoph Schied and Alexander Keller},
    journal = {arXiv:2201.05989},
    year = {2022},
    month = jan
}

Also, thanks to Yen-Chen Lin for the super-useful NeRF-pytorch:

@misc{lin2020nerfpytorch,
  title={NeRF-pytorch},
  author={Yen-Chen, Lin},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished={\url{https://github.com/yenchenlin/nerf-pytorch/}},
  year={2020}
}

If you find this project useful, please consider to cite:

@misc{bhalgat2022hashnerfpytorch,
  title={HashNeRF-pytorch},
  author={Yash Bhalgat},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished={\url{https://github.com/yashbhalgat/HashNeRF-pytorch/}},
  year={2022}
}
Comments
  • Sparsity regulariser not implemented correctly?

    Sparsity regulariser not implemented correctly?

    Here you use the weights of the ray to compute entropy. These weights however are un-normalized, so the calculation of entropy isn't right. I think that either normalizing the weights, or just passing them as logits rather than probs should be correct.

    opened by reynoldscem 8
  • some errors in code

    some errors in code

    File "/data/ytr/conda/envs/direct/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl return forward_call(input, **kwargs) File "/data/ytr/HashNeRF-pytorch/hash_encoding.py", line 63, in forward voxel_min_vertex, voxel_max_vertex, hashed_voxel_indices = get_voxel_vertices(
    File "/data/ytr/HashNeRF-pytorch/utils.py", line 83, in get_voxel_vertices hashed_voxel_indices = hash(voxel_indices, log2_hashmap_size) File "/data/ytr/HashNeRF-pytorch/utils.py", line 18, in hash return ((1<<log2_hashmap_size)-1) & (x
    73856093 ^ y19349663 ^ z83492791) TypeError: unsupported operand type(s) for &: 'int' and 'Tensor'

    Can you tell me how to solve it?

    opened by taoranyi 6
  • TypeError: unsupported operand type(s) for -: 'numpy.ndarray' and 'Tensor'  when runnerf on real images

    TypeError: unsupported operand type(s) for -: 'numpy.ndarray' and 'Tensor' when runnerf on real images

    hi, hashNerf works well on synthetic images, however, it doesn't work on nerf_liff_data, such as fern. I run
    python3 run_nerf.py --config configs/fern.txt --finest_res 512 --log2_hashmap_size 19 --lrate 0.01 --lrate_decay 10 it do can train. but when it supposed to render, i guess, there's an error. TypeError: unsupported operand type(s) for -: 'numpy.ndarray' and 'Tensor'

    Uploading image.png…

    and line 169 is Uploading image.png…

    did anybody meet the same error, help me thanks!!! orz

    opened by Quyans 4
  • Tiny-cuda-nn bindings

    Tiny-cuda-nn bindings

    Hello, thank you for releasing this really usefull code! I have been wondering if you and your team have done any test (or think of any integration) with the tiny-cuda-nn bindings already released by NVIDIA. And if, maybe, it could speed-up performance.

    opened by danperazzo 2
  • fix no_batching=False case error

    fix no_batching=False case error

    Hi @yashbhalgat, Thank you for this great work!

    I just inspect the #10, then figure out is caused by the configuration file doesn't have no_batching = True option. I change the code only for the psnr calculation part because I'm afraid that other change might degrade performance. I check that the learning and rendering is OK now.

    here is a rendered images after 2000 iteration(again, this is really great work!). 000

    Thanks!

    opened by sangminkim-99 1
  • about tv loss

    about tv loss

    new learner here. 2 questions:

    1. what is the suitable weight for tv loss ? (default is 1e-6, I think it is too small)
    2. dose official instant-ngp use tv loss?
    opened by Feynman1999 1
  • TypeError: unsupported operand type(s) for -: 'numpy.ndarray' and 'Tensor'  when runnerf on real images

    TypeError: unsupported operand type(s) for -: 'numpy.ndarray' and 'Tensor' when runnerf on real images

    hi, hashNerf works well on synthetic images, however, it doesn't work on nerf_liff_data, such as fern. I run
    python3 run_nerf.py --config configs/fern.txt --finest_res 512 --log2_hashmap_size 19 --lrate 0.01 --lrate_decay 10 it do can train. but when it supposed to render, i guess, there's an error. TypeError: unsupported operand type(s) for -: 'numpy.ndarray' and 'Tensor'

    Uploading image.png…

    and line 169 is Uploading image.png…

    did anybody meet the same error, help me thanks!!! orz

    opened by Quyans 1
  • Can the checkpoint be loaded in instant-ngp's viewer (or any interactive viewer) ?

    Can the checkpoint be loaded in instant-ngp's viewer (or any interactive viewer) ?

    Hey, Great work on the repo!

    Just wanted to ask if there is any straightforward way to load the checkpoint into the original instant-ngp viewer? or if there are any recommendations for generating novel views as one controllably moves around the scene.

    opened by amanshenoy 1
  • The meaning behind the hash function params and their difference from paper?

    The meaning behind the hash function params and their difference from paper?

    Hi, I noticed the hashing is different from the paper, and the prime numbers are also different from the paper. Could you explain what's the meaning of your implementation and the design choice? torch.tensor((1<<log2_hashmap_size)-1) & (x*73856093 ^ y*19349663 ^ z*83492791)

    opened by Guptajakala 1
  • Instant NGP

    Instant NGP

    Hello, thank for the work. You have mentioned in the paper that this work is orthogonal to works that leverage hybrid or explicit grid-based representations for view synthesis. (Like instant ngp I believe). Did you made some experiments to combine with their codebase? Does it make sense to go in this direction?

    opened by snarb 0
  • Training on real bigger scene

    Training on real bigger scene

    I trained this model on the classroom dataset. It has perfect input view reconstructions but zero generalization. so moving the camera even a tiny step will break the novel view reconstruction. This scene is of size about 10x10x8 meters and the images are omnidirectional. The size of the scene is about 4 times bigger than the synthetic datasets (e.g. the Lego scene) that are used here. Is there something to consider when training on bigger scenes, or is just this model not capable of handing bigger scenes?

    opened by ghasemikasra39 3
  • Another PyTorch only NGP implementation

    Another PyTorch only NGP implementation

    Hey there,

    I've recently come across your work on NGP via an issue at the official instant-ngp. Great work!

    I am working on a very similar project. I have added acceleration structures and dynamic background training in spirit (but not exactly the same) of instant-ngp, however my PSNR results are constantly below the values reported by ngp (on synthetic data: 27dB vs 35dB). Did you experience similar issues?

    Here's my repo in case you want to browse it: https://github.com/cheind/pure-torch-ngp

    opened by cheind 0
  • How does the trilinear interpolation work if the dimensions don't match?

    How does the trilinear interpolation work if the dimensions don't match?

    Hey, i am stuck with this error at the function: def trilinear_interp(self, x, voxel_min_vertex, voxel_max_vertex, voxel_embedds) in class HashEmbedder(nn.Module):

    RuntimeError: The size of tensor a (2) must match the size of tensor b (3) at non-singleton dimension 2 Emerging at the following line:

    c00 = voxel_embedds[:,0]*(1-weights[:,0][:,None]) + voxel_embedds[:,4]*weights[:,0][:,None]

    My input was a 3D coordinate, [X, Y, Z], and I do not see where I can reshape or change, because its the voxel embedding that is the problem:

    Considering an input with 3 dimensions, and the hash embedding having 2 dimensions, how does this even work? The following is the specs in the function itself:

    • x: B x 3
    • voxel_min_vertex: B x 3
    • voxel_max_vertex: B x 3
    • voxel_embedds: B x 8 x 2
    • weights = (x - voxel_min_vertex)/(voxel_max_vertex-voxel_min_vertex) # B x 3

    Please help!

    opened by jayrmh 4
  • SDF experiments of InstantNGP

    SDF experiments of InstantNGP

    I am adding a tracking issue for porting over the SDF experiments from InstantNGP.

    In case there is any interest or people have tried to implement it.

    enhancement help wanted 
    opened by Skylion007 0
  • Training is very slow.

    Training is very slow.

    After training for just 5k iterations (~10 minutes on a single 1050Ti), you start seeing a crisp chair rendering. :).But I trained 5000 times on Tesla M60 6G, which took about an hour in total.Can you help me?

    opened by LaolangOne 2
Owner
Yash Sanjay Bhalgat
The disruptive contributor.
Yash Sanjay Bhalgat
General purpose GPU compute framework for cross vendor graphics cards (AMD, Qualcomm, NVIDIA & friends)

General purpose GPU compute framework for cross vendor graphics cards (AMD, Qualcomm, NVIDIA & friends). Blazing fast, mobile-enabled, asynchronous and optimized for advanced GPU data processing usecases. Backed by the Linux Foundation.

The Kompute Project 1k Jan 6, 2023
Torch-ngp - A pytorch implementation of the hash encoder proposed in instant-ngp

HashGrid Encoder (WIP) A pytorch implementation of the HashGrid Encoder from ins

hawkey 1k Jan 1, 2023
Instant-nerf-pytorch - NeRF trained SUPER FAST in pytorch

instant-nerf-pytorch This is WORK IN PROGRESS, please feel free to contribute vi

null 94 Nov 22, 2022
PyTorch implementation of the Quasi-Recurrent Neural Network - up to 16 times faster than NVIDIA's cuDNN LSTM

Quasi-Recurrent Neural Network (QRNN) for PyTorch Updated to support multi-GPU environments via DataParallel - see the the multigpu_dataparallel.py ex

Salesforce 1.3k Dec 28, 2022
Optimized primitives for collective multi-GPU communication

NCCL Optimized primitives for inter-GPU communication. Introduction NCCL (pronounced "Nickel") is a stand-alone library of standard communication rout

NVIDIA Corporation 2k Jan 9, 2023
NVIDIA Merlin is an open source library providing end-to-end GPU-accelerated recommender systems, from feature engineering and preprocessing to training deep learning models and running inference in production.

NVIDIA Merlin NVIDIA Merlin is an open source library designed to accelerate recommender systems on NVIDIA’s GPUs. It enables data scientists, machine

null 419 Jan 3, 2023
Instant Real-Time Example-Based Style Transfer to Facial Videos

FaceBlit: Instant Real-Time Example-Based Style Transfer to Facial Videos The official implementation of FaceBlit: Instant Real-Time Example-Based Sty

Aneta Texler 131 Dec 19, 2022
Instant-Teaching: An End-to-End Semi-Supervised Object Detection Framework

This repo is the official implementation of "Instant-Teaching: An End-to-End Semi-Supervised Object Detection Framework". @inproceedings{zhou2021insta

null 34 Dec 31, 2022
NPBG++: Accelerating Neural Point-Based Graphics

[CVPR 2022] NPBG++: Accelerating Neural Point-Based Graphics Project Page | Paper This repository contains the official Python implementation of the p

Ruslan Rakhimov 57 Dec 3, 2022
Lunar is a neural network aimbot that uses real-time object detection accelerated with CUDA on Nvidia GPUs.

Lunar Lunar is a neural network aimbot that uses real-time object detection accelerated with CUDA on Nvidia GPUs. About Lunar can be modified to work

Zeyad Mansour 276 Jan 7, 2023
A pure PyTorch implementation of the loss described in "Online Segment to Segment Neural Transduction"

ssnt-loss ℹ️ This is a WIP project. the implementation is still being tested. A pure PyTorch implementation of the loss described in "Online Segment t

張致強 1 Feb 9, 2022
The pure and clear PyTorch Distributed Training Framework.

The pure and clear PyTorch Distributed Training Framework. Introduction Requirements and Usage Dependency Dataset Basic Usage Slurm Cluster Usage Base

WILL LEE 208 Dec 20, 2022
Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data

Real-ESRGAN Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data Ported from https://github.com/xinntao/Real-ESRGAN Depend

Holy Wu 44 Dec 27, 2022
The official PyTorch implementation of recent paper - SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training

This repository is the official PyTorch implementation of SAINT. Find the paper on arxiv SAINT: Improved Neural Networks for Tabular Data via Row Atte

Gowthami Somepalli 284 Dec 21, 2022
This repository contains notebook implementations of the following Neural Process variants: Conditional Neural Processes (CNPs), Neural Processes (NPs), Attentive Neural Processes (ANPs).

The Neural Process Family This repository contains notebook implementations of the following Neural Process variants: Conditional Neural Processes (CN

DeepMind 892 Dec 28, 2022
🦕 NanoSaur is a little tracked robot ROS2 enabled, made for an NVIDIA Jetson Nano

?? nanosaur NanoSaur is a little tracked robot ROS2 enabled, made for an NVIDIA Jetson Nano Website: nanosaur.ai Do you need an help? Discord For tech

NanoSaur 162 Dec 9, 2022
The 1st place solution of track2 (Vehicle Re-Identification) in the NVIDIA AI City Challenge at CVPR 2021 Workshop.

AICITY2021_Track2_DMT The 1st place solution of track2 (Vehicle Re-Identification) in the NVIDIA AI City Challenge at CVPR 2021 Workshop. Introduction

Hao Luo 91 Dec 21, 2022
Tutorial on active learning with the Nvidia Transfer Learning Toolkit (TLT).

Active Learning with the Nvidia TLT Tutorial on active learning with the Nvidia Transfer Learning Toolkit (TLT). In this tutorial, we will show you ho

Lightly 25 Dec 3, 2022
Nvidia Semantic Segmentation monorepo

Paper | YouTube | Cityscapes Score Pytorch implementation of our paper Hierarchical Multi-Scale Attention for Semantic Segmentation. Please refer to t

NVIDIA Corporation 1.6k Jan 4, 2023