NeRF Meta-Learning with PyTorch

Related tags

Deep Learning nerf meta-learning view-synthesis neural-radiance-fields
Overview

NeRF Meta Learning With PyTorch

nerf-meta is a PyTorch re-implementation of NeRF experiments from the paper "Learned Initializations for Optimizing Coordinate-Based Neural Representations". Simply by initializing NeRF with meta-learned weights, we can achieve:

Be sure to check out the original resources from the authors:

Environment

  • Python 3.8
  • PyTorch 1.8
  • NumPy, imageio, imageio-ffmpeg

Photo Tourism

Starting from a meta-initialized NeRF, we can interpolate between camera pose, focal length, aspect ratio and scene appearance. The videos below are generated with a 5 layer only NeRF, trained for ~100k iterations.

BrandenburgGate.mp4
TreviFountain.mp4

Data

Train and Evaluate

  1. Train NeRF on a single landmark scene using Reptile meta-learning: 
    python tourism_train.py --config ./configs/tourism/$landmark.json
  2. Test Photo Tourism performance and generate an interpolation video of the landmark: 
    python tourism_test.py --config ./configs/tourism/$landmark.json --weight-path $meta_weight.pth

View Synthesis from Single Image

Given a single input view, meta-initialized NeRF can generate a 360-degree video. The following ShapeNet video is generated with a class-specific NeRF (5 layers deep), trained for ~100k iterations.

ShapeNet.mp4

Data

Train and Evaluate

  1. Train NeRF on a particular ShapeNet class using Reptile meta-learning: 
    python shapenet_train.py --config ./configs/shapenet/$shape.json
  2. Optimize the meta-trained model on a single view and test on other held-out views. It also generates a 360 video for each test object: 
    python shapenet_test.py --config ./configs/shapenet/$shape.json --weight-path $meta_weight.pth

Acknowledgments

I referenced several open-source NeRF and Meta-Learning code base for this implementation. Specifically, I borrowed/modified code from the following repositories:

Thanks to the authors for releasing their code.

Comments
  • White space

    White space

    Hello, thank you for releasing this implementation. I tried to use the scripts for view synthesis from a single image. I am running with CUDA 10.1 in a Ubuntu 16.04. I am using the lamps set of images and the lamps config script. However, when I end training, the resulting video is blank, with only a white void.

    opened by danperazzo 2
  • Data for Photorealism

    Data for Photorealism

    Hello, thank you for sharing the code for this wonderfull research! Is it possible to share the data for the Photorealism data used during the experiments?

    opened by danperazzo 2
  • pre-trained meta models

    pre-trained meta models

    HI. Thanks for your pytorch code. Can you share the pre-trained meta models? Besides, I found the training is not very stable, sometimes the val PSNR keeps the same number while training and the test results are just some lumps.

    opened by zhangmaoxiansheng 2
  • some question about ShapeNet dataset

    some question about ShapeNet dataset

    Excuse, I would like to ask about the construction process of this dataset. If I want to construct other classes objects of shapenet, what should I do? Thanks!

    opened by LiXinghui-666 1
  • Issue about Single image view synthesis for ShapeNet

    Issue about Single image view synthesis for ShapeNet

    I would like to ask about your experimental part in Single image view synthesis for ShapeNet, you are using only one image in the meta-learning stage under the setting of SV Meta, which means that only one image is used for the whole training process to generate the class 360 degree reconstruction effect for all instances？

    opened by LiXinghui-666 1
  • Some questions about meta learning theory

    Some questions about meta learning theory

    Hi, I'm a beginnger for meta-learning. As I know, it is used for fitting model across different data. In each epoch for meta-learning, there are a batch of tasks and we should make copies of model for each task. For example, to meta-learn the shapenet dataset, in each epoch, we select several cars to process, thus the dimension of dataflow should be [b, ...], while b is the batch size. So, we need to make b copies of basic model so that each model can fit a particular car. However, I find that you only make one copy for the model in the inner loop(copy.deepcopy). Could you please give an explanation here? Do I misunderstand it?

    opened by 1999kevin 0
Owner
Sanowar Raihan
EE Undergrad @ Bangladesh University of Engineering and Technology
Sanowar Raihan
GitHub
Implementation of "Meta-rPPG: Remote Heart Rate Estimation Using a Transductive Meta-Learner"

Meta-rPPG: Remote Heart Rate Estimation Using a Transductive Meta-Learner This repository is the official implementation of Meta-rPPG: Remote Heart Ra

Eugene Lee 137 Dec 13, 2022
Pytorch implementation for A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose

A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose Paper | Website | Data A-NeRF: Articulated Neural Radiance F

Shih-Yang Su 172 Dec 22, 2022
This repository contains a PyTorch implementation of "AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis".

AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis | Project Page | Paper | PyTorch implementation for the paper "AD-NeRF: Audio

null 551 Dec 29, 2022
A PyTorch implementation of NeRF (Neural Radiance Fields) that reproduces the results.

NeRF-pytorch NeRF (Neural Radiance Fields) is a method that achieves state-of-the-art results for synthesizing novel views of complex scenes. Here are

Yen-Chen Lin 3.2k Jan 8, 2023
Unofficial pytorch-lightning implement of Mip-NeRF

mipnerf_pl Unofficial pytorch-lightning implement of Mip-NeRF, Here are some results generated by this repository (pre-trained models are provided bel

Jianxin Huang 159 Dec 23, 2022
This repository contains a pytorch implementation of "HeadNeRF: A Real-time NeRF-based Parametric Head Model (CVPR 2022)".

HeadNeRF: A Real-time NeRF-based Parametric Head Model This repository contains a pytorch implementation of "HeadNeRF: A Real-time NeRF-based Parametr

null 294 Jan 1, 2023
PyTorch implementation of the supervised learning experiments from the paper Model-Agnostic Meta-Learning (MAML)

pytorch-maml This is a PyTorch implementation of the supervised learning experiments from the paper Model-Agnostic Meta-Learning (MAML): https://arxiv

Kate Rakelly 516 Jan 5, 2023
(Arxiv 2021) NeRF--: Neural Radiance Fields Without Known Camera Parameters

NeRF--: Neural Radiance Fields Without Known Camera Parameters Project Page | Arxiv | Colab Notebook | Data Zirui Wang¹, Shangzhe Wu², Weidi Xie², Min

Active Vision Laboratory 411 Dec 26, 2022
PlenOctrees: NeRF-SH Training & Conversion

PlenOctrees Official Repo: NeRF-SH training and conversion This repository contains code to train NeRF-SH and to extract the PlenOctree, constituting

Alex Yu 323 Dec 29, 2022
Unofficial & improved implementation of NeRF--: Neural Radiance Fields Without Known Camera Parameters

[Unofficial code-base] NeRF--: Neural Radiance Fields Without Known Camera Parameters [ Project | Paper | Official code base ] ⬅️ Thanks the original

Jianfei Guo 239 Dec 22, 2022
Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields.

This repository contains the code release for Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields. This implementation is written in JAX, and is a fork of Google's JaxNeRF implementation. Contact Jon Barron if you encounter any issues.

Google 625 Dec 30, 2022
Code release for DS-NeRF (Depth-supervised Neural Radiance Fields)

Depth-supervised NeRF: Fewer Views and Faster Training for Free Project | Paper | YouTube Pytorch implementation of our method for learning neural rad

null 524 Jan 8, 2023
Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis Implementation

Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis Implementation This project attempted to implement the paper Putting NeRF on a

null 254 Dec 27, 2022
NeRF visualization library under construction

NeRF visualization library using PlenOctrees, under construction pip install nerfvis Docs will be at: https://nerfvis.readthedocs.org import nerfvis s

Alex Yu 196 Jan 4, 2023
D-NeRF: Neural Radiance Fields for Dynamic Scenes

D-NeRF: Neural Radiance Fields for Dynamic Scenes [Project] [Paper] D-NeRF is a method for synthesizing novel views, at an arbitrary point in time, of

Albert Pumarola 291 Jan 2, 2023
Code release for NeRF (Neural Radiance Fields)

NeRF: Neural Radiance Fields Project Page | Video | Paper | Data Tensorflow implementation of optimizing a neural representation for a single scene an

null 6.5k Jan 1, 2023
A minimal TPU compatible Jax implementation of NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis

NeRF Minimal Jax implementation of NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. Result of Tiny-NeRF RGB Depth

Soumik Rakshit 11 Jul 24, 2022
Build upon neural radiance fields to create a scene-specific implicit 3D semantic representation, Semantic-NeRF

Semantic-NeRF: Semantic Neural Radiance Fields Project Page | Video | Paper | Data In-Place Scene Labelling and Understanding with Implicit Scene Repr

Shuaifeng Zhi 243 Jan 7, 2023
Point-NeRF: Point-based Neural Radiance Fields

Point-NeRF: Point-based Neural Radiance Fields Project Sites | Paper | Primary c

Qiangeng Xu 662 Jan 1, 2023