Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition - NeurIPS2021

Related tags

Deep Learning Neural-PIL
Overview

Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition

Project Page | Video | Paper

Implementation for Neural-PIL. A novel method which decomposes multiple images into shape, BRDF and illumination with a split-sum preintegrated illumination network.

Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition
Mark Boss1, Varun Jampani2, Raphael Braun1, Ce Liu3, Jonathan T. Barron2, Hendrik P. A. Lensch1
1University of Tübingen, 2Google Research, 3Microsoft Azure AI (work done at Google)

Setup

A conda environment is used for dependency management

conda env create -f environment.yml
conda activate neuralpil

Running

python train_neural_pil.py --datadir [DIR_TO_DATASET_FOLDER] --basedir [TRAIN_DIR] --expname [EXPERIMENT_NAME] --gpu [COMMA_SEPARATED_GPU_LIST]

Specific Arguments per Dataset

Most setup is handled by configurations files located in configs/neural_pil/.

Our Synthethic Scenes

--config configs/neural_pil/blender.txt

NeRF Synthethic Scenes

--config configs/neural_pil/nerf_blender.txt

Real-World

--config configs/neural_pil/real_world.txt

Often objects are captured in a spherical manner and the flag --spherify should be applied for those scenes.

Datasets

All datasets are taken from NeRD.

Run Your Own Data

Mainly camera poses and segmentation masks are required. For the poses the scripts from NeRF are used to prepare the scenes. The dataset then needs to be put in the following file structure:

images/
    [IMG_NAME_1].jpg
    [IMG_NAME_2].jpg
    ...
masks/
    [IMG_NAME_1].jpg
    [IMG_NAME_2].jpg
    ...
poses_bounds.npy

The poses_bounds.npy is generated from the LLFF script.

Evaluation

The train_neural_pil.py can be called with a --render_only flag and the --config flag pointing to the args.txt of the experiments folder.

Citation

@inproceedings{boss2021neuralpil,
  title         = {Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition},
  author        = {Boss, Mark and Jampani, Varun and Braun, Raphael and Liu, Ce and Barron, Jonathan T. and Lensch, Hendrik P.A.},
  booktitle     = {Advances in Neural Information Processing Systems (NeurIPS)},
  year          = {2021},
}
You might also like...
Revisiting Discriminator in GAN Compression: A Generator-discriminator Cooperative Compression Scheme (NeurIPS2021)
Revisiting Discriminator in GAN Compression: A Generator-discriminator Cooperative Compression Scheme (NeurIPS2021)

Revisiting Discriminator in GAN Compression: A Generator-discriminator Cooperative Compression Scheme (NeurIPS2021) Overview Prerequisites Linux Pytho

Shaojie Li 34 Mar 31, 2022
[NeurIPS2021] Code Release of K-Net: Towards Unified Image Segmentation

K-Net: Towards Unified Image Segmentation Introduction This is an official release of the paper K-Net:Towards Unified Image Segmentation. K-Net will a

Wenwei Zhang 423 Jan 2, 2023
PyTorch implementation of Lip to Speech Synthesis with Visual Context Attentional GAN (NeurIPS2021)
PyTorch implementation of Lip to Speech Synthesis with Visual Context Attentional GAN (NeurIPS2021)

Lip to Speech Synthesis with Visual Context Attentional GAN This repository contains the PyTorch implementation of the following paper: Lip to Speech

null 6 Nov 2, 2022
MAU: A Motion-Aware Unit for Video Prediction and Beyond, NeurIPS2021

MAU (NeurIPS2021) Zheng Chang, Xinfeng Zhang, Shanshe Wang, Siwei Ma, Yan Ye, Xinguang Xiang, Wen GAo. Official PyTorch Code for "MAU: A Motion-Aware

ZhengChang 20 Nov 25, 2022
STYLER: Style Factor Modeling with Rapidity and Robustness via Speech Decomposition for Expressive and Controllable Neural Text to Speech
STYLER: Style Factor Modeling with Rapidity and Robustness via Speech Decomposition for Expressive and Controllable Neural Text to Speech

STYLER: Style Factor Modeling with Rapidity and Robustness via Speech Decomposition for Expressive and Controllable Neural Text to Speech Keon Lee, Ky

Keon Lee 114 Dec 12, 2022
This is the code repository implementing the paper
This is the code repository implementing the paper "TreePartNet: Neural Decomposition of Point Clouds for 3D Tree Reconstruction".

TreePartNet This is the code repository implementing the paper "TreePartNet: Neural Decomposition of Point Clouds for 3D Tree Reconstruction". Depende

刘彦超 34 Nov 30, 2022
Functional TensorFlow Implementation of Singular Value Decomposition for paper Fast Graph Learning

tf-fsvd TensorFlow Implementation of Functional Singular Value Decomposition for paper Fast Graph Learning with Unique Optimal Solutions Cite If you f

Sami Abu-El-Haija 14 Nov 25, 2021
[ICLR 2021] Is Attention Better Than Matrix Decomposition?
[ICLR 2021] Is Attention Better Than Matrix Decomposition?

Enjoy-Hamburger 🍔 Official implementation of Hamburger, Is Attention Better Than Matrix Decomposition? (ICLR 2021) Under construction. Introduction T

Gsunshine 271 Dec 29, 2022
official code for dynamic convolution decomposition

Revisiting Dynamic Convolution via Matrix Decomposition (ICLR 2021) A pytorch implementation of DCD. If you use this code in your research please cons

Yunsheng Li 110 Nov 23, 2022
Comments
  • How to generate SGs for relighting

    How to generate SGs for relighting

    Hi,

    Great work! I have the same issue as NeRD issue. May I ask what script/lib you used to convert .hdr or .exr environment lightmap into your input SG parameters? I also have tried PhySG, but the range of output is different.

    Thanks!

    opened by CorneliusHsiao 0
  • Batch size must be greater than zero error

    Batch size must be greater than zero error

    Hi. I see the following error when trying to train Neural-PIL on the Car scene. Any idea what is going wrong?

    Traceback (most recent call last):
  File "train_neural_pil.py", line 933, in <module>
    main(args)
  File "train_neural_pil.py", line 454, in main
    render_test_example(
  File "train_neural_pil.py", line 884, in render_test_example
    coarse_result, fine_result = model.distributed_call(
  File "/esat/topaz/gkouros/repos/nerf-repos/Neural-PIL/models/neural_pil/neural_pil_model.py", line 186, in distributed_call
    tf.data.Dataset.from_tensor_slices((ray_origins, ray_directions))
  File "/users/visics/gkouros/miniconda3/envs/neuralpil/lib/python3.8/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 1746, in batch
    return BatchDataset(self, batch_size, drop_remainder, name=name)
  File "/users/visics/gkouros/miniconda3/envs/neuralpil/lib/python3.8/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 4953, in __init__
    variant_tensor = gen_dataset_ops.batch_dataset_v2(
  File "/users/visics/gkouros/miniconda3/envs/neuralpil/lib/python3.8/site-packages/tensorflow/python/ops/gen_dataset_ops.py", line 779, in batch_dataset_v2
    _ops.raise_from_not_ok_status(e, name)
  File "/users/visics/gkouros/miniconda3/envs/neuralpil/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 7164, in raise_from_not_ok_status
    raise core._status_to_exception(e) from None  # pylint: disable=protected-access
tensorflow.python.framework.errors_impl.InvalidArgumentError: Batch size must be greater than zero. [Op:BatchDatasetV2]

    Thank you in advance.

    opened by gkouros 0
  • How to train illumination_net and brdf_ae?

    How to train illumination_net and brdf_ae?

    According to the code and paper, illumination_net and brdf_ae are trained before training nerual_pil. The repo do have train_illumination_net.py and train_brdf_ae.py, but I didn't find the data to train them. What's the dataset of illumination_net and brdf_ae training? Besides, it would be great if there is some guide of illumination_net and brdf_ae training in README.

    opened by Woolseyyy 1
  • How are the segmentation masks obtained?

    How are the segmentation masks obtained?

    Hello, thank you for the great work!! I want to try Neural-PIL on my own dataset, could you please let me know how were the segmentation masks generated?

    Thank you for your time.

    opened by liu-vis 3
Owner
Computergraphics (University of Tübingen)
Computergraphics (University of Tübingen)
GitHub
Neural Factorization of Shape and Reflectance Under An Unknown Illumination

NeRFactor [Paper] [Video] [Project] This is the authors' code release for: NeRFactor: Neural Factorization of Shape and Reflectance Under an Unknown I

Google 283 Jan 4, 2023
[NeurIPS2021] Exploring Architectural Ingredients of Adversarially Robust Deep Neural Networks

Exploring Architectural Ingredients of Adversarially Robust Deep Neural Networks Code for NeurIPS 2021 Paper "Exploring Architectural Ingredients of A

Hanxun Huang 26 Dec 1, 2022
Lighting the Darkness in the Deep Learning Era: A Survey, An Online Platform, A New Dataset

Lighting the Darkness in the Deep Learning Era: A Survey, An Online Platform, A New Dataset This repository provides a unified online platform, LoLi-P

Chongyi Li 457 Jan 3, 2023
This is the official repository for evaluation on the NoW Benchmark Dataset. The goal of the NoW benchmark is to introduce a standard evaluation metric to measure the accuracy and robustness of 3D face reconstruction methods from a single image under variations in viewing angle, lighting, and common occlusions.

NoW Evaluation This is the official repository for evaluation on the NoW Benchmark Dataset. The goal of the NoW benchmark is to introduce a standard e

Soubhik Sanyal 71 Dec 30, 2022
A new data augmentation method for extreme lighting conditions.

Random Shadows and Highlights This repo has the source code for the paper: Random Shadows and Highlights: A new data augmentation method for extreme l

Osama Mazhar 35 Nov 26, 2022
Official code for the CVPR 2022 (oral) paper "Extracting Triangular 3D Models, Materials, and Lighting From Images".

nvdiffrec Joint optimization of topology, materials and lighting from multi-view image observations as described in the paper Extracting Triangular 3D

NVIDIA Research Projects 1.4k Jan 1, 2023
A Lighting Pytorch Framework for Recommendation System, Easy-to-use and Easy-to-extend.

Torch-RecHub A Lighting Pytorch Framework for Recommendation Models, Easy-to-use and Easy-to-extend. 安装 pip install torch-rechub 主要特性 scikit-learn风格易用

Mincai Lai 67 Jan 4, 2023
LoveDA: A Remote Sensing Land-Cover Dataset for Domain Adaptive Semantic Segmentation (NeurIPS2021 Benchmark and Dataset Track)

LoveDA: A Remote Sensing Land-Cover Dataset for Domain Adaptive Semantic Segmentation by Junjue Wang, Zhuo Zheng, Ailong Ma, Xiaoyan Lu, and Yanfei Zh

Kingdrone 174 Dec 22, 2022
Deep Markov Factor Analysis (NeurIPS2021)

Deep Markov Factor Analysis (DMFA) Codes and experiments for deep Markov factor analysis (DMFA) model accepted for publication at NeurIPS2021: A. Farn

Sarah Ostadabbas 2 Dec 16, 2022
This codebase is the official implementation of Test-Time Classifier Adjustment Module for Model-Agnostic Domain Generalization (NeurIPS2021, Spotlight)

Test-Time Classifier Adjustment Module for Model-Agnostic Domain Generalization This codebase is the official implementation of Test-Time Classifier A

null 47 Dec 28, 2022