[AAAI 2021] EMLight: Lighting Estimation via Spherical Distribution Approximation and [ICCV 2021] Sparse Needlets for Lighting Estimation with Spherical Transport Loss

Overview

EMLight: Lighting Estimation via Spherical Distribution Approximation (AAAI 2021)

Teaser

Update

Teaser

Prerequisites

  • Linux or macOS
  • Python3, PyTorch
  • CPU or NVIDIA GPU + CUDA CuDNN

Dataset Preparation

Laval Indoor HDR Dataset
Thanks to the intellectual property of Laval Indoor dataset, the original datasets and processed training data can not be released from me. Please get access to the dataset by contacting the dataset creator [email protected].

After getting the dataset, the raw illumination map can be processed to generate the training data of the regression network as below:

cd RegressionNetwork/representation/
python3 distribution_representation.py

Pretrained Models

The pretrained regression model of EMLight (96 anchor points, without depth branch) as well as pretrained densenet-121 can be downloaded from Google Drive. Saving the pretrained models in RegressionNetwork/checkpoints. The model parameters should be adjusted accordingly for inference.

Training

Then run the command

cd RegressionNetwork/
python3 train.py

Training tip1: you may overfit the model on a small subset first, then train the model on the full set, to avoid divergence during training.

Training tip2: you can try to reduce the number of anchor points (e.g., 96) in the model, which helps to converge during training.

Virtual Object Insertion & Rendering

To evaluate the performance of lighting estimation, we create a Virtual Object Relighting (VOR) dataset to conduct object insertion & rendering in Blender. The lighting estimaiton performance is evaluated by using the predicted illumination map as the environment light in Blender.

The background scenes of this set include images from Laval Indoor HDR, Fast Spatially-Varying Indoor, and some wild scenes. This dataset can be downloaded from Google Drive.

Teaser

Citation

If you use this code for your research, please cite our papers.

@inproceedings{zhan2021emlight,
  title={EMLight: Lighting Estimation via Spherical Distribution Approximation},
  author={Zhan, Fangneng and Zhang, Changgong and Yu, Yingchen and Chang, Yuan and Lu, Shijian and Ma, Feiying and Xie, Xuansong},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  year={2021}
}
@inproceedings{zhan2021emlight,
  title={Sparse Needlets for Lighting Estimation with Spherical Transport Loss},
  author={Zhan, Fangneng and Zhang, Changgong and Hu, Wenbo and Lu, Shijian and Ma, Feiying and Xie, Xuansong and Shao, Ling},
  booktitle={Proceedings of the IEEE International Conference on Computer Vision},
  year={2021}
}
Comments
  • about supplementary file & daatset file structure

    about supplementary file & daatset file structure

    Hi!

    1. In the paper, you mentioned 'Detailed network structure of neural projector and the training settings are provided in the supplementary file.', but I can not fild the supplementary file. Would you like to tell me where is the supplementary file?

    2. What's more, I tried to train my own dataset with your code for comparison, but I don't know the original file structure so I got some problems. Could you tell me the structure of the original file and the format of the images in the folder?

    Thank you very much!

    opened by liuyuwan233 4
  • depth information

    depth information

    hi ,

    i saw your code in data.py have ['depth'] term, but not in distribution_representation.py pkl's save para. Is the 'depth' used in training? and how can i get the 'depth' information.

    thanks!

    opened by ad45675 3
  • The input parameters for training of the Neural Projector of EMLight

    The input parameters for training of the Neural Projector of EMLight

    Hi fnzhan: Could you please tell me which illumination map parameters you used as input during the training of the Neural Projector of EMlight, the parameters predicted by Regression network or the parameters of Ground Truth ?

    opened by Lingstreasure 3
  • dataset for training

    dataset for training

    Hi,

    congrats for the great work!

    I'm trying to run the training code and I'm getting the following error:

    (base) root@bd14969643f5:~/codes/Illumination-Estimation/RegressionNetwork# CUDA_VISIBLE_DEVICES=2 python train.py
      + Number of params: 9.50M
    0 optim: 0.001
    Traceback (most recent call last):
      File "train.py", line 68, in <module>
        for i, para in enumerate(dataloader):
      File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 435, in __next__
        data = self._next_data()
      File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 475, in _next_data
        data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
      File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
        data = [self.dataset[idx] for idx in possibly_batched_index]
      File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
        data = [self.dataset[idx] for idx in possibly_batched_index]
      File "/root/codes/Illumination-Estimation/RegressionNetwork/data.py", line 68, in __getitem__
        training_pair['depth'] = torch.from_numpy(gt['depth']).float()
    KeyError: 'depth'
    

    I generated the dataset using distribution_representation.py, but I could not find anywhere the depth being added to the list.

    Thanks for the help!

    opened by weberhen 3
  • Question about the train/test split

    Question about the train/test split

    Dear authors,

    Thanks for opening source your code, and it is really helpful. I have a question about the train/test split of the Laval HDR dataset when I read the paper. It says that 19,556 training pairs are generated from 2,100 HDR panoramas, and 200 images are randomly selected as the test set. I am not sure if the 200 images are selected from the 2,100 HDR panoramas, or just from the 19,556 generated images. If they are from the 19,556 images, how to make sure the corresponding HDR panoramas of them are not in the training set or not seen during the training process?

    Best regards, Hao XU

    opened by hxwork 2
  • something about the output of regression

    something about the output of regression

    Forgive me, my level is not enough, the input of the first stage of the model is a local picture, then its output is a local light, but I see from the code it should be global, there is no position information, how to achieve this

    opened by Calsia 2
  • benchmark in GMLight

    benchmark in GMLight

    Hi, fnZhang. Do you reproduce the paper “Fast Spatially-Varying Indoor Lighting Estimation“ as reference, or in any other ways to get the predictions.

    opened by yaodao-x 2
  • Running inference only?

    Running inference only?

    I have been trying to get the code to work on generate a rough HDRI from an image input but have had no luck so far, I have downloaded the pretrained model but I am not entirely sure where to start. Can anyone help me?

    opened by DaLizardWizard 1
  • how to render the three different material sphere with the hdr lighting

    how to render the three different material sphere with the hdr lighting

    hi, I have some question about the evaluations matrix. The matrix in the paper was to caculating the loss of render the three different material sphere with predicted light. how to render the three different material sphere with the hdr lighting??

    opened by ad45675 1
  • Why tone Fov image ?

    Why tone Fov image ?

    Hello!

    I found that you use a TonemapHDR(..) function to tone your images in '\crop' folder. Is it mean that images in '\crop' are just cut from hdr, without any tone process ?

    opened by jxl0131 1
  • regression model training lr

    regression model training lr

    Dear zhan: I am training the regression ,now I happend to the probelms that the intensity output of '*.pth' model is positive number, however convered the onnx model output is negative number. would you like help me sovle the problem?

    opened by cyjouc 1
  • three spheres with different materials

    three spheres with different materials

    hi, i have some question about evaluation.

    The paper mention that the scenes used in evaluations consist of three spheres with different materials including diffuse gray, matte silver and mirror silver.

    Was it bulided with blender?

    if it was builded with blender , can you provide the parameters of three different materials ball?

    opened by ad45675 0
  • How to reconstruct environment map?

    How to reconstruct environment map?

    envmap

    I tried both train and test code on Laval Indoor dataset, and from the test code, I get one result as the image shown above. It seems like the Guassian Map that mentioned in the paper, and I wonder how to reconstruct the environment map from this image.

    Thank you so much for your excellent work and I look forward to any reply.

    opened by a978908609 3
  • evaluation metrix

    evaluation metrix

    HI~I have some question about REMS and si-REMS. I know the matrix, but i don't know which parameter will be calculated. is the illumination map? I also want to ask that how to calculate the Angular Error.

    thanks!

    opened by ad45675 2
  • Why delete the sigmoid in the output layers?

    Why delete the sigmoid in the output layers?

    Thanks for your wonderful work. I wonder why you delete the sigmoid in the output layers in DenseNet.py in the latter version? And I guess it will be reasonable if self.fc_dist(out) is followed by softmax , since the sum of gt_distribution is one .(https://github.com/fnzhan/Illumination-Estimation/blob/master/RegressionNetwork/DenseNet.py)

    opened by tongxueqing 3
  • Some error when I run distribution_representation.py

    Some error when I run distribution_representation.py

    Traceback (most recent call last): File "distribution_representation.py", line 145, in para, map = extractor.compute(hdr) File "distribution_representation.py", line 94, in compute hdr = self.steradian * hdr ValueError: operands could not be broadcast together with shapes (128,256,1) (1024,2048,3)

    When I run distribution_representation.py, I encounter the above error, how can I solve it, thank you!

    opened by xjsxjs 4
Owner
Fangneng Zhan
Computer Vision, Deep Learning.
Fangneng Zhan
Differentiable Neural Computers, Sparse Access Memory and Sparse Differentiable Neural Computers, for Pytorch

Differentiable Neural Computers and family, for Pytorch Includes: Differentiable Neural Computers (DNC) Sparse Access Memory (SAM) Sparse Differentiab

ixaxaar 302 Dec 14, 2022
Official repository of the paper "A Variational Approximation for Analyzing the Dynamics of Panel Data". Mixed Effect Neural ODE. UAI 2021.

Official repository of the paper (UAI 2021) "A Variational Approximation for Analyzing the Dynamics of Panel Data", Mixed Effect Neural ODE. Panel dat

Jurijs Nazarovs 7 Nov 26, 2022
Non-Homogeneous Poisson Process Intensity Modeling and Estimation using Measure Transport

Non-Homogeneous Poisson Process Intensity Modeling and Estimation using Measure Transport This GitHub page provides code for reproducing the results i

Andrew Zammit Mangion 1 Nov 8, 2021
Recall Loss for Semantic Segmentation (This repo implements the paper: Recall Loss for Semantic Segmentation)

Recall Loss for Semantic Segmentation (This repo implements the paper: Recall Loss for Semantic Segmentation) Download Synthia dataset The model uses

null 32 Sep 21, 2022
An implementation for the loss function proposed in Decoupled Contrastive Loss paper.

Decoupled-Contrastive-Learning This repository is an implementation for the loss function proposed in Decoupled Contrastive Loss paper. Requirements P

Ramin Nakhli 71 Dec 4, 2022
CrossNorm and SelfNorm for Generalization under Distribution Shifts (ICCV 2021)

CrossNorm (CN) and SelfNorm (SN) (Accepted at ICCV 2021) This is the official PyTorch implementation of our CNSN paper, in which we propose CrossNorm

null 100 Dec 28, 2022
CrossNorm and SelfNorm for Generalization under Distribution Shifts (ICCV 2021)

CrossNorm (CN) and SelfNorm (SN) (Accepted at ICCV 2021) This is the official PyTorch implementation of our CNSN paper, in which we propose CrossNorm

null 100 Dec 28, 2022
Code for paper "Vocabulary Learning via Optimal Transport for Neural Machine Translation"

**Codebase and data are uploaded in progress. ** VOLT(-py) is a vocabulary learning codebase that allows researchers and developers to automaticaly ge

null 416 Jan 9, 2023
The Official Implementation of the ICCV-2021 Paper: Semantically Coherent Out-of-Distribution Detection.

SCOOD-UDG (ICCV 2021) This repository is the official implementation of the paper: Semantically Coherent Out-of-Distribution Detection Jingkang Yang,

Jake YANG 62 Nov 21, 2022
A bare-bones TensorFlow framework for Bayesian deep learning and Gaussian process approximation

Aboleth A bare-bones TensorFlow framework for Bayesian deep learning and Gaussian process approximation [1] with stochastic gradient variational Bayes

Gradient Institute 127 Dec 12, 2022
Hierarchical Uniform Manifold Approximation and Projection

HUMAP Hierarchical Manifold Approximation and Projection (HUMAP) is a technique based on UMAP for hierarchical non-linear dimensionality reduction. HU

Wilson Estécio Marcílio Júnior 160 Jan 6, 2023
PyTorch implementation of Advantage Actor Critic (A2C), Proximal Policy Optimization (PPO), Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation (ACKTR) and Generative Adversarial Imitation Learning (GAIL).

PyTorch implementation of Advantage Actor Critic (A2C), Proximal Policy Optimization (PPO), Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation (ACKTR) and Generative Adversarial Imitation Learning (GAIL).

Ilya Kostrikov 3k Dec 31, 2022
[ICCV 2021 (oral)] Planar Surface Reconstruction from Sparse Views

Planar Surface Reconstruction From Sparse Views Linyi Jin, Shengyi Qian, Andrew Owens, David F. Fouhey University of Michigan ICCV 2021 (Oral) This re

Linyi Jin 89 Jan 5, 2023
Fast algorithms to compute an approximation of the minimal volume oriented bounding box of a point cloud in 3D.

ApproxMVBB Status Build UnitTests Homepage Fast algorithms to compute an approximation of the minimal volume oriented bounding box of a point cloud in

Gabriel Nützi 390 Dec 31, 2022
Recurrent Scale Approximation (RSA) for Object Detection

Recurrent Scale Approximation (RSA) for Object Detection Codebase for Recurrent Scale Approximation for Object Detection in CNN published at ICCV 2017

Yu Liu (Louis) 239 Dec 28, 2022
Time Dependent DFT in Tamm-Dancoff Approximation

Density Function Theory Program - kspy-tddft(tda) This is an implementation of Time-Dependent Density Functional Theory(TDDFT) using the Tamm-Dancoff

Peter Borthwick 2 Nov 17, 2022
Code for ICCV 2021 paper: ARAPReg: An As-Rigid-As Possible Regularization Loss for Learning Deformable Shape Generators..

ARAPReg Code for ICCV 2021 paper: ARAPReg: An As-Rigid-As Possible Regularization Loss for Learning Deformable Shape Generators.. Installation The cod

Bo Sun 132 Nov 28, 2022
Official implementation for (Show, Attend and Distill: Knowledge Distillation via Attention-based Feature Matching, AAAI-2021)

Show, Attend and Distill: Knowledge Distillation via Attention-based Feature Matching Official pytorch implementation of "Show, Attend and Distill: Kn

Clova AI Research 80 Dec 16, 2022