Generative Modelling of BRDF Textures from Flash Images [SIGGRAPH Asia, 2021]

Overview

Neural Material

Official code repository for the paper:

Generative Modelling of BRDF Textures from Flash Images [SIGGRAPH Asia, 2021]

Henzler, Deschaintre, J. Mitra, Ritschel

[Paper] [Project page]

Rerendering

Data

The dataset is stored under flash_images and contains 306 train folders and 116 test folders (including images from Aitalla et al).

Install dependencies

conda create -n neuralmaterial python=3.8
conda activate neuralmaterial
pip install hydra-core --upgrade
pip install tqdm
conda install pytorch==1.7.1 torchvision==0.8.2 cudatoolkit=10.1 -c pytorch

Training

For training please run

python scripts/train.py

The default config is located at config/config_default.yaml.

Inference

Note, is a relative path in the flash_images/test and is exptected to be located in the trainings/ folder.

Synthesis

In order to synthesise given flash images located in the test folder please run

python scripts/test.py --model 
   
     --test_image_id 
    
      --finetune 
     

     
    
   

Interpolation

Given two images for interpolation please run

python scripts/interpolate.py --model 
   
     --weights1 
    
      --weights2 
     
       --test_image_id1 
      
        --test_image_id2 
        
       
      
     
    
   

If you would like to use fine-tuned weights please run the scripts/test.py command above in order to retrieve them.

Examples

Run the file run_examples.sh to synthesise / interpolate a few examples.

Citation

If you find our work useful in your research, please cite:

@article{henzler2021neuralmaterial,
  title={Generative Modelling of BRDF Textures from Flash Images},
  author={Henzler, Philipp and Deschaintre, Valentin and Mitra, Niloy J and Ritschel, Tobias},
  journal={ACM Trans Graph (Proc. SIGGRAPH Asia)},
  year={2021},
  volume={40},
  number={6},
}

Contact

If you have any questions, please email Philipp Henzler at [email protected].

Comments
  • Scripts do not execute after installation

    Scripts do not execute after installation

    None of the scripts execute after installation as per the instructions. This issue is apparently due to mismatches in dependencies.

    Upon first execution, I got : 'ModuleNotFoundError: No module named 'tensorboard''

    After installing tensorboard via pip, I got: 'ModuleNotFoundError: No module named 'kornia''

    After installing kornia via pip, I got: _op = torch._C.jit_get_operation(qualified_op_name) RuntimeError: No such operator image::read_file

    Installed on Ubuntu 18.04 and 20.04 on a gcp cloud VM with a T4 GPU without success.

    Could you please review that the installation instructions are accurate?

    Thanks

    opened by xxl007 3
  • error loading the pretrained model

    error loading the pretrained model

    Hi,

    I get this error when I try to test the model: "_pickle.UnpicklingError: invalid load key, 'v'." I think it might have something to do with the model being uploaded to github, could you please upload it to google drive? @henzler

    Thanks

    opened by sepideh-srj 2
  • Distorted BRDF maps

    Distorted BRDF maps

    OS: Linux GPU: Tesla K80 (Kepler)

    When running test.py, the reconstructed render seems to come out fine, but the roughness and specular BRDF maps appear to be distorted. The normal map on this particular example seems to be correct, but a few others gave me pause. Running on CPU doesn't seem to make a difference.

    rendering roughness specular normal

    opened by Sam-Izdat 0
  • ModuleNotFoundError: No module named 'lib.core'

    ModuleNotFoundError: No module named 'lib.core'

    After calling trainer.fit(model, data) on Win10:

    (neuralmaterial) E:\projects\ThirdParty\neuralmaterial>python scripts\train.py Working directory : E:\projects\ThirdParty\neuralmaterial\trainings\Neuralmaterial [Data] 25000 train samples [Data] 25000 val samples [INFO] Checkpoint loaded. [INFO] Metadata loaded.

    [Model] decoder | params: 2,778,214 encoder | params: 25,819,304 loss | params: 20,024,384 renderer | params: 0 rotation_encoder | params: 188 total params: 48,622,090

    Epoch: 13 | train step: 0%| | 0/25000 [00:00<?, ?it/s]Traceback (most recent call last): File "", line 1, in File "C:\Users\onepride\Anaconda3\envs\neuralmaterial2\lib\multiprocessing\spawn.py", line 116, in spawn_main exitcode = _main(fd, parent_sentinel) File "C:\Users\onepride\Anaconda3\envs\neuralmaterial2\lib\multiprocessing\spawn.py", line 125, in _main prepare(preparation_data) File "C:\Users\onepride\Anaconda3\envs\neuralmaterial2\lib\multiprocessing\spawn.py", line 236, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\onepride\Anaconda3\envs\neuralmaterial2\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path main_content = runpy.run_path(main_path, File "C:\Users\onepride\Anaconda3\envs\neuralmaterial2\lib\runpy.py", line 268, in run_path return _run_module_code(code, init_globals, run_name, File "C:\Users\onepride\Anaconda3\envs\neuralmaterial2\lib\runpy.py", line 97, in _run_module_code _run_code(code, mod_globals, init_globals, File "C:\Users\onepride\Anaconda3\envs\neuralmaterial2\lib\runpy.py", line 87, in _run_code exec(code, run_globals) File "E:\projects\ThirdParty\neuralmaterial\scripts\train.py", line 8, in from lib.core.trainer import Trainer ModuleNotFoundError: No module named 'lib.core'

    opened by OnePride 0
  • Error loading images when running run_examples.sh

    Error loading images when running run_examples.sh

    Traceback (most recent call last): File "scripts/test.py", line 58, in image = tfm(io.read_image(image_dirs[0]))[None] File "/home/galem/anaconda3/envs/neuralmaterial/lib/python3.8/site-packages/torchvision/io/image.py", line 230, in read_image return decode_image(data) File "/home/galem/anaconda3/envs/neuralmaterial/lib/python3.8/site-packages/torchvision/io/image.py", line 211, in decode_image output = torch.ops.image.decode_image(input) RuntimeError: Unsupported image file. Only jpeg and png are currently supported.

    opened by xxl007 1
Owner
Philipp Henzler
Phd Student in Computer Vision/Graphics and Machine Learning.
Philipp Henzler
The implemetation of Dynamic Nerual Garments proposed in Siggraph Asia 2021

DynamicNeuralGarments Introduction This repository contains the implemetation of Dynamic Nerual Garments proposed in Siggraph Asia 2021. ./GarmentMoti

null 42 Dec 27, 2022
Face Identity Disentanglement via Latent Space Mapping [SIGGRAPH ASIA 2020]

Face Identity Disentanglement via Latent Space Mapping Description Official Implementation of the paper Face Identity Disentanglement via Latent Space

null 150 Dec 7, 2022
A code repository associated with the paper A Benchmark for Rough Sketch Cleanup by Chuan Yan, David Vanderhaeghe, and Yotam Gingold from SIGGRAPH Asia 2020.

A Benchmark for Rough Sketch Cleanup This is the code repository associated with the paper A Benchmark for Rough Sketch Cleanup by Chuan Yan, David Va

null 33 Dec 18, 2022
Implementation for "Manga Filling Style Conversion with Screentone Variational Autoencoder" (SIGGRAPH ASIA 2020 issue)

Manga Filling with ScreenVAE SIGGRAPH ASIA 2020 | Project Website | BibTex This repository is for ScreenVAE introduced in the following paper "Manga F

null 30 Dec 24, 2022
An implementation of "Optimal Textures: Fast and Robust Texture Synthesis and Style Transfer through Optimal Transport"

Optex An implementation of Optimal Textures: Fast and Robust Texture Synthesis and Style Transfer through Optimal Transport for TU Delft CS4240. You c

Hans Brouwer 33 Jan 5, 2023
An image base contains 490 images for learning (400 cars and 90 boats), and another 21 images for testingAn image base contains 490 images for learning (400 cars and 90 boats), and another 21 images for testing

SVM Données Une base d’images contient 490 images pour l’apprentissage (400 voitures et 90 bateaux), et encore 21 images pour fait des tests. Prétrait

Achraf Rahouti 3 Nov 30, 2021
Multiple paper open-source codes of the Microsoft Research Asia DKI group

?? Paper Code Collection (MSRA DKI Group) This repo hosts multiple open-source codes of the Microsoft Research Asia DKI Group. You could find the corr

Microsoft 249 Jan 8, 2023
Code for CVPR2021 paper "Robust Reflection Removal with Reflection-free Flash-only Cues"

Robust Reflection Removal with Reflection-free Flash-only Cues (RFC) Paper | To be released: Project Page | Video | Data Tensorflow implementation for

Chenyang LEI 162 Jan 5, 2023
Pre-trained BERT Models for Ancient and Medieval Greek, and associated code for LaTeCH 2021 paper titled - "A Pilot Study for BERT Language Modelling and Morphological Analysis for Ancient and Medieval Greek"

Ancient Greek BERT The first and only available Ancient Greek sub-word BERT model! State-of-the-art post fine-tuning on Part-of-Speech Tagging and Mor

Pranaydeep Singh 22 Dec 8, 2022
General Virtual Sketching Framework for Vector Line Art (SIGGRAPH 2021)

General Virtual Sketching Framework for Vector Line Art - SIGGRAPH 2021 Paper | Project Page Outline Dependencies Testing with Trained Weights Trainin

Haoran MO 118 Dec 27, 2022
Implementation for "Seamless Manga Inpainting with Semantics Awareness" (SIGGRAPH 2021 issue)

Seamless Manga Inpainting with Semantics Awareness [SIGGRAPH 2021](To appear) | Project Website | BibTex Introduction: Manga inpainting fills up the d

null 101 Jan 1, 2023
Supplementary code for SIGGRAPH 2021 paper: Discovering Diverse Athletic Jumping Strategies

SIGGRAPH 2021: Discovering Diverse Athletic Jumping Strategies project page paper demo video Prerequisites Important Notes We suspect there are bugs i

null 54 Dec 6, 2022
Official implementation of "StyleCariGAN: Caricature Generation via StyleGAN Feature Map Modulation" (SIGGRAPH 2021)

StyleCariGAN in PyTorch Official implementation of StyleCariGAN:Caricature Generation via StyleGAN Feature Map Modulation in PyTorch Requirements PyTo

PeterZhouSZ 49 Oct 31, 2022
Code for HodgeNet: Learning Spectral Geometry on Triangle Meshes, in SIGGRAPH 2021.

HodgeNet | Webpage | Paper | Video HodgeNet: Learning Spectral Geometry on Triangle Meshes Dmitriy Smirnov, Justin Solomon SIGGRAPH 2021 Set-up To ins

Dima Smirnov 61 Nov 27, 2022
Tracing Versus Freehand for Evaluating Computer-Generated Drawings (SIGGRAPH 2021)

Tracing Versus Freehand for Evaluating Computer-Generated Drawings (SIGGRAPH 2021) Zeyu Wang, Sherry Qiu, Nicole Feng, Holly Rushmeier, Leonard McMill

Zach Zeyu Wang 23 Dec 9, 2022
PyTorch implementations for our SIGGRAPH 2021 paper: Editable Free-viewpoint Video Using a Layered Neural Representation.

st-nerf We provide PyTorch implementations for our paper: Editable Free-viewpoint Video Using a Layered Neural Representation SIGGRAPH 2021 Jiakai Zha

Diplodocus 258 Jan 2, 2023
Code for the SIGGRAPH 2021 paper "Consistent Depth of Moving Objects in Video".

Consistent Depth of Moving Objects in Video This repository contains training code for the SIGGRAPH 2021 paper "Consistent Depth of Moving Objects in

Google 203 Jan 5, 2023
ManipNet: Neural Manipulation Synthesis with a Hand-Object Spatial Representation - SIGGRAPH 2021

ManipNet: Neural Manipulation Synthesis with a Hand-Object Spatial Representation - SIGGRAPH 2021 Dataset Code Demos Authors: He Zhang, Yuting Ye, Tak

HE ZHANG 194 Dec 6, 2022
Official implementation of "StyleCariGAN: Caricature Generation via StyleGAN Feature Map Modulation" (SIGGRAPH 2021)

StyleCariGAN: Caricature Generation via StyleGAN Feature Map Modulation This repository contains the official PyTorch implementation of the following

Wonjong Jang 270 Dec 30, 2022