NeuTex: Neural Texture Mapping for Volumetric Neural Rendering

Related tags

Deep Learning NeuTex
Overview

NeuTex: Neural Texture Mapping for Volumetric Neural Rendering

Paper: https://arxiv.org/abs/2103.00762

Running

Run on the provided DTU scene

cd run
bash dtu.sh 114

(Install any missing library from pip)

Further fine tuning for texture after fixing the geometry

bash dtu-freese.sh 114

Run on custom datasets

Similar to the provided DTU scene, you will need to provide a custom data loader similar to data/dtu_dataset.py and modify the dataset arguments in the bash scripts accordingly.

Similar to the dtu_dataset.py, the custom dataset needs to provide the following fields when getting and item:

  • gt_mask, a 0/1 mask for background/foreground.
  • near, the near plane for point sampling on the ray
  • far, the far plane for point sampling on the ray
  • raydir, ray directions
  • gt_image, ground truth pixel colors
  • background_color, color of the image background

The captured scene must be contained in the unit cube centered at world origin.

Citation

@InProceedings{xiang2021neutex,
author = {Xiang, Fanbo and Xu, Zexiang and Hašan, Miloš and Hold-Geoffroy, Yannick and Sunkavalli, Kalyan and Su, Hao},
title = {{N}eu{T}ex: {N}eural {T}exture {M}apping for {V}olumetric {N}eural {R}endering},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
year = {2021}}
You might also like...
NeRViS: Neural Re-rendering for Full-frame Video Stabilization

Neural Re-rendering for Full-frame Video Stabilization

Neural Re-rendering for Full-frame Video Stabilization

NeRViS: Neural Re-rendering for Full-frame Video Stabilization Project Page | Video | Paper | Google Colab Setup Setup environment for [Yu and Ramamoo

Official repo for AutoInt: Automatic Integration for Fast Neural Volume Rendering in CVPR 2021
Official repo for AutoInt: Automatic Integration for Fast Neural Volume Rendering in CVPR 2021

AutoInt: Automatic Integration for Fast Neural Volume Rendering CVPR 2021 Project Page | Video | Paper PyTorch implementation of automatic integration

Generative Query Network (GQN) in PyTorch as described in
Generative Query Network (GQN) in PyTorch as described in "Neural Scene Representation and Rendering"

Update 2019/06/24: A model trained on 10% of the Shepard-Metzler dataset has been added, the following notebook explains the main features of this mod

This repository contains the source code for the paper
This repository contains the source code for the paper "DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks",

DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks Project Page | Video | Presentation | Paper | Data L

The source code of the ICCV2021 paper
The source code of the ICCV2021 paper "PIRenderer: Controllable Portrait Image Generation via Semantic Neural Rendering"

The source code of the ICCV2021 paper "PIRenderer: Controllable Portrait Image Generation via Semantic Neural Rendering"

The source code of the ICCV2021 paper
The source code of the ICCV2021 paper "PIRenderer: Controllable Portrait Image Generation via Semantic Neural Rendering"

Website | ArXiv | Get Start | Video PIRenderer The source code of the ICCV2021 paper "PIRenderer: Controllable Portrait Image Generation via Semantic

A curated list of neural rendering resources.

Awesome-of-Neural-Rendering A curated list of neural rendering and related resources. Please feel free to pull requests or open an issue to add papers

Volsdf - Volume Rendering of Neural Implicit Surfaces
Volsdf - Volume Rendering of Neural Implicit Surfaces

Volume Rendering of Neural Implicit Surfaces Project Page | Paper | Data This re

Comments
  • About cycle consistency loss

    About cycle consistency loss

    Hi @fbxiang , thanks for sharing the codes. I did some simple experiments, and it seems that removing the cycle loss leads to better novel view synthesis (but maybe more distorted uv map). In Fig. 3 of the paper, removing cycle loss will lead to very bad mapping results which didn't happen in my experiments. May I know how to get the results in the last column of Fig. 3? Thanks.

    opened by fnzhan 1
  • How to use custom datasets

    How to use custom datasets

    Hello, Could you give me more detailed guidelines for using custom datasets? I tried to follow your guidelines, but I was confused about how to set these fields: image

    I also wonder should I make all of these files in the 'dtu_dataset.py' following above fields? self.campos = np.load(self.data_dir + "/in_camOrgs.npy") self.camat = np.load(self.data_dir + "/in_camAts.npy") self.focal = np.load(self.data_dir + "/in_camFocal.npy") self.princpt = np.load(self.data_dir + "/in_camPrincpt.npy") self.extrinsics = np.load(self.data_dir + "/in_camExtrinsics.npy") self.point_cloud = trimesh.load(self.data_dir + "/pcd_down_unit.ply") I guessed "pcd_down_unit.ply" is a polygon file of point clouds, but I don't know how to make other files.

    I'd really appreciate it if you reply to my question.

    opened by pianoforte23 1
  • Bad geometry and texture map visualization

    Bad geometry and texture map visualization

    Hi, thanks for the amazing work! I train the network on scan 114 data and got accurate rendering results like this: step-01000000-ray_color However, when I run visualize_nerf_atlas_radiance.py to visualize the geometry and texture map, I found they are strange. point cloud: image mesh: image texture map: cube_view_63 sphere_view_63

    opened by ghy0324 7
Owner
Fanbo Xiang
Fanbo Xiang
SSL_SLAM2: Lightweight 3-D Localization and Mapping for Solid-State LiDAR (mapping and localization separated) ICRA 2021

SSL_SLAM2 Lightweight 3-D Localization and Mapping for Solid-State LiDAR (Intel Realsense L515 as an example) This repo is an extension work of SSL_SL

Wang Han 王晗 1.3k Jan 8, 2023
A PyTorch implementation for V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation

A PyTorch implementation of V-Net Vnet is a PyTorch implementation of the paper V-Net: Fully Convolutional Neural Networks for Volumetric Medical Imag

Matthew Macy 606 Dec 21, 2022
Unofficial implementation of Point-Unet: A Context-Aware Point-Based Neural Network for Volumetric Segmentation

Point-Unet This is an unofficial implementation of the MICCAI 2021 paper Point-Unet: A Context-Aware Point-Based Neural Network for Volumetric Segment

Namt0d 9 Dec 7, 2022
Code for A Volumetric Transformer for Accurate 3D Tumor Segmentation

VT-UNet This repo contains the supported pytorch code and configuration files to reproduce 3D medical image segmentaion results of VT-UNet. Environmen

Himashi Amanda Peiris 7 Dec 3, 2021
An implementation of "Optimal Textures: Fast and Robust Texture Synthesis and Style Transfer through Optimal Transport"

Optex An implementation of Optimal Textures: Fast and Robust Texture Synthesis and Style Transfer through Optimal Transport for TU Delft CS4240. You c

Hans Brouwer 33 Jan 5, 2023
"3D Human Texture Estimation from a Single Image with Transformers", ICCV 2021

Texformer: 3D Human Texture Estimation from a Single Image with Transformers This is the official implementation of "3D Human Texture Estimation from

XiangyuXu 193 Dec 5, 2022
[ICCV'2021] Image Inpainting via Conditional Texture and Structure Dual Generation

[ICCV'2021] Image Inpainting via Conditional Texture and Structure Dual Generation

Xiefan Guo 122 Dec 11, 2022
GANmouflage: 3D Object Nondetection with Texture Fields

GANmouflage: 3D Object Nondetection with Texture Fields Rui Guo1 Jasmine Collins

null 29 Aug 10, 2022
Code for "Layered Neural Rendering for Retiming People in Video."

Layered Neural Rendering in PyTorch This repository contains training code for the examples in the SIGGRAPH Asia 2020 paper "Layered Neural Rendering

Google 154 Dec 16, 2022