NeuTex: Neural Texture Mapping for Volumetric Neural Rendering

Related tags

Text Data & NLP NeuTex
Overview

NeuTex: Neural Texture Mapping for Volumetric Neural Rendering

Paper: https://arxiv.org/abs/2103.00762

Running

Run on the provided DTU scene

cd run
bash dtu.sh 114

(Install any missing library from pip)

Further fine tuning for texture after fixing the geometry

bash dtu-freese.sh 114

Run on custom datasets

Similar to the provided DTU scene, you will need to provide a custom data loader similar to data/dtu_dataset.py and modify the dataset arguments in the bash scripts accordingly.

Similar to the dtu_dataset.py, the custom dataset needs to provide the following fields when getting and item:

  • gt_mask, a 0/1 mask for background/foreground.
  • near, the near plane for point sampling on the ray
  • far, the far plane for point sampling on the ray
  • raydir, ray directions
  • gt_image, ground truth pixel colors
  • background_color, color of the image background

The captured scene must be contained in the unit cube centered at world origin.

Citation

@InProceedings{xiang2021neutex,
author = {Xiang, Fanbo and Xu, Zexiang and Hašan, Miloš and Hold-Geoffroy, Yannick and Sunkavalli, Kalyan and Su, Hao},
title = {{N}eu{T}ex: {N}eural {T}exture {M}apping for {V}olumetric {N}eural {R}endering},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
year = {2021}}
You might also like...
An Open-Source Package for Neural Relation Extraction (NRE)

OpenNRE We have a DEMO website (http://opennre.thunlp.ai/). Try it out! OpenNRE is an open-source and extensible toolkit that provides a unified frame

THUNLP 3.9k Jan 3, 2023
Named-entity recognition using neural networks. Easy-to-use and state-of-the-art results.

NeuroNER NeuroNER is a program that performs named-entity recognition (NER). Website: neuroner.com. This page gives step-by-step instructions to insta

Franck Dernoncourt 1.5k Feb 11, 2021
An assignment on creating a minimalist neural network toolkit for CS11-747

minnn by Graham Neubig, Zhisong Zhang, and Divyansh Kaushik This is an exercise in developing a minimalist neural network toolkit for NLP, part of Car

Graham Neubig 63 Dec 29, 2022
Unsupervised text tokenizer for Neural Network-based text generation.

SentencePiece SentencePiece is an unsupervised text tokenizer and detokenizer mainly for Neural Network-based text generation systems where the vocabu

Google 4.8k Feb 18, 2021
Open Source Neural Machine Translation in PyTorch
Open Source Neural Machine Translation in PyTorch

OpenNMT-py: Open-Source Neural Machine Translation OpenNMT-py is the PyTorch version of the OpenNMT project, an open-source (MIT) neural machine trans

OpenNMT 4.8k Feb 18, 2021
An easier way to build neural search on the cloud
An easier way to build neural search on the cloud

An easier way to build neural search on the cloud Jina is a deep learning-powered search framework for building cross-/multi-modal search systems (e.g

Jina AI 2.3k Feb 18, 2021
Easily train your own text-generating neural network of any size and complexity on any text dataset with a few lines of code.
Easily train your own text-generating neural network of any size and complexity on any text dataset with a few lines of code.

textgenrnn Easily train your own text-generating neural network of any size and complexity on any text dataset with a few lines of code, or quickly tr

Max Woolf 4.3k Feb 18, 2021
✨Fast Coreference Resolution in spaCy with Neural Networks
✨Fast Coreference Resolution in spaCy with Neural Networks

✨ NeuralCoref 4.0: Coreference Resolution in spaCy with Neural Networks. NeuralCoref is a pipeline extension for spaCy 2.1+ which annotates and resolv

Hugging Face 2.2k Feb 18, 2021
A model library for exploring state-of-the-art deep learning topologies and techniques for optimizing Natural Language Processing neural networks
A model library for exploring state-of-the-art deep learning topologies and techniques for optimizing Natural Language Processing neural networks

A Deep Learning NLP/NLU library by Intel® AI Lab Overview | Models | Installation | Examples | Documentation | Tutorials | Contributing NLP Architect

Intel Labs 2.6k Feb 18, 2021
Comments
  • About cycle consistency loss

    About cycle consistency loss

    Hi @fbxiang , thanks for sharing the codes. I did some simple experiments, and it seems that removing the cycle loss leads to better novel view synthesis (but maybe more distorted uv map). In Fig. 3 of the paper, removing cycle loss will lead to very bad mapping results which didn't happen in my experiments. May I know how to get the results in the last column of Fig. 3? Thanks.

    opened by fnzhan 1
  • How to use custom datasets

    How to use custom datasets

    Hello, Could you give me more detailed guidelines for using custom datasets? I tried to follow your guidelines, but I was confused about how to set these fields: image

    I also wonder should I make all of these files in the 'dtu_dataset.py' following above fields? self.campos = np.load(self.data_dir + "/in_camOrgs.npy") self.camat = np.load(self.data_dir + "/in_camAts.npy") self.focal = np.load(self.data_dir + "/in_camFocal.npy") self.princpt = np.load(self.data_dir + "/in_camPrincpt.npy") self.extrinsics = np.load(self.data_dir + "/in_camExtrinsics.npy") self.point_cloud = trimesh.load(self.data_dir + "/pcd_down_unit.ply") I guessed "pcd_down_unit.ply" is a polygon file of point clouds, but I don't know how to make other files.

    I'd really appreciate it if you reply to my question.

    opened by pianoforte23 1
  • Bad geometry and texture map visualization

    Bad geometry and texture map visualization

    Hi, thanks for the amazing work! I train the network on scan 114 data and got accurate rendering results like this: step-01000000-ray_color However, when I run visualize_nerf_atlas_radiance.py to visualize the geometry and texture map, I found they are strange. point cloud: image mesh: image texture map: cube_view_63 sphere_view_63

    opened by ghy0324 7
Owner
Fanbo Xiang
Fanbo Xiang
GitHub
Easy to use, state-of-the-art Neural Machine Translation for 100+ languages

EasyNMT - Easy to use, state-of-the-art Neural Machine Translation This package provides easy to use, state-of-the-art machine translation for more th

Ubiquitous Knowledge Processing Lab 748 Jan 6, 2023
PORORO: Platform Of neuRal mOdels for natuRal language prOcessing

PORORO: Platform Of neuRal mOdels for natuRal language prOcessing pororo performs Natural Language Processing and Speech-related tasks. It is easy to

Kakao Brain 1.2k Dec 21, 2022
Unsupervised text tokenizer for Neural Network-based text generation.

SentencePiece SentencePiece is an unsupervised text tokenizer and detokenizer mainly for Neural Network-based text generation systems where the vocabu

Google 6.4k Jan 1, 2023
Open Source Neural Machine Translation in PyTorch

OpenNMT-py: Open-Source Neural Machine Translation OpenNMT-py is the PyTorch version of the OpenNMT project, an open-source (MIT) neural machine trans

OpenNMT 5.8k Jan 4, 2023
An easier way to build neural search on the cloud

An easier way to build neural search on the cloud Jina is a deep learning-powered search framework for building cross-/multi-modal search systems (e.g

Jina AI 17.1k Jan 9, 2023
:mag: Transformers at scale for question answering & neural search. Using NLP via a modular Retriever-Reader-Pipeline. Supporting DPR, Elasticsearch, HuggingFace's Modelhub...

Haystack is an end-to-end framework for Question Answering & Neural search that enables you to ... ... ask questions in natural language and find gran

deepset 6.4k Jan 9, 2023
Easily train your own text-generating neural network of any size and complexity on any text dataset with a few lines of code.

textgenrnn Easily train your own text-generating neural network of any size and complexity on any text dataset with a few lines of code, or quickly tr

Max Woolf 4.8k Dec 30, 2022
✨Fast Coreference Resolution in spaCy with Neural Networks

✨ NeuralCoref 4.0: Coreference Resolution in spaCy with Neural Networks. NeuralCoref is a pipeline extension for spaCy 2.1+ which annotates and resolv

Hugging Face 2.6k Jan 4, 2023
A model library for exploring state-of-the-art deep learning topologies and techniques for optimizing Natural Language Processing neural networks

A Deep Learning NLP/NLU library by Intel® AI Lab Overview | Models | Installation | Examples | Documentation | Tutorials | Contributing NLP Architect

Intel Labs 2.9k Jan 2, 2023
Sequence-to-sequence framework with a focus on Neural Machine Translation based on Apache MXNet

Sockeye This package contains the Sockeye project, an open-source sequence-to-sequence framework for Neural Machine Translation based on Apache MXNet

Amazon Web Services - Labs 1.1k Dec 27, 2022