Code for Neural Reflectance Surfaces (NeRS)
[arXiv] [Project Page] [Colab Demo] [Bibtex]
This repo contains the code for NeRS: Neural Reflectance Surfaces.
The code was tested with the following dependencies:
- Python 3.8.6
- Pytorch 1.7.0
- Pytorch3d 0.4.0
- CUDA 11.0
Installation
Setup
We recommend using conda to manage dependencies. Make sure to install a cudatoolkit compatible with your GPU.
git clone [email protected]:jasonyzhang/ners.git
conda create -n ners python=3.8
cond activate pytorch3d
conda install -c pytorch pytorch=1.7.0 torchvision cudatoolkit=11.0
pip install -r requirements.txt
Installing Pytorch3d
Here, we list the recommended steps for installing Pytorch3d. Refer to the official installation directions for troubleshooting and additional details.
mkdir -p external
git clone https://github.com/facebookresearch/pytorch3d.git external/pytorch3d
cd external/pytorch3d
conda install -c conda-forge -c fvcore -c iopath fvcore iopath
conda install -c bottler nvidiacub
python setup.py install
If you need to compile for multiple architectures (e.g. Turing for 2080TI and Maxwell for 1080TI), you can pass the architectures as an environment variable, i.e. TORCH_CUDA_ARCH_LIST="Maxwell;Pascal;Turing;Volta" python setup.py install.
If you get a warning about the default C/C++ compiler on your machine, you should compile Pytorch3D using the same compiler that your pytorch installation uses, likely gcc/g++. Try: CC=gcc CXX=g++ python setup.py install.
Acquiring Object Masks
To get object masks, we recommend using PointRend for COCO classes or GrabCut for other categories.
If using GrabCut, you can try this interactive segmentation tool.
Running the Code
Running on MVMC
Coming Soon!
Running on Your Own Objects
We recommend beginning with the demo notebook so that you can visualize the intermediate outputs. The demo notebook generates the 3D reconstruction and illumination prediction for the espresso machine (data included). You can also run the demo script:
python main.py --instance-dir data/espresso --symmetrize --export-mesh --predict-illumination
We also provide a Colab notebook that runs on a single GPU. Note that the Colab demo does not include the view-dependent illumination prediction. At the end of the demo, you can view the turntable NeRS rendering and download the generated mesh as an obj.
To run on your own objects, you will need to acquire images and masks. See data/espresso for an example of the expected directory structure.
We also provide the images and masks for all objects in the paper. All objects except hydrant and robot should have a --symmetrize flag.
gdown https://drive.google.com/uc?id=1JWuofTIlcLJmmzYtZYM2SvZVizJCcOU_
unzip -f misc_objects.zip -d data
Citing NeRS
If you use find this code helpful, please consider citing:
@inproceedings{zhang2021ners,
title={{NeRS}: Neural Reflectance Surfaces for Sparse-view 3D Reconstruction in the Wild},
author={Zhang, Jason Y. and Yang, Gengshan and Tulsiani, Shubham and Ramanan, Deva},
booktitle={Conference on Neural Information Processing Systems},
year={2021}
}
In this, I am taking the inverse of the rotation matrix available with these images.
Any help would be really appreciated!
4 Dec 26, 2021
9 Nov 29, 2022
28 Nov 18, 2022
462 Jan 3, 2023
53 Dec 25, 2022
39 Nov 10, 2022
1 Nov 9, 2021
12 Nov 23, 2022
45 Nov 30, 2022
6 Dec 19, 2022
6 Nov 14, 2022
88 Jan 4, 2023
32 Dec 26, 2022
17 Dec 1, 2022
71 Dec 22, 2022
159 Dec 20, 2022
81 Dec 16, 2022
24 Dec 17, 2022
139 Dec 14, 2022