Rendering color and depth images for ShapeNet models.

Last update: Dec 19, 2022

Overview

Color & Depth Renderer for ShapeNet

This library includes the tools for rendering multi-view color and depth images of ShapeNet models. Physically based rendering (PBR) is featured based on blender2.79.

Outputs

Color image (20 views)

Depth image (20 views)

Point cloud and normals (Back-projected from color & depth images)

Watertight meshes (fused from depth maps)

Install

We recommend to install this repository with conda.

conda env create -f environment.yml
conda activate renderer

Install Pyfusion by

cd ./external/pyfusion
mkdir build
cd ./build
cmake ..
make

Afterwards, compile the Cython code in ./external/pyfusion by

cd ./external/pyfusion
python setup.py build_ext --inplace

Download & Extract blender2.79b, and specify the path of your blender executable file at ./setting.py by
```
g_blender_excutable_path = '../../blender-2.79b-linux-glibc219-x86_64/blender'
```

Usage

Normalize ShapeNet models to a unit cube by
```
python normalize_shape.py
```
The ShapeNetCore.v2 dataset is put in ./datasets/ShapeNetCore.v2. Here we only present some samples in this repository.
Generate multiple camera viewpoints for rendering by
```
python create_viewpoints.py
```
The camera extrinsic parameters will be saved at ./view_points.txt, or you can customize it in this script.
Run renderer to render color and depth images by
```
python run_render.py
```
The rendered images are saved in ./datasets/ShapeNetRenderings. The camera intrinsic and extrinsic parameters are saved in ./datasets/camera_settings. You can change the rendering configurations at ./settings.py, e.g. image sizes and resolution.
The back-projected point cloud and corresponding normals can be visualized by
```
python visualization/draw_pc_from_depth.py
```
Watertight meshes can be obtained by
```
python depth_fusion.py
```
The reconstructed meshes are saved in ./datasets/ShapeNetCore.v2_watertight

Citation

This library is used for data preprocessing in our work SK-PCN. If you find it helpful, please consider citing

@inproceedings{NEURIPS2020_ba036d22,
 author = {Nie, Yinyu and Lin, Yiqun and Han, Xiaoguang and Guo, Shihui and Chang, Jian and Cui, Shuguang and Zhang, Jian.J},
 booktitle = {Advances in Neural Information Processing Systems},
 editor = {H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin},
 pages = {16119--16130},
 publisher = {Curran Associates, Inc.},
 title = {Skeleton-bridged Point Completion: From Global Inference to Local Adjustment},
 url = {https://proceedings.neurips.cc/paper/2020/file/ba036d228858d76fb89189853a5503bd-Paper.pdf},
 volume = {33},
 year = {2020}
}

License

This repository is relased under the MIT License.

Comments

Build correspondence between pixels and points

Hi, thanks for your helpful work. I would like to ask if there is a convenient way of outputing correspondence relationships between pixels on the rendered image and points on the corresponding 3D model. I notice that "visualization/draw_pc_from_depth.py" can generate partial point clouds from specific viewpoints. Is there any function to obtain such "pixel-point" mapping? Look forward to your reply as soon as possible!

opened by KeeganZQJ 1
how to add lighting from all direction

Hi, thanks for the great work, I'm wondering how to add lights from all directions. https://github.com/yinyunie/depth_renderer/blob/57af7716516a04bd012961f2bc283d636660439b/render_all.py#L236-L245

opened by qiminchen 1
Data Preprocessing

Thanks a lot for the awesome work from the authors. But I am a bit confused with data preprocessing. I noticed that in the dataset attached each object has two folders which are respectively image and model. But in the original dataset, each object only has one file which is called XXX.obj. How could I change the XXX.obj to the form of the current dataset?

opened by LLh-lihan 0
No module named 'cyfusion'

`(renderer) root@VGG-V100-LZ-2:~/lkq/depth_renderer# python depth_fusion.py Traceback (most recent call last): File "depth_fusion.py", line 9, in from external import pyfusion

File "/root/lkq/depth_renderer/external/pyfusion/init.py", line 8, in from cyfusion import * ModuleNotFoundError: No module named 'cyfusion'` many thanks for your great work. I follow the install step, but can not run the code. can you give me some help?

opened by UestcJay 3
Points not lying on the surface

Hi,

thanks for open-sourcing this project. I have used it to render some of the shapenet models but have observed that the 3D point do not lie exactly on the surface once back-projected to 3D. The cloud2mesh distance lie in the range of 0.002.

Have you maybe observed the same? Do you know if this is an inherent precision of Blender depth rendering or if there is maybe some precision loss occurring at some point?

Zan

opened by zgojcic 4

Rendering color and depth images for ShapeNet models.

Related tags

Overview

Color & Depth Renderer for ShapeNet

Outputs

Install

Usage

Citation

License

Comments

Build correspondence between pixels and points

how to add lighting from all direction

Data Preprocessing

No module named 'cyfusion'

Points not lying on the surface

Owner

Yinyu Nie

Adversarial Color Enhancement: Generating Unrestricted Adversarial Images by Optimizing a Color Filter

Python script for performing depth completion from sparse depth and rgb images using the msg_chn_wacv20. model in ONNX

Python script for performing depth completion from sparse depth and rgb images using the msg_chn_wacv20. model in Tensorflow Lite.

Monocular Depth Estimation - Weighted-average prediction from multiple pre-trained depth estimation models

An image base contains 490 images for learning (400 cars and 90 boats), and another 21 images for testingAn image base contains 490 images for learning (400 cars and 90 boats), and another 21 images for testing

This repository contains the source code for the paper "DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks",

The implemention of Video Depth Estimation by Fusing Flow-to-Depth Proposals

Monocular Depth Estimation Using Laplacian Pyramid-Based Depth Residuals

Beyond Image to Depth: Improving Depth Prediction using Echoes (CVPR 2021)

Light-weight network, depth estimation, knowledge distillation, real-time depth estimation, auxiliary data.

Data-depth-inference - Data depth inference with python

(CVPR 2022 - oral) Multi-View Depth Estimation by Fusing Single-View Depth Probability with Multi-View Geometry

MetaAvatar: Learning Animatable Clothed Human Models from Few Depth Images

Official Repo for ICCV2021 Paper: Learning to Regress Bodies from Images using Differentiable Semantic Rendering

ViewFormer: NeRF-free Neural Rendering from Few Images Using Transformers

PN-Net a neural field-based framework for depth estimation from single-view RGB images.

Honours project, on creating a depth estimation map from two stereo images of featureless regions