Code for CVPR 2018 paper --- Texture Mapping for 3D Reconstruction with RGB-D Sensor

Overview

G2LTex

This repository contains the implementation of "Texture Mapping for 3D Reconstruction with RGB-D Sensor (CVPR2018)" based on mvs-texturing. Due to the agreement with other company, some parts can only be released in the form of .so files. More information and the paper can be found on our group website and Qingan's homepage.

Publication

If you find this code useful for your research, please cite our work:

Yanping Fu, Qingan Yan, Long Yang, Jie Liao, Chunxia Xiao. Texture Mapping for 3D Reconstruction with RGB-D Sensor. In CVPR. 2018.

@inproceedings{fu2018texture,
  title={Texture Mapping for 3D Reconstruction with RGB-D Sensor},
  author={Fu, Yanping and Yan, Qingan and Yang, Long and Liao, Jie and Xiao, Chunxia},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  pages={4645--4653},
  year={2018},
  organization={IEEE}
}

How to use

1. Run

To test our algorithm. run G2LTex in command line:

./bin/G2LTex [DIR] [PLY] 

Params explanation: -PLY: The reconstructed model for texture mapping. -DIR: The texture image directory, include rgb images, depth images, and camera trajectory.

The parameters of the camera and the system can be set in the config file.

Config/config.yml

How to install and run this code.

git clone https://github.com/fdp0525/G2LTex.git
cd G2LTex/bin
./G2LTex ../Data/bloster/textureimages ../Data/bloster/bloster.ply

We need to modify the configuration file config.yml before running the other datasets.

./G2LTex ../Data/apt0/apt0 ../Data/apt0/apt0.ply

2. Input Format

  • Color frames (color_XX.jpg): RGB, 24-bit, JPG.
  • Depth frames (depth_XX.png): depth (mm), 16-bit, PNG (invalid depth is set to 0).
  • Camera poses (color_XX.cam): world-to-camera [tx, ty, tz, R00, R01, R02, R10, R11, R12, R20, R21, R22].

3. Dependencies

The code has following prerequisites:

  • ubuntu 16.04
  • gcc (5.4.0)
  • OpenCV (2.4.10)
  • Eigen (>3.0)
  • png12
  • jpeg

4. Parameters

All the parameters can be set in the file Config/config.yml as follows:

%YAML:1.0
depth_fx: 540.69
depth_fy: 540.69
depth_cx: 479.75
depth_cy: 269.75
depth_width: 960
depth_height: 540

RGB_fx: 1081.37
RGB_fy: 1081.37
RGB_cx: 959.5
RGB_cy: 539.5
RGB_width: 1920
RGB_height: 1080
.
.
.

5. Results

Some precomputed results can be found in the folder results/.

Comments
  • The number of channels of depth map

    The number of channels of depth map

    Hi, thanks a lot for your great work! I am trying to run this code on my own recording and encountered the same error mentioned in #16 with

    OpenCV Error: Assertion failed (scn == 3 || scn == 4) in cvtColor
    

    I guess this is related to color space conversion in your library. I am wondering whether you enforce the 16bit depth to be saved in some specific shape, e.g. [H, W, C] which must be 3-dimensional even C=1?

    Any suggestion will be appreciated! Thanks

    opened by Xiaoming-Zhao 2
  • question about camera pose file

    question about camera pose file

    Thanks for your work of G2LTex. I am following this work. There is an issue about the data. Would please tell me what the 12 figures in .cam file means? The first nine figures is the rotation matrix? Or the fourth, eighth, and twelfth figure combine as the translation vector?

    I am looking forward for your reply. Thanks for your time.

    opened by eckertzhang 0
  • you are using an empty matrix

    you are using an empty matrix"' failed

    Hello,I meeting a problem as follows,I kept all the input formats consistent with the author. I don't know what could be a bug.I am new at it,if you give me some suggesions ,i will be appricate it . (The example given by the author can run through.)

    The command line information is as follows: ``Reading PLY: 854151 verts... 281965 faces...PLY Loader: Ignoring face with 5 vertices! PLY Loader: Ignoring face with 5 vertices! PLY Loader: Ignoring face with 5 vertices! ... Warning: Zero-length normals detected: 15001 face normals, 45812 vertex normals Generating texture views: -----------tok ext size:12 depth fx:661.75 fy:661.75 cx:382.39 cy:277.98 width:768 height:576 RGB fx:661.75 fy:661.75 cx:382.39 cy:277.98 width:768 height:576 Global depth:100 RGB:1 Local RGB:1 edge:1 Reg:100 Global Iterations:300 Local Iterations:20 -----------tok ext size:12 -----------tok ext size:12 ... num_views:33 0 total edges. 260863 faces have not been seen by a view. Saving model... done. loading face info ... Global iter:0/300 Global iter:50/300 ... local option iteration:18/20 local option iteration:19/20 ----Global Texture Color Correction------ G2LTex: /usr/include/eigen3/Eigen/src/Core/Redux.h:411: typename Eigen::internal::traits::Scalar Eigen::DenseBase::redux(const Func&) const [with BinaryOp = Eigen::internal::scalar_sum_op; Derived = Eigen::Matrix<float, -1, 1>; typename Eigen::internal::traits::Scalar = float]: Assertion `this->rows()>0 && this->cols()>0 && "you are using an empty matrix"' failed. Aborted (core dumped)

    opened by YuzhiChen001 0
  • Assertion `valid_pixel(p1) && valid_pixel(p2) && valid_pixel(p3)' failed. Aborted

    Assertion `valid_pixel(p1) && valid_pixel(p2) && valid_pixel(p3)' failed. Aborted

    Hi. Thank you for your great work. It's really interesting and helpful. I am trying to run your program with your test data and I am getting the following error: num_views:219 982349 total edges. 371352 faces have not been seen by a view. Saving model... done. loading face info ... G2LTex: tex/texture_view.cpp:195: void tex::TextureView::get_face_info(const Vec3f&, const Vec3f&, const Vec3f&, tex::FaceProjectionInfo*, const tex::Settings&) const: Assertion `valid_pixel(p1) && valid_pixel (p2) && valid_pixel(p3)' failed. Aborted What should I do to make everything work smoothly? изображение

    opened by R-285 0
  • Compatibility with Ubuntu 18

    Compatibility with Ubuntu 18

    Is the repo executable on Ubuntu 18? Getting shared library errors on executing the code: ./G2LTex ../Data/bloster/textureimages ../Data/bloster/bloster.ply Errors:

    1. ./G2LTex: error while loading shared libraries: libopencv_core.so.2.4: cannot open shared object file: No such file or directory
    2. ./G2LTex: error while loading shared libraries: libG2LTex.so.1: cannot open shared object file: No such file or directory Is this due to ubuntu version?
    opened by nandinibajaj 4
  • Set texture resolution?

    Set texture resolution?

    Hi, I managed to run G2LTex successfully on my data, but I have a few questions. I end up with several textured meshes with one or more texture files per mesh, the biggest ones being 1024*1024.

    • Is there a way to generate bigger textures, e.g. 2048*2048? Is it a fixed dimension, or does it depend on input data?
    • Also, given the atlas computation is part of your method, I suppose it's not possible to provide my own texture atlas?
    • My mesh is only partly textured, some small parts are left black (not covered by the atlas maybe?), Do you know what might be causing this?
    • Finally, it is not clear to me what the different output meshes are (result_label, G2LTex, and orgTexSelection). Could you provide some explanation? In my case, orgTexSelection gives the best result.

    Thank you for your help.

    opened by Elensil 0
  • Could not load image header of ../Data/bloster/textureimages/color_00.jpg

    Could not load image header of ../Data/bloster/textureimages/color_00.jpg

    Thanks for your work, I have installed libjpeg-dev and libpng12-dev, but it output "Could not load image header of ../Data/bloster/textureimages/color_00.jpg" "Cannot determine image format" when I run the program. Do you know why this happens ?

    opened by dongshengLiu 3
Owner
Fu Yanping(付燕平)
Fu Yanping(付燕平)
SSL_SLAM2: Lightweight 3-D Localization and Mapping for Solid-State LiDAR (mapping and localization separated) ICRA 2021

SSL_SLAM2 Lightweight 3-D Localization and Mapping for Solid-State LiDAR (Intel Realsense L515 as an example) This repo is an extension work of SSL_SL

Wang Han 王晗 1.3k Jan 8, 2023
DSAC* for Visual Camera Re-Localization (RGB or RGB-D)

DSAC* for Visual Camera Re-Localization (RGB or RGB-D) Introduction Installation Data Structure Supported Datasets 7Scenes 12Scenes Cambridge Landmark

Visual Learning Lab 143 Dec 22, 2022
3DMV jointly combines RGB color and geometric information to perform 3D semantic segmentation of RGB-D scans.

3DMV 3DMV jointly combines RGB color and geometric information to perform 3D semantic segmentation of RGB-D scans. This work is based on our ECCV'18 p

Владислав Молодцов 0 Feb 6, 2022
Poisson Surface Reconstruction for LiDAR Odometry and Mapping

Poisson Surface Reconstruction for LiDAR Odometry and Mapping Surfels TSDF Our Approach Table: Qualitative comparison between the different mapping te

Photogrammetry & Robotics Bonn 305 Dec 21, 2022
CoReNet is a technique for joint multi-object 3D reconstruction from a single RGB image.

CoReNet CoReNet is a technique for joint multi-object 3D reconstruction from a single RGB image. It produces coherent reconstructions, where all objec

Google Research 80 Dec 25, 2022
Towards uncontrained hand-object reconstruction from RGB videos

Towards uncontrained hand-object reconstruction from RGB videos Yana Hasson, Gül Varol, Ivan Laptev and Cordelia Schmid Project page Paper Table of Co

Yana 69 Dec 27, 2022
OcclusionFusion: realtime dynamic 3D reconstruction based on single-view RGB-D

OcclusionFusion (CVPR'2022) Project Page | Paper | Video Overview This repository contains the code for the CVPR 2022 paper OcclusionFusion, where we

Wenbin Lin 193 Dec 15, 2022
Research code for CVPR 2021 paper "End-to-End Human Pose and Mesh Reconstruction with Transformers"

MeshTransformer ✨ This is our research code of End-to-End Human Pose and Mesh Reconstruction with Transformers. MEsh TRansfOrmer is a simple yet effec

Microsoft 473 Dec 31, 2022
The code for the CVPR 2021 paper Neural Deformation Graphs, a novel approach for globally-consistent deformation tracking and 3D reconstruction of non-rigid objects.

Neural Deformation Graphs Project Page | Paper | Video Neural Deformation Graphs for Globally-consistent Non-rigid Reconstruction Aljaž Božič, Pablo P

Aljaz Bozic 134 Dec 16, 2022
Source code for CVPR 2021 paper "Riggable 3D Face Reconstruction via In-Network Optimization"

Riggable 3D Face Reconstruction via In-Network Optimization Source code for CVPR 2021 paper "Riggable 3D Face Reconstruction via In-Network Optimizati

null 130 Jan 2, 2023
Demo code for ICCV 2021 paper "Sensor-Guided Optical Flow"

Sensor-Guided Optical Flow Demo code for "Sensor-Guided Optical Flow", ICCV 2021 This code is provided to replicate results with flow hints obtained f

null 10 Mar 16, 2022
3D ResNets for Action Recognition (CVPR 2018)

3D ResNets for Action Recognition Update (2020/4/13) We published a paper on arXiv. Hirokatsu Kataoka, Tenga Wakamiya, Kensho Hara, and Yutaka Satoh,

Kensho Hara 3.5k Jan 6, 2023
StarGAN - Official PyTorch Implementation (CVPR 2018)

StarGAN - Official PyTorch Implementation ***** New: StarGAN v2 is available at https://github.com/clovaai/stargan-v2 ***** This repository provides t

Yunjey Choi 5.1k Jan 4, 2023
PointNetVLAD: Deep Point Cloud Based Retrieval for Large-Scale Place Recognition, CVPR 2018

PointNetVLAD: Deep Point Cloud Based Retrieval for Large-Scale Place Recognition PointNetVLAD: Deep Point Cloud Based Retrieval for Large-Scale Place

Mikaela Uy 294 Dec 12, 2022
Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentation, CVPR 2018

Learning Pixel-level Semantic Affinity with Image-level Supervision This code is deprecated. Please see https://github.com/jiwoon-ahn/irn instead. Int

Jiwoon Ahn 337 Dec 15, 2022
Deep RGB-D Saliency Detection with Depth-Sensitive Attention and Automatic Multi-Modal Fusion (CVPR'2021, Oral)

DSA^2 F: Deep RGB-D Saliency Detection with Depth-Sensitive Attention and Automatic Multi-Modal Fusion (CVPR'2021, Oral) This repo is the official imp

如今我已剑指天涯 46 Dec 21, 2022
PyTorch code for our ECCV 2018 paper "Image Super-Resolution Using Very Deep Residual Channel Attention Networks"

PyTorch code for our ECCV 2018 paper "Image Super-Resolution Using Very Deep Residual Channel Attention Networks"

Yulun Zhang 1.2k Dec 26, 2022
Code for the paper "Adversarially Regularized Autoencoders (ICML 2018)" by Zhao, Kim, Zhang, Rush and LeCun

ARAE Code for the paper "Adversarially Regularized Autoencoders (ICML 2018)" by Zhao, Kim, Zhang, Rush and LeCun https://arxiv.org/abs/1706.04223 Disc

Junbo (Jake) Zhao 399 Jan 2, 2023
Code for paper "Which Training Methods for GANs do actually Converge? (ICML 2018)"

GAN stability This repository contains the experiments in the supplementary material for the paper Which Training Methods for GANs do actually Converg

Lars Mescheder 885 Jan 1, 2023