[ICCV 2021 (oral)] Planar Surface Reconstruction from Sparse Views

Overview

Planar Surface Reconstruction From Sparse Views

Linyi Jin, Shengyi Qian, Andrew Owens, David F. Fouhey
University of Michigan
ICCV 2021 (Oral)


This repo contains code for our paper. Our model is implemented in Detectron2.

Given two RGB images with an unknown relationship, our system produces a single, coherent planar surface reconstruction of the scene in terms of 3D planes and relative camera poses.

model-architecture We use a ResNet50-FPN to detect planes and predict probabilities of relative camera poses, and use a two-step optimization to generate a coherent planar reconstruction. (a) For each plane, we predict a segmentation mask, plane parameters, and an appearance feature. (b) Concurrently, we pass image features from the detection backbone through the attention layer and predict the camera transformation between views. (c) Our discrete optimization fuses the prediction of the separate heads to select the best camera pose and plane correspondence. (d) Finally, we use continuous optimization to update the camera and plane parameters.

Usage Instructions

  1. How to setup your environment?
  2. How to inference the code on a pair of images?
  3. How to process the dataset?
  4. How to train your model?
  5. How to evaluate your model?

Citation

If you find this code useful, please consider citing:

@inproceedings{jin2021planar,
      title={Planar Surface Reconstruction from Sparse Views}, 
      author={Linyi Jin and Shengyi Qian and Andrew Owens and David F. Fouhey},
      booktitle = {ICCV},
      year={2021}
}

Acknowledgment

We thank Dandan Shan, Mohamed El Banani, Nilesh Kulkarni, Richard Higgins for helpful discussions. Toyota Research Institute ("TRI") provided funds to assist the authors with their research but this article solely reflects the opinions and conclusions of its authors and not TRI or any other Toyota entity.

Comments
  • how many images I can input ?

    how many images I can input ?

    Hello nice job) Have you tested this on more than two images ? --img-list /path/to/img_list.txt \

    img_list.txt:

    img1.png img2.png img3.png ...... what about structure img_list.txt

    Can I add 3 or 10 images ?

    opened by KirillHiddleston 4
  • Dataset download links seem extremely slow

    Dataset download links seem extremely slow

    Hi, thanks for sharing this amazing work!

    The dataset download link are extremely slow for me. I tried on multiple networks.

    Is there an alternative link to download the dataset?

    Thanks, Pushkar

    opened by Pushkar2307 3
  • Training details

    Training details

    Hi,thanks for your great work!

    I home some trouble for training the network on step1(the detect plane stage) because I am not sure when to stop! I am training the network on Scannet dataset using your pre-trained model,and my loss is continuing to decline slowly during 40k iteration. Could you give me some advice?Thank you! 2022-03-12 16-27-06 的屏幕截图

    opened by zhirui-gao 3
  • ValueError: Textures do not match the dimensions of Meshes.

    ValueError: Textures do not match the dimensions of Meshes.

    Hi, Thanks for your great job. After running "python ./tools/inference_sparse_plane.py --config-file ./tools/demo/config.yaml --input ./tools/demo/teaser --output ./debug", I met an error as follows, do you know what caused it and how to solve it? Traceback (most recent call last): File "./tools/inference_sparse_plane.py", line 766, in main() File "./tools/inference_sparse_plane.py", line 762, in main inference_pair(output_dir, model, dis_opt, con_opt, im0, im1) File "./tools/inference_sparse_plane.py", line 727, in inference_pair webvis=True, File "./tools/inference_sparse_plane.py", line 627, in save_pair_objects cam_meshes = get_camera_meshes(cam_list) File "/home/jia/workspace/plane_resc_210617/SparsePlanes/sparsePlane/sparseplane/utils/mesh_utils.py", line 378, in get_camera_meshes meshes = Meshes(verts=verts_list, faces=faces_list, textures=tex) File "/home/jia/anaconda3/envs/sparseplane/lib/python3.7/site-packages/pytorch3d/structures/meshes.py", line 436, in init raise ValueError(msg) ValueError: Textures do not match the dimensions of Meshes. Screenshot from 2021-09-08 14-11-52

    opened by SJingjia 3
  • requirements

    requirements

    No module named 'KMSolver' No module named 'seaborn' ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject u need uninstall numpy 1.19.2 and change to pip install numpy==1.20.0 and no problem install seaborn

    How to install KMSolver ????

    opened by KirillHiddleston 2
  • Looking forward to the evaluation code.

    Looking forward to the evaluation code.

    Hi, Great work! Hope to see the evaluation code soon. I have some confusion about the evaluation process for the final results. Although I find some metric codes in the project, it seems that they are used for validation. Best.

    opened by IceTTTb 1
  • Custom dataset

    Custom dataset

    "plane": # plane parameters Can u explain hot to get "plane" - plane parameters from depth and segmentation maps ? maybe u have some utils.py ?

    opened by KirillHiddleston 1
Owner
Linyi Jin
CS Ph.D. student working on Computer Vision
Linyi Jin
git《Learning Pairwise Inter-Plane Relations for Piecewise Planar Reconstruction》(ECCV 2020) GitHub:

Learning Pairwise Inter-Plane Relations for Piecewise Planar Reconstruction Code for the ECCV 2020 paper by Yiming Qian and Yasutaka Furukawa Getting

null 37 Dec 4, 2022
Indoor Panorama Planar 3D Reconstruction via Divide and Conquer

HV-plane reconstruction from a single 360 image Code for our paper in CVPR 2021: Indoor Panorama Planar 3D Reconstruction via Divide and Conquer (pape

sunset 36 Jan 3, 2023
Stereo Radiance Fields (SRF): Learning View Synthesis for Sparse Views of Novel Scenes

Stereo Radiance Fields (SRF): Learning View Synthesis for Sparse Views of Novel Scenes

null 111 Dec 29, 2022
[CVPR 2022 Oral] Crafting Better Contrastive Views for Siamese Representation Learning

Crafting Better Contrastive Views for Siamese Representation Learning (CVPR 2022 Oral) 2022-03-29: The paper was selected as a CVPR 2022 Oral paper! 2

null 249 Dec 28, 2022
Implementation of CVPR'2022:Reconstructing Surfaces for Sparse Point Clouds with On-Surface Priors

Reconstructing Surfaces for Sparse Point Clouds with On-Surface Priors (CVPR 2022) Personal Web Pages | Paper | Project Page This repository contains

null 151 Dec 26, 2022
Poisson Surface Reconstruction for LiDAR Odometry and Mapping

Poisson Surface Reconstruction for LiDAR Odometry and Mapping Surfels TSDF Our Approach Table: Qualitative comparison between the different mapping te

Photogrammetry & Robotics Bonn 305 Dec 21, 2022
Implementation for the "Surface Reconstruction from 3D Line Segments" paper.

Surface Reconstruction from 3D Line Segments Surface reconstruction from 3d line segments. Langlois, P. A., Boulch, A., & Marlet, R. In 2019 Internati

null 85 Jan 4, 2023
Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance

Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance Project Page | Paper | Data This repository contains an implementatio

Lior Yariv 521 Dec 30, 2022
The official implementation code of "PlantStereo: A Stereo Matching Benchmark for Plant Surface Dense Reconstruction."

PlantStereo This is the official implementation code for the paper "PlantStereo: A Stereo Matching Benchmark for Plant Surface Dense Reconstruction".

Wang Qingyu 14 Nov 28, 2022
ViSER: Video-Specific Surface Embeddings for Articulated 3D Shape Reconstruction

ViSER: Video-Specific Surface Embeddings for Articulated 3D Shape Reconstruction. NeurIPS 2021.

Gengshan Yang 59 Nov 25, 2022
Deep Surface Reconstruction from Point Clouds with Visibility Information

Data, code and pretrained models for the paper Deep Surface Reconstruction from Point Clouds with Visibility Information.

Raphael Sulzer 23 Jan 4, 2023
Implementation of CVPR'2022:Surface Reconstruction from Point Clouds by Learning Predictive Context Priors

Surface Reconstruction from Point Clouds by Learning Predictive Context Priors (CVPR 2022) Personal Web Pages | Paper | Project Page This repository c

null 136 Dec 12, 2022
Python and C++ implementation of "MarkerPose: Robust real-time planar target tracking for accurate stereo pose estimation". Accepted at LXCV @ CVPR 2021.

MarkerPose: Robust real-time planar target tracking for accurate stereo pose estimation This is a PyTorch and LibTorch implementation of MarkerPose: a

Jhacson Meza 47 Nov 18, 2022
(CVPR 2022 Oral) Official implementation for "Surface Representation for Point Clouds"

RepSurf - Surface Representation for Point Clouds [CVPR 2022 Oral] By Haoxi Ran* , Jun Liu, Chengjie Wang ( * : corresponding contact) The pytorch off

Haoxi Ran 264 Dec 23, 2022
Code for "NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video", CVPR 2021 oral

NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video Project Page | Paper NeuralRecon: Real-Time Coherent 3D Reconstruction from Mon

ZJU3DV 1.4k Dec 30, 2022
Differentiable Neural Computers, Sparse Access Memory and Sparse Differentiable Neural Computers, for Pytorch

Differentiable Neural Computers and family, for Pytorch Includes: Differentiable Neural Computers (DNC) Sparse Access Memory (SAM) Sparse Differentiab

ixaxaar 302 Dec 14, 2022
Planar Prior Assisted PatchMatch Multi-View Stereo

ACMP [News] The code for ACMH is released!!! [News] The code for ACMM is released!!! About This repository contains the code for the paper Planar Prio

Qingshan Xu 127 Dec 31, 2022
This package is for running the semantic SLAM algorithm using extracted planar surfaces from the received detection

Semantic SLAM This package can perform optimization of pose estimated from VO/VIO methods which tend to drift over time. It uses planar surfaces extra

Hriday Bavle 125 Dec 2, 2022
PyTorch implementation of HDN(Homography Decomposition Networks) for planar object tracking

Homography Decomposition Networks for Planar Object Tracking This project is the offical PyTorch implementation of HDN(Homography Decomposition Networ

CaptainHook 48 Dec 15, 2022