MVSDF - Learning Signed Distance Field for Multi-view Surface Reconstruction

Last update: Dec 20, 2022

Related tags

Deep Learning MVSDF

Overview

MVSDF - Learning Signed Distance Field for Multi-view Surface Reconstruction

This is the official implementation for the ICCV 2021 paper Learning Signed Distance Field for Multi-view Surface Reconstruction

In this work, we introduce a novel neural surface reconstruction framework that leverages the knowledge of stereo matching and feature consistency to optimize the implicit surface representation. More specifically, we apply a signed distance field (SDF) and a surface light field to represent the scene geometry and appearance respectively. The SDF is directly supervised by geometry from stereo matching, and is refined by optimizing the multi-view feature consistency and the fidelity of rendered images. Our method is able to improve the robustness of geometry estimation and support reconstruction of complex scene topologies. Extensive experiments have been conducted on DTU, EPFL and Tanks and Temples datasets. Compared to previous state-of-the-art methods, our method achieves better mesh reconstruction in wide open scenes without masks as input.

How to Use

Environment Setup

The code is tested in the following environment (manually installed packages only). The newer version of the packages should also be fine.

dependencies:
  - cudatoolkit=10.2.89
  - numpy=1.19.2
  - python=3.8.8
  - pytorch=1.7.1
  - tqdm=4.60.0
  - pip:
    - cvxpy==1.1.12
    - gputil==1.4.0
    - imageio==2.9.0
    - open3d==0.13.0
    - opencv-python==4.5.1.48
    - pyhocon==0.3.57
    - scikit-image==0.18.3
    - scikit-learn==0.24.2
    - trimesh==3.9.13
    - pybind11==2.9.0

Data Preparation

Download preprocessed DTU datasets from here

Training

cd code
python training/exp_runner.py --data_dir <DATA_DIR>/scan<SCAN>/imfunc4 --batch_size 8 --nepoch 1800 --expname dtu_<SCAN>

The results will be written in exps/mvsdf_dtu_.

Trained Models

Download trained models and put them in exps folder. This set of models achieve the following results.

	Chamfer	PSNR
24	0.846	24.67
37	1.894	20.15
40	0.895	25.15
55	0.435	23.19
63	1.067	26.24
65	0.903	26.9
69	0.746	26.54
83	1.241	25.15
97	1.009	25.71
105	1.320	26.48
106	0.867	28.81
110	0.842	23.16
114	0.340	27.51
118	0.472	28.46
122	0.466	27.71
Mean	0.890	25.72

Testing

python evaluation/eval.py --data_dir <DATA_DIR>/scan<SCAN>/imfunc4 --expname dtu_<SCAN> [--eval_rendering]

add --eval_rendering flag to generate and evaluate rendered images. The results will be written in evals/mvsdf_dtu_.

Trimming

cd mesh_cut
python setup.py build_ext -i  # compile
python mesh_cut.py 
    
    
      [--thresh 15 --smooth 10]

Note that this part of code can only be used for research purpose. Please refer to mesh_cut/IBFS/license.txt

Evaluation

Apart from the official implementation, you can also use my re-implemented evaluation script.

Citation

If you find our work useful in your research, please kindly cite

@article{zhang2021learning,
	title={Learning Signed Distance Field for Multi-view Surface Reconstruction},
	author={Zhang, Jingyang and Yao, Yao and Quan, Long},
	journal={International Conference on Computer Vision (ICCV)},
	year={2021}
}

Comments

Question about tank and tamples dataset

Thanks again for sharing this great work. I have found that the results on tank and temples dataset only contains the horse and statues, while the floor and trees are not reconstructed. I am curious how you got this results. Do you have mask for those Interesting part or you remove floor and trees after reconstruction ?

opened by gaobodaxinwen 4
Gradients of the feature consistency loss

Hi, thanks for releasing this work.

I have a question about equation 7 in your paper, where you derive the gradients of the feature consistency loss with respect to network parameters. Why is it necessary to compute them explicitly? Intuitively, I would think that PyTorch autograd will figure them out automatically and in your code I couldn't find the corresponding explicit computation. If I missed it, can you point me to the correct line?

Thank you in advance!

opened by morsingher 2
DTU evaluation 结果

作者你好，我发现在你的两篇文章里（MVSDF和Vis-MVSNet），对于Vis-MVSNet在DTU上的evlauation的结果的差距比较大，如下图所示，一个是0.88，一个是0.365。如果我没有理解错的话，Chamfer（L1）就是Acc. Comp.的平均数，也就是overall。如果两个指标应该是同一个意思，为什么在两篇论文里会差距这么大呢？请多多指教，谢谢！

opened by decai-chen 2
question about evaluation on EPFL dataset

Thanks for sharing this great work. I am curious about the evaluation on EPFL fountain and Herzjesu dataset. It seems that you use this script to evaluate on the DTU dataset. Do you use the same script to evaluate on EPFL dataset ? What's more, could you please provide the masks of images and gt point clouds of EPFL dataset?

opened by gaobodaxinwen 2
Custom data training/testing
Hi,

Are there any guides for custom data training or testing?

I am confused about :

the meaning of the last four numbers in the camera files, e.g. the last line "425.0 1.875 256.0 903.125" in data/scan105/cam_00000000_flow3.txt;

The difference between 'mask_hd' and 'pmask';

Would you kindly provide the definitions or the converted scripts for them?

Thanks.
opened by mikirui 1

Owner

GitHub

A-SDF: Learning Disentangled Signed Distance Functions for Articulated Shape Representation (ICCV 2021)

A-SDF: Learning Disentangled Signed Distance Functions for Articulated Shape Representation (ICCV 2021) This repository contains the official implemen

81 Dec 14, 2022

Learning Continuous Signed Distance Functions for Shape Representation

DeepSDF This is an implementation of the CVPR '19 paper "DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation" by Park et a

1.1k Jan 1, 2023

(CVPR 2022 - oral) Multi-View Depth Estimation by Fusing Single-View Depth Probability with Multi-View Geometry

Multi-View Depth Estimation by Fusing Single-View Depth Probability with Multi-View Geometry Official implementation of the paper Multi-View Depth Est

138 Dec 28, 2022

Hand-distance-measurement-game - Hand Distance Measurement Game

Hand Distance Measurement Game This is program is made to calculate the distance

2 Jan 12, 2022

the code of the paper: Recurrent Multi-view Alignment Network for Unsupervised Surface Registration (CVPR 2021)

RMA-Net This repo is the implementation of the paper: Recurrent Multi-view Alignment Network for Unsupervised Surface Registration (CVPR 2021). Paper

205 Nov 9, 2022

Implementation of CVPR'2022:Surface Reconstruction from Point Clouds by Learning Predictive Context Priors

Surface Reconstruction from Point Clouds by Learning Predictive Context Priors (CVPR 2022) Personal Web Pages | Paper | Project Page This repository c

136 Dec 12, 2022

Blender add-on: Add to Cameras menu: View → Camera, View → Add Camera, Camera → View, Previous Camera, Next Camera

Blender add-on: Camera additions In 3D view, it adds these actions to the View|Cameras menu: View → Camera : set the current camera to the 3D view Vie

11 Feb 8, 2022

"MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction" (CVPRW 2022) & (Winner of NTIRE 2022 Challenge on Spectral Reconstruction from RGB)

MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction (CVPRW 2022) Yuanhao Cai, Jing Lin, Zudi Lin, Haoqian Wang, Yulun Z

274 Jan 5, 2023

A Planar RGB-D SLAM which utilizes Manhattan World structure to provide optimal camera pose trajectory while also providing a sparse reconstruction containing points, lines and planes, and a dense surfel-based reconstruction.

ManhattanSLAM Authors: Raza Yunus, Yanyan Li and Federico Tombari ManhattanSLAM is a real-time SLAM library for RGB-D cameras that computes the camera

117 Dec 28, 2022

MVSDF - Learning Signed Distance Field for Multi-view Surface Reconstruction

Related tags

Overview

MVSDF - Learning Signed Distance Field for Multi-view Surface Reconstruction

How to Use

Environment Setup

Data Preparation

Training

Trained Models

Testing

Trimming

Evaluation

Citation

Comments

Question about tank and tamples dataset

Gradients of the feature consistency loss

DTU evaluation 结果

question about evaluation on EPFL dataset

Custom data training/testing

Owner

A-SDF: Learning Disentangled Signed Distance Functions for Articulated Shape Representation (ICCV 2021)

Learning Continuous Signed Distance Functions for Shape Representation

(CVPR 2022 - oral) Multi-View Depth Estimation by Fusing Single-View Depth Probability with Multi-View Geometry

Hand-distance-measurement-game - Hand Distance Measurement Game

the code of the paper: Recurrent Multi-view Alignment Network for Unsupervised Surface Registration (CVPR 2021)

Implementation of CVPR'2022:Surface Reconstruction from Point Clouds by Learning Predictive Context Priors

Blender add-on: Add to Cameras menu: View → Camera, View → Add Camera, Camera → View, Previous Camera, Next Camera

"MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction" (CVPRW 2022) & (Winner of NTIRE 2022 Challenge on Spectral Reconstruction from RGB)

Poisson Surface Reconstruction for LiDAR Odometry and Mapping

Implementation for the "Surface Reconstruction from 3D Line Segments" paper.

[ICCV 2021 (oral)] Planar Surface Reconstruction from Sparse Views

Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance

The official implementation code of "PlantStereo: A Stereo Matching Benchmark for Plant Surface Dense Reconstruction."

ViSER: Video-Specific Surface Embeddings for Articulated 3D Shape Reconstruction

Deep Surface Reconstruction from Point Clouds with Visibility Information

[ICCV'21] UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction

Self-Supervised Monocular 3D Face Reconstruction by Occlusion-Aware Multi-view Geometry Consistency[ECCV 2020]

PN-Net a neural field-based framework for depth estimation from single-view RGB images.

A Planar RGB-D SLAM which utilizes Manhattan World structure to provide optimal camera pose trajectory while also providing a sparse reconstruction containing points, lines and planes, and a dense surfel-based reconstruction.