Deep Implicit Moving Least-Squares Functions for 3D Reconstruction

Overview

DeepMLS: Deep Implicit Moving Least-Squares Functions for 3D Reconstruction

This repository contains the implementation of the paper:

Deep Implicit Moving Least-Squares Functions for 3D Reconstruction [arXiv]
Shi-Lin Liu, Hao-Xiang Guo, Hao Pan, Pengshuai Wang, Xin Tong, Yang Liu.

If you find our code or paper useful, please consider citing

@inproceedings{Liu2021MLS,
 author =  {Shi-Lin Liu, Hao-Xiang Guo, Hao Pan, Pengshuai Wang, Xin Tong, Yang Liu},
 title = {Deep Implicit Moving Least-Squares Functions for 3D Reconstruction},
 year = {2021}}

Installation

First you have to make sure that you have all dependencies in place. The simplest way to do so, is to use anaconda.

You can create an anaconda environment called deep_mls using

conda env create -f environment.yml
conda activate deep_mls

Next, a few customized tensorflow modules should be installed:

O-CNN Module

O-CNN is an octree-based convolution module, please take the following steps to install:

cd Octree && git clone https://github.com/microsoft/O-CNN/
cd O-CNN/octree/external && git clone --recursive https://github.com/wang-ps/octree-ext.git
cd .. && mkdir build && cd build
cmake ..  && cmake --build . --config Release
export PATH=`pwd`:$PATH
cd ../../tensorflow/libs && python build.py --cuda /usr/local/cuda-10.0
cp libocnn.so ../../../ocnn-tf/libs

Efficient Neighbor Searching Ops

Neighbor searching is intensively used in DeepMLS. For efficiency reasons, we provide several customized neighbor searching ops:

cd points3d-tf/points3d
bash build.sh

In this step, some errors like this may occur:

tensorflow_core/include/tensorflow/core/util/gpu_kernel_helper.h:22:10: fatal error: third_party/gpus/cuda/include/cuda_fp16.h: No such file or directory
 #include "third_party/gpus/cuda/include/cuda_fp16.h"

For solving this, please refer to issue.
Basically, We need to edit the codes in tensorflow framework, please modify

#include "third_party/gpus/cuda/include/cuda_fp16.h"

in "site-packages/tensorflow_core/include/tensorflow/core/util/gpu_kernel_helper.h" to

#include "cuda_fp16.h"

and

#include "third_party/gpus/cuda/include/cuComplex.h"
#include "third_party/gpus/cuda/include/cuda.h"

in "site-packages/tensorflow_core/include/tensorflow/core/util/gpu_device_functions.h" to

#include "cuComplex.h"
#include "cuda.h"

Modified Marching Cubes Module

We have modified the PyMCubes to get a more efficient marching cubes method for extract 0-isosurface defined by mls points.
To install:

git clone https://github.com/Andy97/PyMCubes
cd PyMCubes && python setup.py install

Datasets

Preprocessed ShapeNet Dataset

We have provided the processed tfrecords file. This can be used directly.

Our training data is available now! (total 130G+)
Please download all zip files for extraction.
ShapeNet_points_all_train.zip.001
ShapeNet_points_all_train.zip.002
ShapeNet_points_all_train.zip.003
After extraction, please modify the "train_data" field in experiment config json file with this tfrecords name.

Build the Dataset

If you want to build the dataset from your own data, please follow:

Step 1: Get Watertight Meshes

To acquire a watertight mesh, we first preprocess each mesh follow the preprocess steps of Occupancy Networks.

Step 2: Get the groundtruth sdf pair

From step 1, we have already gotten the watertight version of each model. Then, we utilize OpenVDB library to get the sdf values and gradients for training.
For details, please refer to here.

Usage

Inference using pre-trained model

We have provided pretrained models which can be used to inference:

#first download the pretrained models
cd Pretrained && python download_models.py
#then we can use either of the pretrained model to do the inference
cd .. && python DeepMLS_Generation.py Pretrained/Config_d7_1p_pretrained.json --test

The input for the inference is defined in here.
Your can replace it with other point cloud files in examples or your own data.

Extract Isosurface from MLS Points

After inference, now we have network predicted mls points. The next step is to extract the surface:

python mls_marching_cubes.py --i examples/d0fa70e45dee680fa45b742ddc5add59.ply.xyz --o examples/d0fa70e45dee680fa45b742ddc5add59_mc.obj --scale

Training

Our code supports single and multiple gpu training. For details, please refer to the config json file.

python DeepMLS_Generation.py examples/Config_g2_bs32_1p_d6.json

Evaluation

For evaluation of results, ConvONet has provided a great script. Please refer to here.

Comments
  • tensorflow.python.framework.errors_impl.NotFoundError: libcudart.so.10.0: cannot open shared object file: No such file or directory

    tensorflow.python.framework.errors_impl.NotFoundError: libcudart.so.10.0: cannot open shared object file: No such file or directory

    When using Python build.py -- cuda-/usr/local/cuda-10.0, I encountered the following problem:tensorflow.python.framework.errors_impl.NotFoundError: libcudart.so.10.0: cannot open shared object file: No such file or directory

    opened by novauto-nju 3
  • TFRecords preprocessing

    TFRecords preprocessing

    Hi, I want to generate tfrecords training data from another mesh dataset (which has already been watertight), but there seems to be no instruction on this step. I have read this istruction, but the code seems to compute sdf values only and have no help with my question. I found there are fields in the tfrecords data that I don't know what they means and where they come from, such as "sdf_samples_depth6", "sdf_samples_depth7", "depth6_coord" and "depth7_coord". Can you give a more detailed instruction on how to generate the right-format tfrecords data? Thanks a lot!

    opened by bluestyle97 2
  • /.conda/envs/deep_mls/lib/python3.7/site-packages/tensorflow_core/include/absl/strings/string_view.h(495): warning: expression has no effect

    /.conda/envs/deep_mls/lib/python3.7/site-packages/tensorflow_core/include/absl/strings/string_view.h(495): warning: expression has no effect

    Hi~ Complete TensorFlow after modifying #include "cuComplex.h" #include "cuda.h" ,I code bash build.sh again.But I encountered some tough problems´╝Ü

    .conda/envs/deep_mls/lib/python3.7/site-packages/tensorflow_core/include/absl/strings/string_view.h(495): warning: expression has no effect

    .conda/envs/deep_mls/lib/python3.7/site-packages/tensorflow_core/include/absl/types/optional.h(425): warning: expression has no effect detected during instantiation of "const T &absl::optional::operator*() const & [with T=stream_executor::dnn::AlgorithmDesc]"

    .conda/envs/deep_mls/lib/python3.7/site-packages/tensorflow_core/include/tensorflow/stream_executor/dnn.h(802): here

    .conda/envs/deep_mls/lib/python3.7/site-packages/tensorflow_core/include/absl/types/optional.h(425): warning: expression has no effect detected during instantiation of "const T &absl::optional::operator*() const & [with T=size_t]" .conda/envs/deep_mls/lib/python3.7/site-packages/tensorflow_core/include/tensorflow/stream_executor/dnn.h(856): here

    Do you know how to solve it?

    opened by novauto-nju 2
  • what does the ground truth shape mean?

    what does the ground truth shape mean?

    Dear Andy, In your paper, "each input point cloud contains 3000 points, randomly sampled from the ground-truth shape and Gaussian noise with standard deviation set to 0.005 of the maximum bounding box side length of the shape."

    I want to ask what you mean "the ground truth shape". The original CAD model in obj file or the mesh from 256**3 voxel followed by the marching cube. Thanks a lot. @Andy97

    opened by THHHomas 1
  • Unable to download dataset

    Unable to download dataset

    Thank you for your work. When I tried to download the dataset, the google drive reminded me that the file does not exist. Could you provide other links for downloading?

    opened by TianshuoMa 0
  • About datasets downloading

    About datasets downloading

    hello andy , i tried to download the datasets you provided in readme, but google driver said i need your permission to access ,i have send my application to you .wish you can allow me to download datasets, thanks. : )

    opened by fusheng-ji 0
Owner
null
DeepLM: Large-scale Nonlinear Least Squares on Deep Learning Frameworks using Stochastic Domain Decomposition (CVPR 2021)

DeepLM DeepLM: Large-scale Nonlinear Least Squares on Deep Learning Frameworks using Stochastic Domain Decomposition (CVPR 2021) Run Please install th

Jingwei Huang 130 Dec 2, 2022
Official PyTorch implementation of the paper "Deep Constrained Least Squares for Blind Image Super-Resolution", CVPR 2022.

Deep Constrained Least Squares for Blind Image Super-Resolution [Paper] This is the official implementation of 'Deep Constrained Least Squares for Bli

MEGVII Research 141 Dec 30, 2022
T-LOAM: Truncated Least Squares Lidar-only Odometry and Mapping in Real-Time

T-LOAM: Truncated Least Squares Lidar-only Odometry and Mapping in Real-Time The first Lidar-only odometry framework with high performance based on tr

Pengwei Zhou 183 Dec 1, 2022
MonoRec: Semi-Supervised Dense Reconstruction in Dynamic Environments from a Single Moving Camera

MonoRec: Semi-Supervised Dense Reconstruction in Dynamic Environments from a Single Moving Camera

Felix Wimbauer 494 Jan 6, 2023
Neural implicit reconstruction experiments for the Vector Neuron paper

Neural Implicit Reconstruction with Vector Neurons This repository contains code for the neural implicit reconstruction experiments in the paper Vecto

Congyue Deng 35 Jan 2, 2023
[ICCV'21] UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction

UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction Project Page | Paper | Supplementary | Video This reposit

null 331 Dec 28, 2022
A customisable game where you have to quickly click on black tiles in order of appearance while avoiding clicking on white squares.

W.I.P-Aim-Memory-Game A customisable game where you have to quickly click on black tiles in order of appearance while avoiding clicking on white squar

dE_soot 1 Dec 8, 2021
null 5 Jan 5, 2023
Rasterize with the least efforts for researchers.

utils3d Rasterize and do image-based 3D transforms with the least efforts for researchers. Based on numpy and OpenGL. It could be helpful when you wan

Ruicheng Wang 8 Dec 15, 2022
A Planar RGB-D SLAM which utilizes Manhattan World structure to provide optimal camera pose trajectory while also providing a sparse reconstruction containing points, lines and planes, and a dense surfel-based reconstruction.

ManhattanSLAM Authors: Raza Yunus, Yanyan Li and Federico Tombari ManhattanSLAM is a real-time SLAM library for RGB-D cameras that computes the camera

null 117 Dec 28, 2022
"MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction" (CVPRW 2022) & (Winner of NTIRE 2022 Challenge on Spectral Reconstruction from RGB)

MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction (CVPRW 2022) Yuanhao Cai, Jing Lin, Zudi Lin, Haoqian Wang, Yulun Z

Yuanhao Cai 274 Jan 5, 2023
Unofficial Tensorflow 2 implementation of the paper Implicit Neural Representations with Periodic Activation Functions

Siren: Implicit Neural Representations with Periodic Activation Functions The unofficial Tensorflow 2 implementation of the paper Implicit Neural Repr

Seyma Yucer 2 Jun 27, 2022
DeFMO: Deblurring and Shape Recovery of Fast Moving Objects (CVPR 2021)

Evaluation, Training, Demo, and Inference of DeFMO DeFMO: Deblurring and Shape Recovery of Fast Moving Objects (CVPR 2021) Denys Rozumnyi, Martin R. O

Denys Rozumnyi 139 Dec 26, 2022
Moving Object Segmentation in 3D LiDAR Data: A Learning-based Approach Exploiting Sequential Data

LiDAR-MOS: Moving Object Segmentation in 3D LiDAR Data This repo contains the code for our paper: Moving Object Segmentation in 3D LiDAR Data: A Learn

Photogrammetry & Robotics Bonn 394 Dec 29, 2022
performing moving objects segmentation using image processing techniques with opencv and numpy

Moving Objects Segmentation On this project I tried to perform moving objects segmentation using background subtraction technique. the introduced meth

Mohamed Magdy 15 Dec 12, 2022
Code for the SIGGRAPH 2021 paper "Consistent Depth of Moving Objects in Video".

Consistent Depth of Moving Objects in Video This repository contains training code for the SIGGRAPH 2021 paper "Consistent Depth of Moving Objects in

Google 203 Jan 5, 2023
A hand tracking demo made with mediapipe where you can control lights with pinching your fingers and moving your hand up/down.

HandTrackingBrightnessControl A hand tracking demo made with mediapipe where you can control lights with pinching your fingers and moving your hand up

Teemu Laurila 19 Feb 12, 2022
Simulation of moving particles under microscopic imaging

Simulation of moving particles under microscopic imaging Install scipy numpy scikit-image tiffile Run python simulation.py Read result https://imagej

Zehao Wang 2 Dec 14, 2021
Keyhole Imaging: Non-Line-of-Sight Imaging and Tracking of Moving Objects Along a Single Optical Path

Keyhole Imaging Code & Dataset Code associated with the paper "Keyhole Imaging: Non-Line-of-Sight Imaging and Tracking of Moving Objects Along a Singl

Stanford Computational Imaging Lab 20 Feb 3, 2022