[NeurIPS'21] Shape As Points: A Differentiable Poisson Solver

Overview

Shape As Points (SAP)

Paper | Project Page | Short Video (6 min) | Long Video (12 min)

This repository contains the implementation of the paper:

Shape As Points: A Differentiable Poisson Solver
Songyou Peng, Chiyu "Max" Jiang, Yiyi Liao, Michael Niemeyer, Marc Pollefeys and Andreas Geiger
NeurIPS 2021 (Oral)

If you find our code or paper useful, please consider citing

@inproceedings{Peng2021SAP,
 author    = {Peng, Songyou and Jiang, Chiyu "Max" and Liao, Yiyi and Niemeyer, Michael and Pollefeys, Marc and Geiger, Andreas},
 title     = {Shape As Points: A Differentiable Poisson Solver},
 booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
 year      = {2021}}

Installation

First you have to make sure that you have all dependencies in place. The simplest way to do so, is to use anaconda.

You can create an anaconda environment called sap using

conda env create -f environment.yaml
conda activate sap

Now, you can install PyTorch3D 0.6.0 from the official instruction as follows

pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/py38_cu102_pyt190/download.html

And install PyTorch Scatter:

conda install pytorch-scatter -c pyg

Demo - Quick Start

First, run the script to get the demo data:

bash scripts/download_demo_data.sh

Optimization-based 3D Surface Reconstruction

You can now quickly test our code on the data shown in the teaser. To this end, simply run:

python optim_hierarchy.py configs/optim_based/teaser.yaml

This script should create a folder out/demo_optim where the output meshes and the optimized oriented point clouds under different grid resolution are stored.

To visualize the optimization process on the fly, you can set o3d_show: Frue in configs/optim_based/teaser.yaml.

Learning-based 3D Surface Reconstruction

You can also test SAP on another application where we can reconstruct from unoriented point clouds with either large noises or outliers with a learned network.

For the point clouds with large noise as shown above, you can run:

python generate.py configs/learning_based/demo_large_noise.yaml

The results can been found at out/demo_shapenet_large_noise/generation/vis.

As for the point clouds with outliers, you can run:

python generate.py configs/learning_based/demo_outlier.yaml

You can find the reconstrution on out/demo_shapenet_outlier/generation/vis.

Dataset

We have different dataset for our optimization-based and learning-based settings.

Dataset for Optimization-based Reconstruction

Here we consider the following dataset:

Please cite the corresponding papers if you use the data.

You can download the processed dataset (~200 MB) by running:

bash scripts/download_optim_data.sh

Dataset for Learning-based Reconstruction

We train and evaluate on ShapeNet. You can download the processed dataset (~220 GB) by running:

bash scripts/download_shapenet.sh

After, you should have the dataset in data/shapenet_psr folder.

Alternatively, you can also preprocess the dataset yourself. To this end, you can:

Usage for Optimization-based 3D Reconstruction

For our optimization-based setting, you can consider running with a coarse-to-fine strategy:

python optim_hierarchy.py configs/optim_based/CONFIG.yaml

We start from a grid resolution of 32^3, and increase to 64^3, 128^3 and finally 256^3.

Alternatively, you can also run on a single resolution with:

python optim.py configs/optim_based/CONFIG.yaml

You might need to modify the CONFIG.yaml accordingly.

Usage for Learning-based 3D Reconstruction

Mesh Generation

To generate meshes using a trained model, use

python generate.py configs/learning_based/CONFIG.yaml

where you replace CONFIG.yaml with the correct config file.

Use a pre-trained model

The easiest way is to use a pre-trained model. You can do this by using one of the config files with postfix _pretrained.

For example, for 3D reconstruction from point clouds with outliers using our model with 7x offsets, you can simply run:

python generate.py configs/learning_based/outlier/ours_7x_pretrained.yaml

The script will automatically download the pretrained model and run the generation. You can find the outputs in the out/.../generation_pretrained folders.

Note config files are only for generation, not for training new models: when these configs are used for training, the model will be trained from scratch, but during inference our code will still use the pretrained model.

We provide the following pretrained models:

noise_small/ours.pt
noise_large/ours.pt
outlier/ours_1x.pt
outlier/ours_3x.pt
outlier/ours_5x.pt
outlier/ours_7x.pt
outlier/ours_3plane.pt

Evaluation

To evaluate a trained model, we provide the script eval_meshes.py. You can run it using:

python eval_meshes.py configs/learning_based/CONFIG.yaml

The script takes the meshes generated in the previous step and evaluates them using a standardized protocol. The output will be written to .pkl and .csv files in the corresponding generation folder that can be processed using pandas.

Training

Finally, to train a new network from scratch, simply run:

python train.py configs/learning_based/CONFIG.yaml

For available training options, please take a look at configs/default.yaml.

Comments
  • Mesh Evaluation for the Optimization-based part

    Mesh Evaluation for the Optimization-based part

    Hi Songyou,

    Thanks again for sharing the code for your excellent work! I found that the script "eval_meshes.py" is prepared for the learning-based method, but I want to evaluate the mesh result produced by the optimization-based part. Could you please give some suggestions for this?

    Looking for your reply. :)

    Best, Runsong

    opened by Runsong123 9
  • Fail to setup pytorch3d 0.6.0 : a typo?

    Fail to setup pytorch3d 0.6.0 : a typo?

    In section Installation " Now, you can install PyTorch3D 0.6.0 from the official instruction as follows pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/py38_cu102_pyt190/download.html " but I can only get pytorch3d 0.0.1 to 0.3.0 from the above address. I want to check whether we should install pytorch3d 0.3.0 with the above command or manually install pytorch 0.6.0 from official instruction?

    P.S. I tried pytorch3d 0.3.0 but it ends up with ImportError: libcudart.so.10.1: cannot open shared object file: No such file or directory

    Thanks!

    opened by xiazeqing 6
  • Load DTU dataset. uv_creation is missing

    Load DTU dataset. uv_creation is missing

    Hi Songyou,

    Thanks again for sharing the code for your excellent work! It helps a lot!

    I am trying to implement optimization-based multi-view images reconstruction. I used your PixelNeRFDTUDataset to load DTU. However, I found there is a function called uv_creation is missing. And I am kind of stuck here. Could you please kindly provide your code for that function?

    Thanks a lot!

    opened by jiangsijia9495 4
  • question about interpolation method

    question about interpolation method

    Thanks for sharing this great work! It seems that the interpolation method used in the point_rasterize and grid_interp function is based on the absolute distance of the point to the adjacent grid point. Have you ever tried other interpolation method such as trilinear interpolation? Do different interpolation methods affect the results?

    opened by l1346792580123 3
  • Pretrained model for the optimization-based version

    Pretrained model for the optimization-based version

    Hi Songyou,

    Thanks again for sharing the code for your excellent work! Could you please provide the pretrained model of optimization-based version that are reported in your paper (there are three datasets: dfaust, thingi, deep_geometric_prior_data).

    Looking for your reply. :)

    Best, Runsong

    opened by Runsong123 2
  • question about rasterize function

    question about rasterize function

    Thanks for sharing this great work! I am a little confused about the rasterize function. It seems that the weight is the product of the absolute distance, and the value on the gird is the weighted sum of all the relevant points. Does the grid value need to be normalized? Because I have found that when I rasterize the parameters of the points to the grid, the value on the grid increases as the number of the sampled points increases. But when I normalize the value on the grid, there is an error of marching cube. Do you have any advice or is there any relationship between the number of sampling points and the grid size?

    opened by l1346792580123 2
  • Input all Points for Optimization-based method?

    Input all Points for Optimization-based method?

    Hi Songyou,

    Thanks for sharing your code for the excellent work! I am curious about the optimization-based part and I have two questions. Can you help me? thanks a lot in advance. :)

    • Do you use the whole point cloud? For example, the "daratech" shape(SRB dataset) contains 71265 point cloud, you put all the point clouds for your methods or you downsample them first.
    • Can you provide the parameter for Poisson Surface Reconstruction(e.g. depth) and parameter for minimum spanning tree method if possible.

    Looking forward to your reply!

    Best, Runsong

    opened by Runsong123 2
  • The datasets can`t be fetched.

    The datasets can`t be fetched.

    Hello, I want to download the demo data,but falied to establish SSL connection. Could you please upload a copy to GoogleDrive at the same time?
    Thanks!

    wget https://s3.eu-central-1.amazonaws.com/avg-projects/shape_as_points/data/demo.zip
    
    --2021-11-10 10:33:21--  https://s3.eu-central-1.amazonaws.com/avg-projects/shape_as_points/data/demo.zip
    Proxy tunneling failed: Bad GatewayUnable to establish SSL connection.
    
    opened by ZipengFeng 2
  • What is train_overfit.lst and test_overfit.lst?

    What is train_overfit.lst and test_overfit.lst?

    Thank you for great work

    I'm curious about what train_overfit.lst and test_overfit.lst mean in the preprocessed dataset which can be downloaded in Dataset for Learning-based Reconstruction?

    I cannot find any clue in scripts/preprocess_shapenet.py.

    Thank you

    opened by Kitsunetic 1
  • weird results for depth map input

    weird results for depth map input

    I use SAP to reconstruct mesh from depth map input, while the result is somekind weird. It seems that SAP try to reconstruct closed shape. For those only one side oriented point clouds extracted from depth map SAP additionally generates some discrete faces. Is there any way to remove these faces?

    opened by l1346792580123 1
  • Optimization-based vs. Learning-based results

    Optimization-based vs. Learning-based results

    Hey, Thank you for publishing the code and congratulations on getting the NiPS oral! Could you please provide some comparison of the Optimization-based vs. Learning-based results or at least provide some insight into what works better under what circumstances. (I looked through the paper and SM and couldn't find this, but if it exists and I've missed it, could you please point me in the right direction?)

    Thanks, Eliahu

    opened by eliahuhorwitz 1
  • Segmentation fault (core dumped)

    Segmentation fault (core dumped)

    got error when trying to make parameter o3d_show as True in yaml file.

    aga/res_32 --train:lr_pcl 0.002000 --data:object_id -1 Changing model:grid_res ---- 256 to 32 Changing model:psr_sigma ---- 2 to 2 Changing train:input_mesh ---- to None Changing train:total_epochs ---- 4000 to 1000 Changing train:out_dir ---- /dsk1/shape_as_points/after/aga to /dsk1/shape_as_points/after/aga/res_32 Changing train:lr_pcl ---- 2e-2 to 0.002000 Changing data:object_id ---- 0 to -1 /dsk1/shape_as_points/after/aga/res_32 Segmentation fault (core dumped)

    Couldn't find the right solution to the questions, how can this be fixed?

    opened by Siziff 2
  • The question is about the model and training.

    The question is about the model and training.

    Good afternoon, I have a few questions, I am trying to do a point cloud processing cycle for my master's work at the university and I will be very glad if you answer my questions:

    1. From your article, I did not quite understand what exactly is being trained in the model, could you clarify which parameters are being improved in the learning process?
    2. Do I understand correctly that you trained your model on all three datasets? I am trying to optimize a point cloud in the form of an airplane, and the result of your model does not turn out to be as high-quality as I would like. I want to train the model on 3D figures of planes, BUT there are already planes in the ShapeNet dataset on which you trained your model, and despite this, the result is still not very good (the result is attached to the cart). Does it make sense to retrain your model ONLY on airplanes to improve the result ? The original number of points in the aircraft model is about 60000+-.

    image image

    opened by Siziff 0
This repository is the official implementation of Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning (NeurIPS21).

Core-tuning This repository is the official implementation of ``Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regular

vanint 18 Dec 17, 2022
LBK 35 Dec 26, 2022
[ICLR 2022] Contact Points Discovery for Soft-Body Manipulations with Differentiable Physics

CPDeform Code and data for paper Contact Points Discovery for Soft-Body Manipulations with Differentiable Physics at ICLR 2022 (Spotlight). @InProceed

(Lester) Sizhe Li 29 Nov 29, 2022
Sudoku solver - A sudoku solver with python

sudoku_solver A sudoku solver What is Sudoku? Sudoku (Japanese: 数独, romanized: s

Sikai Lu 0 May 22, 2022
Simple Linear 2nd ODE Solver GUI - A 2nd constant coefficient linear ODE solver with simple GUI using euler's method

Simple_Linear_2nd_ODE_Solver_GUI Description It is a 2nd constant coefficient li

:) 4 Feb 5, 2022
2D Time independent Schrodinger equation solver for arbitrary shape of well

Schrodinger Well Python Python solver for timeless Schrodinger equation for well with arbitrary shape https://imgur.com/a/jlhK7OZ Pictures of circular

WeightAn 24 Nov 18, 2022
Differentiable Neural Computers, Sparse Access Memory and Sparse Differentiable Neural Computers, for Pytorch

Differentiable Neural Computers and family, for Pytorch Includes: Differentiable Neural Computers (DNC) Sparse Access Memory (SAM) Sparse Differentiab

ixaxaar 302 Dec 14, 2022
Poisson Surface Reconstruction for LiDAR Odometry and Mapping

Poisson Surface Reconstruction for LiDAR Odometry and Mapping Surfels TSDF Our Approach Table: Qualitative comparison between the different mapping te

Photogrammetry & Robotics Bonn 305 Dec 21, 2022
Implements an infinite sum of poisson-weighted convolutions

An infinite sum of Poisson-weighted convolutions Kyle Cranmer, Aug 2018 If viewing on GitHub, this looks better with nbviewer: click here Consider a v

Kyle Cranmer 26 Dec 7, 2022
Copy Paste positive polyp using poisson image blending for medical image segmentation

Copy Paste positive polyp using poisson image blending for medical image segmentation According poisson image blending I've completely used it for bio

Phạm Vũ Hùng 2 Oct 19, 2021
Non-Homogeneous Poisson Process Intensity Modeling and Estimation using Measure Transport

Non-Homogeneous Poisson Process Intensity Modeling and Estimation using Measure Transport This GitHub page provides code for reproducing the results i

Andrew Zammit Mangion 1 Nov 8, 2021
This is the repository for the NeurIPS-21 paper [Contrastive Graph Poisson Networks: Semi-Supervised Learning with Extremely Limited Labels].

CGPN This is the repository for the NeurIPS-21 paper [Contrastive Graph Poisson Networks: Semi-Supervised Learning with Extremely Limited Labels]. Req

null 10 Sep 12, 2022
Finite difference solution of 2D Poisson equation. Can handle Dirichlet, Neumann and mixed boundary conditions.

Poisson-solver-2D Finite difference solution of 2D Poisson equation Current version can handle Dirichlet, Neumann, and mixed (combination of Dirichlet

Mohammad Asif Zaman 34 Dec 23, 2022
A fast poisson image editing implementation that can utilize multi-core CPU or GPU to handle a high-resolution image input.

Poisson Image Editing - A Parallel Implementation Jiayi Weng (jiayiwen), Zixu Chen (zixuc) Poisson Image Editing is a technique that can fuse two imag

Jiayi Weng 110 Dec 27, 2022
Image morphing without reference points by applying warp maps and optimizing over them.

Differentiable Morphing Image morphing without reference points by applying warp maps and optimizing over them. Differentiable Morphing is machine lea

Alex K 380 Dec 19, 2022
Code for Iso-Points: Optimizing Neural Implicit Surfaces with Hybrid Representations

Implementation for Iso-Points (CVPR 2021) Official code for paper Iso-Points: Optimizing Neural Implicit Surfaces with Hybrid Representations paper |

Yifan Wang 66 Nov 8, 2022
(CVPR 2021) Back-tracing Representative Points for Voting-based 3D Object Detection in Point Clouds

BRNet Introduction This is a release of the code of our paper Back-tracing Representative Points for Voting-based 3D Object Detection in Point Clouds,

null 86 Oct 5, 2022
CenterPoint 3D Object Detection and Tracking using center points in the bird-eye view.

CenterPoint 3D Object Detection and Tracking using center points in the bird-eye view. Center-based 3D Object Detection and Tracking, Tianwei Yin, Xin

Tianwei Yin 134 Dec 23, 2022
Official PyTorch implementation of "Adversarial Reciprocal Points Learning for Open Set Recognition"

Adversarial Reciprocal Points Learning for Open Set Recognition Official PyTorch implementation of "Adversarial Reciprocal Points Learning for Open Se

Guangyao Chen 78 Dec 28, 2022