Code for Iso-Points: Optimizing Neural Implicit Surfaces with Hybrid Representations

Overview

Implementation for Iso-Points (CVPR 2021)

Official code for paper Iso-Points: Optimizing Neural Implicit Surfaces with Hybrid Representations

paper | supplementary material | project page

Overview

Iso-points are well-distributed points which lie on the neural iso-surface, they are an explicit form of representation of the implicit surfaces. We propose using iso-points to augment the optimization of implicit neural surfaces. The implicit and explicit surface representations are coupled, i.e. the implicit model determines the locations and normals of iso-points, whereas the iso-points can be utilized to control the optimization of the implicit model.

The implementation of the key steps for iso-points extraction is in levelset_sampling.py and utils/point_processing.py. To demonstrate the utilisation of iso-points, we provide scripts for multiple applications and scenarios:

Demo

Installation

This code is built as an extension of out Differentiable Surface Splatting pytorch library (DSS), which depends on pytorch3d, torch_cluster. Currently we support up to pytorch 1.6.

git clone --recursive https://github.com/yifita/iso-points.git
cd iso-points

# conda environment and dependencies
# update conda
conda update -n base -c defaults conda
# install requirements
conda env create --name DSS -f environment.yml
conda activate DSS

# build additional dependencies of DSS
# FRNN - fixed radius nearest neighbors
cd external/FRNN/external
git submodule update --init --recursive
cd prefix_sum
python setup.py install
cd ../..
python setup.py install

# build batch-svd
cd ../torch-batch-svd
python setup.py install

# build DSS itself
cd ../..
python setup.py develop

prepare data

Download data

cd data
wget https://igl.ethz.ch/projects/iso-points/data.zip
unzip data.zip
rm data.zip

Including subset of masked DTU data (courtesy of Yariv et.al.), synthetic rendered multiview data, and masked furu stereo reconstruction of DTU dataset.

multiview reconstruction

sampling-with-iso-points

# train baseline implicit representation only using ray-tracing
python train_mvr.py configs/compressor_implicit.yml --exit-after 6000

# train with uniform iso-points
python train_mvr.py configs/compressor_uni.yml --exit-after 6000

# train with iso-points distributed according to loss value (hard example mining)
python train_mvr.py configs/compressor_uni_lossS.yml --exit-after 6000

sampling result

DTU-data

python train_mvr.py configs/dtu55_iso.yml

dtu mvr result

implicit surface to noisy point cloud

python test_dtu_points.py data/DTU_furu/scan122.ply --use_off_normal_loss -o exp/points_3d_outputs/scan122_ours

cite

Please cite us if you find the code useful!

@inproceedings{yifan2020isopoints,
      title={Iso-Points: Optimizing Neural Implicit Surfaces with Hybrid Representations},
      author={Wang Yifan and Shihao Wu and Cengiz Oztireli and Olga Sorkine-Hornung},
      year={2020},
      booktitle = {CVPR},
      year = {2020},
}

Acknowledgement

We would like to thank Viviane Yang for her help with the point2surf code. This work was supported in parts by Apple scholarship, SWISSHEART Failure Network (SHFN), and UKRI Future Leaders Fellowship [grant number MR/T043229/1]

You might also like...
Code for
Code for "Diffusion is All You Need for Learning on Surfaces"

Source code for "Diffusion is All You Need for Learning on Surfaces", by Nicholas Sharp Souhaib Attaiki Keenan Crane Maks Ovsjanikov NOTE: the linked

Convert game ISO and archives to CD CHD for emulation on Linux.

tochd Convert game ISO and archives to CD CHD for emulation. Author: Tuncay D. Source: https://github.com/thingsiplay/tochd Releases: https://github.c

Official PyTorch implementation of Synergies Between Affordance and Geometry: 6-DoF Grasp Detection via Implicit Representations

Synergies Between Affordance and Geometry: 6-DoF Grasp Detection via Implicit Representations Zhenyu Jiang, Yifeng Zhu, Maxwell Svetlik, Kuan Fang, Yu

This package is for running the semantic SLAM algorithm using extracted planar surfaces from the received detection
This package is for running the semantic SLAM algorithm using extracted planar surfaces from the received detection

Semantic SLAM This package can perform optimization of pose estimated from VO/VIO methods which tend to drift over time. It uses planar surfaces extra

Python implementation of 3D facial mesh exaggeration using the techniques described in the paper: Computational Caricaturization of Surfaces.
Python implementation of 3D facial mesh exaggeration using the techniques described in the paper: Computational Caricaturization of Surfaces.

Python implementation of 3D facial mesh exaggeration using the techniques described in the paper: Computational Caricaturization of Surfaces.

Implementation of CVPR'2022:Reconstructing Surfaces for Sparse Point Clouds with On-Surface Priors
Implementation of CVPR'2022:Reconstructing Surfaces for Sparse Point Clouds with On-Surface Priors

Reconstructing Surfaces for Sparse Point Clouds with On-Surface Priors (CVPR 2022) Personal Web Pages | Paper | Project Page This repository contains

BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search
BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search

BossNAS This repository contains PyTorch evaluation code, retraining code and pretrained models of our paper: BossNAS: Exploring Hybrid CNN-transforme

Hybrid Neural Fusion for Full-frame Video Stabilization

FuSta: Hybrid Neural Fusion for Full-frame Video Stabilization Project Page | Video | Paper | Google Colab Setup Setup environment for [Yu and Ramamoo

A hybrid framework (neural mass model + ML) for SC-to-FC prediction

The current workflow simulates brain functional connectivity (FC) from structural connectivity (SC) with a neural mass model. Gradient descent is applied to optimize the parameters in the neural mass model.

Comments
  • No module named 'im2mesh'

    No module named 'im2mesh'

    When I'm trying to run test_dtu_points.py, it comes out that:

    Traceback (most recent call last): File "test_dtu_points.py", line 11, in from im2mesh.checkpoints import CheckpointIO ModuleNotFoundError: No module named 'im2mesh'

    Could you help me with this?

    opened by Linxius 3
  • About inv_sigma_spatial

    About inv_sigma_spatial

    According to the paper Eq.4, the density bandwidth is set to be 16D/|Qt|, is https://github.com/yifita/iso-points/blob/6faca4a09ddf1f4cae4f5d8af054f6aecd98aa8c/DSS/models/levelset_sampling.py#L256 for this?

    Or how should I understand this line, I cant find descriptions in the paper.

    Many thanks!

    opened by csyhping 0
  • arange_pixels missing function for project

    arange_pixels missing function for project

    Hi, I try to run the code, and go through most of conda install, and find a problem:

    python train_mvr.py configs/compressor_implicit.yml --exit-after 6000
    Traceback (most recent call last):
      File "train_mvr.py", line 7, in <module>
        import config
      File "/home/lir0b/Code/NeuralRep/iso-points/config.py", line 12, in <module>
        from DSS.training.trainer import Trainer
      File "/home/lir0b/Code/NeuralRep/iso-points/DSS/training/trainer.py", line 27, in <module>
        from ..utils import slice_dict, scaler_to_color, check_weights, arange_pixels, sample_patch_points
    ImportError: cannot import name 'arange_pixels' from 'DSS.utils' (/home/lir0b/Code/NeuralRep/iso-points/DSS/utils/__init__.py)
    

    I try to search arange_pixels, but it is still not such function in source code, could you please check the github files, I think there is something missing.

    opened by arthurlirui 0
  • pre-trained model

    pre-trained model

    Hi,yifita: Thanks for your work. Did you provide a pre-trained model to help quickly reproduce the work of surface reconstruction from sparse point cloud ? I'm very grateful for your help.

    opened by mabaorui 0
Owner
Yifan Wang
PhD student @ ETH Zurich
Yifan Wang
Volsdf - Volume Rendering of Neural Implicit Surfaces

Volume Rendering of Neural Implicit Surfaces Project Page | Paper | Data This re

Lior Yariv 221 Jan 7, 2023
Image morphing without reference points by applying warp maps and optimizing over them.

Differentiable Morphing Image morphing without reference points by applying warp maps and optimizing over them. Differentiable Morphing is machine lea

Alex K 380 Dec 19, 2022
Hybrid CenterNet - Hybrid-supervised object detection / Weakly semi-supervised object detection

Hybrid-Supervised Object Detection System Object detection system trained by hybrid-supervision/weakly semi-supervision (HSOD/WSSOD): This project is

null 5 Dec 10, 2022
Code for the paper "Implicit Representations of Meaning in Neural Language Models"

Implicit Representations of Meaning in Neural Language Models Preliminaries Create and set up a conda environment as follows: conda create -n state-pr

Belinda Li 39 Nov 3, 2022
An Efficient Implementation of Analytic Mesh Algorithm for 3D Iso-surface Extraction from Neural Networks

AnalyticMesh Analytic Marching is an exact meshing solution from neural networks. Compared to standard methods, it completely avoids geometric and top

Karbo 45 Dec 21, 2022
Pytorch implementation of COIN, a framework for compression with implicit neural representations 🌸

COIN ?? This repo contains a Pytorch implementation of COIN: COmpression with Implicit Neural representations, including code to reproduce all experim

Emilien Dupont 104 Dec 14, 2022
Unofficial Tensorflow 2 implementation of the paper Implicit Neural Representations with Periodic Activation Functions

Siren: Implicit Neural Representations with Periodic Activation Functions The unofficial Tensorflow 2 implementation of the paper Implicit Neural Repr

Seyma Yucer 2 Jun 27, 2022
PyTorch implementation of paper: HPNet: Deep Primitive Segmentation Using Hybrid Representations.

HPNet This repository contains the PyTorch implementation of paper: HPNet: Deep Primitive Segmentation Using Hybrid Representations. Installation The

Siming Yan 42 Dec 7, 2022
Neural-Pull: Learning Signed Distance Functions from Point Clouds by Learning to Pull Space onto Surfaces(ICML 2021)

Neural-Pull: Learning Signed Distance Functions from Point Clouds by Learning to Pull Space onto Surfaces(ICML 2021) This repository contains the code

null 149 Dec 15, 2022
This repository contains the code for the CVPR 2020 paper "Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision"

Differentiable Volumetric Rendering Paper | Supplementary | Spotlight Video | Blog Entry | Presentation | Interactive Slides | Project Page This repos

null 697 Jan 6, 2023