Code for the SIGGRAPH 2022 paper "DeltaConv: Anisotropic Operators for Geometric Deep Learning on Point Clouds."

Overview

DeltaConv

[Paper] [Project page]

Code for the SIGGRAPH 2022 paper "DeltaConv: Anisotropic Operators for Geometric Deep Learning on Point Clouds" by Ruben Wiersma, Ahmad Nasikun, Elmar Eisemann, and Klaus Hildebrandt.

Anisotropic convolution is a central building block of CNNs but challenging to transfer to surfaces. DeltaConv learns combinations and compositions of operators from vector calculus, which are a natural fit for curved surfaces. The result is a simple and robust anisotropic convolution operator for point clouds with state-of-the-art results.

Top: unlike images, surfaces have no global coordinate system. Bottom: DeltaConv learns both scalar and vector features using geometric operators.

Contents

Installation

  1. Clone this repository:
git clone https://github.com/rubenwiersma/deltaconv.git
  1. Create a conda environment from the environment.yml:
conda env create -n deltaconv -f environment.yml

Done!

Manual installation

If you wish to install DeltaConv in your own environment, proceed as follows.

  1. Make sure that you have installed:

  2. Install DeltaConv:

pip install deltaconv

Building DeltaConv for yourself

  1. Make sure you clone the repository with submodules:
git clone --recurse-submodules https://github.com/rubenwiersma/deltaconv.git

If you have already cloned the repository without submodules, you can fix it with git submodule update --init --recursive.

  1. Install from folder:
cd [root_folder]
pip install

Replicating the experiments

See the README.md in replication_scripts for instructions on replicating the experiments and using the pre-trained weights (available in experiments/pretrained_weights).

In short, you can run bash scripts to replicate our experiments. For example, evaluating pre-trained weights on ShapeNet:

cd [root_folder]
conda activate deltaconv
bash replication_scripts/pretrained/shapenet.sh

You can also directly run the python files in experiments:

python experiments/train_shapenet.py

Use the -h or --help flag to find out which arguments can be passed to the training script:

python experiments/train_shapenet.py -h

You can keep track of the training process with tensorboard:

tensorboard logdir=experiments/runs/shapenet_all

Anisotropic Diffusion

The code that was used to generate Figure 2 from the paper and Figure 2 and 3 from the supplement is a notebook in the folder experiments/anisotropic_diffusion.

Data

The training scripts assume that you have a data folder in experiments. ModelNet40 and ShapeNet download the datasets from a public repository. Instructions to download the data for human body shape segmentation, SHREC, and ScanObjectNN are given in the training scripts.

Tests

In the paper, we make statements about a number of properties of DeltaConv that are either a result of prior work or due to the implementation. We created a test suite to ensure that these properties hold for the implementation, along with unit tests for each module. For example:

  • Section 3.6, 3.7: Vector MLPs are equivariant to norm-preserving transformations, or coordinate-independent (rotations, reflections)
    • test/nn/test_mlp.py
    • test/nn/test_nonlin.py
  • Section 3.7: DeltaConv is coordinate-independent, a forward pass on a shape with one choice of bases leads to the same output and weight updates when run with different bases
    • test/nn/test_deltaconv.py
  • Introduction, section 3.2: The operators are robust to noise and outliers.
    • test/geometry/test_grad_div.py
  • Supplement, section 1: Vectors can be mapped between points with equation (15).
    • test/geometry/test_grad_div.py

Citations

Please cite our paper if this code contributes to an academic publication:

@Article{Wiersma2022DeltaConv,
  author    = {Ruben Wiersma, Ahmad Nasikun, Elmar Eisemann, Klaus Hildebrandt},
  journal   = {Transactions on Graphics},
  title     = {DeltaConv: Anisotropic Operators for Geometric Deep Learning on Point Clouds},
  year      = {2022},
  month     = jul,
  number    = {4},
  volume    = {41},
  doi       = {10.1145/3528223.3530166},
  publisher = {ACM},
}

The farthest point sampling code relies on Geometry Central:

@misc{geometrycentral,
  title = {geometry-central},
  author = {Nicholas Sharp and Keenan Crane and others},
  note = {www.geometry-central.net},
  year = {2019}
}

And we make use of PyG (and underlying packages) to load point clouds, compute sparse matrix products, and compute nearest neighbors:

@inproceedings{Fey/Lenssen/2019,
  title={Fast Graph Representation Learning with {PyTorch Geometric}},
  author={Fey, Matthias and Lenssen, Jan E.},
  booktitle={ICLR Workshop on Representation Learning on Graphs and Manifolds},
  year={2019},
}
You might also like...
General Virtual Sketching Framework for Vector Line Art (SIGGRAPH 2021)
General Virtual Sketching Framework for Vector Line Art (SIGGRAPH 2021)

General Virtual Sketching Framework for Vector Line Art - SIGGRAPH 2021 Paper | Project Page Outline Dependencies Testing with Trained Weights Trainin

Implementation for
Implementation for "Seamless Manga Inpainting with Semantics Awareness" (SIGGRAPH 2021 issue)

Seamless Manga Inpainting with Semantics Awareness [SIGGRAPH 2021](To appear) | Project Website | BibTex Introduction: Manga inpainting fills up the d

Implementation for "Manga Filling Style Conversion with Screentone Variational Autoencoder" (SIGGRAPH ASIA 2020 issue)

Manga Filling with ScreenVAE SIGGRAPH ASIA 2020 | Project Website | BibTex This repository is for ScreenVAE introduced in the following paper "Manga F

TilinGNN: Learning to Tile with Self-Supervised Graph Neural Network (SIGGRAPH 2020)
TilinGNN: Learning to Tile with Self-Supervised Graph Neural Network (SIGGRAPH 2020)

TilinGNN: Learning to Tile with Self-Supervised Graph Neural Network (SIGGRAPH 2020) About The goal of our research problem is illustrated below: give

Official implementation of "StyleCariGAN: Caricature Generation via StyleGAN Feature Map Modulation" (SIGGRAPH 2021)

StyleCariGAN in PyTorch Official implementation of StyleCariGAN:Caricature Generation via StyleGAN Feature Map Modulation in PyTorch Requirements PyTo

Tracing Versus Freehand for Evaluating Computer-Generated Drawings (SIGGRAPH 2021)
Tracing Versus Freehand for Evaluating Computer-Generated Drawings (SIGGRAPH 2021)

Tracing Versus Freehand for Evaluating Computer-Generated Drawings (SIGGRAPH 2021) Zeyu Wang, Sherry Qiu, Nicole Feng, Holly Rushmeier, Leonard McMill

ManipNet: Neural Manipulation Synthesis with a Hand-Object Spatial Representation - SIGGRAPH 2021
ManipNet: Neural Manipulation Synthesis with a Hand-Object Spatial Representation - SIGGRAPH 2021

ManipNet: Neural Manipulation Synthesis with a Hand-Object Spatial Representation - SIGGRAPH 2021 Dataset Code Demos Authors: He Zhang, Yuting Ye, Tak

Official implementation of
Official implementation of "StyleCariGAN: Caricature Generation via StyleGAN Feature Map Modulation" (SIGGRAPH 2021)

StyleCariGAN: Caricature Generation via StyleGAN Feature Map Modulation This repository contains the official PyTorch implementation of the following

[SIGGRAPH Asia 2021] DeepVecFont: Synthesizing High-quality Vector Fonts via Dual-modality Learning.

DeepVecFont This is the homepage for "DeepVecFont: Synthesizing High-quality Vector Fonts via Dual-modality Learning". Yizhi Wang and Zhouhui Lian. WI

Comments
  • possibly redundant

    possibly redundant "`from tkinter import W`" in transforms/__init__.py

    In transforms/__init__.py, there is from tkinter import W. This import seems out of place. Can you please confirm?

    This actually caused a issue when using in a docker container with python 3.8.

      File "/home/algorithm/.local/lib/python3.8/site-packages/deltaconv/__init__.py", line 4, in <module> 
        from . import transforms
      File "/home/algorithm/.local/lib/python3.8/site-packages/deltaconv/transforms/__init__.py", line 1, in <module>
        from tkinter import W
      File "/opt/conda/lib/python3.8/tkinter/__init__.py", line 36, in <module>
        import _tkinter # If this fails your Python may not be configured for Tk
    ImportError: libX11.so.6: cannot open shared object file: No such file or directory
    
    opened by msrepo 2
  • Implementation of the derivatives of \gamma(u, v) in Eq (15)

    Implementation of the derivatives of \gamma(u, v) in Eq (15)

    Hi @rubenwiersma, Thx for sharing this excellent work. While I was going through the paper details, I got confused by the implementation of $\partial_{u} \Gamma\left(u_{j}, v_{j}\right) \text{ and } \partial_{v} \Gamma\left(u_{j}, v_{j}\right)$ in these two lines L173-174. Why is the implementation different from that of grad? Can u plz give detailed mathematical expressions for this?

    opened by Crane-YU 2
  • Can it be designed to judge whether the point cloud is perpendicular to a certain plane

    Can it be designed to judge whether the point cloud is perpendicular to a certain plane

    This work is very interesting, I think if I have a group of point clouds with different shapes but all perpendicular to the xy plane (similar to walls with various shapes perpendicular to the ground), can I use your algorithm directly to determine how much they are perpendicular to the ground (some walls are not exactly perpendicular to the ground)

    opened by Longyouspongecake 1
Owner
null
Supplementary code for SIGGRAPH 2021 paper: Discovering Diverse Athletic Jumping Strategies

SIGGRAPH 2021: Discovering Diverse Athletic Jumping Strategies project page paper demo video Prerequisites Important Notes We suspect there are bugs i

null 52 Aug 31, 2022
Code for the SIGGRAPH 2021 paper "Consistent Depth of Moving Objects in Video".

Consistent Depth of Moving Objects in Video This repository contains training code for the SIGGRAPH 2021 paper "Consistent Depth of Moving Objects in

Google 193 Sep 21, 2022
Imposter-detector-2022 - HackED 2022 Team 3IQ - 2022 Imposter Detector

HackED 2022 Team 3IQ - 2022 Imposter Detector By Aneeljyot Alagh, Curtis Kan, Jo

Joshua Ji 3 Aug 20, 2022
PyTorch implementations for our SIGGRAPH 2021 paper: Editable Free-viewpoint Video Using a Layered Neural Representation.

st-nerf We provide PyTorch implementations for our paper: Editable Free-viewpoint Video Using a Layered Neural Representation SIGGRAPH 2021 Jiakai Zha

Diplodocus 247 Sep 29, 2022
Code for HodgeNet: Learning Spectral Geometry on Triangle Meshes, in SIGGRAPH 2021.

HodgeNet | Webpage | Paper | Video HodgeNet: Learning Spectral Geometry on Triangle Meshes Dmitriy Smirnov, Justin Solomon SIGGRAPH 2021 Set-up To ins

Dima Smirnov 58 Sep 18, 2022
Code release for Local Light Field Fusion at SIGGRAPH 2019

Local Light Field Fusion Project | Video | Paper Tensorflow implementation for novel view synthesis from sparse input images. Local Light Field Fusion

null 997 Sep 28, 2022
Code release for Local Light Field Fusion at SIGGRAPH 2019

Local Light Field Fusion Project | Video | Paper Tensorflow implementation for novel view synthesis from sparse input images. Local Light Field Fusion

null 748 Nov 27, 2021
[CVPR 2022] CoTTA Code for our CVPR 2022 paper Continual Test-Time Domain Adaptation

CoTTA Code for our CVPR 2022 paper Continual Test-Time Domain Adaptation Prerequisite Please create and activate the following conda envrionment. To r

Qin Wang 73 Sep 22, 2022
Face Identity Disentanglement via Latent Space Mapping [SIGGRAPH ASIA 2020]

Face Identity Disentanglement via Latent Space Mapping Description Official Implementation of the paper Face Identity Disentanglement via Latent Space

null 147 Sep 18, 2022