Source code for the paper "TearingNet: Point Cloud Autoencoder to Learn Topology-Friendly Representations"

Overview

TearingNet: Point Cloud Autoencoder to Learn Topology-Friendly Representations

Created by Jiahao Pang, Duanshun Li, and Dong Tian from InterDigital

framework

Introduction

This repository contains the implementation of our TearingNet paper accepted in CVPR 2021. Given a point cloud dataset containing objects with various genera, or scenes with multiple objects, we propose the TearingNet, which is an autoencoder tackling the challenging task of representing the point clouds using a fixed-length descriptor. Unlike existing works directly deforming predefined primitives of genus zero (e.g., a 2D square patch) to an object-level point cloud, our TearingNet is characterized by a proposed Tearing network module and a Folding network module interacting with each other iteratively. Particularly, the Tearing network module learns the point cloud topology explicitly. By breaking the edges of a primitive graph, it tears the graph into patches or with holes to emulate the topology of a target point cloud, leading to faithful reconstructions.

Installation

  • We use Python 3.6, PyTorch 1.3.1 and CUDA 10.0, example commands to set up a virtual environment with anaconda are:
conda create tearingnet python=3.6
conda activate tearingnet
conda install pytorch=1.3.1 torchvision=0.4.2 cudatoolkit=10.0 -c pytorch 
conda install -c open3d-admin open3d
conda install -c conda-forge tensorboardx
conda install -c anaconda h5py

Data Preparation

KITTI Multi-Object Dataset

  • Our KITTI Multi-Object (KIMO) Dataset is constructed with kitti_dataset.py of PCDet (commit 95d2ab5). Please clone and install PCDet, then prepare the KITTI dataset according to their instructions.
  • Assume the name of the cloned folder is PCDet, please replace the create_groundtruth_database() function in kitti_dataset.py by our modified one provided in TearingNet/util/pcdet_create_grouth_database.py.
  • Prepare the KITTI dataset, then generate the data infos according to the instructions in the README.md of PCDet.
  • Create the folders TearingNet/dataset and TearingNet/dataset/kittimulobj then put the newly-generated folder PCDet/data/kitti/kitti_single under TearingNet/dataset/kittimulobj. Also, put the newly-generated file PCDet/data/kitti/kitti_dbinfos_object.pkl under the TearingNet/dataset/kittimulobj folder.
  • Instead of assembling several single-object point clouds together and write down as a multi-object point cloud, we generate the parameters that parameterize the multi-object point clouds then assemble them on the fly during training/testing. To obtain the parameters, run our prepared scripts as follows under the TearingNet folder. These scripts generate the training and testing splits of the KIMO-5 dataset:
./scripts/launch.sh ./scripts/gen_data/gen_kitti_mulobj_train_5x5.sh
./scripts/launch.sh ./scripts/gen_data/gen_kitti_mulobj_test_5x5.sh
  • The file structure of the KIMO dataset after these steps becomes:
kittimulobj
      ├── kitti_dbinfos_object.pkl
      ├── kitti_mulobj_param_test_5x5_2048.pkl
      ├── kitti_mulobj_param_train_5x5_2048.pkl
      └── kitti_single
              ├── 0_0_Pedestrian.bin
              ├── 1000_0_Car.bin
              ├── 1000_1_Car.bin
              ├── 1000_2_Van.bin
              ...

CAD Model Multi-Object Dataset

dataset
    ├── cadmulobj
    ├── kittimulobj
    ├── modelnet40
    │       └── modelnet40_ply_hdf5_2048
    │                   ├── ply_data_test0.h5
    │                   ├── ply_data_test_0_id2file.json
    │                   ├── ply_data_test1.h5
    │                   ├── ply_data_test_1_id2file.json
    │                   ...
    └── shapenet_part
            ├── shapenetcore_partanno_segmentation_benchmark_v0
            │   ├── 02691156
            │   │   ├── points
            │   │   │   ├── 1021a0914a7207aff927ed529ad90a11.pts
            │   │   │   ├── 103c9e43cdf6501c62b600da24e0965.pts
            │   │   │   ├── 105f7f51e4140ee4b6b87e72ead132ed.pts
            ...
  • Extract the "person", "car", "cone" and "plant" models from ModelNet40, and the "motorbike" models from the ShapeNet part dataset, by running the following Python script under the TearingNet folder:
python util/cad_models_collector.py
  • The previous step generates the file TearingNet/dataset/cadmulobj/cad_models.npy, based on which we generate the parameters for the CAMO dataset. To do so, launch the following scripts:
./scripts/launch.sh ./scripts/gen_data/gen_cad_mulobj_train_5x5.sh
./scripts/launch.sh ./scripts/gen_data/gen_cad_mulobj_test_5x5.sh
  • The file structure of the CAMO dataset after these steps becomes:
cadmulobj
    ├── cad_models.npy
    ├── cad_mulobj_param_test_5x5.npy
    └── cad_mulobj_param_train_5x5.npy

Experiments

Training

We employ a two-stage training strategy to train the TearingNet. The first step is to train a FoldingNet (E-Net & F-Net in paper). Take the KIMO dataset as an example, launch the following scripts under the TearingNet folder:

./scripts/launch.sh ./scripts/experiments/train_folding_kitti.sh

Having finished the first step, a pretrained model will be saved in TearingNet/results/train_folding_kitti. To load the pretrained FoldingNet into a TearingNet configuration and perform training, launch the following scripts:

./scripts/launch.sh ./scripts/experiments/train_tearing_kitti.sh

To see the meanings of the parameters in train_folding_kitti.sh and train_tearing_kitti.sh, check the Python script TearinNet/util/option_handler.py.

Reconstruction

To perform the reconstruction experiment with the trained model, launch the following scripts:

./scripts/launch.sh ./scripts/experiments/reconstruction.sh

One may write down the reconstructions in PLY format by setting a positive PC_WRITE_FREQ value. Again, please refer to TearinNet/util/option_handler.py for the meanings of individual parameters.

Counting

To perform the counting experiment with the trained model, launch the following scripts:

./scripts/launch.sh ./scripts/experiments/counting.sh

Citing this Work

Please cite our work if you find it useful for your research:

@inproceedings{pang2021tearingnet, 
    title={TearingNet: Point Cloud Autoencoder to Learn Topology-Friendly Representations}, 
    author={Pang, Jiahao and Li, Duanshun, and Tian, Dong}, 
    booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, 
    year={2021}
}

Related Projects

torus interpolation

You might also like...
Source code for CsiNet and CRNet using Fully Connected Layer-Shared feedback architecture.

FCS-applications Source code for CsiNet and CRNet using the Fully Connected Layer-Shared feedback architecture. Introduction This repository contains

Guide to using pre-trained large language models of source code
Guide to using pre-trained large language models of source code

Large Models of Source Code I occasionally train and publicly release large neural language models on programs, including PolyCoder. Here, I describe

Code of paper: A Recurrent Vision-and-Language BERT for Navigation

Recurrent VLN-BERT Code of the Recurrent-VLN-BERT paper: A Recurrent Vision-and-Language BERT for Navigation Yicong Hong, Qi Wu, Yuankai Qi, Cristian

Code for the paper "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer"

T5: Text-To-Text Transfer Transformer The t5 library serves primarily as code for reproducing the experiments in Exploring the Limits of Transfer Lear

Code for the paper "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer"

T5: Text-To-Text Transfer Transformer The t5 library serves primarily as code for reproducing the experiments in Exploring the Limits of Transfer Lear

Code associated with the "Data Augmentation using Pre-trained Transformer Models" paper

Data Augmentation using Pre-trained Transformer Models Code associated with the Data Augmentation using Pre-trained Transformer Models paper Code cont

Code for CVPR 2021 paper: Revamping Cross-Modal Recipe Retrieval with Hierarchical Transformers and Self-supervised Learning

Revamping Cross-Modal Recipe Retrieval with Hierarchical Transformers and Self-supervised Learning This is the PyTorch companion code for the paper: A

This repository will contain the code for the CVPR 2021 paper
This repository will contain the code for the CVPR 2021 paper "GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields"

GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields Project Page | Paper | Supplementary | Video | Slides | Blog | Talk If

Code for ACL 2021 main conference paper
Code for ACL 2021 main conference paper "Conversations are not Flat: Modeling the Intrinsic Information Flow between Dialogue Utterances".

Conversations are not Flat: Modeling the Intrinsic Information Flow between Dialogue Utterances This repository contains the code and pre-trained mode

Comments
  • Reconstruction experiment

    Reconstruction experiment

    Hi,

    First of all, I would like to congratulate you on your work and thank you for sharing its implementation.

    I've been able to successfully train your TearingNet model using the provided scripts (or at least I didn't get any errors when running them) but I'm struggling with the reconstruction test. I needed to modify the 6th line of the script since the TearingNet training only ran only for 480 epochs, but now I think the results I've obtained are too low compared with the results shown in your paper:

    ----------------- Options ---------------
                     exp_name: results/rec_tearing_kitti     	[default: experiment_name]
                        phase: test                          	[default: train]
                      encoder: pointnetvanilla               
                      decoder: foldingnetvanilla             
                   checkpoint: /home/daviddiaz/PointCloud/TearingNet/results/train_tearing_kitti/epoch_479.pth	[default: ]
             load_weight_only: False                         
                xyz_loss_type: 0                             
           xyz_chamfer_weight: 1                             
                   batch_size: 32                            	[default: 8]
                   num_points: 2048                          
                 dataset_name: kitti_mulobj                  	[default: ]
       config_from_checkpoint: True                          	[default: False]
                   tf_summary: True                          
                   test_split: test_5x5                      	[default: test]
                   print_freq: 5                             	[default: 20]
                pc_write_freq: -1                            	[default: 1000]
                     gt_color: [0.2, 0.2, 0.2]               	[default: [0, 0, 0]]
                   write_mesh: True                          	[default: False]
                  graph_thres: 0.0                           	[default: -1]
             graph_edge_color: [0.6, 0.6, 0.6]               	[default: [0.5, 0.5, 0.5]]
      graph_delete_point_mode: 0                             	[default: -1]
       graph_delete_point_eps: 0.08                          	[default: -1]
                   thres_edge: 3                             
         point_color_as_index: False                         
            pointnet_mlp_dims: [3, 64, 128, 128, 1024]       
             pointnet_fc_dims: [1024, 512, 512, 512]         
      pointnet_mlp_dolastrelu: False                         
                    grid_dims: [45, 45]                      	[default: None]
                folding1_dims: [514, 512, 512, 3]            
                folding2_dims: [515, 512, 512, 3]            
    ----------------- End -------------------
    1 GPUs will be used for testing.
    
    Model configuration loaded from checkpoint /home/daviddiaz/PointCloud/TearingNet/results/train_tearing_kitti/epoch_479.pth.
    Existing model /home/daviddiaz/PointCloud/TearingNet/results/train_tearing_kitti/epoch_479.pth loaded.
    
    
    Testing...
        batch_no: 0/313, ch_dist: 0.120451, ch^2_dist: 0.075922
        batch_no: 5/313, ch_dist: 0.108537, ch^2_dist: 0.060739
        batch_no: 10/313, ch_dist: 0.108818, ch^2_dist: 0.054396
        batch_no: 15/313, ch_dist: 0.108697, ch^2_dist: 0.057873
        batch_no: 20/313, ch_dist: 0.111949, ch^2_dist: 0.053711
        batch_no: 25/313, ch_dist: 0.110766, ch^2_dist: 0.052050
        batch_no: 30/313, ch_dist: 0.107380, ch^2_dist: 0.049375
        batch_no: 35/313, ch_dist: 0.109659, ch^2_dist: 0.051643
        batch_no: 40/313, ch_dist: 0.113601, ch^2_dist: 0.053893
        batch_no: 45/313, ch_dist: 0.110499, ch^2_dist: 0.053097
        batch_no: 50/313, ch_dist: 0.114762, ch^2_dist: 0.054872
        batch_no: 55/313, ch_dist: 0.116591, ch^2_dist: 0.060483
        batch_no: 60/313, ch_dist: 0.109698, ch^2_dist: 0.050469
        batch_no: 65/313, ch_dist: 0.109198, ch^2_dist: 0.051559
        batch_no: 70/313, ch_dist: 0.114969, ch^2_dist: 0.051424
        batch_no: 75/313, ch_dist: 0.114839, ch^2_dist: 0.055455
        batch_no: 80/313, ch_dist: 0.113275, ch^2_dist: 0.049001
        batch_no: 85/313, ch_dist: 0.115835, ch^2_dist: 0.051997
        batch_no: 90/313, ch_dist: 0.109653, ch^2_dist: 0.047442
        batch_no: 95/313, ch_dist: 0.111587, ch^2_dist: 0.052127
        batch_no: 100/313, ch_dist: 0.115906, ch^2_dist: 0.057076
        batch_no: 105/313, ch_dist: 0.113487, ch^2_dist: 0.051748
        batch_no: 110/313, ch_dist: 0.112954, ch^2_dist: 0.051023
        batch_no: 115/313, ch_dist: 0.116102, ch^2_dist: 0.051404
        batch_no: 120/313, ch_dist: 0.114349, ch^2_dist: 0.050175
        batch_no: 125/313, ch_dist: 0.116523, ch^2_dist: 0.052869
        batch_no: 130/313, ch_dist: 0.116486, ch^2_dist: 0.050731
        batch_no: 135/313, ch_dist: 0.117292, ch^2_dist: 0.051647
        batch_no: 140/313, ch_dist: 0.115285, ch^2_dist: 0.052137
        batch_no: 145/313, ch_dist: 0.116890, ch^2_dist: 0.053940
        batch_no: 150/313, ch_dist: 0.116629, ch^2_dist: 0.048998
        batch_no: 155/313, ch_dist: 0.113302, ch^2_dist: 0.051695
        batch_no: 160/313, ch_dist: 0.118178, ch^2_dist: 0.053083
        batch_no: 165/313, ch_dist: 0.116427, ch^2_dist: 0.051726
        batch_no: 170/313, ch_dist: 0.118368, ch^2_dist: 0.053872
        batch_no: 175/313, ch_dist: 0.120092, ch^2_dist: 0.056463
        batch_no: 180/313, ch_dist: 0.114862, ch^2_dist: 0.052044
        batch_no: 185/313, ch_dist: 0.117937, ch^2_dist: 0.051447
        batch_no: 190/313, ch_dist: 0.115581, ch^2_dist: 0.049403
        batch_no: 195/313, ch_dist: 0.117704, ch^2_dist: 0.051687
        batch_no: 200/313, ch_dist: 0.118300, ch^2_dist: 0.054128
        batch_no: 205/313, ch_dist: 0.116107, ch^2_dist: 0.049980
        batch_no: 210/313, ch_dist: 0.117240, ch^2_dist: 0.049790
        batch_no: 215/313, ch_dist: 0.117319, ch^2_dist: 0.049634
        batch_no: 220/313, ch_dist: 0.119937, ch^2_dist: 0.052074
        batch_no: 225/313, ch_dist: 0.117204, ch^2_dist: 0.052434
        batch_no: 230/313, ch_dist: 0.120750, ch^2_dist: 0.056373
        batch_no: 235/313, ch_dist: 0.115643, ch^2_dist: 0.049918
        batch_no: 240/313, ch_dist: 0.117935, ch^2_dist: 0.050043
        batch_no: 245/313, ch_dist: 0.119937, ch^2_dist: 0.053037
        batch_no: 250/313, ch_dist: 0.121604, ch^2_dist: 0.054612
        batch_no: 255/313, ch_dist: 0.115947, ch^2_dist: 0.048929
        batch_no: 260/313, ch_dist: 0.121110, ch^2_dist: 0.053670
        batch_no: 265/313, ch_dist: 0.122228, ch^2_dist: 0.055334
        batch_no: 270/313, ch_dist: 0.124175, ch^2_dist: 0.059319
        batch_no: 275/313, ch_dist: 0.122733, ch^2_dist: 0.056860
        batch_no: 280/313, ch_dist: 0.123875, ch^2_dist: 0.054965
        batch_no: 285/313, ch_dist: 0.123566, ch^2_dist: 0.056253
        batch_no: 290/313, ch_dist: 0.124758, ch^2_dist: 0.057836
        batch_no: 295/313, ch_dist: 0.123366, ch^2_dist: 0.056187
        batch_no: 300/313, ch_dist: 0.123768, ch^2_dist: 0.056015
        batch_no: 305/313, ch_dist: 0.126508, ch^2_dist: 0.058711
        batch_no: 310/313, ch_dist: 0.124168, ch^2_dist: 0.057548
    avg_ch_dist: 0.116116    avg_ch^2_dist: 0.052880
    
    Done!
    

    It also shocked me that in the 5th line it says decoder: foldingnetvanilla when I would have expected decoder: tearingnetgraph.

    I don't know if I've made any mistakes when running the scripts or if the results are correct and I'm just misunderstanding them. Could you provide me some advice?

    Best regards, David

    EDIT: Sorry, I forgot to say that, when I modified the 6th line of the reconstruction script, I changed it by: CHECKPOINT="${HOME_DIR}/results/train_tearing_kitti/epoch_479.pth"

    opened by DavidDiazGuerra 4
  • PCDet repo missing

    PCDet repo missing

    Am i lacking critical information, or does the repo to constrcut the kitti_dataset no longer exist? I am unable to find that the user 3bobo has a repo named PCDet, and all links refering to it are dead for me.

    opened by JohanYe 1
Owner
InterDigital
InterDigital
source code for paper: WhiteningBERT: An Easy Unsupervised Sentence Embedding Approach.

WhiteningBERT Source code and data for paper WhiteningBERT: An Easy Unsupervised Sentence Embedding Approach. Preparation git clone https://github.com

null 49 Dec 17, 2022
(ACL 2022) The source code for the paper "Towards Abstractive Grounded Summarization of Podcast Transcripts"

Towards Abstractive Grounded Summarization of Podcast Transcripts We provide the source code for the paper "Towards Abstractive Grounded Summarization

null 10 Jul 1, 2022
Code to use Augmented Shapiro Wilks Stopping, as well as code for the paper "Statistically Signifigant Stopping of Neural Network Training"

This codebase is being actively maintained, please create and issue if you have issues using it Basics All data files are included under losses and ea

Justin Terry 32 Nov 9, 2021
This is the source code of RPG (Reward-Randomized Policy Gradient)

RPG (Reward-Randomized Policy Gradient) Zhenggang Tang*, Chao Yu*, Boyuan Chen, Huazhe Xu, Xiaolong Wang, Fei Fang, Simon Shaolei Du, Yu Wang, Yi Wu (

null 40 Nov 25, 2022
Source code for AAAI20 "Generating Persona Consistent Dialogues by Exploiting Natural Language Inference".

Generating Persona Consistent Dialogues by Exploiting Natural Language Inference Source code for RCDG model in AAAI20 Generating Persona Consistent Di

null 16 Oct 8, 2022
This repository contains all the source code that is needed for the project : An Efficient Pipeline For Bloom’s Taxonomy Using Natural Language Processing and Deep Learning

Pipeline For NLP with Bloom's Taxonomy Using Improved Question Classification and Question Generation using Deep Learning This repository contains all

Rohan Mathur 9 Jul 17, 2021
The source code of HeCo

HeCo This repo is for source code of KDD 2021 paper "Self-supervised Heterogeneous Graph Neural Network with Co-contrastive Learning". Paper Link: htt

Nian Liu 106 Dec 27, 2022
Open source code for AlphaFold.

AlphaFold This package provides an implementation of the inference pipeline of AlphaFold v2.0. This is a completely new model that was entered in CASP

DeepMind 9.7k Jan 2, 2023
GPT-Code-Clippy (GPT-CC) is an open source version of GitHub Copilot, a language model

GPT-Code-Clippy (GPT-CC) is an open source version of GitHub Copilot, a language model -- based on GPT-3, called GPT-Codex -- that is fine-tuned on publicly available code from GitHub.

Nathan Cooper 2.3k Jan 1, 2023