[PyTorch] Official implementation of CVPR2021 paper "PointDSC: Robust Point Cloud Registration using Deep Spatial Consistency". https://arxiv.org/abs/2103.05465

Overview

PointDSC repository

PyTorch implementation of PointDSC for CVPR'2021 paper "PointDSC: Robust Point Cloud Registration using Deep Spatial Consistency", by Xuyang Bai, Zixin Luo, Lei Zhou, Hongkai Chen, Lei Li, Zeyu Hu, Hongbo Fu and Chiew-Lan Tai.

This paper focus on outlier rejection for 3D point clouds registration. If you find this project useful, please cite:

@article{bai2021pointdsc,
  title={{PointDSC}: {R}obust {P}oint {C}loud {R}egistration using {D}eep {S}patial {C}onsistency},
  author={Xuyang Bai, Zixin Luo, Lei Zhou, Hongkai Chen, Lei Li, Zeyu Hu, Hongbo Fu and Chiew-Lan Tai},
  journal={CVPR},
  year={2021}
}

Introduction

Removing outlier correspondences is one of the critical steps for successful feature-based point cloud registration. Despite the increasing popularity of introducing deep learning techniques in this field, spatial consistency, which is essentially established by a Euclidean transformation between point clouds, has received almost no individual attention in existing learning frameworks. In this paper, we present PointDSC, a novel deep neural network that explicitly incorporates spatial consistency for pruning outlier correspondences. First, we propose a nonlocal feature aggregation module, weighted by both feature and spatial coherence, for feature embedding of the input correspondences. Second, we formulate a differentiable spectral matching module, supervised by pairwise spatial compatibility, to estimate the inlier confidence of each correspondence from the embedded features. With modest computation cost, our method outperforms the state-of-the-art hand-crafted and learning-based outlier rejection approaches on several real-world datasets by a significant margin. We also show its wide applicability by combining PointDSC with different 3D local descriptors.

fig0

Requirements

If you are using conda, you may configure PointDSC as:

conda env create -f environment.yml
conda activate pointdsc

If you also want to use FCGF as the 3d local descriptor, please install MinkowskiEngine v0.5.0 and download the FCGF model (pretrained on 3DMatch) from here.

Demo

We provide a small demo to extract dense FPFH descriptors for two point cloud, and register them using PointDSC. The ply files are saved in the demo_data folder, which can be replaced by your own data. Please use model pretrained on 3DMatch for indoor RGB-D scans and model pretrained on KITTI for outdoor LiDAR scans. To try the demo, please run

python demo_registration.py --chosen_snapshot [PointDSC_3DMatch_release/PointDSC_KITTI_release] --descriptor [fcgf/fpfh]

For challenging cases, we recommend to use learned feature descriptors like FCGF or D3Feat.

Dataset Preprocessing

3DMatch

The raw point clouds of 3DMatch can be downloaded from FCGF repo. The test set point clouds and the ground truth poses can be downloaded from 3DMatch Geometric Registration website. Please make sure the data folder contains the following:

.                          
├── fragments                 
│   ├── 7-scene-redkitechen/       
│   ├── sun3d-home_at-home_at_scan1_2013_jan_1/      
│   └── ...                
├── gt_result                   
│   ├── 7-scene-redkitechen-evaluation/   
│   ├── sun3d-home_at-home_at_scan1_2013_jan_1-evaluation/
│   └── ...         
├── threedmatch            
│   ├── *.npz
│   └── *.txt                            

To reduce the training time, we pre-compute the 3D local descriptors (FCGF or FPFH) so that we can directly build the input correspondence using NN search during training. Please use misc/cal_fcgf.py or misc/cal_fpfh.py to extract FCGF or FPFH descriptors. Here we provide the pre-computed descriptors for the 3DMatch test set.

KITTI

The raw point clouds can be download from KITTI Odometry website. Please follow the similar steps as 3DMatch dataset for pre-processing.

Augmented ICL-NUIM

Data can be downloaded from Redwood website. Details can be found in multiway/README.md

Pretrained Model

We provide the pre-trained model of 3DMatch in snapshot/PointDSC_3DMatch_release and KITTI in snapshot/PointDSC_KITTI_release.

Instructions to training and testing

3DMatch

The training and testing on 3DMatch dataset can be done by running

python train_3dmatch.py

python evaluation/test_3DMatch.py --chosen_snapshot [exp_id] --use_icp False

where the exp_id should be replaced by the snapshot folder name for testing (e.g. PointDSC_3DMatch_release). The testing results will be saved in logs/. The training config can be changed in config.py. We also provide the scripts to test the traditional outlier rejection baselines on 3DMatch in baseline_scripts/baseline_3DMatch.py.

KITTI

Similarly, the training and testing of KITTI data set can be done by running

python train_KITTI.py

python evaluation/test_KITTI.py --chosen_snapshot [exp_id] --use_icp False

We also provide the scripts to test the traditional outlier rejection baselines on KITTI in baseline_scripts/baseline_KITTI.py.

Augmemented ICL-NUIM

The detailed guidance of evaluating our method in multiway registration tasks can be found in multiway/README.md

3DLoMatch

We also evaluate our method on a recently proposed benchmark 3DLoMatch following OverlapPredator,

python evaluation/test_3DLoMatch.py --chosen_snapshot [exp_id] --descriptor [fcgf/predator] --num_points 5000

If you want to evaluate predator descriptor with PointDSC, you first need to follow the offical instruction of OverlapPredator to extract the features.

Contact

If you run into any problems or have questions, please create an issue or contact [email protected]

Acknowledgments

We thank the authors of

for open sourcing their methods.

Comments
  • cal_fcgf process_kitti error

    cal_fcgf process_kitti error

    when i run cal_fcgf of kitti dataset processing , it turns out to be an error in the function process_kitti ,

                    xyz0_t = apply_transform(xyz0[sel0], M)
                    pcd0 = make_point_cloud(xyz0_t)
                    pcd1 = make_point_cloud(xyz1[sel1])
    ```when i change the index to     apply_transform(xyz0[sel0[1]], M)  , it  goes well  but   error  become like this
    `zlib.error: Error -3 while decompressing data: invalid distance code`
    
    so   I  want to  know  is  it  exactly  correct  of   the  code:    xyz0_t = apply_transform(xyz0[sel0], M)
    
    opened by Gardlin 7
  • A question about 3DRegNet loss

    A question about 3DRegNet loss

    Sorry, I closed the last question by mistake. I have one last question about 3DRegNet. In your article, you only used classified loss to train 3DRegNet, namely BEC loss of PointDSC, right?

    opened by QWTforGithub 6
  • the demo_registration is low RAM efficient and low gpu memory efficient

    the demo_registration is low RAM efficient and low gpu memory efficient

    Thanks for your excellent work first,i get some problems here. 1.Run i run demo_registration.py with kitti like pcd files,use fcgf i can get a large feature which shape is [45793,3]。 When run the code below to calculate corr_pos,the process will be killed due to a huge memory usage which up to 16GB!

    # distance = np.sqrt(2 - 2 * (src_features @ tgt_features.T) + 1e-6)
    # source_idx = np.argmin(distance, axis=1)
    # source_dis = np.min(distance, axis=1)
    # corr = np.concatenate([np.arange(source_idx.shape[0])[:, None], source_idx[:, None]], axis=-1)
    # src_keypts = src_pts[corr[:,0]]
    # tgt_keypts = tgt_pts[corr[:,1]]
    # corr_pos = np.concatenate([src_keypts, tgt_keypts], axis=-1)
    # corr_pos = corr_pos - corr_pos.mean(0)
    

    This problems has been solved by using fcgf_feature_matching function which is borrowed from DGR instead of the code above. 2.Then i downsample the pcd,now the feature shape is [25000,3],when start to run the code below,the process is been killed again due to a huge gpu memory use which up to 20GB!

    res = model(data)
    

    I have register the feature which has shape [45793,3] successfully using DGR with only 8GB gpu memory usage,which is quite fast.is that normal this repo could use 20GB gpu memory?I need your help to solve the problems above!

    opened by lfxx 5
  • FileNotFoundError:[Error 2] No such file or directory: 'redwood_simulated/livingroom1-simulated.json'.

    FileNotFoundError:[Error 2] No such file or directory: 'redwood_simulated/livingroom1-simulated.json'.

    when I tring to Reproduce your paper PointDsc, I encounter a FileNotFoundError:[Error 2] No such file or directory: 'redwood_simulated/livingroom1-simulated.json'. And I've already found all directorys ,but failed to find it. so maybe you forgot upload it or other situation I don't konw hope your reply

    opened by lian0809 5
  • I want to know which version of your pytorch ,cuda and ME.

    I want to know which version of your pytorch ,cuda and ME.

    Thanks your work!!! I want to know which version of your pytorch ,cuda and ME. Because I have this error, with the old version of ME when i use the FCGF before: Traceback (most recent call last): File "demo_registration.py", line 10, in from misc.fcgf import ResUNetBN2C as FCGF File "/Bill/Project/PointDSC/misc/fcgf.py", line 77, in region_type=ME.RegionType.HYPER_CUBE, File "/opt/conda/lib/python3.7/enum.py", line 349, in getattr raise AttributeError(name) from None AttributeError: HYPER_CUBE

    Thanks

    opened by BiaoBiaoLi 5
  • A question about the result of 3DRegNet.

    A question about the result of 3DRegNet.

    Hi, thank you very much for your well-organized pointDSC code. I noticed the experimental result of 3DRegNet on 3DMatch in your paper. Would you like to provide the code of 3DRegNet on 3DMatch? Thank you very much!

    opened by QWTforGithub 4
  • What tools to plot the FIgure 6.

    What tools to plot the FIgure 6.

    Hi Xuyang,

    What tool to plot the Figure 6 in your paper(feature similarity distribution of inlier and non-inlier pairs). It is very descriptive and vivid.

    Thanks.

    opened by Ggs1mida 2
  • question about the number of input points

    question about the number of input points

    the number of input points between train and test is different,and each one in the test set is different.If i scale it to 5000,will it effect the final result?

    opened by dreamcubeblock 2
  • Extract FPFH feature on KITTI dataset

    Extract FPFH feature on KITTI dataset

    Hi,

    Thanks for sharing valuable code. However, in the cal_fpfh.py script, I do not find how to extract features on KITTI dataset? Should I process the raw point cloud similar to other datasets with voxel_size=0.05?

    Thanks a lot for your attention and look forward to your reply.

    Best regards

    opened by rann1018 2
  • Do you have any processed data for training?

    Do you have any processed data for training?

    Hello,I'm trying to reproduce this code, but I'm having some problems.

    I have the catalogue ready: .
    ├── fragments
    │ ├── 7-scene-redkitechen/
    │ ├── sun3d-home_at-home_at_scan1_2013_jan_1/
    │ └── ...
    ├── gt_result
    │ ├── 7-scene-redkitechen-evaluation/
    │ ├── sun3d-home_at-home_at_scan1_2013_jan_1-evaluation/ │ └── ...
    ├── threedmatch
    │ ├── *.npz │ └── *.txt

    But Error when I run training code"python train_3DMatch.py": "AssertionError: Make sure that the path /data/3DMatch has data sun3d-brown_bm_1-brown_bm_1*0.30.txt" Did I do something wrong? And my files failed to install"MinkowskiEngine". Could you please provide this folder "/3DMatch" used for training ? Thank you!

    opened by Alicebash 2
  • Traceback (most recent call last):   File run(config) File "multiway/make_fragments.py", line 168, in run if config["python_multi_threading"].lower() == "true": AttributeError: 'bool' object has no attribute 'lower'">

    Traceback (most recent call last): File "multiway/make_fragments.py", line 200, in run(config) File "multiway/make_fragments.py", line 168, in run if config["python_multi_threading"].lower() == "true": AttributeError: 'bool' object has no attribute 'lower'

    when I run " python multiway/make_fragments.py redwood_simulated/livingroom1-simulated.json" , I meet the problem like above .Looking forward for your reply

    opened by Feegooddddd 1
  • Download the pre-computed descriptors for the 3DMatch test set

    Download the pre-computed descriptors for the 3DMatch test set

    Hello! Thanks for your open source code and data!

    I find the link to download the pre-computed descriptors for the 3DMatch test set is not aviliable now. Could you please give me a new link? image

    What's more, I have another question. Do you know where is the link that I can download the pre-computed descriptors (FCGF and Predator) for the 3DLoMatch?

    I'm looking forward to your reply.

    opened by Xinyi-tum 1
  • About testing the 3DLoMatch

    About testing the 3DLoMatch

    Hi, I have seen your presentation video of PointDSC, it's beautiful! Besides, I want to know something about the training dataset of the experiments of the 'predator+pointdsc',
    Did you use the original 3DMatch(>30% overlaps) dataset or use the 3DLoMatch(10%-30% and >30% overlaps) to train the pointdsc?

    opened by chalth 1
Owner
PhD candidate at HKUST.
null
Supplementary code for the paper "Meta-Solver for Neural Ordinary Differential Equations" https://arxiv.org/abs/2103.08561

Meta-Solver for Neural Ordinary Differential Equations Towards robust neural ODEs using parametrized solvers. Main idea Each Runge-Kutta (RK) solver w

Julia Gusak 25 Aug 12, 2021
Unofficial implementation of "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" (https://arxiv.org/abs/2103.14030)

Swin-Transformer-Tensorflow A direct translation of the official PyTorch implementation of "Swin Transformer: Hierarchical Vision Transformer using Sh

null 52 Dec 29, 2022
This repo provides the official code for TransBTS: Multimodal Brain Tumor Segmentation Using Transformer (https://arxiv.org/pdf/2103.04430.pdf).

TransBTS: Multimodal Brain Tumor Segmentation Using Transformer This repo is the official implementation for TransBTS: Multimodal Brain Tumor Segmenta

Raymond 247 Dec 28, 2022
Official implementation of the paper Image Generators with Conditionally-Independent Pixel Synthesis https://arxiv.org/abs/2011.13775

CIPS -- Official Pytorch Implementation of the paper Image Generators with Conditionally-Independent Pixel Synthesis Requirements pip install -r requi

Multimodal Lab @ Samsung AI Center Moscow 201 Dec 21, 2022
This is an official implementation of our CVPR 2021 paper "Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression" (https://arxiv.org/abs/2104.02300)

Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression Introduction In this paper, we are interested in the bottom-up paradigm of estima

HRNet 367 Dec 27, 2022
Non-Official Pytorch implementation of "Face Identity Disentanglement via Latent Space Mapping" https://arxiv.org/abs/2005.07728 Using StyleGAN2 instead of StyleGAN

Face Identity Disentanglement via Latent Space Mapping - Implement in pytorch with StyleGAN 2 Description Pytorch implementation of the paper Face Ide

Daniel Roich 58 Dec 24, 2022
Official Implementation for "ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement" https://arxiv.org/abs/2104.02699

ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement Recently, the power of unconditional image synthesis has significantly advanced th

null 967 Jan 4, 2023
Unofficial Tensorflow-Keras implementation of Fastformer based on paper [Fastformer: Additive Attention Can Be All You Need](https://arxiv.org/abs/2108.09084).

Fastformer-Keras Unofficial Tensorflow-Keras implementation of Fastformer based on paper Fastformer: Additive Attention Can Be All You Need. Tensorflo

Yam Peleg 10 Jan 30, 2022
Pytorch implementation of Each Part Matters: Local Patterns Facilitate Cross-view Geo-localization https://arxiv.org/abs/2008.11646

[TCSVT] Each Part Matters: Local Patterns Facilitate Cross-view Geo-localization LPN [Paper] NEWs Prerequisites Python 3.6 GPU Memory >= 8G Numpy > 1.

null 46 Dec 14, 2022
A PyTorch implementation of EventProp [https://arxiv.org/abs/2009.08378], a method to train Spiking Neural Networks

Spiking Neural Network training with EventProp This is an unofficial PyTorch implemenation of EventProp, a method to compute exact gradients for Spiki

Pedro Savarese 35 Jul 29, 2022
Unofficial implementation of Alias-Free Generative Adversarial Networks. (https://arxiv.org/abs/2106.12423) in PyTorch

alias-free-gan-pytorch Unofficial implementation of Alias-Free Generative Adversarial Networks. (https://arxiv.org/abs/2106.12423) This implementation

Kim Seonghyeon 502 Jan 3, 2023
Pytorch implementation of Distributed Proximal Policy Optimization: https://arxiv.org/abs/1707.02286

Pytorch-DPPO Pytorch implementation of Distributed Proximal Policy Optimization: https://arxiv.org/abs/1707.02286 Using PPO with clip loss (from https

Alexis David Jacq 163 Dec 26, 2022
PyTorch implementation of Asymmetric Siamese (https://arxiv.org/abs/2204.00613)

Asym-Siam: On the Importance of Asymmetry for Siamese Representation Learning This is a PyTorch implementation of the Asym-Siam paper, CVPR 2022: @inp

Meta Research 89 Dec 18, 2022
Code for paper "A Critical Assessment of State-of-the-Art in Entity Alignment" (https://arxiv.org/abs/2010.16314)

A Critical Assessment of State-of-the-Art in Entity Alignment This repository contains the source code for the paper A Critical Assessment of State-of

Max Berrendorf 16 Oct 14, 2022
Code for the paper: Learning Adversarially Robust Representations via Worst-Case Mutual Information Maximization (https://arxiv.org/abs/2002.11798)

Representation Robustness Evaluations Our implementation is based on code from MadryLab's robustness package and Devon Hjelm's Deep InfoMax. For all t

Sicheng 19 Dec 7, 2022
Source code for models described in the paper "AudioCLIP: Extending CLIP to Image, Text and Audio" (https://arxiv.org/abs/2106.13043)

AudioCLIP Extending CLIP to Image, Text and Audio This repository contains implementation of the models described in the paper arXiv:2106.13043. This

null 458 Jan 2, 2023
Minimal implementation of PAWS (https://arxiv.org/abs/2104.13963) in TensorFlow.

PAWS-TF ?? Implementation of Semi-Supervised Learning of Visual Features by Non-Parametrically Predicting View Assignments with Support Samples (PAWS)

Sayak Paul 43 Jan 8, 2023
Tensorflow implementation of Semi-supervised Sequence Learning (https://arxiv.org/abs/1511.01432)

Transfer Learning for Text Classification with Tensorflow Tensorflow implementation of Semi-supervised Sequence Learning(https://arxiv.org/abs/1511.01

DONGJUN LEE 82 Oct 22, 2022