PyTorch implementation of NeurIPS 2021 paper: "CoFiNet: Reliable Coarse-to-fine Correspondences for Robust Point Cloud Registration"

Related tags

Deep Learning Coarse-to-fine-correspondences
Overview

CoFiNet: Reliable Coarse-to-fine Correspondences for Robust Point Cloud Registration (NeurIPS 2021)

PyTorch implementation of the paper:

CoFiNet: Reliable Coarse-to-fine Correspondences for Robust Point Cloud Registration by:

Hao Yu, Fu Li, Mahdi Saleh, Benjamin Busam and Slobodan Ilic.

Introduction

We study the problem of extracting correspondences between a pair of point clouds for registration. For correspondence retrieval, existing works benefit from matching sparse keypoints detected from dense points but usually struggle to guarantee their repeatability. To address this issue, we present CoFiNet - Coarse-to-Fine Network which extracts hierarchical correspondences from coarse to fine without keypoint detection. On a coarse scale and guided by a weighting scheme, our model firstly learns to match down-sampled nodes whose vicinity points share more overlap, which significantly shrinks the search space of a consecutive stage. On a finer scale, node proposals are consecutively expanded to patches that consist of groups of points together with associated descriptors. Point correspondences are then refined from the overlap areas of corresponding patches, by a density-adaptive matching module capable to deal with varying point density. Extensive evaluation of CoFiNet on both indoor and outdoor standard benchmarks shows our superiority over existing methods. Especially on 3DLoMatch where point clouds share less overlap, CoFiNet significantly outperforms state-of-the-art approaches by at least 5% on Registration Recall, with at most two-third of their parameters.

image

News

  • 28.10.2021: Paper available on arxiv.

  • 27.10.2021: Release training and testing code of 3DMatch and 3DLoMatch.

Installation

  • Clone the repository:

    git clone https://github.com/haoyu94/Coarse-to-fine-correspondences.git
cd Coarse-to-fine-correspondences

  • Create conda environment and install requirements:

    conda create -n {environment name} python=3.8
pip install -r requirements.txt

  • Compile C++ and CUDA scripts:

    cd cpp_wrappers
sh compile_wrappers.sh
cd ..

Demo

TBD

3DMatch & 3DLoMatch

Pretrained model

Pretrained model is given in weights/.

Prepare datasets

sh scripts/download_data.sh

Train

sh scripts/train_3dmatch.sh

Test

  • Point correspondences are first extracted by running:
sh scripts/test_3dmatch.sh

and stored on snapshot/tdmatch_enc_dec_test/3DMatch/.

  • To evaluate on 3DLoMatch, please change the benchmark keyword in configs/tdmatch/tdmatch_test.yaml from 3DMatch to 3DLoMatch.

  • The evaluation of extracted correspondences and relative poses estimated by RANSAC can be done by running:

sh scripts/run_ransac.sh

  • The final results are stored in est_traj/3DMatch/{number of correspondences}/result and the results evaluated on our computer have been provided in est_traj/.

  • To evaluate on 3DLoMatch, please change 3DMatch in scripts/run_ransac.sh to 3DLoMatch.

KITTI

TBD

Acknowledgments

The code is heavily borrowed from PREDATOR.

Our backbone network is from KPConv.

We use the Transformer implementation in SuperGlue.

Sinkhorn implementation is from SuperGlue and RPM-Net.

Citiation

TBD

Comments
  • Question about ransac pose estimation

    Question about ransac pose estimation

    Hello, I very appreciate your excellent work. I have some confusion about ransac_pose_estimation_correspondences in lib/tester.py when reading your code I find checkers in o3d.registration.registration_ransac_based_on_correspondence can speed up code compared with settings without checkers I also find checkers is only used in o3d.registration.registration_ransac_based_on_feature_matching in OverlapPredator Code. I am curious about how this setting works?

    opened by Ericgeese 2
  • How to get the result of kitti

    How to get the result of kitti

    I've test the parameters in kitti.yaml but found it can't work, then I refer to the parameter used for 3dMatch and successfully run the code,but the result of recall is only 6.1%, what should I do if I want to get a better result in kitti?

    opened by ljnaaa 2
  • How to draw figure-3?

    How to draw figure-3?

    Hi, this is amazing work. I am trying to reproduce the results in your paper. But I encountered a problem when trying to draw Figure 3 in the paper. I learned the knowledge of open3D, but I still don't know how to display the two registered point clouds on one map and connect the corresponding points with lines. I would like to ask if you can provide the source code of Figure 3, or tell me what method and function I can use to draw the effect of Figure 3. As a beginner in the field, I know this question may be simple, but I'd still appreciate to get your help. Thanks！ 0

    opened by Ruye-aa 2
  • Question about the generation of ground truth confidence matrix at coarse scale

    Question about the generation of ground truth confidence matrix at coarse scale

    Hello! I have a question about the equation-(4) in the paper which states about the generation of the gt-confidence matrix at coarse scale: image However, during your code implementation: https://github.com/haoyu94/Coarse-to-fine-correspondences/blob/dd899ab612ffb15d551aaf4490f1db32cde029fd/lib/utils.py#L148-L158 In my understanding, the r(i',j') is actually defined as: image Also the final calculation of W'(i',j') is defined as the multiplication of r(i',j') and r(i'), which is different from the equation-(5) in the paper, could you give some hints about it?

    Thanks very much for your help.

    opened by qsisi 1
  • Questions about the understanding of sinkhorn operations

    Questions about the understanding of sinkhorn operations

    Hello! Thanks for open-sourcing this amazing work. Here I got a question about the sinkhorn operation utilized in the paper.

    I noticed that there are two versions of sinkhorn operations used in the paper. One is the version from SuperGlue, the other is from the RPMNet. The first version is used to process the soft assignment of coarse-level matching, and the second is for the fine-level matching.

    The two implementations actually differs a lot in implementations, the most obvious one is the choice of padding, one is to pad the score matrix with learnable parameters, while the other chooses the fixed zero scalar.

    Could you give some hints about the behind reasons for the two sinkhorn choices when you designed the network? Or what is your understanding of these two implementations of the sinkhorn operations above? That would be so helpful.

    Thank you very much for your help.

    opened by qsisi 1
  • How to increase batch_size?

    How to increase batch_size?

    Thank you for your outstanding work, I have some questions when running the code.

    When I successfully run the program, I found that an epoch takes a long time to run. I tried to increase the batchsize，and change the batchsize from 1 to 2. But I encountered a bug here： File "/home/aiyang/anaconda3/lib/python3.7/site-packages/torch/_utils.py", line 429, in reraise raise self.exc_type(msg) AssertionError: Caught AssertionError in DataLoader worker process 0. Original Traceback (most recent call last): File "/home/aiyang/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 202, in _worker_loop data = fetcher.fetch(index) File "/home/aiyang/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch return self.collate_fn(data) File "/home/aiyang/Coarse-to-fine-correspondences/model/KPConv/preprocessing.py", line 72, in collate_fn_descriptor assert len(list_data) == 1 AssertionError

    I wanna to know if there is any way to solve this problem, or if you have any way to increase the training speed during training. Thanks

    opened by Ruye-aa 1
Owner
null
GitHub
Official implementation of NeurIPS 2021 paper "One Loss for All: Deep Hashing with a Single Cosine Similarity based Learning Objective"

Official implementation of NeurIPS 2021 paper "One Loss for All: Deep Hashing with a Single Cosine Similarity based Learning Objective"

Ng Kam Woh 71 Dec 22, 2022
Official implementation of NeurIPS 2021 paper "Contextual Similarity Aggregation with Self-attention for Visual Re-ranking"

CSA: Contextual Similarity Aggregation with Self-attention for Visual Re-ranking PyTorch training code for CSA (Contextual Similarity Aggregation). We

Hui Wu 19 Oct 21, 2022
The official implementation of NeurIPS 2021 paper: Finding Optimal Tangent Points for Reducing Distortions of Hard-label Attacks

The official implementation of NeurIPS 2021 paper: Finding Optimal Tangent Points for Reducing Distortions of Hard-label Attacks

machen 11 Nov 27, 2022
Official implementation of NeurIPS'2021 paper TransformerFusion

TransformerFusion: Monocular RGB Scene Reconstruction using Transformers Project Page | Paper | Video TransformerFusion: Monocular RGB Scene Reconstru

Aljaz Bozic 118 Dec 25, 2022
Pytorch implementation of RED-SDS (NeurIPS 2021).

Recurrent Explicit Duration Switching Dynamical Systems (RED-SDS) This repository contains a reference implementation of RED-SDS, a non-linear state s

Abdul Fatir 10 Dec 2, 2022
Official Pytorch implementation of "Unbiased Classification Through Bias-Contrastive and Bias-Balanced Learning (NeurIPS 2021)

Unbiased Classification Through Bias-Contrastive and Bias-Balanced Learning (NeurIPS 2021) Official Pytorch implementation of Unbiased Classification

Youngkyu 17 Jan 1, 2023
The PyTorch implementation of Directed Graph Contrastive Learning (DiGCL), NeurIPS-2021

Directed Graph Contrastive Learning The PyTorch implementation of Directed Graph Contrastive Learning (DiGCL). In this paper, we present the first con

Tong Zekun 28 Jan 8, 2023
This is an official PyTorch implementation of Task-Adaptive Neural Network Search with Meta-Contrastive Learning (NeurIPS 2021, Spotlight).

NeurIPS 2021 (Spotlight): Task-Adaptive Neural Network Search with Meta-Contrastive Learning This is an official PyTorch implementation of Task-Adapti

Wonyong Jeong 15 Nov 21, 2022
Official Pytorch implementation for Deep Contextual Video Compression, NeurIPS 2021

Introduction Official Pytorch implementation for Deep Contextual Video Compression, NeurIPS 2021 Prerequisites Python 3.8 and conda, get Conda CUDA 11

null 51 Dec 3, 2022
This is a pytorch implementation of the NeurIPS paper GAN Memory with No Forgetting.

GAN Memory for Lifelong learning This is a pytorch implementation of the NeurIPS paper GAN Memory with No Forgetting. Please consider citing our paper

Miaoyun Zhao 43 Dec 27, 2022
Pytorch Implementation for NeurIPS (oral) paper: Pixel Level Cycle Association: A New Perspective for Domain Adaptive Semantic Segmentation

Pixel-Level Cycle Association This is the Pytorch implementation of our NeurIPS 2020 Oral paper Pixel-Level Cycle Association: A New Perspective for D

null 87 Oct 19, 2022
Code for our NeurIPS 2021 paper Mining the Benefits of Two-stage and One-stage HOI Detection

CDN Code for our NeurIPS 2021 paper "Mining the Benefits of Two-stage and One-stage HOI Detection". Contributed by Aixi Zhang*, Yue Liao*, Si Liu, Mia

null 71 Dec 14, 2022
Code to reproduce the experiments from our NeurIPS 2021 paper " The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspective"

Code To run: python runner.py new --save <SAVE_NAME> --data <PATH_TO_DATA_DIR> --dataset <DATASET> --model <model_name> [options] --n 1000 - train - t

Geoff Pleiss 5 Dec 12, 2022
Companion code for the paper "An Infinite-Feature Extension for Bayesian ReLU Nets That Fixes Their Asymptotic Overconfidence" (NeurIPS 2021)

ReLU-GP Residual (RGPR) This repository contains code for reproducing the following NeurIPS 2021 paper: @inproceedings{kristiadi2021infinite, title=

Agustinus Kristiadi 4 Dec 26, 2021
Code for our NeurIPS 2021 paper 'Exploiting the Intrinsic Neighborhood Structure for Source-free Domain Adaptation'

Exploiting the Intrinsic Neighborhood Structure for Source-free Domain Adaptation (NeurIPS 2021) Code for our NeurIPS 2021 paper 'Exploiting the Intri

Shiqi Yang 53 Dec 25, 2022
This GitHub repository contains code used for plots in NeurIPS 2021 paper 'Stochastic Multi-Armed Bandits with Control Variates.'

About Repository This repository contains code used for plots in NeurIPS 2021 paper 'Stochastic Multi-Armed Bandits with Control Variates.' About Code

Arun Verma 1 Nov 9, 2021
Source code of NeurIPS 2021 Paper ''Be Confident! Towards Trustworthy Graph Neural Networks via Confidence Calibration''

CaGCN This repo is for source code of NeurIPS 2021 paper "Be Confident! Towards Trustworthy Graph Neural Networks via Confidence Calibration". Paper L

null 6 Dec 19, 2022
Code for NeurIPS 2021 paper: Invariant Causal Imitation Learning for Generalizable Policies

Invariant Causal Imitation Learning for Generalizable Policies Ioana Bica, Daniel Jarrett, Mihaela van der Schaar Neural Information Processing System

Ioana Bica 17 Dec 1, 2022
Official implementation of "Open-set Label Noise Can Improve Robustness Against Inherent Label Noise" (NeurIPS 2021)

Open-set Label Noise Can Improve Robustness Against Inherent Label Noise NeurIPS 2021: This repository is the official implementation of ODNL. Require

Hongxin Wei 12 Dec 7, 2022