PointNetVLAD: Deep Point Cloud Based Retrieval for Large-Scale Place Recognition, CVPR 2018

Overview

PointNetVLAD: Deep Point Cloud Based Retrieval for Large-Scale Place Recognition

PointNetVLAD: Deep Point Cloud Based Retrieval for Large-Scale Place Recognition CVPR 2018, Salt Lake City, USA

Mikaela Angelina Uy and Gim Hee Lee

National University of Singapore

pic-network

Introduction

The PointNetVLAD is a deep network that addresses the problem of large-scale place recognition through point cloud based retrieval. The arXiv version of PointNetVLAD can be found here.

@inproceedings{uy2018pointnetvlad,
      title={PointNetVLAD: Deep Point Cloud Based Retrieval for Large-Scale Place Recognition},
      author={Uy, Mikaela Angelina and Lee, Gim Hee},
      booktitle={The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
      year={2018}
}

Benchmark Datasets

The benchmark datasets introdruced in this work can be downloaded here.

  • All submaps are in binary file format
  • Ground truth GPS coordinate of the submaps are found in the corresponding csv files for each run
  • Filename of the submaps are their timestamps which is consistent with the timestamps in the csv files
  • Use CSV files to define positive and negative point clouds
  • All submaps are preprocessed with the road removed and downsampled to 4096 points

Oxford Dataset

  • 45 sets in total of full and partial runs
  • Used both full and partial runs for training but only used full runs for testing/inference
  • Training submaps are found in the folder "pointcloud_20m_10overlap/" and its corresponding csv file is "pointcloud_locations_20m_10overlap.csv"
  • Training submaps are not mutually disjoint per run
  • Each training submap ~20m of car trajectory and subsequent submaps are ~10m apart
  • Test/Inference submaps found in the folder "pointcloud_20m/" and its corresponding csv file is "pointcloud_locations_20m.csv"
  • Test/Inference submaps are mutually disjoint

NUS (Inhouse) Datasets

  • Each inhouse dataset has 5 runs
  • Training submaps are found in the folder "pointcloud_25m_10/" and its corresponding csv file is "pointcloud_centroids_10.csv"
  • Test/Infenrence submaps are found in the folder "pointcloud_25m_25/" and its corresponding csv file is "pointcloud_centroids_25.csv"
  • Training submaps are not mutually disjoint per run but test submaps are

Project Code

Pre-requisites

  • Python
  • CUDA
  • Tensorflow
  • Scipy
  • Pandas
  • Sklearn

Code was tested using Python 3 on Tensorflow 1.4.0 with CUDA 8.0

sudo apt-get install python3-pip python3-dev python-virtualenv
virtualenv --system-site-packages -p python3 ~/tensorflow
source ~/tensorflow/bin/activate
easy_install -U pip
pip3 install --upgrade tensorflow-gpu==1.4.0
pip install scipy, pandas, sklearn

Dataset set-up

Download the zip file of the benchmark datasets found here. Extract the folder on the same directory as the project code. Thus, on that directory you must have two folders: 1) benchmark_datasets/ and 2) pointnetvlad/

Generate pickle files

We store the positive and negative point clouds to each anchor on pickle files that are used in our training and evaluation codes. The files only need to be generated once. The generation of these files may take a few minutes.

cd generating_queries/ 

# For training tuples in our baseline network
python generate_training_tuples_baseline.py

# For training tuples in our refined network
python generate_training_tuples_refine.py

# For network evaluation
python generate_test_sets.py

Model Training and Evaluation

To train our network, run the following command:

python train_pointnetvlad.py

To evaluate the model, run the following command:

python evaluate.py

Pre-trained Models

The pre-trained models for both the baseline and refined networks can be downloaded here

Submap generation

Added the rough MATLAB code that was used for submap generation upon requests. Some functions are gotten from the toolbox of Oxford Robotcar.

Some clarification: The voxel grid filter was used to downsample the cloud to 4096, which was done by selecting a leaf size that initially downsamples the cloud close to 4096 points, after which we randomly add points to make the cloud have exactly 4096 points. Please feel free to send me an email ([email protected]) for any further questions.

License

This repository is released under MIT License (see LICENSE file for details).

Comments
  • Issue with triplet loss

    Issue with triplet loss

    I think your implementation of best_pos_distance (which is used for every loss in your code) should use reduce_max instead of reduce_min (following the original hard triplet loss paper). I re-implemented your code in PyTorch, and with max pos_distance i got an average top 1% recall on oxford baseline (no refine) of 84%

    opened by cattaneod 9
  • what the size of RAM did you use?

    what the size of RAM did you use?

    Hi, i was running the code using a computer with 16GB memory and NVIDIA GeForce GTX2070. However, the memory was filled when I run the generate_training_tuples_baselineand.py & evaluate.py. The process was killed by the system. It seems that the size of the memory is not enough. Therefore, I would like to know how large the RAM did you use?

    opened by roycyh 5
  • Is the loss trend normal?

    Is the loss trend normal?

    @mikacuy Hi,mika. First of all, thanks for your great work. I'm new to place recognition tasks and trying to retrain your network as a tutorial. I used to retrain some other networks and the losses go almost down directly while the loss defined in your network keeps going up and down. Here is a screenshot which I log out the batch losses. So does it go the right way as you did before? Tell me if I miss any details. Thanks image

    Claude Cui

    opened by cloudcjf 4
  • A question about trasplanting the model to KITTI dataset.

    A question about trasplanting the model to KITTI dataset.

    Hi! Thanks for your great work and code release. Recently I am also doing some research on 3d-relocalization. After reading your paper and part of code, I am trying to convert the kitti data to the format which this code can run. And I wonder how strong is the generalization ability of this model? Will the recall decrease much? Have you tested before? Great thanks!!

    opened by songjiaxina 4
  • Question about the benchmark_datasets

    Question about the benchmark_datasets

    Hi, thanks for your great job! But I am trying to visualize your submaps, and the following picture is from one of the .bin file 图片 However, the max value is up to 10^38. I have no idea what is the visualization result represent for. Is there something wrong with the submap dataset? Or I miss something?

    opened by ahandsomeperson 3
  • A simple question about the dataset

    A simple question about the dataset

    Hi, thanks for your code and dataset. Could you please tell me the difference between the folder called pointcloud_20m_10overlap and the folder called pointcloud_20m?

    opened by songjiaxina 2
  • "Failed to find any matching files for log/model.ckpt"

    Hi, @mikacuy may I ask how to solve this problem? It seems there is no file named model.ckpt. I only found model_baseline.ckpt.index and model_refine.ckpt.index from the downloaded pre-trained model.

    2019-12-21 18:46:13.067473: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for log/model.ckpt

    Thanks.

    opened by roycyh 1
  • Different timestamps for Oxford dataset when generating submaps

    Different timestamps for Oxford dataset when generating submaps

    Hi, thanks for your great work! I am trying to use the script generate_submaps.m to recreate the benchmark dataset for the Oxford Robotcar dataset. However, I seem to get different pointclouds (as described by the timestamps) than the ones found in the uploaded benchmarks folder on google drive.

    For example, for the 2015-08-17-10-26-38 folder, the 1439803802060497.bin pointcloud cannot be found after generating the submaps, with the closest generated timestamps being: 1439803802982745.bin and 1439803801699224.bin.

    There is some overlap between the generated and posted pointclouds for the same 2015-08-17-10-26-38 folder, such as: 1439803985022188.bin and 1439803989723175.bin.

    I have not modified the uploaded script except for path changes, and am using the default values for the following parameters:

    submap_cover_distance=20.
    laser_reading_distance=0.025;
    laser_reading_angle=30;
    dist_start_next_frame=10.0;
    

    Are there any parameter changes that need to be made to get the generated timestamps & pointclouds to correspond? Thank you!

    opened by sashaDoubov 1
  • Cannot find groundplane fitting code

    Cannot find groundplane fitting code

    Hi! Thanks for sharing your code! I couldn't find the source code of removing ground plane (e.g. pcfitplane). Could you give me some guidance? Is there something wrong? Thanks! Looking forward to your reply!

    opened by kxhit 1
  • Question about generating the dataset

    Question about generating the dataset

    Hi, thanks for your code. I have a question regarding the datasets you are using to generate the training tuples. In the file generate_training_tuples_baseline.py, line 21, index_list=range(len(all_folders)-1) , it seems that you are using all runs except for the last sequence. Any reason for this? Or did I miss something? Thanks for your help in advance!

    opened by JuanDuGit 1
  • global variable

    global variable "HARD_NEGATIVES={}" is never updated

    Thanks for your wonderful work, but i find in the file "train_pointnetvlad", the global variable "HARD_NEGATIVES={}" is never updated which means it is useless, is there something wrong here?

    opened by Master-cai 0
  • About reverse direction loops

    About reverse direction loops

    Dear Author,

    It seems like PointNetVLAD couldn't not perform well for reverse direction loops in my experiments although there is T-Net... Have you tried to recognize reverse direction localization with PointNetVLAD? I'm not sure if my experiments went wrong.

    Thanks.

    opened by BIT-MJY 0
  • Cannot find

    Cannot find "pointcloud_20m/" and "pointcloud_locations_20m.csv"

    Cannot find the following folder and file:

    • Test/Inference submaps found in the folder "pointcloud_20m/" and its corresponding csv file is "pointcloud_locations_20m.csv"
    opened by chenpengxin 0
  • About training and test data split

    About training and test data split

    Hi, thanks for your code! I have a question about training and test data split in generate_training_tuples_baseline.py. Line 30, you set p1 = [5735712.768124,620084.402381], p2 = [5735611.299219,620540.270327], p3 = [5735237.358209,620543.094379], and p4 = [5734749.303802,619932.693364]. I think they are the mean value of "northing" and "easting". I just want to know how to get the four different mean values. And why you choose four mean values not five or more. Thanks for your help in advance!

    opened by PSYZ1234 2
  • How can I use your PointNetVLAD code for localization (place retrieval)

    How can I use your PointNetVLAD code for localization (place retrieval)

    I have read your paper about PointNetVLAD and run the code you provide on the Github. It is really a great work.

    Now, I would like to use PointNetVLAD for place localization (or point cloud retrieval) using the global descriptor. In fact, I see a lot of similar work you have done in your paper. For example, the work offigure 4. You used the query point clouds to retrieve the corresponding point clouds.

    Now, I would like to perform the similar task as you did in figure 4. However, the code provided on the github is more likely to evaluate the performance of the network.

    I would like to know what code you used to realize the above function (figure 4 in paper) so that it can get a retrieved point cloud with a input one. Otherwise, how can I use the existing code on github to fulfill this task?

    In fact, I have sent this message to your email [email protected]. It seems that the address has expired. You may update it if you do not mind.

    Thanks.

    opened by roycyh 2
Owner
Mikaela Uy
CS PhD Student
Mikaela Uy
Style-based Point Generator with Adversarial Rendering for Point Cloud Completion (CVPR 2021)

Style-based Point Generator with Adversarial Rendering for Point Cloud Completion (CVPR 2021) An efficient PyTorch library for Point Cloud Completion.

Microsoft 119 Jan 2, 2023
Point Cloud Denoising input segmentation output raw point-cloud valid/clear fog rain de-noised Abstract Lidar sensors are frequently used in environme

Point Cloud Denoising input segmentation output raw point-cloud valid/clear fog rain de-noised Abstract Lidar sensors are frequently used in environme

null 75 Nov 24, 2022
Code for "FPS-Net: A convolutional fusion network for large-scale LiDAR point cloud segmentation".

FPS-Net Code for "FPS-Net: A convolutional fusion network for large-scale LiDAR point cloud segmentation", accepted by ISPRS journal of Photogrammetry

null 15 Nov 30, 2022
Perturbed Self-Distillation: Weakly Supervised Large-Scale Point Cloud Semantic Segmentation (ICCV2021)

Perturbed Self-Distillation: Weakly Supervised Large-Scale Point Cloud Semantic Segmentation (ICCV2021) This is the implementation of PSD (ICCV 2021),

null 12 Dec 12, 2022
Official pytorch implementation of "DSPoint: Dual-scale Point Cloud Recognition with High-frequency Fusion"

DSPoint Official implementation of "DSPoint: Dual-scale Point Cloud Recognition with High-frequency Fusion". Paper link: https://arxiv.org/abs/2111.10

Ziyao Zeng 10 Nov 24, 2021
This repo contains the official code of our work SAM-SLR which won the CVPR 2021 Challenge on Large Scale Signer Independent Isolated Sign Language Recognition.

Skeleton Aware Multi-modal Sign Language Recognition By Songyao Jiang, Bin Sun, Lichen Wang, Yue Bai, Kunpeng Li and Yun Fu. Smile Lab @ Northeastern

Isen (Songyao Jiang) 128 Dec 8, 2022
Pytorch implementation for "Large-Scale Long-Tailed Recognition in an Open World" (CVPR 2019 ORAL)

Large-Scale Long-Tailed Recognition in an Open World [Project] [Paper] [Blog] Overview Open Long-Tailed Recognition (OLTR) is the author's re-implemen

Zhongqi Miao 761 Dec 26, 2022
Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking

Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking We revisit and address issues with Oxford 5k and Paris 6k image retrieval benchm

Filip Radenovic 188 Dec 17, 2022
DeepLM: Large-scale Nonlinear Least Squares on Deep Learning Frameworks using Stochastic Domain Decomposition (CVPR 2021)

DeepLM DeepLM: Large-scale Nonlinear Least Squares on Deep Learning Frameworks using Stochastic Domain Decomposition (CVPR 2021) Run Please install th

Jingwei Huang 130 Dec 2, 2022
3D ResNets for Action Recognition (CVPR 2018)

3D ResNets for Action Recognition Update (2020/4/13) We published a paper on arXiv. Hirokatsu Kataoka, Tenga Wakamiya, Kensho Hara, and Yutaka Satoh,

Kensho Hara 3.5k Jan 6, 2023
Not All Points Are Equal: Learning Highly Efficient Point-based Detectors for 3D LiDAR Point Clouds (CVPR 2022, Oral)

Not All Points Are Equal: Learning Highly Efficient Point-based Detectors for 3D LiDAR Point Clouds (CVPR 2022, Oral) This is the official implementat

Yifan Zhang 259 Dec 25, 2022
Implementation of the "PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences" paper.

PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences Introduction Point cloud sequences are irregular and unordered in the spatial dimen

Hehe Fan 63 Dec 9, 2022
Implementation of the "Point 4D Transformer Networks for Spatio-Temporal Modeling in Point Cloud Videos" paper.

Point 4D Transformer Networks for Spatio-Temporal Modeling in Point Cloud Videos Introduction Point cloud videos exhibit irregularities and lack of or

Hehe Fan 101 Dec 29, 2022
Synthetic LiDAR sequential point cloud dataset with point-wise annotations

SynLiDAR dataset: Learning From Synthetic LiDAR Sequential Point Cloud This is official repository of the SynLiDAR dataset. For technical details, ple

null 78 Dec 27, 2022
[ICCV 2021 Oral] SnowflakeNet: Point Cloud Completion by Snowflake Point Deconvolution with Skip-Transformer

This repository contains the source code for the paper SnowflakeNet: Point Cloud Completion by Snowflake Point Deconvolution with Skip-Transformer (ICCV 2021 Oral). The project page is here.

AllenXiang 65 Dec 26, 2022
Code for the CVPR2021 paper "Patch-NetVLAD: Multi-Scale Fusion of Locally-Global Descriptors for Place Recognition"

Patch-NetVLAD: Multi-Scale Fusion of Locally-Global Descriptors for Place Recognition This repository contains code for the CVPR2021 paper "Patch-NetV

QVPR 368 Jan 6, 2023
Compute descriptors for 3D point cloud registration using a multi scale sparse voxel architecture

MS-SVConv : 3D Point Cloud Registration with Multi-Scale Architecture and Self-supervised Fine-tuning Compute features for 3D point cloud registration

null 42 Jul 25, 2022
This repository allows the user to automatically scale a 3D model/mesh/point cloud on Agisoft Metashape

Metashape-Utils This repository allows the user to automatically scale a 3D model/mesh/point cloud on Agisoft Metashape, given a set of 2D coordinates

INSCRIBE 4 Nov 7, 2022
🏆 The 1st Place Submission to AICity Challenge 2021 Natural Language-Based Vehicle Retrieval Track (Alibaba-UTS submission)

AI City 2021: Connecting Language and Vision for Natural Language-Based Vehicle Retrieval ?? The 1st Place Submission to AICity Challenge 2021 Natural

null 82 Dec 29, 2022