This repository contains the implementation of the following paper: Cross-Descriptor Visual Localization and Mapping

Related tags

Deep Learning cross-descriptor-vis-loc-map
Overview

Cross-Descriptor Visual Localization and Mapping

This repository contains the implementation of the following paper:

"Cross-Descriptor Visual Localization and Mapping".
M. Dusmanu, O. Miksik, J.L. Schönberger, and M. Pollefeys. ICCV 2021.

[Paper on arXiv]

Requirements

COLMAP

We use COLMAP for DoG keypoint extraction as well as localization and mapping. Please follow the installation instructions available on the official webpage. Before proceeding, we recommend setting an environmental variable to the COLMAP executable folder by running export COLMAP_PATH=path_to_colmap_executable_folder.

Python

The environment can be set up directly using conda:

conda env create -f env.yml
conda activate cross-descriptor-vis-loc-map

Training data

We provide a script for downloading the raw training data:

bash scripts/download_training_data.sh

Evaluation data

We provide a script for downloading the LFE dataset along with the GT used for evaluation as well as the Aachen Day-Night dataset:

bash scripts/download_evaluation_data.sh

Training

Data preprocessing

First step is extracting keypoints and descriptors on the training data downloaded above.

bash scripts/process_training_data.sh

Alternatively, you can directly download the processed training data by running:

bash scripts/download_processed_training_data.sh

Training

To run training with the default architecture and hyper-parameters, execute the following:

python train.py \
    --dataset_path data/train/colmap \
    --features brief sift-kornia hardnet sosnet

Pretrained models

We provide two pretrained models trained on descriptors extracted from COLMAP SIFT and OpenCV SIFT keypoints, respectively. These models can be downloaded by running:

bash scripts/download_checkpoints.sh

Evaluation

Demo Notebook

Click for details...

Local Feature Evaluation Benchmark

Click for details...

First step is extracting descriptors on all datasets:

bash scripts/process_LFE_data.sh

We provide examples below for running reconstruction on Madrid Metrpolis in each different evaluation scenario.

Reconstruction using a single descriptor (standard)

python local-feature-evaluation/reconstruction_pipeline_progressive.py \
    --dataset_path data/eval/LFE-release/Madrid_Metropolis \
    --colmap_path $COLMAP_PATH \
    --features sift-kornia \
    --exp_name sift-kornia-single

Reconstruction using the progressive approach (ours)

python local-feature-evaluation/reconstruction_pipeline_progressive.py \
    --dataset_path data/eval/LFE-release/Madrid_Metropolis \
    --colmap_path $COLMAP_PATH \
    --features brief sift-kornia hardnet sosnet \
    --exp_name progressive

Reconstruction using the joint embedding approach (ours)

python local-feature-evaluation/reconstruction_pipeline_embed.py \
    --dataset_path data/eval/LFE-release/Madrid_Metropolis \
    --colmap_path $COLMAP_PATH \
    --features brief sift-kornia hardnet sosnet \
    --exp_name embed

Reconstruction using a single descriptor on the associated split (real-world)

python local-feature-evaluation/reconstruction_pipeline_subset.py \
    --dataset_path data/eval/LFE-release/Madrid_Metropolis/ \
    --colmap_path $COLMAP_PATH \
    --features brief sift-kornia hardnet sosnet \
    --feature sift-kornia \
    --exp_name sift-kornia-subset

Evaluation of a reconstruction w.r.t. metric pseudo-ground-truth

python local-feature-evaluation/align_and_compare.py \
    --colmap_path $COLMAP_PATH \
    --reference_model_path data/eval/LFE-release/Madrid_Metropolis/sparse-reference/filtered-metric/ \
    --model_path data/eval/LFE-release/Madrid_Metropolis/sparse-sift-kornia-single/0/

Aachen Day-Night

Click for details...

BibTeX

If you use this code in your project, please cite the following paper:

@InProceedings{Dusmanu2021Cross,
    author = {Dusmanu, Mihai and Miksik, Ondrej and Sch\"onberger, Johannes L. and Pollefeys, Marc},
    title = {{Cross Descriptor Visual Localization and Mapping}},
    booktitle = {Proceedings of the International Conference on Computer Vision},
    year = {2021}
}
You might also like...
Repository for the paper
Repository for the paper "Online Domain Adaptation for Occupancy Mapping", RSS 2020

RSS 2020 - Online Domain Adaptation for Occupancy Mapping Repository for the paper "Online Domain Adaptation for Occupancy Mapping", Robotics: Science

Anthony 26 Sep 22, 2022
A 2D Visual Localization Framework based on Essential Matrices [ICRA2020]
A 2D Visual Localization Framework based on Essential Matrices [ICRA2020]

A 2D Visual Localization Framework based on Essential Matrices This repository provides implementation of our paper accepted at ICRA: To Learn or Not

Qunjie Zhou 27 Nov 7, 2022
DSAC* for Visual Camera Re-Localization (RGB or RGB-D)
DSAC* for Visual Camera Re-Localization (RGB or RGB-D)

DSAC* for Visual Camera Re-Localization (RGB or RGB-D) Introduction Installation Data Structure Supported Datasets 7Scenes 12Scenes Cambridge Landmark

Visual Learning Lab 143 Dec 22, 2022
Weak-supervised Visual Geo-localization via Attention-based Knowledge Distillation

Weak-supervised Visual Geo-localization via Attention-based Knowledge Distillation Introduction WAKD is a PyTorch implementation for our ICPR-2022 pap

null 2 Oct 20, 2022
Re-implementation of the Noise Contrastive Estimation algorithm for pyTorch, following "Noise-contrastive estimation: A new estimation principle for unnormalized statistical models." (Gutmann and Hyvarinen, AISTATS 2010)

Noise Contrastive Estimation for pyTorch Overview This repository contains a re-implementation of the Noise Contrastive Estimation algorithm, implemen

Denis Emelin 42 Nov 24, 2022
The implementation of the paper "A Deep Feature Aggregation Network for Accurate Indoor Camera Localization".

A Deep Feature Aggregation Network for Accurate Indoor Camera Localization This is the PyTorch implementation of our paper "A Deep Feature Aggregation

null 9 Dec 9, 2022
This repository contains a re-implementation of the code for the CVPR 2021 paper
This repository contains a re-implementation of the code for the CVPR 2021 paper "Omnimatte: Associating Objects and Their Effects in Video."

Omnimatte in PyTorch This repository contains a re-implementation of the code for the CVPR 2021 paper "Omnimatte: Associating Objects and Their Effect

Erika Lu 728 Dec 28, 2022
This repository contains numerical implementation for the paper Intertemporal Pricing under Reference Effects: Integrating Reference Effects and Consumer Heterogeneity.

This repository contains numerical implementation for the paper Intertemporal Pricing under Reference Effects: Integrating Reference Effects and Consumer Heterogeneity.

Hansheng Jiang 6 Nov 18, 2022
An image base contains 490 images for learning (400 cars and 90 boats), and another 21 images for testingAn image base contains 490 images for learning (400 cars and 90 boats), and another 21 images for testing
An image base contains 490 images for learning (400 cars and 90 boats), and another 21 images for testingAn image base contains 490 images for learning (400 cars and 90 boats), and another 21 images for testing

SVM Données Une base d’images contient 490 images pour l’apprentissage (400 voitures et 90 bateaux), et encore 21 images pour fait des tests. Prétrait

Achraf Rahouti 3 Nov 30, 2021
Comments
  • Update README.md

    Update README.md

    Should bash scripts/preprocess_training_data.sh be bash scripts/process_training_data.sh ?

    Or is the preprocess_training_data.sh script file missing from the scripts directory?

    opened by mikkosaarisalo 0
  • Question about the feature convertion

    Question about the feature convertion

    Dear Professor: Thanks for your wonderful work. However, there are a question which bothering me a lot. Is it possible to use this algorithm to transform the vgg feature descriptor to the SIFT descriptor？ Yours

    opened by Gatsby23 0
Owner
Mihai Dusmanu
PhD Student at ETH Zurich. Computer Vision + Deep Learning. Feature detection / description / matching, 3D reconstruction.
Mihai Dusmanu
GitHub
This repository contains notebook implementations of the following Neural Process variants: Conditional Neural Processes (CNPs), Neural Processes (NPs), Attentive Neural Processes (ANPs).

The Neural Process Family This repository contains notebook implementations of the following Neural Process variants: Conditional Neural Processes (CN

DeepMind 892 Dec 28, 2022
Implementation of the method proposed in the paper "Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation"

Neural Descriptor Fields (NDF) PyTorch implementation for training continuous 3D neural fields to represent dense correspondence across objects, and u

null 167 Jan 6, 2023
This reporistory contains the test-dev data of the paper "xGQA: Cross-lingual Visual Question Answering".

This reporistory contains the test-dev data of the paper "xGQA: Cross-lingual Visual Question Answering".

AdapterHub 18 Dec 9, 2022
Python scripts performing class agnostic object localization using the Object Localization Network model in ONNX.

ONNX Object Localization Network Python scripts performing class agnostic object localization using the Object Localization Network model in ONNX. Ori

Ibai Gorordo 15 Oct 14, 2022
PyTorch implementation of SIFT descriptor

This is an differentiable pytorch implementation of SIFT patch descriptor. It is very slow for describing one patch, but quite fast for batch. It can

Dmytro Mishkin 150 Dec 24, 2022
Open source implementation of "A Self-Supervised Descriptor for Image Copy Detection" (SSCD).

A Self-Supervised Descriptor for Image Copy Detection (SSCD) This is the open-source codebase for "A Self-Supervised Descriptor for Image Copy Detecti

Meta Research 68 Jan 4, 2023
Pytorch implementation of Each Part Matters: Local Patterns Facilitate Cross-view Geo-localization https://arxiv.org/abs/2008.11646

[TCSVT] Each Part Matters: Local Patterns Facilitate Cross-view Geo-localization LPN [Paper] NEWs Prerequisites Python 3.6 GPU Memory >= 8G Numpy > 1.

null 46 Dec 14, 2022
Official Implementation of Few-shot Visual Relationship Co-localization

VRC Official implementation of the Few-shot Visual Relationship Co-localization (ICCV 2021) paper project page | paper Requirements Use python >= 3.8.

null 22 Oct 13, 2022
LVI-SAM: Tightly-coupled Lidar-Visual-Inertial Odometry via Smoothing and Mapping

LVI-SAM This repository contains code for a lidar-visual-inertial odometry and mapping system, which combines the advantages of LIO-SAM and Vins-Mono

Tixiao Shan 1.1k Dec 27, 2022
COVINS -- A Framework for Collaborative Visual-Inertial SLAM and Multi-Agent 3D Mapping

COVINS -- A Framework for Collaborative Visual-Inertial SLAM and Multi-Agent 3D Mapping Version 1.0 COVINS is an accurate, scalable, and versatile vis

ETHZ V4RL 183 Dec 27, 2022