Pip-package for trajectory benchmarking from "Be your own Benchmark: No-Reference Trajectory Metric on Registered Point Clouds", ECMR'21

Overview

Map Metrics for Trajectory Quality

Map metrics toolkit provides a set of metrics to quantitatively evaluate trajectory quality via estimating consistency of the map aggregated from point clouds.

GPS or Motion Capture systems are not always available in perception systems, or their quality is not enough (GPS on small-scale distances) for use as ground truth trajectory. Thus, common full-reference trajectory metrics (APE, RPE, and their modifications) could not be applied to evaluate trajectory quality. When 3D sensing technologies (depth camera, LiDAR) are available on the perception system, one can alternatively assess trajectory quality --- estimate the consistency of the map from registered point clouds via the trajectory.

Documentation: https://map-metrics.readthedocs.io.

Documentation Status Updates

Features

Our toolkit provides implementation of the next metrics:

  • Mean Map Entropy (MME), Mean Plane Variance(MPV) [1] [2]
  • Mutually Orthogonal Metric (MOM) [3] -- has strong correlation with RPE

Citation

If you use this toolkit or MOM-metric results, please, cite our work:

@misc{kornilova2021benchmark,
    title={Be your own Benchmark: No-Reference Trajectory Metric on Registered Point Clouds},
    author={Anastasiia Kornilova and Gonzalo Ferrer},
    year={2021},
    eprint={2106.11351},
    archivePrefix={arXiv},
    primaryClass={cs.RO}
}

Credits

This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.

Links

[1] Droeschel, David, Jörg Stückler, and Sven Behnke. "Local multi-resolution representation for 6D motion estimation and mapping with a continuously rotating 3D laser scanner." 2014 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2014.
[2] Razlaw, Jan, et al. "Evaluation of registration methods for sparse 3D laser scans." 2015 European Conference on Mobile Robots (ECMR). IEEE, 2015.
[3] Kornilova, Anastasiia, and Gonzalo Ferrer. "Be your own Benchmark: No-Reference Trajectory Metric on Registered Point Clouds." arXiv preprint arXiv:2106.11351 (2021).
Comments
  • Cross-Platform Wheels

    Cross-Platform Wheels

    Linux and Windows32-64 wheels.

    MacOS is running into from map_metrics import map_metrics E ImportError: dlopen(/Users/runner/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/map_metrics/map_metrics.cpython-36m-darwin.so, 2): Library not loaded: @rpath/libboost_graph-mt-x64.dylib E Referenced from: /Users/runner/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/map_metrics/map_metrics.cpython-36m-darwin.so E Reason: image not found

    opened by achains 1
  • Speed up Github Actions workflow

    Speed up Github Actions workflow

    It takes an hour to build Open3D

    Possible solutions:

    • [ ] Add -j flag
    • [ ] Specify Open3D build options
    • [ ] Store Open3D files in Github Actions Cache
    enhancement 
    opened by achains 1
  • Python version interface improvements

    Python version interface improvements

    Tasks:

    • [x] Break down large functions
    • [x] Break functions into modules
    • [x] Build pip-package, configure CI
    • [x] Add more functions, e.g. I/O processing
    opened by achains 0
  • pip-package. MPV. MME

    pip-package. MPV. MME

    Overview

    Python package wheels with implemented basic metrics (MPV, MME) on C++

    Platform tags

    • manylinux2010_x86_64
    • macosx_10_14_x86_64
    • win32 / win_amd64

    Python version

    • Python >=3.6
    opened by achains 0
  • Cosmetic fixes

    Cosmetic fixes

    What was done:

    • Removed redundant comments
    • Implemented general interface for methods
    • Added missing headers
    • Unsigned long -> int
    • All functions have CamelCase now
    opened by achains 0
  • CMake project structure. Baseline MME and MPV

    CMake project structure. Baseline MME and MPV

    Progress:

    • Configured CMake files
    • Baseline of mme and mpv algorithms

    Notes (Tasks for next PR)

    • Algorithms need to be tested
    • CI should be configured
    opened by achains 0
  • A drawback of mom when dealing drafting walls in indoor envs.

    A drawback of mom when dealing drafting walls in indoor envs.

    • Map Metrics 0.0.1:
    • Python version 3.8:
    • Operating System win10: 1

    Description

    It tired the metirc on a small scale data, likely a room with differential chassis and a RS-16. When the point cloud map consists of multiple wall (actually one) caused by drifting, the mom metric may have a smaller value than a normal map. I think it is caused by the orthogonal walls (both the real one and drifted one) and floor.

    Specifically, Fig.1 is generated by LOAM (without drift), and Fig. 2 is gererated by Lio-sam (with draft).

    Fig.1

    Fig.2

    I also considered to downsample the two maps into the same scale with voxel downsampling. The results are listed below.

    1640311688(1)

    I think this kind of problem is very familiar for indoor, I have seen many drafting wall during daily usage of lidar based slam, without semantic labels, it is very hard for distinguishing them automatically.

    What I Did

    I will try the 0.0.2 version later.

    Paste the command(s) you ran and the output.
    If there was a crash, please include the traceback here.
    
    opened by hahakid 2
  • Add C++ executable with console interface

    Add C++ executable with console interface

    It will be useful for quick testing and performance measurments if library is able to run from command-line interface.

    For example, ./map-metrics --pc="data/kitti_00" --tj="data/Tj_0" --metric="mme" [--"some common hyperparameters"]

    It's build should be disabled by default

    enhancement 
    opened by achains 0
  • Support 10.9 MacOS

    Support 10.9 MacOS

    • Map Metrics version: 0.0.3
    • Python version: >=3.6
    • Operating System: MacOS

    Description

    Pip-package supports MacOS >=10.14 due to C++17 features. We can downgrade to C++14, but a bit later :)

    enhancement 
    opened by achains 0
  • editoring error

    editoring error

    A \delta seems missing in Eq.6 of the paper. Have you tested on all the sequences and other dataset. Recommend to ref LIBRE: The Multiple 3D LiDAR Dataset. VLP64 may have a low precise for long distance measurement. clipboard

    opened by hahakid 1
Owner
Mobile Robotics Lab. at Skoltech
Mobile Robotics Lab. at Skoltech
SE3 Pose Interp - Interpolate camera pose or trajectory in SE3, pose interpolation, trajectory interpolation

SE3 Pose Interpolation Pose estimated from SLAM system are always discrete, and

Ran Cheng 4 Dec 15, 2022
pip install python-office

?? python for office ?? http://www.python4office.cn/ ?? ?? English Documentation ?? 简介 Python-office 是一个 Python 自动化办公第三方库,能解决大部分自动化办公的问题。而且每个功能只需一行代码,

程序员晚枫 272 Dec 29, 2022
Revisiting, benchmarking, and refining Heterogeneous Graph Neural Networks.

Heterogeneous Graph Benchmark Revisiting, benchmarking, and refining Heterogeneous Graph Neural Networks. Roadmap We organize our repo by task, and on

THUDM 176 Dec 17, 2022
FedScale: Benchmarking Model and System Performance of Federated Learning

FedScale: Benchmarking Model and System Performance of Federated Learning (Paper) This repository contains scripts and instructions of building FedSca

null 268 Jan 1, 2023
PyTorch code for EMNLP 2021 paper: Don't be Contradicted with Anything! CI-ToD: Towards Benchmarking Consistency for Task-oriented Dialogue System

Don’t be Contradicted with Anything!CI-ToD: Towards Benchmarking Consistency for Task-oriented Dialogue System This repository contains the PyTorch im

Libo Qin 25 Sep 6, 2022
RobustART: Benchmarking Robustness on Architecture Design and Training Techniques

The first comprehensive Robustness investigation benchmark on large-scale dataset ImageNet regarding ARchitecture design and Training techniques towards diverse noises.

null 132 Dec 23, 2022
PyTorch code for EMNLP 2021 paper: Don't be Contradicted with Anything! CI-ToD: Towards Benchmarking Consistency for Task-oriented Dialogue System

PyTorch code for EMNLP 2021 paper: Don't be Contradicted with Anything! CI-ToD: Towards Benchmarking Consistency for Task-oriented Dialogue System

Libo Qin 12 Sep 26, 2021
Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking

Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking We revisit and address issues with Oxford 5k and Paris 6k image retrieval benchm

Filip Radenovic 188 Dec 17, 2022
Molecular Sets (MOSES): A benchmarking platform for molecular generation models

Molecular Sets (MOSES): A benchmarking platform for molecular generation models Deep generative models are rapidly becoming popular for the discovery

Neelesh C A 3 Oct 14, 2022
Official codebase for "B-Pref: Benchmarking Preference-BasedReinforcement Learning" contains scripts to reproduce experiments.

B-Pref Official codebase for B-Pref: Benchmarking Preference-BasedReinforcement Learning contains scripts to reproduce experiments. Install conda env

null 48 Dec 20, 2022
Molecular Sets (MOSES): A Benchmarking Platform for Molecular Generation Models

Molecular Sets (MOSES): A benchmarking platform for molecular generation models Deep generative models are rapidly becoming popular for the discovery

MOSES 656 Dec 29, 2022
ColossalAI-Benchmark - Performance benchmarking with ColossalAI

Benchmark for Tuning Accuracy and Efficiency Overview The benchmark includes our

HPC-AI Tech 31 Oct 7, 2022
Code for the paper "Benchmarking and Analyzing Point Cloud Classification under Corruptions"

ModelNet-C Code for the paper "Benchmarking and Analyzing Point Cloud Classification under Corruptions". For the latest updates, see: sites.google.com

Jiawei Ren 45 Dec 28, 2022
Evaluation and Benchmarking of Speech Super-resolution Methods

Speech Super-resolution Evaluation and Benchmarking What this repo do: A toolbox for the evaluation of speech super-resolution algorithms. Unify the e

Haohe Liu (刘濠赫) 84 Dec 20, 2022
Repo for "Benchmarking Robustness of 3D Point Cloud Recognition against Common Corruptions" https://arxiv.org/abs/2201.12296

Benchmarking Robustness of 3D Point Cloud Recognition against Common Corruptions This repo contains the dataset and code for the paper Benchmarking Ro

Jiachen Sun 168 Dec 29, 2022
Create animations for the optimization trajectory of neural nets

Animating the Optimization Trajectory of Neural Nets loss-landscape-anim lets you create animated optimization path in a 2D slice of the loss landscap

Logan Yang 81 Dec 25, 2022
Code + pre-trained models for the paper Keeping Your Eye on the Ball Trajectory Attention in Video Transformers

Motionformer This is an official pytorch implementation of paper Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers. In this rep

Facebook Research 192 Dec 23, 2022
Learning trajectory representations using self-supervision and programmatic supervision.

Trajectory Embedding for Behavior Analysis (TREBA) Implementation from the paper: Jennifer J. Sun, Ann Kennedy, Eric Zhan, David J. Anderson, Yisong Y

null 58 Jan 6, 2023
A selection of State Of The Art research papers (and code) on human locomotion (pose + trajectory) prediction (forecasting)

A selection of State Of The Art research papers (and code) on human trajectory prediction (forecasting). Papers marked with [W] are workshop papers.

Karttikeya Manglam 40 Nov 18, 2022