Learning to Track with Object Permanence
A video-based MOT approach capable of tracking through full occlusions: 
Learning to Track with Object Permanence,
Pavel Tokmakov, Jie Li, Wolfram Burgard, Adrien Gaidon,
arXiv technical report (arXiv 2103.14258)
@inproceedings{tokmakov2021learning,
title={Learning to Track with Object Permanence},
author={Tokmakov, Pavel and Li, Jie and Burgard, Wolfram and Gaidon, Adrien},
booktitle={ICCV},
year={2021}
}
Abstract
Tracking by detection, the dominant approach for online multi-object tracking, alternates between localization and association steps. As a result, it strongly depends on the quality of instantaneous observations, often failing when objects are not fully visible. In contrast, tracking in humans is underlined by the notion of object permanence: once an object is recognized, we are aware of its physical existence and can approximately localize it even under full occlusions. In this work, we introduce an end-to-end trainable approach for joint object detection and tracking that is capable of such reasoning. We build on top of the recent CenterTrack architecture, which takes pairs of frames as input, and extend it to videos of arbitrary length. To this end, we augment the model with a spatio-temporal, recurrent memory module, allowing it to reason about object locations and identities in the current frame using all the previous history. It is, however, not obvious how to train such an approach. We study this question on a new, large-scale, synthetic dataset for multi-object tracking, which provides ground truth annotations for invisible objects, and propose several approaches for supervising tracking behind occlusions. Our model, trained jointly on synthetic and real data, outperforms the state of the art on KITTI and MOT17 datasets thanks to its robustness to occlusions.
Installation
Please refer to INSTALL.md for installation instructions.
Benchmark Evaluation and Training
After installation, follow the instructions in DATA.md to setup the datasets. Then check GETTING_STARTED.md to reproduce the results in the paper. We provide scripts for all the experiments in the experiments folder.
License
PermaTrack is developed upon CenterTrack. Both codebases are released under MIT License themselves. Some code of CenterTrack are from third-parties with different licenses, please check the CenterTrack repo for details. In addition, this repo uses py-motmetrics for MOT evaluation, nuscenes-devkit for nuScenes evaluation and preprocessing, and TAO codebase for computing Track AP. ConvGRU implementation is adopted from this repo. See NOTICE for detail. Please note the licenses of each dataset. Most of the datasets we used in this project are under non-commercial licenses.
305 Dec 16, 2022
23 Nov 5, 2022
443 Dec 6, 2022
53 Dec 28, 2022
26 Nov 4, 2022
49 Oct 24, 2022
23 Jan 25, 2022
478 Jan 1, 2023
78 Dec 29, 2022
82 Dec 29, 2022
121 Dec 30, 2022
51 Dec 17, 2022
99 Dec 27, 2022
174 Dec 22, 2022
2 Oct 20, 2021
29 Nov 13, 2022
7 Jul 15, 2022
249 Jan 3, 2023
6 Dec 21, 2022