a project for 3D multi-object tracking

Last update: Jan 4, 2023

Related tags

Deep Learning 3D-Multi-Object-Tracker

Overview

3D Multi-Object Tracker

This project is developed for tracking multiple objects in 3D scene. The visualization code is from here.

Features

Fast: currently, the codes can achieve 700 FPS using only CPU (not include detection and data op), can perform tracking on all kitti val sequence in several seconds.
Support both online and global implementation. The overall framework of design is shown below:

Kitti Results

Results on the Kitti tracking val seq [1,6,8,10,12,13,14,15,16,18,19] using second-iou and point-rcnn detections. We followed the HOTA metric, and tuned the parameters in this code by firstly considering the HOTA performance.

Detector	HOTA	DetA	AssA	DetRe	DetPr	AssRe	AssPr	LocA	MOTA
second-iou	78.787	74.482	83.611	80.665	84.72	89.022	88.575	88.63	85.129
point-rcnn	78.91	75.814	82.406	83.489	82.185	87.209	87.586	87.308	88.412

Prepare data

You can download the Kitti tracking pose data from here, and you can find the point-rcnn and second-iou detections from here.

To run this code, you should organize Kitti tracking dataset as below:

# Kitti Tracking Dataset       
└── kitti_tracking
       ├── testing 
       |      ├──calib
       |      |    ├──0000.txt
       |      |    ├──....txt
       |      |    └──0028.txt
       |      ├──image_02
       |      |    ├──0000
       |      |    ├──....
       |      |    └──0028
       |      ├──pose
       |      |    ├──0000
       |      |    |    └──pose.txt
       |      |    ├──....
       |      |    └──0028
       |      |         └──pose.txt
       |      ├──label_02
       |      |    ├──0000.txt
       |      |    ├──....txt
       |      |    └──0028.txt
       |      └──velodyne
       |           ├──0000
       |           ├──....
       |           └──0028      
       └── training # the structure is same as testing set
              ├──calib
              ├──image_02
              ├──pose
              ├──label_02
              └──velodyne

Detections

└── point-rcnn
       ├── training
       |      ├──0000
       |      |    ├──000001.txt
       |      |    ├──....txt
       |      |    └──000153.txt
       |      ├──...
       |      └──0020
       └──testing

Requirements

python3
numpy
opencv
yaml

Quick start

Please modify the dataset path and detections path in the yaml file to your own path.
Then run python3 kitti_3DMOT.py config/point_rcnn_mot.yaml
The results are automatically saved to evaluation\results\sha_key\data, and evaluated by HOTA metrics.

Notes

The evaluation codes are copied from Kitti.

Comments

Why the source code is different from the published paper?
In your paper, three costs (appearance cost, motion cost and geometry cost) are used. But in the source code, it seems that only position differences are used. dis = (all_detections[...,0:3]-all_predictions[...,0:3])**2 Does it mean that other costs are actullay not so important for tracking?

The prediction confidence is not updated as descripted in the paper (Equition 19), why？

So I wonder whether I can get the same performance decribed in README with the code. If not, can you make some time to updated the code？

Thank you for your good method and looking forword to your reply.
opened by fjc523 5
The pose file is obtained from the oxts files provided by KITTI tracking dataset using the official matlab tools.
The pose file is obtained from the oxts files provided by KITTI tracking dataset using the official matlab tools.

Originally posted by @hailanyi in https://github.com/hailanyi/3D-Multi-Object-Tracker/issues/11#issuecomment-1279639990

Thank you for your great work！I have a question about pose.txt you provided:

Using official matlab tools directly only seems to get the pose in IMU/GPS coordinate system, so did you convert it to lidar coordinate system? Because I found that your code seems to use only the external parameter between lidar and camera, i.e. velo_to_cam, but not use external parameter between lidar and imu.

So is the pose.txt you provided in the lidar coordinate system or in the IMU/GPS coordinate system?
opened by Sonne-Zhu 3
Gain no way to get testing label

Hi, I can't find any link to download the testing label, at least not on Kitti's official website, Can you please provide me the link to testing label ? Thx a lot =。=

opened by HaohaoNJU 2
The detection score you provide is not normalized?

I find that the detection score you use is not normalized (not in the range of [0, 1]). Which layer or where did you extract those scores from? Are those score the logits before softmax?

Car -1 -1 -8.0728 299.5648 166.8018 457.8472 299.3745 2.0278 1.8372 4.4899 -4.5203 1.9349 13.4348 -8.3923 6.8229 Cyclist -1 -1 -8.3277 743.3280 157.8975 945.0159 374.0000 1.7928 0.7382 1.7908 1.6935 1.6931 5.7445 -8.0567 6.3583 Pedestrian -1 -1 -8.9000 1086.0386 169.0511 1223.0736 325.3153 1.7003 0.7000 1.0835 6.3164 1.6586 8.4887 -8.2776 3.9818 Car -1 -1 -2.0002 0.0000 198.3879 22.4742 238.5690 1.4643 1.6507 3.9074 -26.4072 2.5741 29.8174 -2.7207 -0.4804 Car -1 -1 -7.6264 284.9278 190.2203 347.9479 230.2715 1.5499 1.6856 4.1383 -12.9598 2.3717 31.9821 -8.0090 -0.7022 Car -1 -1 -4.6372 136.7996 189.1622 176.2292 219.5773 1.5594 1.6092 4.0767 -26.0653 2.5465 41.5085 -5.1953 -1.3563 Car -1 -1 -2.5441 0.0000 195.0718 83.9934 227.2724 1.4837 1.6807 3.8615 -28.5618 2.6095 35.5869 -3.2169 -2.0098 Car -1 -1 -4.4626 347.0251 177.7920 392.8701 208.4988 1.6766 1.6258 3.8540 -13.9505 1.9773 42.0110 -4.7817 -2.1874

opened by a43992899 2

trackeval.utils.TrackEvalException: no seqmap found: evaluate_tracking.seqmap.val

I have prepared the data and ran kitti_3DMOT.py according to readme.md, the following error has thrown. Did I missing something?

Traceback (most recent call last):
  File "/home/stardust/yrb/3D-Multi-Object-Tracker/kitti_3DMOT.py", line 167, in <module>
    tracking_val_seq(args)
  File "/home/stardust/yrb/3D-Multi-Object-Tracker/kitti_3DMOT.py", line 160, in tracking_val_seq
    eval_kitti()
  File "/home/stardust/yrb/3D-Multi-Object-Tracker/evaluation_HOTA/scripts/run_kitti.py", line 80, in eval_kitti
    dataset_list = [trackeval.datasets.Kitti2DBox(dataset_config)]
  File "/home/stardust/yrb/3D-Multi-Object-Tracker/evaluation_HOTA/trackeval/datasets/kitti_2d_box.py", line 71, in __init__
    raise TrackEvalException('no seqmap found: ' + os.path.basename(seqmap_file))
trackeval.utils.TrackEvalException: no seqmap found: evaluate_tracking.seqmap.val

On line 70, the missing seqmap_file was pointing to /home/stardust/yrb/3D-Multi-Object-Tracker/evaluation_HOTA/../evaluation/data/tracking/evaluate_tracking.seqmap.val

opened by callzhang 2

Is this method online MOT?

Hi, thanks for your interesting work! In your code, before saving results, the post_processing seems to filter out all trajectories with a score lower than the config.avg_score. In my understanding, this process makes your method unable to run online. Looking forward to your reply. Thanks.

opened by zouhaoa 2
License?

Hello @hailanyi

Amazing work with this project! (and the visualizer + Cas-A)

I noticed you didn't provide any license. I'd like to feature some of your work in a course I'm building about 3D Computer Vision. Is that cool if I do so?

Thank you! Jeremy

opened by Jeremy26 1
Some questions about the evaluation of experiment results in your paper, especially related to AB3DMOT
Thanks for your great work, but I still have a few questions about the experiment part of the paper.

in Table II, the results are obtained on kitti training data, how did you get the AB3DMOT results, did you evaluate them yourself by running the open source code of AB3DMOT?

If the answer to question 1 is yes, do you still converted 3D tracking results onto a 2D image plane to get the evaluation result?

For every tracked object, its BBs IoU with ground truth is required to be above a threshold in order to be considered as a successful match. What was the IoU threshold set in the experiments in Table II, was it 0.5?

Is the MOTA result in Table II the average of all 21 sequences on the traing dataset, or just the average of some of theses 21 equences .

If the answer to question 2 is yes, have you evaluated the 3d MOTA using different 3d IOU thresholds (0.25 0.5 0.7) on the traing sequences as in the AB3DMOT paper and compared with the AB3DMOT results in its paper?

I am looking forward to your reply, thank you very much!
opened by Sonne-Zhu 1
How to visualize these result files？

Thank for you great work. evaluation/results/sha_key/data **.txt , **.txt……How to visualize these result files？Same effect as this https://github.com/hailanyi/3D-Detection-Tracking-Viewer？Can you help me more specifically?Thank you

opened by hz-ants 1
a request

I'm so happy that author write the A guide to submit results to KITTI test. This is the 3d method. Now I want to get the method of 2d results submit to the kitti . Can you help me?

opened by dodeokwangyooo 1
How to deal with the results output through the "OpenPCDet" framework？

{'name': array(['Car', 'Car', 'Cyclist', 'Car', 'Car', 'Car', 'Cyclist', 'Pedestrian', 'Pedestrian', 'Cyclist'], dtype='<U10'), 'truncated': array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]), 'occluded': array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]), 'alpha': array([-4.515572 , -4.082801 , -8.211994 , -4.2358727, -6.37723 , -4.212622 , -7.653681 , -7.828194 , -3.5104022, -7.7533026], dtype=float32), 'bbox': array([[ 390.46756, 182.59967, 420.16034, 202.18625], [ 93.72412, 207.40984, 192.874 , 245.46086], [ 676.26434, 162.27713, 691.895 , 190.43402], [ 207.97661, 185.72922, 272.2654 , 217.085 ], [ 771.51385, 172.55157, 896.3316 , 212.16136], [ 217.41855, 188.42061, 271.71057, 213.47649], [ 338.7873 , 181.46661, 348.29086, 203.7352 ], [ 953.0622 , 152.42534, 984.1048 , 240.13443], [1197.7362 , 145.42426, 1241. , 231.93535], [ 214.69026, 177.82062, 222.83852, 203.2955 ]], dtype=float32), 'dimensions': array([[3.8072727 , 1.4813445 , 1.6003532 ], [3.52273 , 1.3748183 , 1.8758411 ], [1.6750405 , 1.752761 , 0.42281026], [3.6912992 , 1.555141 , 1.5810864 ], [4.483073 , 1.4696931 , 1.6517544 ], [3.673051 , 1.4601415 , 1.5243837 ], [1.5081377 , 1.6859876 , 0.39957428], [0.8074811 , 1.77992 , 0.5690844 ], [0.60090435, 1.7681558 , 0.5539663 ], [1.5440738 , 1.6429834 , 0.31740588]], dtype=float32), 'location': array([[-16.576267 , 2.297428 , 58.48187 ], [-20.094868 , 2.954785 , 31.181479 ], [ 4.6577272, 1.0940528, 45.77933 ], [-20.028645 , 2.286567 , 39.15602 ], [ 8.54361 , 1.4581716, 28.02376 ], [-23.375515 , 2.4971836, 46.22146 ], [-20.705027 , 2.3634393, 56.004333 ], [ 7.4708343, 1.3650522, 15.131747 ], [ 12.740014 , 1.2072334, 15.104981 ], [-25.844282 , 1.9757288, 47.61044 ]], dtype=float32), 'rotation_y': array([-4.790928 , -4.6516724, -8.111552 , -4.706223 , -6.0847206, -4.678815 , -8.006577 , -7.3778024, -2.8195329, -8.248504 ], dtype=float32), 'score': array([0.62366086, 0.5092147 , 0.45878437, 0.4128577 , 0.40351897, 0.3037016 , 0.15691371, 0.12470149, 0.10985158, 0.10319255], dtype=float32), 'boxes_lidar': array([[ 58.771694 , 16.60492 , -0.842473 , 3.8072727 , 1.6003532 , 1.4813445 , 3.2201314 ], [ 31.478838 , 20.126875 , -1.8011765 , 3.52273 , 1.8758411 , 1.3748183 , 3.0808759 ], [ 46.06226 , -4.6421843 , 0.13941461, 1.6750405 , 0.42281026, 1.752761 , 6.540756 ], [ 39.445976 , 20.054586 , -0.960226 , 3.6912992 , 1.5810864 , 1.555141 , 3.1354265 ], [ 28.312376 , -8.526211 , -0.5928153 , 4.483073 , 1.6517544 , 1.4696931 , 4.513924 ], [ 46.512444 , 23.404373 , -1.1091216 , 3.673051 , 1.5243837 , 1.4601415 , 3.1080184 ], [ 56.294018 , 20.733837 , -0.788435 , 1.5081377 , 0.39957428, 1.6859876 , 6.4357805 ], [ 15.419843 , -7.4560823 , -0.46799874, 0.8074811 , 0.5690844 , 1.77992 , 5.807006 ], [ 15.392667 , -12.72664 , -0.37202018, 0.60090435, 0.5539663 , 1.7681558 , 1.2487365 ], [ 47.89532 , 25.867668 , -0.45570797, 1.5440738 , 0.31740588, 1.6429834 , 6.6777077 ]], dtype=float32), 'frame_id': '000001'}

The output is shown above,I hope to get your help,thank you.

opened by stidk 1

Owner

GitHub

Object Detection and Multi-Object Tracking

1.6k Jan 4, 2023

SiamMOT is a region-based Siamese Multi-Object Tracking network that detects and associates object instances simultaneously.

SiamMOT: Siamese Multi-Object Tracking

432 Dec 17, 2022

TSDF++: A Multi-Object Formulation for Dynamic Object Tracking and Reconstruction

TSDF++: A Multi-Object Formulation for Dynamic Object Tracking and Reconstruction TSDF++ is a novel multi-object TSDF formulation that can encode mult

130 Dec 29, 2022

Multi-Object Tracking in Satellite Videos with Graph-Based Multi-Task Modeling

TGraM Multi-Object Tracking in Satellite Videos with Graph-Based Multi-Task Modeling, Qibin He, Xian Sun, Zhiyuan Yan, Beibei Li, Kun Fu Abstract Rece

6 Nov 25, 2022

a project for 3D multi-object tracking

155 Jan 4, 2023

Python package for multiple object tracking research with focus on laboratory animals tracking.

motutils is a Python package for multiple object tracking research with focus on laboratory animals tracking. Features loads: MOTChallenge CSV, sleap

2 Sep 5, 2022

Object tracking and object detection is applied to track golf puts in real time and display stats/games.

Putting_Game Object tracking and object detection is applied to track golf puts in real time and display stats/games. Works best with the Perfect Prac

1 Dec 29, 2021

TrackFormer: Multi-Object Tracking with Transformers

TrackFormer: Multi-Object Tracking with Transformers This repository provides the official implementation of the TrackFormer: Multi-Object Tracking wi

321 Dec 29, 2022

FairMOT - A simple baseline for one-shot multi-object tracking

3.6k Jan 8, 2023

Official code for "EagerMOT: 3D Multi-Object Tracking via Sensor Fusion" [ICRA 2021]

EagerMOT: 3D Multi-Object Tracking via Sensor Fusion Read our ICRA 2021 paper here. Check out the 3 minute video for the quick intro or the full prese

276 Dec 30, 2022

ByteTrack(Multi-Object Tracking by Associating Every Detection Box)のPythonでのONNX推論サンプル

ByteTrack-ONNX-Sample ByteTrack(Multi-Object Tracking by Associating Every Detection Box)のPythonでのONNX推論サンプルです。 ONNXに変換したモデルも同梱しています。変換自体を試したい方はByteT

16 Oct 26, 2022

Implementation of MeMOT - Multi-Object Tracking with Memory - in Pytorch

MeMOT - Pytorch (wip) Implementation of MeMOT - Multi-Object Tracking with Memory - in Pytorch. This paper is just one in a line of work, but importan

15 May 9, 2022

The official repo for OC-SORT: Observation-Centric SORT on video Multi-Object Tracking. OC-SORT is simple, online and robust to occlusion/non-linear motion.

OC-SORT Observation-Centric SORT (OC-SORT) is a pure motion-model-based multi-object tracker. It aims to improve tracking robustness in crowded scenes

325 Jan 5, 2023

Project for music generation system based on object tracking and CGAN

Project for music generation system based on object tracking and CGAN The project was inspired by MIDINet: A Convolutional Generative Adversarial Netw

1 Nov 21, 2021

Joint detection and tracking model named DEFT, or ``Detection Embeddings for Tracking.

DEFT: Detection Embeddings for Tracking DEFT: Detection Embeddings for Tracking, Mohamed Chaabane, Peter Zhang, J. Ross Beveridge, Stephen O'Hara

253 Dec 18, 2022

Tracking code for the winner of track 1 in the MMP-Tracking Challenge at ICCV 2021 Workshop.

Tracking Code for the winner of track1 in MMP-Trakcing challenge This repository contains our tracking code for the Multi-camera Multiple People Track

29 Nov 13, 2022

Tracking Pipeline helps you to solve the tracking problem more easily

Tracking_Pipeline Tracking_Pipeline helps you to solve the tracking problem more easily I integrate detection algorithms like: Yolov5, Yolov4, YoloX,

32 Dec 21, 2022

Quadruped-command-tracking-controller - Quadruped command tracking controller (flat terrain)

Quadruped command tracking controller (flat terrain) Prepare Install RAISIM link

4 Oct 20, 2022

Tools to create pixel-wise object masks, bounding box labels (2D and 3D) and 3D object model (PLY triangle mesh) for object sequences filmed with an RGB-D camera.

Tools to create pixel-wise object masks, bounding box labels (2D and 3D) and 3D object model (PLY triangle mesh) for object sequences filmed with an RGB-D camera. This project prepares training and testing data for various deep learning projects such as 6D object pose estimation projects singleshotpose, as well as object detection and instance segmentation projects.

305 Dec 16, 2022