Official PyTorch implementation of Joint Object Detection and Multi-Object Tracking with Graph Neural Networks

Related tags

Deep Learning GSDT
Overview

GSDT

Joint Object Detection and Multi-Object Tracking with Graph Neural Networks

This is the official PyTorch implementation of our paper: "Joint Object Detection and Multi-Object Tracking with Graph Neural Networks". Our project website and video demos are here. If you find our work useful, we'd appreciate you citing our paper as follows:

@article{Wang2020_GSDT, 
author = {Wang, Yongxin and Kitani, Kris and Weng, Xinshuo}, 
journal = {arXiv:2006.13164}, 
title = {{Joint Object Detection and Multi-Object Tracking with Graph Neural Networks}}, 
year = {2020} 
}

Introduction

Object detection and data association are critical components in multi-object tracking (MOT) systems. Despite the fact that the two components are dependent on each other, prior work often designs detection and data association modules separately which are trained with different objectives. As a result, we cannot back-propagate the gradients and optimize the entire MOT system, which leads to sub-optimal performance. To address this issue, recent work simultaneously optimizes detection and data association modules under a joint MOT framework, which has shown improved performance in both modules. In this work, we propose a new instance of joint MOT approach based on Graph Neural Networks (GNNs). The key idea is that GNNs can model relations between variable-sized objects in both the spatial and temporal domains, which is essential for learning discriminative features for detection and data association. Through extensive experiments on the MOT15/16/17/20 datasets, we demonstrate the effectiveness of our GNN-based joint MOT approach and show the state-of-the-art performance for both detection and MOT tasks.

Usage

Dependencies

We recommend using anaconda for managing dependency and environments. You may follow the commands below to setup your environment.

conda create -n dev python=3.6
conda activate dev
pip install -r requirements.txt

We use the PyTorch Geometric package for the implementation of our Graph Neural Network based architecture.

bash install_pyg.sh   # we used CUDA_version=cu101 

Build Deformable Convolutional Networks V2 (DCNv2)

cd ./src/lib/models/networks/DCNv2
bash make.sh

To automatically generate output tracking as videos, please install ffmpeg

conda install ffmpeg=4.2.2

Data preperation

We follow the same dataset setup as in JDE. Please refer to their DATA ZOO for data download and preperation.

To prepare 2DMOT15 and MOT20 data, you can directly download from the MOT Challenge website, and format each directory as follows:

MOT15
   |——————images
   |        └——————train
   |        └——————test
   └——————labels_with_ids
            └——————train(empty)
MOT20
   |——————images
   |        └——————train
   |        └——————test
   └——————labels_with_ids
            └——————train(empty)

Then change the seq_root and label_root in src/gen_labels_15.py and src/gen_labels_20.py accordingly, and run:

cd src
python gen_labels_15.py
python gen_labels_20.py

This will generate the desired label format of 2DMOT15 and MOT20. The seqinfo.ini files are required for 2DMOT15 and can be found here [Google], [Baidu],code:8o0w.

Inference

Download and save the pretrained weights for each dataset by following the links below:

Dataset Model
2DMOT15 model_mot15.pth
MOT17 model_mot17.pth
MOT20 model_mot20.pth

Run one of the following command to reproduce our paper's tracking performance on the MOT Challenge.

cd ./experiments
track_gnn_mot_AGNNConv_RoIAlign_mot15.sh 
track_gnn_mot_AGNNConv_RoIAlign_mot17.sh 
track_gnn_mot_AGNNConv_RoIAlign_mot20.sh 

To clarify, currently we directly used the MOT17 results as MOT16 results for submission. That is, our MOT16 and MOT17 results and models are identical.

Training

We are currently in the process of cleaning the training code. We'll release as soon as we can. Stay tuned!

Performance on MOT Challenge

You can refer to MOTChallenge website for performance of our method. For your convenience, we summarize results below:

Dataset MOTA IDF1 MT ML IDS
2DMOT15 60.7 64.6 47.0% 10.5% 477
MOT16 66.7 69.2 38.6% 19.0% 959
MOT17 66.2 68.7 40.8% 18.3% 3318
MOT20 67.1 67.5 53.1% 13.2% 3133

Acknowledgement

A large part of the code is borrowed from FairMOT. We appreciate their great work!

Comments
  • Remove pycache from repo

    Remove pycache from repo

    Hi, great work!

    Pycache stores temporary files created when execution of the python code is triggered. It is not meant to be distributed, so is better to remove from the repo.

    Andrea Garritano

    opened by andrea-garritano 0
  • RuntimeError: CUDA driver error: a PTX JIT compilation failed

    RuntimeError: CUDA driver error: a PTX JIT compilation failed

    RuntimeError: CUDA driver error: a PTX JIT compilation failed

    (S) [email protected]:/home1/liuqigui/project/GSDT-main/experiments$ sh track_gnn_mot_AGNNConv_RoIAlign_mot17.sh Fix size testing. training chunk_sizes: [12, 12] The output will be saved to /home1/liuqigui/project/GSDT-main/src/lib/../../exp/gnn_mot/default heads {'hm': 1, 'wh': 2, 'id': 512, 'reg': 2} 2022-11-02 16:40:12 [INFO]: start seq: MOT17-01-SDP Creating model... loaded model_mot17.pth, epoch 10 Drop parameter loss.0.s_det.If you see this, your model does not fully load the pre-trained weight. Please make sure you have c orrectly specified --arch xxx or set the correct --num_classes for your own dataset.Drop parameter loss.0.s_id.If you see this, your model does not fully load the pre-trained weight. Please make sure you have co rrectly specified --arch xxx or set the correct --num_classes for your own dataset.Drop parameter loss.0.classifier.weight.If you see this, your model does not fully load the pre-trained weight. Please make sur e you have correctly specified --arch xxx or set the correct --num_classes for your own dataset.Drop parameter loss.0.classifier.bias.If you see this, your model does not fully load the pre-trained weight. Please make sure you have correctly specified --arch xxx or set the correct --num_classes for your own dataset.Drop parameter loss.1.s_det.If you see this, your model does not fully load the pre-trained weight. Please make sure you have c orrectly specified --arch xxx or set the correct --num_classes for your own dataset.Drop parameter loss.1.s_id.If you see this, your model does not fully load the pre-trained weight. Please make sure you have co rrectly specified --arch xxx or set the correct --num_classes for your own dataset.Drop parameter loss.1.classifier.weight.If you see this, your model does not fully load the pre-trained weight. Please make sur e you have correctly specified --arch xxx or set the correct --num_classes for your own dataset.Drop parameter loss.1.classifier.bias.If you see this, your model does not fully load the pre-trained weight. Please make sure you have correctly specified --arch xxx or set the correct --num_classes for your own dataset.2022-11-02 16:40:20 [INFO]: Processing frame 0 (100000.00 fps) Traceback (most recent call last): File "track_gnn.py", line 384, in save_videos=opt.save_videos) File "track_gnn.py", line 219, in main save_dir=output_dir, show_image=show_image, frame_rate=frame_rate, conf_thres=conf) File "track_gnn.py", line 88, in eval_seq p_imgs=p_img.unsqueeze(0).cuda(), conf_thres=conf_thres) File "/home1/liuqigui/project/GSDT-main/src/lib/tracker/multitracker.py", line 266, in update output = self.model(im_blob, p_crops, p_crops_lengths, edge_index, p_imgs=p_imgs)[gnn_output_layer] File "/home/wguo/anaconda3/envs/S/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "/home1/liuqigui/project/GSDT-main/src/lib/models/networks/pose_dla_dcn.py", line 471, in forward gnn_out = gnn(gnn_feat, graph.edge_index) File "/home/wguo/anaconda3/envs/S/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "/home/wguo/anaconda3/envs/S/lib/python3.7/site-packages/torch_geometric/nn/conv/agnn_conv.py", line 70, in forward return self.propagate(edge_index, x=x, x_norm=x_norm, size=None) File "/home/wguo/anaconda3/envs/S/lib/python3.7/site-packages/torch_geometric/nn/conv/message_passing.py", line 237, in propa gate out = self.message(**msg_kwargs) File "/home/wguo/anaconda3/envs/S/lib/python3.7/site-packages/torch_geometric/nn/conv/agnn_conv.py", line 76, in message alpha = softmax(alpha, index, ptr, size_i) RuntimeError: CUDA driver error: a PTX JIT compilation failed

    opened by liuqigui 0
  • Train code release

    Train code release

    Hi and thanks for your great work!

    You have mentioned that the training code is being cleaned up to be released. I would like to ask when this code (even part of it or its latest version) could be published.

    Thanks again!

    opened by Livhk 1
  • Dear author, have you sorted out the training code of this project? Can you send it to Guan Mobai to read it? I really hope to get your reply. No matter what the result is, thank you very much.

    Dear author, have you sorted out the training code of this project? Can you send it to Guan Mobai to read it? I really hope to get your reply. No matter what the result is, thank you very much.

    Dear author, have you sorted out the training code of this project? Can you send it to Guan Mobai to read it? I really hope to get your reply. No matter what the result is, thank you very much.

    opened by LSGL-LLW 0
  • ImportError: libcudart.so.10.0: cannot open shared object file: No such file or directory

    ImportError: libcudart.so.10.0: cannot open shared object file: No such file or directory

    (gnn) [email protected]:~/GNN/GSDT/GSDT-main/experiments$ bash track_gnn_mot_AGNNConv_RoIAlign_mot15.sh /home/llw/GNN/GSDT/GSDT-main/model_mot15 Traceback (most recent call last): File "track_gnn.py", line 20, in from tracker.multitracker import GNNTracker File "/home/llw/GNN/GSDT/GSDT-main/src/lib/tracker/multitracker.py", line 8, in from models.model import create_model, load_model File "/home/llw/GNN/GSDT/GSDT-main/src/lib/models/model.py", line 9, in from .networks.pose_dla_dcn import get_pose_net_with_gnn as get_dla_dcn_gnn File "/home/llw/GNN/GSDT/GSDT-main/src/lib/models/networks/pose_dla_dcn.py", line 18, in from .DCNv2.dcn_v2 import DCN File "/home/llw/GNN/GSDT/GSDT-main/src/lib/models/networks/DCNv2/dcn_v2.py", line 13, in import _ext as _backend ImportError: libcudart.so.10.0: cannot open shared object file: No such file or directory

    I don't know where the problem is. Has the author or others encountered this problem

    opened by LSGL-LLW 2
  • The uploaded weights

    The uploaded weights

    Could you please tell me, the pretrained weights you uploaded (model_mot15.pth, model_mot17.pth, model_mot20.pth) are used for AGNNConv with 3 layers (as said in paper), or it is for AGNNConv with 1 layer? I found some mismatches in parameters and did not get the results on the paper when running val_mot15.

    opened by Leo63963 0
Owner
Richard Wang
Richard Wang
Joint detection and tracking model named DEFT, or ``Detection Embeddings for Tracking.

DEFT: Detection Embeddings for Tracking DEFT: Detection Embeddings for Tracking, Mohamed Chaabane, Peter Zhang, J. Ross Beveridge, Stephen O'Hara

Mohamed Chaabane 250 Nov 12, 2022
Multi-Object Tracking in Satellite Videos with Graph-Based Multi-Task Modeling

TGraM Multi-Object Tracking in Satellite Videos with Graph-Based Multi-Task Modeling, Qibin He, Xian Sun, Zhiyuan Yan, Beibei Li, Kun Fu Abstract Rece

Qibin He 5 Oct 19, 2022
This is the official implementation of 3D-CVF: Generating Joint Camera and LiDAR Features Using Cross-View Spatial Feature Fusion for 3D Object Detection, built on SECOND.

3D-CVF This is the official implementation of 3D-CVF: Generating Joint Camera and LiDAR Features Using Cross-View Spatial Feature Fusion for 3D Object

YecheolKim 98 Nov 14, 2022
Drone-based Joint Density Map Estimation, Localization and Tracking with Space-Time Multi-Scale Attention Network

DroneCrowd Paper Detection, Tracking, and Counting Meets Drones in Crowds: A Benchmark. Introduction This paper proposes a space-time multi-scale atte

VisDrone 98 Nov 16, 2022
Pytorch implementation of "Attention-Based Recurrent Neural Network Models for Joint Intent Detection and Slot Filling"

RNN-for-Joint-NLU Pytorch implementation of "Attention-Based Recurrent Neural Network Models for Joint Intent Detection and Slot Filling"

Kim SungDong 193 Nov 16, 2022
Object Detection and Multi-Object Tracking

Object Detection and Multi-Object Tracking

Bobby Chen 1.6k Nov 23, 2022
Official Implementation of "LUNAR: Unifying Local Outlier Detection Methods via Graph Neural Networks"

LUNAR Official Implementation of "LUNAR: Unifying Local Outlier Detection Methods via Graph Neural Networks" Adam Goodge, Bryan Hooi, Ng See Kiong and

Adam Goodge 22 Nov 22, 2022
Official implementation of the ICCV 2021 paper "Joint Inductive and Transductive Learning for Video Object Segmentation"

JOINT This is the official implementation of Joint Inductive and Transductive learning for Video Object Segmentation, to appear in ICCV 2021. @inproce

Yunyao 35 Oct 16, 2022
Implementation of MeMOT - Multi-Object Tracking with Memory - in Pytorch

MeMOT - Pytorch (wip) Implementation of MeMOT - Multi-Object Tracking with Memory - in Pytorch. This paper is just one in a line of work, but importan

Phil Wang 15 May 9, 2022
The official repo for OC-SORT: Observation-Centric SORT on video Multi-Object Tracking. OC-SORT is simple, online and robust to occlusion/non-linear motion.

OC-SORT Observation-Centric SORT (OC-SORT) is a pure motion-model-based multi-object tracker. It aims to improve tracking robustness in crowded scenes

Jinkun Cao 306 Nov 29, 2022
ByteTrack(Multi-Object Tracking by Associating Every Detection Box)のPythonでのONNX推論サンプル

ByteTrack-ONNX-Sample ByteTrack(Multi-Object Tracking by Associating Every Detection Box)のPythonでのONNX推論サンプルです。 ONNXに変換したモデルも同梱しています。 変換自体を試したい方はByteT

KazuhitoTakahashi 16 Oct 26, 2022
PyTorch implementation of HDN(Homography Decomposition Networks) for planar object tracking

Homography Decomposition Networks for Planar Object Tracking This project is the offical PyTorch implementation of HDN(Homography Decomposition Networ

CaptainHook 43 Nov 22, 2022
TSDF++: A Multi-Object Formulation for Dynamic Object Tracking and Reconstruction

TSDF++: A Multi-Object Formulation for Dynamic Object Tracking and Reconstruction TSDF++ is a novel multi-object TSDF formulation that can encode mult

ETHZ ASL 124 Oct 17, 2022
Object tracking and object detection is applied to track golf puts in real time and display stats/games.

Putting_Game Object tracking and object detection is applied to track golf puts in real time and display stats/games. Works best with the Perfect Prac

Max 1 Dec 29, 2021
An official implementation of "Exploiting a Joint Embedding Space for Generalized Zero-Shot Semantic Segmentation" (ICCV 2021) in PyTorch.

Exploiting a Joint Embedding Space for Generalized Zero-Shot Semantic Segmentation This is an official implementation of the paper "Exploiting a Joint

CV Lab @ Yonsei University 35 Oct 26, 2022
CoReNet is a technique for joint multi-object 3D reconstruction from a single RGB image.

CoReNet CoReNet is a technique for joint multi-object 3D reconstruction from a single RGB image. It produces coherent reconstructions, where all objec

Google Research 77 Nov 22, 2022
Official code for "EagerMOT: 3D Multi-Object Tracking via Sensor Fusion" [ICRA 2021]

EagerMOT: 3D Multi-Object Tracking via Sensor Fusion Read our ICRA 2021 paper here. Check out the 3 minute video for the quick intro or the full prese

Aleksandr Kim 264 Nov 21, 2022
Python package for multiple object tracking research with focus on laboratory animals tracking.

motutils is a Python package for multiple object tracking research with focus on laboratory animals tracking. Features loads: MOTChallenge CSV, sleap

Matěj Šmíd 2 Sep 5, 2022