Track to Detect and Segment: An Online Multi-Object Tracker (CVPR 2021)

Overview

Track to Detect and Segment: An Online Multi-Object Tracker (CVPR 2021)

Track to Detect and Segment: An Online Multi-Object Tracker
Jialian Wu, Jiale Cao, Liangchen Song, Yu Wang, Ming Yang, Junsong Yuan
In CVPR, 2021. [Paper] [Project Page] [Demo (YouTube)]

Many thanks to CenterTrack authors for their great framework!

Installation

Please refer to INSTALL.md for installation instructions.

Run Demo

We reuse the demo script from CenterTrack. Before run the demo, first download our trained models: CrowdHuman model (2D tracking), MOT model (2D tracking) or nuScenes model (3D tracking). Then, put the models in TraDeS_ROOT/models/ and cd TraDeS_ROOT/src/. The demo result will be saved as a video in TraDeS_ROOT/results/.

2D Tracking Demo

Demo for a video clip from MOT dataset: Run the demo (using the MOT model):

python demo.py tracking --dataset mot --load_model ../models/mot_half.pth --demo ../videos/mot_mini.mp4 --pre_hm --ltrb_amodal --pre_thresh 0.5 --track_thresh 0.4 --inference --clip_len 3 --trades --save_video --resize_video --input_h 544 --input_w 960

Demo for a video clip which we randomly selected from YouTube: Run the demo (using the CrowdHuman model):

python demo.py tracking --load_model ../models/crowdhuman.pth --num_class 1 --demo ../videos/street_2d.mp4 --pre_hm --ltrb_amodal --pre_thresh 0.5 --track_thresh 0.5 --inference --clip_len 2 --trades --save_video --resize_video --input_h 480 --input_w 864

Demo for your own video or image folder: Please specify the file path after --demo and run (using the CrowdHuman model):

python demo.py tracking --load_model ../models/crowdhuman.pth --num_class 1 --demo $path to your video or image folder$ --pre_hm --ltrb_amodal --pre_thresh 0.5 --track_thresh 0.5 --inference --clip_len 2 --trades --save_video --resize_video --input_h $your_input_h$ --input_w $your_input_w$

(Some Notes: (i) For 2D tracking, the models are only used for person tracking, since our method is only trained on CrowdHuman or MOT. You may train a model on COCO or your own dataset for multi-category 2D object tracking. (ii) --clip_len is set to 3 for MOT; otherwise, it should be 2. You may refer to our paper for this detail. (iii) The CrowdHuman model is more able to generalize to real world scenes than the MOT model. Note that both datasets are in non-commercial licenses. (iii) input_h and input_w shall be evenly divided by 32.)

3D Tracking Demo

Demo for a video clip from nuScenes dataset: Run the demo (using the nuScenes model):

python demo.py tracking,ddd --dataset nuscenes --load_model ../models/nuscenes.pth --demo ../videos/nuscenes_mini.mp4 --pre_hm --track_thresh 0.1 --inference --clip_len 2 --trades --save_video --resize_video --input_h 448 --input_w 800 --test_focal_length 633

(You will need to specify test_focal_length for monocular 3D tracking demo to convert the image coordinate system back to 3D. The value 633 is half of a typical focal length (~1266) in nuScenes dataset in input resolution 1600x900. The mini demo video is in an input resolution of 800x448, so we need to use a half focal length. You don't need to set the test_focal_length when testing on the original nuScenes data.)

You can also refer to CenterTrack for the usage of webcam demo (code is available in this repo, but we have not tested yet).

Benchmark Evaluation and Training

Please refer to Data.md for dataset preparation.

2D Object Tracking

MOT17 Val MOTA↑ IDF1↑ IDS↓
Our Baseline 64.8 59.5 1055
CenterTrack 66.1 64.2 528
TraDeS (ours) 68.2 71.7 285

Test on MOT17 validation set: Place the MOT model in $TraDeS_ROOT/models/ and run:

sh experiments/mot17_test.sh

Train on MOT17 halftrain set: Place the pretrained model in $TraDeS_ROOT/models/ and run:

sh experiments/mot17_train.sh

3D Object Tracking

nuScenes Val AMOTA↑ AMOTP↓ IDSA↓
Our Baseline 4.3 1.65 1792
CenterTrack 6.8 1.54 813
TraDeS (ours) 11.8 1.48 699

Test on nuScenes validation set: Place the nuScenes model in $TraDeS_ROOT/models/. You need to change the MOT and nuScenes dataset API versions due to their conflicts. The default installed versions are for MOT dataset. For experiments on nuScenes dataset, please run:

sh nuscenes_switch_version.sh

sh experiments/nuScenes_test.sh

To switch back to the API versions for MOT experiments, you can run:

sh mot_switch_version.sh

Train on nuScenes train set: Place the pretrained model in $TraDeS_ROOT/models/ and run:

sh experiments/nuScenes_train.sh

Train on Static Images

We follow CenterTrack which uses CrowdHuman to pretrain 2D object tracking model. Only the training set is used.

sh experiments/crowdhuman.sh

The trained model is available at CrowdHuman model.

Instance Segmentation Tracking

Code will be released later on after we clean it up. Our implementation is based on here.

Citation

If you find it useful in your research, please consider citing our paper as follows:

@inproceedings{Wu2021TraDeS,
title={Track to Detect and Segment: An Online Multi-Object Tracker},
author={Wu, Jialian and Cao, Jiale and Song, Liangchen and Wang, Yu and Yang, Ming and Yuan, Junsong},
booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2021}}
Comments
  • 2 heads at the same time

    2 heads at the same time

    Hello! Thank you for this great work! I faced with small problem. I want to use 2 heads at the same time (to predict segmentation mask and 3D bbox). Now I can run it separately - or 3d bbox, or segmentation head. I am ready to take your trained weights and turn on 2 heads at the same time. But I don't understand where in code you turn on segmentation head? Please help.

    best wishes, Ilya

    opened by Ilyabasharov 10
  • How to run demo.py in pycharm

    How to run demo.py in pycharm

    I am a python novice, sorry to disturb everyone, as the question shows, how to run the following program in pycharm, I can only run successfully in cmdpython demo.py tracking,ddd --load_model ../models/nuScenes_3Dtracking.pth --dataset nuscenes --pre_hm --track_thresh 0.1 --demo ../videos/nuscenes_mini.mp4 --test_focal_length 633

    opened by xiaoweidao 8
  • Feature aggregration

    Feature aggregration

    Hi

    Just one question about the features used here. It seems that, for the inference_prehm here, which stores the previous heatmaps that would be used for feature aggregation in the proposed MFW. The code used as follows:

    https://github.com/JialianW/TraDeS/blob/3eafd249ca0f18af8000d5798d4c552a0bd627ec/src/lib/detector.py#L351

    https://github.com/JialianW/TraDeS/blob/3eafd249ca0f18af8000d5798d4c552a0bd627ec/src/lib/detector.py#L354

    And it looks like that the same heatmaps from the previous frame (t-1) are used twice. And by debug, it has been proved.

    I personally think that, in TraDeS pipeline, you would like to use the features and corresponding heatmaps from (t-1) and (t-2), instead of repeated (t-1), for the aggregations of features. However, it looks that it is not.

    Just kindly ask, are there some hidden tricks here? Many thanks.

    opened by Leo63963 6
  • Question regarding

    Question regarding "class agnostic center heatmap" P(t-tao)_agn

    Hi there, thank you for sharing your great work! I have a question here, please do correct me if I got it wrong:

    We use a "class agnostic center heatmap"(of previous feat) P_agn times the previous feat, then passing it to the DCN along with the offset OD to get the "center attentive feature", which can be used to enhance the current feat. In your paper, it said P_agn was obtained from the heatmap output of CenterNet, but in the base_model section in your training code I found that the P_agn, aka 'pre_hm_i', was always generated directly from the ground truth.

    base_model.py: 1 trainer.py: 2 generic_dataset.py, getitem: 3 _get_pre_dets: 5

    I also read your code for test, the Detector seemed ok since it used the previous heatmap result. Have I misunderstood anything here? Many thanks!

    opened by NosremeC 6
  • TraDeS for Instance Segmentation

    TraDeS for Instance Segmentation

    Thank you for your awesome research work. I have a question related to segmentation task by this repo. I followed your instructions to run validation on YoutubeVIS 2019, but the results is in tracking format. Does this repo support segmentation format? Thank you.

    opened by phuocnguyen2008 5
  • demo parameters

    demo parameters

    I have tried the demo (with default parameters) on a very simple video which is tracking only 1 person, and found 2 following problems

    1. If it fails to detect the person at the current frame, it will assign a new track ID at the next frame
    2. Sometimes there are 2 overlapped detection boxes (with different track IDs) with very high IOU at the same place How can I fine-tune the default parameters to overcome these problems?
    opened by pqviet 5
  • Label

    Label

    Hi Could you please tell me the reason form in your code below: https://github.com/JialianW/TraDeS/blob/3eafd249ca0f18af8000d5798d4c552a0bd627ec/src/lib/dataset/generic_dataset.py#L679 https://github.com/JialianW/TraDeS/blob/3eafd249ca0f18af8000d5798d4c552a0bd627ec/src/lib/dataset/generic_dataset.py#L680 which are used for making the labels of cost volume. It seems that in the code: https://github.com/JialianW/TraDeS/blob/3eafd249ca0f18af8000d5798d4c552a0bd627ec/src/lib/dataset/generic_dataset.py#L677 https://github.com/JialianW/TraDeS/blob/3eafd249ca0f18af8000d5798d4c552a0bd627ec/src/lib/dataset/generic_dataset.py#L678 the labels for h-hm and w-hm as already made.

    opened by Leo63963 4
  • Question about the code

    Question about the code

    Hi Thanks for the work, really adimre that

    There is a question about the code, in the follow: https://github.com/JialianW/TraDeS/blob/3eafd249ca0f18af8000d5798d4c552a0bd627ec/src/lib/utils/tracker.py#L196

    Why you invalid the last embedding while measuring the similarity? I tried run the code without this, and the IDF1 drops (0.1), which means it is important.

    Many thanks.

    opened by Leo63963 4
  • Crashed building DCNv2

    Crashed building DCNv2

    When I compiled DCNv2 with gcc 7.5.0, it crashed because of some function undeclared errors. Does anybody meets similar errors?

    TraDeS-master/src/lib/model/networks/DCNv2/src/cpu/dcn_v2_cpu.cpp:224:9: error: ‘THFloatBlas_gemv’ was not declared in this scope
             THFloatBlas_gemv('t', k_, m_, 1.0f,
             ^~~~~~~~~~~~~~~~
    TraDeS-master/src/lib/model/networks/DCNv2/src/cpu/dcn_v2_cpu.cpp:224:9: note: suggested alternative: ‘THFloatBlas_gemm’
             THFloatBlas_gemv('t', k_, m_, 1.0f,
             ^~~~~~~~~~~~~~~~
             THFloatBlas_gemm
    error: command 'g++' failed with exit status 1
    
    opened by hxk11111 3
  • About the Head Networks

    About the Head Networks

    Thanks for your working.

    We try to train on mot dataset. We don't know where Head Network(such as 2D detection, 3D detection and Instace segmentation) plays, can you tell me where this part is in the code.

    Thanks a lot.

    opened by Jx-Tan 2
  • How to set the number of epoch when training my model?

    How to set the number of epoch when training my model?

    I have used sh experiments/mot17_test.sh to train my own model. And I can set the batch size. Because of the limitation of GPU performance, I changed it from 32 into 4. But I cannot see where to set the number of epoch. BTW, the notification said the program have to go through 140 epochs. I got the noti by running the command.

    opened by anthonyweidai 2
  • about baseline in paper

    about baseline in paper

    Hi, thanks for your works. When reading your paper, i have few questions about the baseline:

    1. Baseline is build based on CenterNet with an extra head predicting tracking offset map O_B. As mention in #36 , backbone network just take a single image (frame at timestamp t) as input instead of previous_frame + current_frame + previous_detection_heatmap like CenterTrack? So I wonder how to predict offset map O_B with just one frame. My initial guess is that feature map of frame t and t-T is calculated independently, but attaching offset-map head on an aggregated feature map (e.g simple stack/average/subtraction) of t-T and t?
    2. In Fig. 4, since Baseline doesn't have O_C (w/o CVA), does it means Baseline == Baseline + CVA or O_C == O_B in the baseline context?
    opened by dangnh0611 1
  • Low performance on the nuscenes validation set

    Low performance on the nuscenes validation set

    Hello @JialianW I test the performance of the model you provided (nuscenes.pth)on the nuscenes validation set following the readme. But I got low performance which is different from the results in the paper.

    ### Final results ###
    
    Per-class results:
    		AMOTA	AMOTP	RECALL	MOTAR	GT	MOTA	MOTP	MT	ML	FAF	TP	FP	FN	IDS	FRAG	TID	LGD
    bicycle 	0.000	1.495	0.552	0.000	1993	0.000	0.793	27	38	683.2	864	35219	893	236	90	1.07	1.82
    bus     	0.062	1.420	0.224	0.394	2112	0.088	0.811	9	87	17.5	470	285	1638	4	19	0.70	4.02
    car     	0.088	0.893	0.114	0.520	58317	0.059	0.486	133	3379	55.1	6582	3158	51646	89	166	2.04	4.35
    motorcy 	0.000	1.579	0.545	0.000	1977	0.000	0.851	29	24	494.5	819	22430	899	259	118	1.14	2.32
    pedestr 	0.000	1.394	0.639	0.000	25423	0.000	0.873	486	198	2454.6	10887	143963	9168	5368	1778	0.60	1.73
    trailer 	0.000	1.614	0.543	0.000	2425	0.000	1.084	26	29	2907.8	869	151552	1108	448	169	1.32	2.16
    truck   	0.003	1.341	0.093	0.139	9650	0.013	0.939	23	509	21.2	884	761	8756	10	30	0.70	3.08
    
    Aggregated results:
    AMOTA	0.022
    AMOTP	1.391
    RECALL	0.387
    MOTAR	0.150
    GT	14556
    MOTA	0.023
    MOTP	0.834
    MT	733
    ML	4264
    FAF	947.7
    TP	21375
    FP	357368
    FN	74108
    IDS	6414
    FRAG	2370
    TID	1.08
    LGD	2.78
    Eval time: 3056.8s 
    

    I also tested the results from CenterTrack and got good performance. So I ruled out the issues of data preparation and nuScenes dataset API. What anything else should I do to get the same performance as you provided? Thank you.

    opened by lhiceu 1
  • How to transfer results.json to coco-style

    How to transfer results.json to coco-style

    Thanks for your working. I test it on dataset and get a "results.json".But the competition needs different json style just Like { "video_id" : int, "category_id" : int, "segmentations" : [RLE or [polygon] or None], "score" : float, } How can I convet it?Thanks a lot.

    opened by lpwMei 0
  •  Questions about demo testing under CrowdHuman model

    Questions about demo testing under CrowdHuman model

    Hello,many thanks for your work. As you can see from the following two pictures ,when person2, person6 and person7 overlap, their serial numbers are exchanged. Is there any way to solve this problem? Thanks again! 2021-12-27 16-56-33屏幕截图 2021-12-27 16-56-54屏幕截图

    opened by 0-Maxwei-0 1
Owner
Jialian Wu
Ph.D. Candidate at SUNY Buffalo
Jialian Wu
A simple script that can be used to track real time that user was online in telegram

TG_OnlineTracker A simple script that can be used to track real time that user was online in telegram Join @DaisySupport_Official ?? for help ??‍♂️ Ea

Inuka Asith 15 Oct 23, 2022
This app is providing you to track some online products' prices via GMAIL.

Price Tracking App variables and descriptions of that code is in Turkish language. but we're working on translate them into English. This app is provi

Abdullah Aslan 1 Dec 11, 2021
Tracker to check the covid shot slot availability in India and send mobile alerts via Twilio Messaging Service.

Cowin-Slot-Tracker Tracker to check the covid vaccine slot availability in India and send mobile notifications through Twilio Messaging Service. Requi

invalid username 27 Nov 12, 2022
Small Python Tracker clone of Electra

Discord Bot Tracker - Python Simply Track your Bots (Status) to get notified when one of those go offline/online. Paste IDs into the config.py files,

Koni 2 Nov 23, 2021
WhatsApp Status Tracker With Python

Warning!! This Repo is Purly educational purpose Don't use this to stalk on others, which is subjective to crime Pre-Req: Telegram bot of your own wit

Vignesh Karunagaran 10 Dec 9, 2022
Bitcoin tracker hecho con python.

Bitcoin Tracker Precio del Bitcoin en tiempo real. Script simple hecho con python. Rollercoin RollerCoin es un juego en el que puedes ganar bitcoin (y

biyivi 3 Jan 4, 2022
An Amazon Price Tracker app helps you to buy which product you want within sale price by sending an E-Mail.

Amazon Price Tracker An Amazon Price Tracker app helps you to buy which product you want within sale price by sending an E-Mail. Installing Download t

Aytaç Kaşoğlu 2 Feb 10, 2022
Discord Bot for League of Legends live match tracker

SABot Dicord Bot for League of Legends match auto tracker Features: Search Summoners statistics in League of Legends. Auto-notifications provide when

Jungyu Choi 4 Sep 27, 2022
RequestTrackerBot - Request Tracker Bot With Python

Request Tracker Bot This is a Request Tracker Bot repo, It is for those who uplo

Prince Jaiswal 1 Dec 30, 2021
Webb-Tracker-Bot - This is a discord bot that displays current progress of the James Webb Space Telescope.

Webb-Tracker-Bot - This is a discord bot that displays current progress of the James Webb Space Telescope.

Copperbotte 1 Jan 5, 2022
Weather Tracker, made with Python using Open Weather API

Weather Tracker Weather Tracker, made with Python using Open Weather API

Sahil Kumar 1 Feb 7, 2022
streamlit translator is used to detect and translate between languages created using gTTS, googletrans, pillow and streamlit python packages

Streamlit Translator Streamlit Translator is a simple translator app to detect and translate between languages. Streamlit Translator gets text and lan

Siva Prakash 5 Apr 5, 2022
Track live sentiment for stocks from Reddit and Twitter and identify growing stocks

Market Sentiment About This repository can mainly be used for two things. a. Tracking the live sentiment of stocks from Reddit and Twitter b. Tracking

Market Sentiment 345 Dec 17, 2022
Official python API for Phish.AI public and private API to detect zero-day phishing websites

phish-ai-api Summary Official python API for Phish.AI public and private API to detect zero-day phishing websites How it Works (TLDR) Essentially we h

Phish.AI 168 May 17, 2022
Cloudkeeper is “housekeeping for clouds” - find leaky resources, manage quota limits, detect drift and clean up.

Cloudkeeper Housekeeping for Clouds! Table of contents Overview Docker based quick start Cloning this repository Component list Contact License Overvi

Some Engineering 1.2k Jan 3, 2023
A discord bot that can detect Nitro Scam Links and delete them to protect other users

A discord bot that can detect Nitro Scam Links and delete them to protect other users. Add it to your server from here.

Kanak Mittal 9 Oct 20, 2022
A management system designed for the employees of MIRAS (Art Gallery). It is used to sell/cancel tickets, book/cancel events and keeps track of all upcoming events.

Art-Galleria-Management-System Its a management system designed for the employees of MIRAS (Art Gallery). Backend : Python Frontend : Django Database

Areesha Tahir 8 Nov 30, 2022
A simple Discord bot wrote with Python. Kizmeow let you track your NFT project and display some useful information

Kizmeow-OpenSea-and-Etherscan-Discord-Bot 中文版 | English Ver A Discord bot wrote with Python. Kizmeow let you track your NFT project and display some u

Xeift 93 Dec 31, 2022
Wordle-bot: A Discord bot to track you and your friends' Wordle scores.

wordle-bot A Discord bot to track you and your friends' Wordle scores, so you can see who's the best! To submit a score to wordle-bot, just paste the

Spencer Murray 8 Feb 16, 2022