A general python framework for single object tracking in LiDAR point clouds, based on PyTorch Lightning.

Related tags

Deep Learning Open3DSOT
Overview

Open3DSOT

A general python framework for single object tracking in LiDAR point clouds, based on PyTorch Lightning.

The official code release of BAT and MM Track.

Features

  • Modular design. It is easy to config the model and training/testing behaviors through just a .yaml file.
  • DDP support for both training and testing.
  • Support all common tracking datasets (KITTI, NuScenes, Waymo Open Dataset).

๐Ÿ“ฃ One tracking paper is accepted by CVPR2022 (Oral)! ๐Ÿ‘‡

Trackers

This repository includes the implementation of the following models:

MM-Track (CVPR2022 Oral)

[Paper] [Project Page]

MM-Track is the first motion-centric tracker in LiDAR SOT, which robustly handles distractors and drastic appearance changes in complex driving scenes. Unlike previous methods, MM-Track is a matching-free two-stage tracker which localizes the targets by explicitly modeling the "relative target motion" among frames.

BAT (ICCV2021)

[Paper] [Results]

Official implementation of BAT. BAT uses the BBox information to compensate the information loss of incomplete scans. It augments the target template with box-aware features that efficiently and effectively improve appearance matching.

P2B (CVPR2020)

[Paper] [Official implementation]

Third party implementation of P2B. Our implementation achieves better results than the official code release. P2B adapts SiamRPN to 3D point clouds by integrating a pointwise correlation operator with a point-based RPN (VoteNet).

Setup

Installation

  • Create the environment

    git clone https://github.com/Ghostish/Open3DSOT.git
cd Open3DSOT
conda create -n Open3DSOT  python=3.6
conda activate Open3DSOT

  • Install pytorch

    conda install pytorch==1.4.0 torchvision==0.5.0 cudatoolkit=10.1 -c pytorch

    Our code is well tested with pytorch 1.4.0 and CUDA 10.1. But other platforms may also work. Follow this to install another version of pytorch. Note: In order to reproduce the reported results with the provided checkpoints, please use CUDA 10.x.

  • Install other dependencies:

    pip install -r requirement.txt

    Install the nuscenes-devkit if you use want to use NuScenes dataset:

    pip install nuscenes-devkit

KITTI dataset

  • Download the data for velodyne, calib and label_02 from KITTI Tracking.
  • Unzip the downloaded files.
  • Put the unzipped files under the same folder as following. 
    [Parent Folder]
--> [calib]
    --> {0000-0020}.txt
--> [label_02]
    --> {0000-0020}.txt
--> [velodyne]
    --> [0000-0020] folders with velodynes .bin files

NuScenes dataset

  • Download the dataset from the download page
  • Extract the downloaded files and make sure you have the following structure: 
    [Parent Folder]
  samples	-	Sensor data for keyframes.
  sweeps	-	Sensor data for intermediate frames.
  maps	        -	Folder for all map files: rasterized .png images and vectorized .json files.
  v1.0-*	-	JSON tables that include all the meta data and annotations. Each split (trainval, test, mini) is provided in a separate folder.

Note: We use the train_track split to train our model and test it with the val split. Both splits are officially provided by NuScenes. During testing, we ignore the sequences where there is no point in the first given bbox.

Waymo dataset

  • Download and prepare dataset by the instruction of CenterPoint. 
    [Parent Folder]
  tfrecord_training	                    
  tfrecord_validation	                 
  train 	                                    -	all training frames and annotations 
  val   	                                    -	all validation frames and annotations 
  infos_train_01sweeps_filter_zero_gt.pkl
  infos_val_01sweeps_filter_zero_gt.pkl
  • Prepare SOT dataset. Data from specific category and split will be merged (e.g., sot_infos_vehicle_train.pkl).
  python datasets/generate_waymo_sot.py

Quick Start

Training

To train a model, you must specify the .yaml file with --cfg argument. The .yaml file contains all the configurations of the dataset and the model. Currently, we provide four .yaml files under the cfgs directory. Note: Before running the code, you will need to edit the .yaml file by setting the path argument as the correct root of the dataset.

python main.py --gpu 0 1 --cfg cfgs/BAT_Car.yaml  --batch_size 50 --epoch 60 --preloading

After you start training, you can start Tensorboard to monitor the training process:

tensorboard --logdir=./ --port=6006

By default, the trainer runs a full evaluation on the full test split after training every epoch. You can set --check_val_every_n_epoch to a larger number to speed up the training. The --preloading flag is used to preload the training samples into the memory to save traning time. Remove this flag if you don't have enough memory.

Testing

To test a trained model, specify the checkpoint location with --checkpoint argument and send the --test flag to the command.

python main.py --gpu 0 1 --cfg cfgs/BAT_Car.yaml  --checkpoint /path/to/checkpoint/xxx.ckpt --test

Reproduction

Model Category Success Precision Checkpoint
BAT-KITTI Car 65.37 78.88 pretrained_models/bat_kitti_car.ckpt
BAT-NuScenes Car 40.73 43.29 pretrained_models/bat_nuscenes_car.ckpt
BAT-KITTI Pedestrian 45.74 74.53 pretrained_models/bat_kitti_pedestrian.ckpt

Three trained BAT models for KITTI and NuScenes datasets are provided in the pretrained_models directory. To reproduce the results, simply run the code with the corresponding .yaml file and checkpoint. For example, to reproduce the tracking results on KITTI Car, just run:

python main.py --gpu 0 1 --cfg cfgs/BAT_Car.yaml  --checkpoint ./pretrained_models/bat_kitti_car.ckpt --test

Acknowledgment

  • This repo is built upon P2B and SC3D.
  • Thank Erik Wijmans for his pytorch implementation of PointNet++

License

This repository is released under MIT License (see LICENSE file for details).

Comments
  • Hi author, can you kindly provide a direct copy of pre-trained NuScenes model?

    Hi author, can you kindly provide a direct copy of pre-trained NuScenes model?

    Congratulation on the great work of M2Track. In the paper, there's comparison of P2B, BAT, and M2Track on the NuScenes in 5 categories. Can you kindly release all 15 trained model? Thank you!

    opened by Xiaoqiong-Liu 6
  • Memory problem while testing nuscences

    Memory problem while testing nuscences

    Hi๏ผŒwhile testing in nuscenes dataset,my memory usage keeping rising to 100% and the process will bekilled.My memory is 64 GB.I want to know your memory size.And how can I modify the code to make it run normally๏ผŸLooking forward to your reply!Thank you. image

    opened by freshwk 6
  • How to encoder the time (frame id) information?

    How to encoder the time (frame id) information?

    How do you set the time information to construct the point cloud (Nx4) in the final temporal dimension as you said in your paper? There may be several choices: (1) The true frame id like 400 for the 400-th frame; (2) Normalized frame id like 0-1 for all frame ids, then 0.5 will indidate the half frames (200-th frame for a 400-frame video) (3) just use 0 for all the templates and 1 for all search frames.

    opened by sallymmx 5
  • the number of points in the first frameโ€™s car

    the number of points in the first frameโ€™s car

    Would you please tell me how you compute the number of points in the first frameโ€™s car.My result is quite different from the result in your paper. My code are as follows: image

    opened by freshwk 5
  • Testing InNuScenes

    Testing InNuScenes

    Hi, Thanks for your work.

    I try to test the results on Nuscenes, but the total number of frame is different from your value reported in paper. For example, "Bus" includes 2953 frames when testing, and 2215 in paper. I don't why. Can you give me some tips? image

    Thanks.

    opened by StrivedTye 4
  • validation by RTX-3090

    validation by RTX-3090

    Hi, Maybe I can provide another validation by using the provided pre-trained bat_kitti_cat.cpkt model:

    DATALOADER:0 VALIDATE RESULTS {'precision/test': 76.09471893310547, 'precision/test_epoch': 76.09471893310547, 'success/test': 63.18337321281433, 'success/test_epoch': 63.18337321281433}

    Environment: Cuda-11.0; PyTorch: 1.7.0; RTX 3090;

    It seems that the success result is better and the precision is slightly inferior to the original result reported in Table 1 of the paper. Is it correct?

    opened by jimmy-dq 4
  • More details about NuScenes dataset

    More details about NuScenes dataset

    Thanks for your work! I want to ask for some information about NuScenes dataset.

    1. you said you use the training set of NuScenes for training and test on its validation set. Have you used the two splits all? As I know, there are some tracking sequences whose first frame has no points, do you test them too?
    2. I see you test Car, Truck, Trailer, Bus in NuScenes, how about the other categories? such as "pedestrian", "motorcycle"? Thanks again for your work!
    opened by 3bobo 4
  • Cannot reproduce results on KITTI pedestrian

    Cannot reproduce results on KITTI pedestrian

    Hi, Thanks for your contribution to 3D SOT research area. However, I met some problems when I used your code. It's so weird that I can reproduce the result of KITTI car while I cannot reproduce the result of KITTI pedestrian using pretrained models.

    CUDA_VISIBLE_DEVICES=2,3 python main.py --gpu 0 1 --cfg cfgs/BAT_Car.yaml  --checkpoint ./pretrained_models/bat_kitti_car.ckpt --test
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/2
initializing distributed: GLOBAL_RANK: 1, MEMBER: 2/2
----------------------------------------------------------------------------------------------------
distributed_backend=nccl
All distributed processes registered. Starting with 2 processes
----------------------------------------------------------------------------------------------------

LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [2,3]
LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [2,3]
Validating: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 60/60 [02:55<00:00,  1.43s/it]-
--------------------------------------------------------------------------------
DATALOADER:0 VALIDATE RESULTS
{'precision/test': 78.87921142578125,
 'precision/test_epoch': 78.87921142578125,
 'success/test': 65.34402370452881,
 'success/test_epoch': 65.34402370452881}

    CUDA_VISIBLE_DEVICES=2,3 python main.py --gpu 0 1 --cfg cfgs/BAT_Pedestrian.yaml  --checkpoint ./pretrained_models/bat_kitti_pedestrian.ckpt --test                                                                                                                                                                                                                                  
GPU available: True, used: True                                                                                                                                                             
TPU available: False, using: 0 TPU cores                                                                                                                                                    
IPU available: False, using: 0 IPUs                                                                                                                                                         
initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/2                                                                                                                                       
initializing distributed: GLOBAL_RANK: 1, MEMBER: 2/2                                                                                                                                       
----------------------------------------------------------------------------------------------------                                                                                        
distributed_backend=nccl                                                                                                                                                                    
All distributed processes registered. Starting with 2 processes                                                                                                                             
----------------------------------------------------------------------------------------------------

LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [2,3]
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [2,3]
Validating: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 31/31 [03:01<00:00,  3.21s/it]
--------------------------------------------------------------------------------
DATALOADER:0 VALIDATE RESULTS
{'precision/test': 71.10257720947266,
 'precision/test_epoch': 71.10257720947266,
 'success/test': 43.36358308792114,
 'success/test_epoch': 43.36358308792114}
--------------------------------------------------------------------------------

    I evaluate BAT model with cuda10.2 and pytorch 1.8, thus I suppose this phenomenon is not caused by the platform. Do you also meet with such a phenomenon? Look forward to your explanation.

    Thanks~

    opened by slothfulxtx 3
  • Predict or Inference using BAT model

    Predict or Inference using BAT model

    Nice work! I wonder how I can achieve a simple inference step using your model since you used Pytorch_Lighting and your current configuration is aimed to train a model and validate it. I think sampling process [here] is required in all modes even in inference mode, right?

    Thanks

    opened by YoushaaMurhij 3
  • Van and Cyclist

    Van and Cyclist

    Hi,if i want to train and test Van and Cyclist category ,what do I need to modify in the BAT_Car.yaml? I am so sorry to bother you again! I am following your work and I need to do a complete experiment to test my idea.It would be better if you have time to upload the config file to github.Thank you!

    opened by freshwk 3
  • Questions about mean result?

    Questions about mean result?

    Dear authors,

    Thanks for your work, I suffer from a question when following your work.

    The paper reports result for multi-category and reports a mean result.

    I wonder if the result for per-category is obtained by training the model on the data of the specified category.

    Thanks!

    opened by WangJun-ZJUT 2
  • Testing on NuScenes Validation

    Testing on NuScenes Validation

    Hi authors,

    Great work!

    I have a question about the testing results on NuScenes. In the paper "V2B", they also evaluate the trackers using the keyframes of the validation set, but you have a different number of the final testing frames. For example, you have 64159 cars, V2B has only 15578 cars. Could you give some tips if you know the reason behind it?

    Thanks!

    opened by StrivedTye 1
  • About the running speed of M2Track

    About the running speed of M2Track

    Hi,thank you for your contribution. In your paper, you report the running speed of M2Track is 57 FPS. However, i run your model on a single 3090 GPU, it only takes 92s to test all 6424 frames in the Car category. It achieves 69.8fps. I want to know how do you compute the Speed metric, thank you! image

    opened by freshwk 1
  • Reproduction of M2track

    Reproduction of M2track

    Dear authors,

    I am not able to reproduce the results reported in the paper using the given config at M2_track_kitti.yaml

    The paper reports 61.5/88.2 for the pedestrian class on KITTI but I have 52.1/76.5.

    I wonder if the same config used for generating the results in the paper is provided. Or is there anything else we can take note of to reproduce the results?

    Thanks!

    opened by st724586 10
  • Hi author, can you provide a direct copy of the Waymo SOT dataset that has been processed?

    Hi author, can you provide a direct copy of the Waymo SOT dataset that has been processed?

    Hi author, can you provide a direct copy of the Waymo SOT dataset that has been processed? I am confused about which version of the data is available on the official website and the subsequent processing of the data.

    opened by 2311762665 4
Owner
Kangel Zenn
Ph.D. Student in CUHKSZ.
Kangel Zenn
GitHub
MOT-Tracking-by-Detection-Pipeline - For Tracking-by-Detection format MOT (Multi Object Tracking), is it a framework that separates Detection and Tracking processes?

MOT-Tracking-by-Detection-Pipeline Tracking-by-DetectionๅฝขๅผใฎMOT(Multi Object Trac

KazuhitoTakahashi 41 Nov 23, 2022
The official implementation of ICCV paper "Box-Aware Feature Enhancement for Single Object Tracking on Point Clouds".

Box-Aware Tracker (BAT) Pytorch-Lightning implementation of the Box-Aware Tracker. Box-Aware Feature Enhancement for Single Object Tracking on Point C

Kangel Zenn 5 Mar 26, 2022
Fog Simulation on Real LiDAR Point Clouds for 3D Object Detection in Adverse Weather

LiDAR fog simulation Created by Martin Hahner at the Computer Vision Lab of ETH Zurich. This is the official code release of the paper Fog Simulation

Martin Hahner 110 Dec 30, 2022
Uncertainty-aware Semantic Segmentation of LiDAR Point Clouds for Autonomous Driving

SalsaNext: Fast, Uncertainty-aware Semantic Segmentation of LiDAR Point Clouds for Autonomous Driving Abstract In this paper, we introduce SalsaNext f

null 308 Jan 4, 2023
A general framework for deep learning experiments under PyTorch based on pytorch-lightning

torchx Torchx is a general framework for deep learning experiments under PyTorch based on pytorch-lightning. TODO list gan-like training wrapper text

Yingtian Liu 6 Mar 17, 2022
Official PyTorch implementation of CAPTRA: CAtegory-level Pose Tracking for Rigid and Articulated Objects from Point Clouds

CAPTRA: CAtegory-level Pose Tracking for Rigid and Articulated Objects from Point Clouds Introduction This is the official PyTorch implementation of o

Yijia Weng 96 Dec 7, 2022
Code for "PV-RAFT: Point-Voxel Correlation Fields for Scene Flow Estimation of Point Clouds", CVPR 2021

PV-RAFT This repository contains the PyTorch implementation for paper "PV-RAFT: Point-Voxel Correlation Fields for Scene Flow Estimation of Point Clou

Yi Wei 43 Dec 5, 2022
Point Cloud Denoising input segmentation output raw point-cloud valid/clear fog rain de-noised Abstract Lidar sensors are frequently used in environme

Point Cloud Denoising input segmentation output raw point-cloud valid/clear fog rain de-noised Abstract Lidar sensors are frequently used in environme

null 75 Nov 24, 2022
Synthetic LiDAR sequential point cloud dataset with point-wise annotations

SynLiDAR dataset: Learning From Synthetic LiDAR Sequential Point Cloud This is official repository of the SynLiDAR dataset. For technical details, ple

null 78 Dec 27, 2022
(CVPR 2021) Back-tracing Representative Points for Voting-based 3D Object Detection in Point Clouds

BRNet Introduction This is a release of the code of our paper Back-tracing Representative Points for Voting-based 3D Object Detection in Point Clouds,

null 86 Oct 5, 2022
Code for "CloudAAE: Learning 6D Object Pose Regression with On-line Data Synthesis on Point Clouds" @ICRA2021

CloudAAE This is an tensorflow implementation of "CloudAAE: Learning 6D Object Pose Regression with On-line Data Synthesis on Point Clouds" Files log:

Gee 35 Nov 14, 2022
Code for the paper SphereRPN: Learning Spheres for High-Quality Region Proposals on 3D Point Clouds Object Detection, ICIP 2021.

SphereRPN Code for the paper SphereRPN: Learning Spheres for High-Quality Region Proposals on 3D Point Clouds Object Detection, ICIP 2021. Authors: Th

Thang Vu 15 Dec 2, 2022
BADet: Boundary-Aware 3D Object Detection from Point Clouds (Pattern Recognition 2022)

BADet: Boundary-Aware 3D Object Detection from Point Clouds (Pattern Recognition

Rui Qian 17 Dec 12, 2022
Voxel Set Transformer: A Set-to-Set Approach to 3D Object Detection from Point Clouds (CVPR 2022)

Voxel Set Transformer: A Set-to-Set Approach to 3D Object Detection from Point Clouds (CVPR2022)[paper] Authors: Chenhang He, Ruihuang Li, Shuai Li, L

Billy HE 141 Dec 30, 2022
An efficient 3D semantic segmentation framework for Urban-scale point clouds like SensatUrban, Campus3D, etc.

An efficient 3D semantic segmentation framework for Urban-scale point clouds like SensatUrban, Campus3D, etc.

Zou 33 Jan 3, 2023
Code Release for ICCV 2021 (oral), "AdaFit: Rethinking Learning-based Normal Estimation on Point Clouds"

AdaFit: Rethinking Learning-based Normal Estimation on Point Clouds (ICCV 2021 oral) **Project Page | Arxiv ** Runsong Zhuยน, Yuan Liuยฒ, Zhen Dongยน, Te

null 40 Dec 30, 2022
Attention-based Transformation from Latent Features to Point Clouds (AAAI 2022)

Attention-based Transformation from Latent Features to Point Clouds This repository contains a PyTorch implementation of the paper: Attention-based Tr

null 12 Nov 11, 2022
This repo is a PyTorch implementation for Paper "Unsupervised Learning for Cuboid Shape Abstraction via Joint Segmentation from Point Clouds"

Unsupervised Learning for Cuboid Shape Abstraction via Joint Segmentation from Point Clouds This repository is a PyTorch implementation for paper: Uns

Kaizhi Yang 42 Dec 9, 2022
CLOCs: Camera-LiDAR Object Candidates Fusion for 3D Object Detection

CLOCs is a novel Camera-LiDAR Object Candidates fusion network. It provides a low-complexity multi-modal fusion framework that improves the performance of single-modality detectors. CLOCs operates on the combined output candidates of any 3D and any 2D detector, and is trained to produce more accurate 3D and 2D detection results.

Su Pang 254 Dec 16, 2022