Proposal, Tracking and Segmentation (PTS): A Cascaded Network for Video Object Segmentation
By Qiang Zhou*, Zilong Huang*, Lichao Huang, Han Shen, Yongchao Gong, Chang Huang, Wenyu Liu, Xinggang Wang.(* means equal contribution)
This code is the implementation mainly for DAVIS 2017 dataset. For more detail, please refer to our paper.
Architecture
Overview of our proposed PTSNet for video object segmentation. OPN is designed for generating proposals of the interested objects and OTN aims to distinguish which one of the proposals is the best. Finally, DRSN does the final pixel level tracking(segmentation) task. Note in our implementation we couple OPN and OTN as a whole network, and spearate DRSN out under engineering consideration.
 |
 |
 |
 |
 |
 |
Usage
Preparation
-
Install PyTorch 1.0 and necessary libraries like opencv, PIL etc.
-
There are some native CUDA implementations, InPlace-ABN and MaskRCNN Operators, which must be compiled at the very start.
# Before you compile, you need to figure out several things: # - The CUDA kernels supported by your GPU, here we use `sm_52`, `sm_61` and `sm_70` for NVIDIA Titan V. # - `cuda` and `nvcc` paths in your operating system, which exist usually in `/usr/local/cuda` and `/usr/local/cuda/bin/nvcc` respectively. # InPlace-ABN_0.4 (PyTorch 0.4) cd model/inplace_ABN_0.4 bash build.sh # OR you could choose the 1.0 version of inplace ABN. # InPlace-ABN_1.0 (PyTorch 1.0) cd model/inplace_ABN # It is dynamically compiled when running (gcc > 4.9) # MaskRCNN Operators (PyTorch 0.4) cd coupled_otn_opn/tracking/maskrcnn/lib bash make.sh
-
You can train PTSNet from scratch or just evaluate our pretrained model.
-
Train it from scratch, you need to download:
# DRSN: wget "https://download.pytorch.org/models/resnet50-19c8e357.pth" -O drsn/init_models/resnet50-19c8e357.pth # OPN: wget "https://drive.google.com/open?id=1ma1fNmEvS9dJLOIcm1FRzYofVS_t3aI3" -O coupled_otn_opn/tracking/maskrcnn/data/X-152-32x8d-IN5k.pkl # If you want to use our pretrained OTN: # wget https://drive.google.com/open?id=12bF1dRlEUZoQz3Qcr2WD3ojqNHzbCrjf, put it into `coupled_otn_opn/models/mdnet_davis_50cyche.pth` # Else please modify from py-MDNet(https://github.com/HyeonseobNam/py-MDNet) to train OTN on DAVIS by yourself.
-
If you want to use our pretrained model to do the evaluation, you need to download:
# DRSN: https://drive.google.com/open?id=116yXnqX43BZ7kEgdzUhIeTSn1dbvcE2F, put it into `drsn/snapshots/drsn_yvos_10w_davis_3p5w.pth` # OPN: wget "https://drive.google.com/open?id=1ma1fNmEvS9dJLOIcm1FRzYofVS_t3aI3" -O coupled_otn_opn/tracking/maskrcnn/data/X-152-32x8d-IN5k.pkl # OTN: https://drive.google.com/open?id=12bF1dRlEUZoQz3Qcr2WD3ojqNHzbCrjf, put it into `coupled_otn_opn/models/mdnet_davis_50cycle.pth`
-
-
Dataset
- YouTube-VOS: Download from YouTube-VOS, note we only need the training part(
train_all_frames.zip), totally about 41G. Unzip, move and rename it todrsn/dataset/yvos. - DAVIS: Download from DAVIS, note we only need the 480p version(
DAVIS-2017-trainval-480p.zip). Unzip, move and rename it todrsn/dataset/DAVIS/trainvalandcoupled_otn_opn/DAVIS/trainval. Here you need to make a subdirectory oftrainvaldirectory to store the dataset.
And make sure to put the files as the following structure:
. ├── drsn │ ├── dataset │ │ ├── DAVIS │ │ │ └── trainval │ │ │ ├── Annotations │ │ │ ├── ImageSets │ │ │ └── JPEGImages │ │ └── yvos │ │ └── train_all_frames │ ├── init_model │ │ └── resnet50-19c8e357.pth │ └── snapshots │ └── drsn_yvos_10w_davis_3p5w.pth └── coupled_otn_opn ├── DAVIS │ └── trainval ├── models │ └── mdnet_davis_50cycle.pth └── tracking └── maskrcnn └── data └── X-152-32x8d-FPN-IN5k.pkl - YouTube-VOS: Download from YouTube-VOS, note we only need the training part(
Train and Evaluate
- Firstly, check the directory of
coupled_otn_opnand follow theREADME.mdinside to generate our proposals. You can also skip this step for we have provided generated proposals indrsn/dataset/result_davisdirectory. - Secondly, enter
drsnand checkdo_train_eval.shto train and evaluate. - Finally, we also provide result masks by our PTSNet in result-masks-GoogleDrive. The quantitative results are measured by DAVIS official matlab toolbox.
| J Mean | F Mean | G Mean | |
|---|---|---|---|
| Avg | 71.6 | 77.7 | 74.7 |
Acknowledgment
The work was mainly done during an internship at Horizon Robotics.
Citing PTSNet
If you find PTSNet useful in your research, please consider citing:
@article{ptsnet2019,
title={Proposal, Tracking and Segmentation (PTS): A Cascaded Network for Video Object Segmentation},
author={Zhou, Qiang and Huang, Zilong and Huang, Lichao and Han, Shen and Gong, Yongchao and Huang, Chang and Liu, Wenyu and Wang, Xinggang},
journal = {arXiv preprint arXiv:1907.01203v2},
year={2019}
}
1 Nov 22, 2021
1.5k Dec 27, 2022
233 Dec 29, 2022
432 Dec 17, 2022
3 Oct 7, 2022
63 Sep 27, 2022
276 Dec 31, 2022
63 Mar 27, 2022
2 Sep 5, 2022
1 Dec 29, 2021
443 Dec 6, 2022
1.6k Jan 4, 2023
130 Dec 29, 2022
325 Jan 5, 2023
62 Dec 12, 2022
11 Oct 8, 2022
290 Dec 24, 2022
76 Dec 14, 2022
72 Dec 14, 2022