Learning to Segment Instances in Videos with Spatial Propagation Network

Related tags

Deep Learning Seg-with-SPN
Overview

Learning to Segment Instances in Videos with Spatial Propagation Network

alt text

This paper is available at the 2017 DAVIS Challenge website.

Check our results in this video.

Contact: Jingchun Cheng (chengjingchun at gmail dot com)

Cite the Paper

If you find that our method is useful in your research, please cite:

@article{DAVIS2017-6th,
  author = {J. Cheng and S. Liu and Y.-H. Tsai and W.-C. Hung and S. Gupta and J. Gu and J. Kautz and S. Wang and M.-H. Yang}, 
  title = {Learning to Segment Instances in Videos with Spatial Propagation Network}, 
  journal = {The 2017 DAVIS Challenge on Video Object Segmentation - CVPR Workshops}, 
  year = {2017}
}

About the Code

  • The code released here mainly consistes of two parts in the paper: foreground segmentation and instance recognition.

  • It contains the parent net for foreground segmentation and training codes for instance recognition networks.

  • The matlab_code folder contains a simple version of our CRAF step for segmentation refinement.

Requirements

Training

  • Train the per-object recognition model.
    cd training
    python solve.py PATH_OF_MODEL PATH_OF_SOLVER
    Foe example, on the 'choreography' video for the 1st object, run:
    python solve.py ../pretrained/PN_ResNetF.caffemodel ../ResNetF/testnet_per_obj/choreography/solver_1.prototxt

Testing

  • Test the general foreground/backgroung model.
    python infer_test_fgbg.py PATH_OF_MODEL PATH_OF_RESULT VIDEO_NAME
    Foe example, on the 'lions' video, run:
    python infer_test_fgbg.py pretrained/PN_ResNetF.caffemodel results/fgbg lions

  • Test the object instance model.
    python infer_test_perobj.py MODEL_ITERATION VIDEO_NAME OBJECT_ID
    For example, on the 'lions' video for the 2nd object, run:
    python infer_test_perobj.py 3000 lions 2

  • Run example_CRAF.m in the matlab_code folder for a demo on CRAF segmentation refinement.

Download Our Segmentation Results on 2017 DAVIS Challenge

  • General foreground/background segmentation here
  • Instance-level object segmentation without refinement here
  • Final instance-level object segmentation with refinement here

Note

The model and code are available for non-commercial research purposes only.

  • 09/2017: code and model released
  • 03/2018: pre-trained model updated
Comments
  • Does this repo contain the implementation of SPN

    Does this repo contain the implementation of SPN

    Hi Dear Authors,

    I've read your paper about this project, and thanks a lot for sharing your code! I have a question: does this repository contain an implementation on SPN? It seems missing, right? Thanks.

    opened by glhfgg1024 4
  • compatible caffe installation

    compatible caffe installation

    Hi, I am facing the following issue : Check failed: ReadProtoFromBinaryFile(param_file, param) Failed to parse NetParameter file: pretrained/PN_ResNetF.caffemodel

    I guess, its an issue with the caffe installation. I have tried both, the latest build of caffe and the one you used for SegFlow(which I think was based on flownet2) and the issue remains the same. Could you provide some pointers to solve the issue or share the caffe version/build? Thanks

    opened by amankh 1
  • about SPN network in paper

    about SPN network in paper

    The paper proposes the spatial propagation network (SPN) as a refinement module, which learns the affinity entities and refine the coarse mask. But I didn't find the SPN code and pretrained SPN model. Could you submit the corresponding code for us.

    opened by zhui064 1
  • Davis annotation

    Davis annotation

    Hi ,How do you handle the multiple instances tag for Davis 2017 dataset? Is it converted into two categories? How do you visualize Davis 2017 data sets?Because Davis is not a different category with different colors. For example, the label color of bear is the same as that of bicycle.Thank you!

    opened by InstantWindy 0
  • Pre-trained prototxt

    Pre-trained prototxt

    Thank you very much for the source code.I am facing a issue when I was training the parent network for foreground segmentation,If you can release the training details of parent model that network's training prototxt and solver prototxt, I will be very grateful to you.

    opened by EternityZY 0
Owner
Jingchun Cheng
Jingchun Cheng
GitHub
SiamMOT is a region-based Siamese Multi-Object Tracking network that detects and associates object instances simultaneously.

SiamMOT: Siamese Multi-Object Tracking

null 432 Dec 17, 2022
【ACMMM 2021】DSANet: Dynamic Segment Aggregation Network for Video-Level Representation Learning

DSANet: Dynamic Segment Aggregation Network for Video-Level Representation Learning (ACMMM 2021) Overview We release the code of the DSANet (Dynamic S

Wenhao Wu 46 Dec 27, 2022
Novel Instances Mining with Pseudo-Margin Evaluation for Few-Shot Object Detection

Novel Instances Mining with Pseudo-Margin Evaluation for Few-Shot Object Detection (NimPme) The official implementation of Novel Instances Mining with

null 12 Sep 8, 2022
YOLOX-CondInst - Implement CondInst which is a instances segmentation method on YOLOX

YOLOX CondInst -- YOLOX 实例分割 前言 本项目是自己学习实例分割时，复现的代码. 通过自己编程，让自己对实例分割有更进一步的了解。 若想

DDGRCF 16 Nov 18, 2022
Graph Self-Attention Network for Learning Spatial-Temporal Interaction Representation in Autonomous Driving

GSAN Introduction Code for paper GSAN: Graph Self-Attention Network for Learning Spatial-Temporal Interaction Representation in Autonomous Driving, wh

YE Luyao 6 Oct 27, 2022
Code for "Learning to Segment Rigid Motions from Two Frames".

rigidmask Code for "Learning to Segment Rigid Motions from Two Frames". ** This is a partial release with inference and evaluation code.

Gengshan Yang 157 Nov 21, 2022
[CVPR2021] DoDNet: Learning to segment multi-organ and tumors from multiple partially labeled datasets

DoDNet This repo holds the pytorch implementation of DoDNet: DoDNet: Learning to segment multi-organ and tumors from multiple partially labeled datase

null 116 Dec 12, 2022
Pytorch implementation of paper "Learning Co-segmentation by Segment Swapping for Retrieval and Discovery"

SegSwap Pytorch implementation of paper "Learning Co-segmentation by Segment Swapping for Retrieval and Discovery" [PDF] [Project page] If our project

xshen 41 Dec 10, 2022
meProp: Sparsified Back Propagation for Accelerated Deep Learning

meProp The codes were used for the paper meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting (ICML 2017) [pdf]

LancoPKU 107 Nov 18, 2022
meProp: Sparsified Back Propagation for Accelerated Deep Learning (ICML 2017)

meProp The codes were used for the paper meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting (ICML 2017) [pdf]

LancoPKU 107 Nov 18, 2022
Some tentative models that incorporate label propagation to graph neural networks for graph representation learning in nodes, links or graphs.

Some tentative models that incorporate label propagation to graph neural networks for graph representation learning in nodes, links or graphs.

zshicode 1 Nov 18, 2021
data/code repository of "C2F-FWN: Coarse-to-Fine Flow Warping Network for Spatial-Temporal Consistent Motion Transfer"

C2F-FWN data/code repository of "C2F-FWN: Coarse-to-Fine Flow Warping Network for Spatial-Temporal Consistent Motion Transfer" (https://arxiv.org/abs/

EKILI 46 Dec 14, 2022
a reimplementation of Optical Flow Estimation using a Spatial Pyramid Network in PyTorch

pytorch-spynet This is a personal reimplementation of SPyNet [1] using PyTorch. Should you be making use of this work, please cite the paper according

Simon Niklaus 269 Jan 2, 2023
Code for paper Decoupled Dynamic Spatial-Temporal Graph Neural Network for Traffic Forecasting

Decoupled Spatial-Temporal Graph Neural Networks Code for our paper: Decoupled Dynamic Spatial-Temporal Graph Neural Network for Traffic Forecasting.

S22 43 Jan 4, 2023
A Text Attention Network for Spatial Deformation Robust Scene Text Image Super-resolution (CVPR2022)

A Text Attention Network for Spatial Deformation Robust Scene Text Image Super-resolution (CVPR2022) https://arxiv.org/abs/2203.09388 Jianqi Ma, Zheto

MA Jianqi, shiki 104 Jan 5, 2023
code for `Look Closer to Segment Better: Boundary Patch Refinement for Instance Segmentation`

Look Closer to Segment Better: Boundary Patch Refinement for Instance Segmentation (CVPR 2021) Introduction PBR is a conceptually simple yet effective

H.Chen 143 Jan 5, 2023
Official Tensorflow implementation of "M-LSD: Towards Light-weight and Real-time Line Segment Detection"

M-LSD: Towards Light-weight and Real-time Line Segment Detection Official Tensorflow implementation of "M-LSD: Towards Light-weight and Real-time Line

NAVER/LINE Vision 357 Jan 4, 2023
Pytorch implementation of "M-LSD: Towards Light-weight and Real-time Line Segment Detection"

M-LSD: Towards Light-weight and Real-time Line Segment Detection Pytorch implementation of "M-LSD: Towards Light-weight and Real-time Line Segment Det

null 123 Jan 4, 2023
LETR: Line Segment Detection Using Transformers without Edges

LETR: Line Segment Detection Using Transformers without Edges Introduction This repository contains the official code and pretrained models for Line S

mlpc-ucsd 157 Jan 6, 2023