Joint-task Self-supervised Learning for Temporal Correspondence (NeurIPS 2019)

Related tags

Deep Learning UVC
Overview

Joint-task Self-supervised Learning for Temporal Correspondence

Project | Paper

Overview

Joint-task Self-supervised Learning for Temporal Correspondence

Xueting Li*, Sifei Liu*, Shalini De Mello, Xiaolong Wang, Jan Kautz, Ming-Hsuan Yang.

(* equal contributions)

In Neural Information Processing Systems (NeurIPS), 2019.

Citation

If you use our code in your research, please use the following BibTex:

@inproceedings{uvc_2019,
    Author = {Xueting Li and Sifei Liu and Shalini De Mello and Xiaolong Wang and Jan Kautz and Ming-Hsuan Yang},
    Title = {Joint-task Self-supervised Learning for Temporal Correspondence},
    Booktitle = {NeurIPS},
    Year = {2019},
}

Instance segmentation propagation on DAVIS2017

Method J_mean J_recall J_decay F_mean F_recall F_decay
Ours 0.563 0.650 0.289 0.592 0.641 0.354
Ours - track 0.577 0.683 0.263 0.613 0.698 0.324

Prerequisites

The code is tested in the following environment:

  • Ubuntu 16.04
  • Pytorch 1.1.0, tqdm, scipy 1.2.1

Testing on DAVIS2017

Testing without tracking

To test on DAVIS2017 for instance segmentation mask propagation, please run:

python test.py -d /workspace/DAVIS/ -s 480

Important parameters:

  • -c: checkpoint path.
  • -o: results path.
  • -d: DAVIS 2017 dataset path.
  • -s: test resolution, all results in the paper are tested on 480p images, i.e. -s 480.

Please check the test.py file for other parameters.

Testing with tracking

To test on DAVIS2017 by tracking & propagation, please run:

python test_with_track.py -d /workspace/DAVIS/ -s 480

Similar parameters as test.py, please see the test_with_track.py for details.

Testing on the VIP dataset

To test on VIP, please run the following command with your own VIP path:

python test_mask_vip.py -o results/VIP/category/ --scale_size 560 560 --pre_num 1 -d /DATA/VIP/VIP_Fine/Images/ --val_txt /DATA/VIP/VIP_Fine/lists/val_videos.txt -c weights/checkpoint_latest.pth.tar

and then:

python eval_vip.py -g DATA/VIP/VIP_Fine/Annotations/Category_ids/ -p results/VIP/category/

Testing on the JHMDB dataset

Please check out this branch. The code is borrowed from TimeCycle.

Training on Kinetics

Dataset

We use the kinetics dataset for training.

Training command

python track_match_v1.py --wepoch 10 --nepoch 30 -c match_track_switch --batchsize 40 --coord_switch 0 --lc 0.3

Acknowledgements

Comments
  • can't find variable 'Fp_tar'

    can't find variable 'Fp_tar'

    In model.py, line 187: self.grid_flat_crop = create_flat_grid(Fp_tar.size()).permute(0,2,1).detach() I can't find the declaration of the variable 'Fp_tar', I wonder what the variable refers to? Thanks a lot !

    opened by ruoqi77 5
  • ImportError: cannot import name 'norm_mask'

    ImportError: cannot import name 'norm_mask'

    Thanks for your excellent work. I have tried to run the test code (python test_with_track.py -s 480), and find an error:

    Traceback (most recent call last): File "test_with_track.py", line 21, in from libs.vis_utils import norm_mask ImportError: cannot import name 'norm_mask'

    could you fix the test code? Thanks :-)

    opened by swrdZWJ 2
  • AP metric for the VIP dataset

    AP metric for the VIP dataset

    Thanks for releasing this great work!

    Can I get the AP^{r}_{vol} score for the VIP dataset using "eval_vip.py"? It seems that only the mIoU score is computed in this file. Do I miss something? Thanks for your help.

    opened by 594422814 2
  • ModuleNotFoundError: No module named 'libs.net_utils'

    ModuleNotFoundError: No module named 'libs.net_utils'

    Good work. I have tried to run the training code, and find an error: from model import track_match_comb as Model File "/raid/codes/UVC/model.py", line 4, in from libs.net_utils import NLM, NLM_dot, NLM_woSoft ModuleNotFoundError: No module named 'libs.net_utils

    So, can you upload the missing net_utils.py. Thank you very much.

    opened by carrierlxk 0
  • Can't find concentration with a truncated loss mentioned in paper (Section:Moving as a unit.Eq.(6))

    Can't find concentration with a truncated loss mentioned in paper (Section:Moving as a unit.Eq.(6))

    Hi, thank you for providing your project. In this code, I can't find concentration regularization with a truncated loss in region tracking process mentioned in paper (Section:Moving as a unit.Eq.(6)). Could you please tell me where it is? Or if you did'nt upload the part of concentration regularization with a truncated loss, could you please upload this part? image

    opened by Hevans123 4
Owner
Sifei Liu
Sifei Liu
GitHub
Learning Correspondence from the Cycle-consistency of Time (CVPR 2019)

TimeCycle Code for Learning Correspondence from the Cycle-consistency of Time (CVPR 2019, Oral). The code is developed based on the PyTorch framework,

Xiaolong Wang 706 Nov 29, 2022
Patch Rotation: A Self-Supervised Auxiliary Task for Robustness and Accuracy of Supervised Models

Patch-Rotation(PatchRot) Patch Rotation: A Self-Supervised Auxiliary Task for Robustness and Accuracy of Supervised Models Submitted to Neurips2021 To

null 4 Jul 12, 2021
The Self-Supervised Learner can be used to train a classifier with fewer labeled examples needed using self-supervised learning.

Published by SpaceML • About SpaceML • Quick Colab Example Self-Supervised Learner The Self-Supervised Learner can be used to train a classifier with

SpaceML 92 Nov 30, 2022
Multi-Task Temporal Shift Attention Networks for On-Device Contactless Vitals Measurement (NeurIPS 2020)

MTTS-CAN: Multi-Task Temporal Shift Attention Networks for On-Device Contactless Vitals Measurement Paper Xin Liu, Josh Fromm, Shwetak Patel, Daniel M

Xin Liu 106 Dec 30, 2022
Official Pytorch implementation of 'GOCor: Bringing Globally Optimized Correspondence Volumes into Your Neural Network' (NeurIPS 2020)

Official implementation of GOCor This is the official implementation of our paper : GOCor: Bringing Globally Optimized Correspondence Volumes into You

Prune Truong 71 Nov 18, 2022
《Dual-Resolution Correspondence Network》(NeurIPS 2020)

Dual-Resolution Correspondence Network Dual-Resolution Correspondence Network, NeurIPS 2020 Dependency All dependencies are included in asset/dualrcne

Active Vision Laboratory 45 Nov 21, 2022
Self-supervised learning on Graph Representation Learning (node-level task)

graph_SSL Self-supervised learning on Graph Representation Learning (node-level task) How to run the code To run GRACE, sh run_GRACE.sh To run GCA, sh

Namkyeong Lee 3 Dec 31, 2021
Code for the paper "Spatio-temporal Self-Supervised Representation Learning for 3D Point Clouds" (ICCV 2021)

Spatio-temporal Self-Supervised Representation Learning for 3D Point Clouds This is the official code implementation for the paper "Spatio-temporal Se

Hesper 63 Jan 5, 2023
Self-supervised spatio-spectro-temporal represenation learning for EEG analysis

EEG-Oriented Self-Supervised Learning and Cluster-Aware Adaptation This repository provides a tensorflow implementation of a submitted paper: EEG-Orie

Wonjun Ko 4 Jun 9, 2022
PyTorch implementation of CVPR 2020 paper (Reference-Based Sketch Image Colorization using Augmented-Self Reference and Dense Semantic Correspondence) and pre-trained model on ImageNet dataset

Reference-Based-Sketch-Image-Colorization-ImageNet This is a PyTorch implementation of CVPR 2020 paper (Reference-Based Sketch Image Colorization usin

Yuzhi ZHAO 11 Jul 28, 2022
DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervision

The Official PyTorch Implementation of DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervision

Shiyi Lan 3 Oct 15, 2021
The PyTorch implementation of DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervision.

DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervision The PyTorch implementation of DiscoBox: Weakly Supe

Shiyi Lan 1 Oct 23, 2021
Apply Graph Self-Supervised Learning methods to graph-level task(TUDataset, MolculeNet Datset)

Graphlevel-SSL Overview Apply Graph Self-Supervised Learning methods to graph-level task(TUDataset, MolculeNet Dataset). It is unified framework to co

JunSeok 8 Oct 15, 2021
PyToch implementation of A Novel Self-supervised Learning Task Designed for Anomaly Segmentation

Self-Supervised Anomaly Segmentation Intorduction This is a PyToch implementation of A Novel Self-supervised Learning Task Designed for Anomaly Segmen

WuFan 2 Jan 27, 2022
Unified Pre-training for Self-Supervised Learning and Supervised Learning for ASR

UniSpeech The family of UniSpeech: UniSpeech (ICML 2021): Unified Pre-training for Self-Supervised Learning and Supervised Learning for ASR UniSpeech-

Microsoft 282 Jan 9, 2023
Code for MarioNette: Self-Supervised Sprite Learning, in NeurIPS 2021

MarioNette | Webpage | Paper | Video MarioNette: Self-Supervised Sprite Learning Dmitriy Smirnov, Michaël Gharbi, Matthew Fisher, Vitor Guizilini, Ale

Dima Smirnov 28 Nov 18, 2022
CVPR2021: Temporal Context Aggregation Network for Temporal Action Proposal Refinement

Temporal Context Aggregation Network - Pytorch This repo holds the pytorch-version codes of paper: "Temporal Context Aggregation Network for Temporal

Zhiwu Qing 63 Sep 27, 2022
Implementation of temporal pooling methods studied in [ICIP'20] A Comparative Evaluation Of Temporal Pooling Methods For Blind Video Quality Assessment

Implementation of temporal pooling methods studied in [ICIP'20] A Comparative Evaluation Of Temporal Pooling Methods For Blind Video Quality Assessment

Zhengzhong Tu 5 Sep 16, 2022
Cascaded Deep Video Deblurring Using Temporal Sharpness Prior and Non-local Spatial-Temporal Similarity

This repository is the official PyTorch implementation of Cascaded Deep Video Deblurring Using Temporal Sharpness Prior and Non-local Spatial-Temporal Similarity

hippopmonkey 4 Dec 11, 2022