Clockwork Convnets for Video Semantic Segmentation

Related tags

Deep Learning clockwork-fcn
Overview

Clockwork Convnets for Video Semantic Segmentation

This is the reference implementation of arxiv:1608.03609:

Clockwork Convnets for Video Semantic Segmentation
Evan Shelhamer*, Kate Rakelly*, Judy Hoffman*, Trevor Darrell
arXiv:1605.06211

This project reproduces results from the arxiv and demonstrates how to execute staged fully convolutional networks (FCNs) on video in Caffe by controlling the net through the Python interface. In this way this these experiments are a proof-of-concept implementation of clockwork, and further development is needed to achieve peak efficiency (such as pre-fetching video data layers, threshold GPU layers, and a native Caffe library edition of the staged forward pass for pipelining).

For simple reference, refer to these (display only) editions of the experiments:

Contents

  • notebooks: interactive code and documentation that carries out the experiments (in jupyter/ipython format).
  • nets: the net specification of the various FCNs in this work, and the pre-trained weights (see installation instructions).
  • caffe: the Caffe framework, included as a git submodule pointing to a compatible version
  • datasets: input-output for PASCAL VOC, NYUDv2, YouTube-Objects, and Cityscapes
  • lib: helpers for executing networks, scoring metrics, and plotting

License

This project is licensed for open non-commercial distribution under the UC Regents license; see LICENSE. Its dependencies, such as Caffe, are subject to their own respective licenses.

Requirements & Installation

Caffe, Python, and Jupyter are necessary for all of the experiments. Any installation or general Caffe inquiries should be directed to the caffe-users mailing list.

  1. Install Caffe. See the installation guide and try Caffe through Docker (recommended). Make sure to configure pycaffe, the Caffe Python interface, too.
  2. Install Python, and then install our required packages listed in requirements.txt. For instance, for x in $(cat requirements.txt); do pip install $x; done should do.
  3. Install Jupyter, the interface for viewing, executing, and altering the notebooks.
  4. Configure your PYTHONPATH as indicated by the included .envrc so that this project dir and pycaffe are included.
  5. Download the model weights for this project and place them in nets.

Now you can explore the notebooks by firing up Jupyter.

Comments
  • CamVid dataset code for clockwork-fcn

    CamVid dataset code for clockwork-fcn

    Hi, shelhamer. I have test your code and it works fine with cityscapes. I'm using CamVid dataset and i want to use your network on this dataset. Could you provide caffemodel for this dataset? Thank you.

    opened by yacineDZ44 3
  • how to get the label set of NYU-Depth v2 dataset?

    how to get the label set of NYU-Depth v2 dataset?

    In this paper, "Clockwork Convnets for Video Semantic Segmentation", The NYUDv2 dataset [6] collects short RGB-D clips and includes a segmentation benchmark with high-quality but temporally sparse pixel annotations (every tenth video frame is labeled). We run on video from the “raw” clips subsampled 10X and evaluate on every labeled frame. But i am confused how to get the every 10th labeled ground truth, maybe I have to annotate them by myself, because I have not found the download link in http://cs.nyu.edu/~silberman/datasets/nyu_depth_v2.

    opened by s5248 2
  • how to train clockwork-fcn

    how to train clockwork-fcn

    Hi, shelhamer. I have test your code and it works fine. While I haven't find the scripts or steps about how to train this clockwork-fcn network. Would you give me a hand? Thanks a lot.

    opened by tonysy 2
  • ImportError: No module named labels

    ImportError: No module named labels

    When I was testing the code on cityscapes, I encountered an error : File "datasets\cityscapes.py", line 17, in init labels = import('labels') ImportError: No module named labels. How can I get this labels module? Thank you

    opened by SiyingLiang 1
  • train.prototxt and solver.prototxt

    train.prototxt and solver.prototxt

    Hi, thank you for sharing this great work. I want to training this network,but I facing some problem about training detail. If you release the train.prototxt and solver.prototxt, I will be very grateful to you. Thank you !

    opened by EternityZY 1
  • where can I find all the .prototxt files ?

    where can I find all the .prototxt files ?

    First for the installation it says to install everything in "requirements.txt" but file is missing.

    Also, I am trying to test your code but I need the "prostage-cityscapes-fcn8s.prototxt". In general if I want to test all the prototxt are missing, can you provide them?

    Thanks

    opened by citlag 1
  • some typos

    some typos

    I have tried the "pascal-translate-exp.ipynb", and find some typos,

    1. in shell [2], "caffe.set_device(1)", I think it's better to use "caffe.set_device(0)".
    2. in shell [5], '../nets/fcn-pool3-pascal.caffemodel', '../nets/fcn-pool4-pascal.caffemodel', but in your "clockwork-nets" package, the corresponding file is "voc-fcn-pool3.caffemodel" and "voc-fcn-pool4.caffemodel". I think it's better to use the same names.
    opened by stormkingz 1
Owner
Evan Shelhamer
Evan Shelhamer
GitHub
Recall Loss for Semantic Segmentation (This repo implements the paper: Recall Loss for Semantic Segmentation)

Recall Loss for Semantic Segmentation (This repo implements the paper: Recall Loss for Semantic Segmentation) Download Synthia dataset The model uses

null 32 Sep 21, 2022
RepVGG: Making VGG-style ConvNets Great Again

RepVGG: Making VGG-style ConvNets Great Again (PyTorch) This is a super simple ConvNet architecture that achieves over 80% top-1 accuracy on ImageNet

null 2.8k Jan 4, 2023
RepVGG: Making VGG-style ConvNets Great Again

This repository is the code that needs to be submitted for OpenMMLab Algorithm Ecological Challenge，the paper is RepVGG: Making VGG-style ConvNets Great Again

Ty Feng 62 May 21, 2022
PyTorch implementation of spectral graph ConvNets, NIPS’16

Graph ConvNets in PyTorch October 15, 2017 Xavier Bresson http://www.ntu.edu.sg/home/xbresson https://github.com/xbresson https://twitter.com/xbresson

Xavier Bresson 287 Jan 4, 2023
Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentation, CVPR 2018

Learning Pixel-level Semantic Affinity with Image-level Supervision This code is deprecated. Please see https://github.com/jiwoon-ahn/irn instead. Int

Jiwoon Ahn 337 Dec 15, 2022
Segmentation in Style: Unsupervised Semantic Image Segmentation with Stylegan and CLIP

Segmentation in Style: Unsupervised Semantic Image Segmentation with Stylegan and CLIP Abstract: We introduce a method that allows to automatically se

Daniil Pakhomov 134 Dec 19, 2022
TorchDistiller - a collection of the open source pytorch code for knowledge distillation, especially for the perception tasks, including semantic segmentation, depth estimation, object detection and instance segmentation.

This project is a collection of the open source pytorch code for knowledge distillation, especially for the perception tasks, including semantic segmentation, depth estimation, object detection and instance segmentation.

yifan liu 147 Dec 3, 2022
Mae segmentation - Reproduction of semantic segmentation using masked autoencoder (mae)

ADE20k Semantic segmentation with MAE Getting started Install the mmsegmentation

null 97 Dec 17, 2022
Video-Captioning - A machine Learning project to generate captions for video frames indicating the relationship between the objects in the video

Video-Captioning - A machine Learning project to generate captions for video frames indicating the relationship between the objects in the video

null 1 Jan 23, 2022
A curated list of awesome resources related to Semantic Search🔎 and Semantic Similarity tasks.

A curated list of awesome resources related to Semantic Search?? and Semantic Similarity tasks.

null 224 Jan 4, 2023
Siamese-nn-semantic-text-similarity - A repository containing comprehensive Neural Networks based PyTorch implementations for the semantic text similarity task

Siamese Deep Neural Networks for Semantic Text Similarity PyTorch A repository c

Shahrukh Khan 32 Dec 15, 2022
Build upon neural radiance fields to create a scene-specific implicit 3D semantic representation, Semantic-NeRF

Semantic-NeRF: Semantic Neural Radiance Fields Project Page | Video | Paper | Data In-Place Scene Labelling and Understanding with Implicit Scene Repr

Shuaifeng Zhi 243 Jan 7, 2023
Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers

Segmentation Transformer Implementation of Segmentation Transformer in PyTorch, a new model to achieve SOTA in semantic segmentation while using trans

Abhay Gupta 161 Dec 8, 2022
Exploring Cross-Image Pixel Contrast for Semantic Segmentation

Exploring Cross-Image Pixel Contrast for Semantic Segmentation Exploring Cross-Image Pixel Contrast for Semantic Segmentation, Wenguan Wang, Tianfei Z

Tianfei Zhou 510 Jan 2, 2023
Unsupervised Semantic Segmentation by Contrasting Object Mask Proposals.

Unsupervised Semantic Segmentation by Contrasting Object Mask Proposals This repo contains the Pytorch implementation of our paper: Unsupervised Seman

Wouter Van Gansbeke 335 Dec 28, 2022
Implementation of SETR model, Original paper: Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers.

SETR - Pytorch Since the original paper (Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers.) has no official

zhaohu xing 112 Dec 16, 2022
Prototypical Pseudo Label Denoising and Target Structure Learning for Domain Adaptive Semantic Segmentation (CVPR 2021)

Prototypical Pseudo Label Denoising and Target Structure Learning for Domain Adaptive Semantic Segmentation (CVPR 2021, official Pytorch implementatio

Microsoft 247 Dec 25, 2022
Shape-aware Semi-supervised 3D Semantic Segmentation for Medical Images

SASSnet Code for paper: Shape-aware Semi-supervised 3D Semantic Segmentation for Medical Images(MICCAI 2020) Our code is origin from UA-MT You can fin

klein 125 Jan 3, 2023
A Comprehensive Analysis of Weakly-Supervised Semantic Segmentation in Different Image Domains (IJCV submission)

wsss-analysis The code of: A Comprehensive Analysis of Weakly-Supervised Semantic Segmentation in Different Image Domains, arXiv pre-print 2019 paper.

Lyndon Chan 48 Dec 18, 2022