《Fst Lerning of Temporl Action Proposl vi Dense Boundry Genertor》(AAAI 2020)

Related tags

Deep Learning ActionDetection-DBG
Overview

Update

  • 2020.03.13: Release tensorflow-version and pytorch-version DBG complete code.
  • 2019.11.12: Release tensorflow-version DBG inference code.
  • 2019.11.11: DBG is accepted by AAAI2020.
  • 2019.11.08: Our ensemble DBG ranks No.1 on ActivityNet

Introduction

In this repo, we propose a novel and unified action detection framework, named DBG, with superior performance over the state-of-the-art action detectors BSN and BMN. You can use the code to evaluate our DBG for action proposal generation or action detection. For more details, please refer to our paper Fast Learning of Temporal Action Proposal via Dense Boundary Generator!

Contents

Paper Introduction

image

This paper introduces a novel and unified temporal action proposal generator named Dense Boundary Generator (DBG). In this work, we propose dual stream BaseNet to generate two different level and more discriminative features. We then adopt a temporal boundary classification module to predict precise temporal boundaries, and an action-aware completeness regression module to provide reliable action completeness confidence.

ActivityNet1.3 Results

image

THUMOS14 Results

image

Qualitative Results

Prerequisites

  • Tensorflow == 1.9.0 or PyTorch == 1.1
  • Python == 3.6
  • NVIDIA GPU == Tesla P40
  • Linux CUDA 9.0 CuDNN
  • gcc 5

Getting Started

Installation

Clone the github repository. We will call the cloned directory as $DBG_ROOT.

cd $DBG_ROOT

Firstly, you should compile our proposal feature generation layers.

Please compile according to the framework you need.

Compile tensorflow-version proposal feature generation layers:

cd tensorflow/custom_op
make

Compile pytorch-version proposal feature generation layers:

cd pytorch/custom_op
python setup.py install

Download Datasets

Prepare ActivityNet 1.3 dataset. You can use official ActivityNet downloader to download videos from the YouTube. Some videos have been deleted from YouTube，and you can also ask for the whole dataset by email.

Extract visual feature, we adopt TSN model pretrained on the training set of ActivityNet, Please refer this repo TSN-yjxiong to extract frames and optical flow and refer this repo anet2016-cuhk to find pretrained TSN model.

For convenience of training and testing, we rescale the feature length of all videos to same length 100, and we provide the 19993 rescaled feature at here Google Cloud or 微云. Then put the features to data/tsn_anet200 directory.

For generating the video features, scripts in ./tools will help you to start from scrach.

Testing of DBG

If you don't want to train the model, you can run the testing code directly using the pretrained model.

Pretrained model is included in output/pretrained_model and set parameters on config/config_pretrained.yaml. Please check the feat_dir in config/config_pretrained.yaml and use scripts to run DBG.

# TensorFlow version (AUC result = 68.37%):
python tensorflow/test.py config/config_pretrained.yaml
python post_processing.py output/result/ results/result_proposals.json
python eval.py results/result_proposals.json

# PyTorch version (AUC result = 68.26%):
python pytorch/test.py config/config_pretrained.yaml
python post_processing.py output/result/ results/result_proposals.json
python eval.py results/result_proposals.json

Training of DBG

We also provide training code of tensorflow and pytorch version. Please check the feat_dir in config/config.yaml and follow these steps to train your model:

1. Training

# TensorFlow version:
python tensorflow/train.py config/config.yaml

# PyTorch version:
python pytorch/train.py config/config.yaml

2. Testing

# TensorFlow version:
python tensorflow/test.py config/config.yaml

# PyTorch version:
python pytorch/test.py config/config.yaml

3. Postprocessing

python post_processing.py output/result/ results/result_proposals.json

4. Evaluation

python eval.py results/result_proposals.json

Citation

If you find DBG useful in your research, please consider citing:

@inproceedings{DBG2020arXiv,
  author    = {Chuming Lin*, Jian Li*, Yabiao Wang, Ying Tai, Donghao Luo, Zhipeng Cui, Chengjie Wang, Jilin Li, Feiyue Huang, Rongrong Ji},
  title     = {Fast Learning of Temporal Action Proposal via Dense Boundary Generator},
  booktitle   = {AAAI Conference on Artificial Intelligence},
  year      = {2020},
}

Contact

For any question, please file an issue or contact

Jian Li: [email protected]
Chuming Lin: [email protected]
Comments
  • Segmentation fault when define the whole model with PFG layer compiled

    Segmentation fault when define the whole model with PFG layer compiled

    I have compiled the PFG layer to prop_tcfg.so. After that, i'm going to train a new network in which T == 500, for long videos. However, when i define the model, "Segmentation fault" breakouted, and this is the all error description. Could u please tell what the problem is? Thanks!

    error: WARNING:tensorflow:From ***************/DBG/model.py:19: conv1d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version. Instructions for updating: Use tf.keras.layers.Conv1D instead. WARNING:tensorflow:From ****************/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.init (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version. Instructions for updating: Call initializer instance with the dtype argument instead of passing it to the constructor Segmentation fault

    opened by tyziaaaa 2
  • Question about PFG layer

    Question about PFG layer

    Brilliant work and thanks for the open source code.

    I'm now reading the paper and code about the DBG model and I have a question about the PFG layer.

    According to the paper and the code, the parameter w_l and w_r for the the PFG layer's output at location (t_s, t_e, n, c), which corresponding to the w, h, t, c in the code if I have understood correctly, can be directly calculated under the formula (1)(2)(3)(4) in the "Proposal feature generation layer" section. But both the paper and code shows that this layer is a trainable layer. So will the w_l and w_r be updated during the backward? Since it looks like a variable that don't need to be trained rather than a trainable parameter to me and I'm confused about this. Could you please explain this? Thank you very much!

    opened by BlueDing101 2
  • Which optical flow algorithm is used?

    Which optical flow algorithm is used?

    Great work! Thanks for sharing!!

    To generate optical flow, do you use dense flow from https://github.com/yjxiong/anet2016-cuhk use denseFlow or https://github.com/feichtenhofer/gpu_flow same as BSN author? Wonder which one is better.

    opened by happyCodingSusan 2
  • When will train.py be released?

    When will train.py be released?

    Thanks for sharing the project! Recently, I was learning DBG papers and codes, really want to try training with my own dataset. May i ask the time schedule of releasing the traing codes (train.py)?

    opened by Light-- 2
  • [run error] No OpKernel was registered to support Op 'PropTcfg' with these attrs

    [run error] No OpKernel was registered to support Op 'PropTcfg' with these attrs

    Hi, I had follow the instructions to installed tf1.9.0 and python3.6, and compiled "Proposal Feature Generation Layer" successfully, which generate "prop_tcfg.so". but when run "bash auto_run.sh", following errors were occured:

    loading config file: ./config.yaml loading config file: ./config.yaml 2019-11-23 17:13:54.840138: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA 2019-11-23 17:13:54.841548: I tensorflow/core/common_runtime/process_util.cc:69] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance. Runing DBG model ... 0%| | 0/296 [00:00<?, ?it/s] Traceback (most recent call last): File "/home/anaconda2/envs/tf19/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1322, in _do_call return fn(*args) File "/home/anaconda2/envs/tf19/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1307, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "/home/anaconda2/envs/tf19/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.InvalidArgumentError: No OpKernel was registered to support Op 'PropTcfg' with these attrs. Registered devices: [CPU], Registered kernels: device='GPU'; T in [DT_FLOAT]

     [[Node: model/PropTcfg = PropTcfg[T=DT_FLOAT, center_num=16, end_num=8, mode=0, start_num=8](model/strided_slice)]]

    It seems like the Op 'PropTcfg' was not registered correctly, but How can I fix it? Thank you!

    opened by nevin007 2
  • Can you release the THUMOS14 or ActivityNet1.3 features?

    Can you release the THUMOS14 or ActivityNet1.3 features?

    Hello! Thanks for your work. Existing link is dead. I wanna kown if you can release the features extracted from THUMOS14 or ActivityNet1.3 dataset, which will help me a lot.

    opened by Traveler-Pass 1
  • THUMOS14 code can you share ?

    THUMOS14 code can you share ?

    I admire your experiment very much,but I have encountered some troubles when trying to implement the algorithm related to Thumos14. can you send me your code and the features on Thumos14? my email address : [email protected]

    opened by little-mumu 0
  • Thumos code

    Thumos code

    Hello! Thanks for your work.could you send me your code on Thumos14 please? it will help me a lot. thank you very much!!! my email address : [email protected]

    opened by aiohejiajun 0
  • Can you release the THUMOS14 features?

    Can you release the THUMOS14 features?

    Hello! Thanks for your work, I have been looking for the features extracted from THUMOS14 dataset. While I haven't seen anyone released it. I wanna kown if you can release the features, which will help me a lot.

    opened by whytheworldisbeautiful 1
  • proposal

    proposal

    Can someone give me a proposal.txt or proposal.json of DBG?My Laboratory Conditions can't make it. If you can generate proposal,can you send me one? Thanks. My email is [email protected]

    opened by xumh-9 1
  • run error

    run error

    Thanks for your work. But I met an error when I run this code:python pytorch/test.py config/config_pretrained.yaml .The error : Traceback (most recent call last): File "pytorch/test.py", line 11, in from model import DBG File "/mnt/songyan/10519_xuminhuang/ActionDetection-DBG-master/pytorch/model.py", line 4, in from custom_op.prop_tcfg_op import PropTcfg File "/mnt/songyan/10519_xuminhuang/ActionDetection-DBG-master/pytorch/custom_op/prop_tcfg_op.py", line 4, in import prop_tcfg_cuda ImportError: /mnt/songyan/10519_xuminhuang/anaconda3/lib/python3.7/site-packages/prop_tcfg_cuda-0.0.0-py3.7-linux-x86_64.egg/prop_tcfg_cuda.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __cudaRegisterFatBinaryEnd

    my tool: pytorch1.1 cuda9 python3.7 gcc5.4 Can someone help me solve this problem?thanks

    opened by xumh-9 0
Owner
Tencent
Tencent
GitHub
Allows including an action inside another action (by preprocessing the Yaml file). This is how composite actions should have worked.

actions-includes Allows including an action inside another action (by preprocessing the Yaml file). Instead of using uses or run in your action step,

Tim Ansell 70 Nov 4, 2022
Official implementation of ACTION-Net: Multipath Excitation for Action Recognition (CVPR'21).

ACTION-Net Official implementation of ACTION-Net: Multipath Excitation for Action Recognition (CVPR'21). Getting Started EgoGesture data folder struct

V-Sense 171 Dec 26, 2022
Official Pytorch Implementation of 'Learning Action Completeness from Points for Weakly-supervised Temporal Action Localization' (ICCV-21 Oral)

Learning-Action-Completeness-from-Points Official Pytorch Implementation of 'Learning Action Completeness from Points for Weakly-supervised Temporal A

Pilhyeon Lee 67 Jan 3, 2023
Human Action Controller - A human action controller running on different platforms.

Human Action Controller (HAC) Goal A human action controller running on different platforms. Fun Easy-to-use Accurate Anywhere Fun Examples Mouse Cont

null 27 Jul 20, 2022
The official TensorFlow implementation of the paper Action Transformer: A Self-Attention Model for Short-Time Pose-Based Human Action Recognition

Action Transformer A Self-Attention Model for Short-Time Human Action Recognition This repository contains the official TensorFlow implementation of t

PIC4SeRCentre 20 Jan 3, 2023
git《Commonsense Knowledge Base Completion with Structural and Semantic Context》(AAAI 2020) GitHub: [fig1]

Commonsense Knowledge Base Completion with Structural and Semantic Context Code for the paper Commonsense Knowledge Base Completion with Structural an

AI2 96 Nov 5, 2022
Scalable Attentive Sentence-Pair Modeling via Distilled Sentence Embedding (AAAI 2020) - PyTorch Implementation

Scalable Attentive Sentence-Pair Modeling via Distilled Sentence Embedding PyTorch implementation for the Scalable Attentive Sentence-Pair Modeling vi

Microsoft 25 Dec 2, 2022
Learning Dense Representations of Phrases at Scale (Lee et al., 2020)

DensePhrases DensePhrases provides answers to your natural language questions from the entire Wikipedia in real-time. While it efficiently searches th

Princeton Natural Language Processing 540 Dec 30, 2022
PyTorch implementation of CVPR 2020 paper (Reference-Based Sketch Image Colorization using Augmented-Self Reference and Dense Semantic Correspondence) and pre-trained model on ImageNet dataset

Reference-Based-Sketch-Image-Colorization-ImageNet This is a PyTorch implementation of CVPR 2020 paper (Reference-Based Sketch Image Colorization usin

Yuzhi ZHAO 11 Jul 28, 2022
Spatial Action Maps for Mobile Manipulation (RSS 2020)

spatial-action-maps Update: Please see our new spatial-intention-maps repository, which extends this work to multi-agent settings. It contains many ne

Jimmy Wu 27 Nov 30, 2022
[EMNLP 2020] Keep CALM and Explore: Language Models for Action Generation in Text-based Games

Contextual Action Language Model (CALM) and the ClubFloyd Dataset Code and data for paper Keep CALM and Explore: Language Models for Action Generation

Princeton Natural Language Processing 43 Dec 16, 2022
UDP++ (ECCVW 2020 Oral), (Winner of COCO 2020 Keypoint Challenge).

UDP-Pose This is the pytorch implementation for UDP++, which won the Fisrt place in COCO Keypoint Challenge at ECCV 2020 Workshop. Top-Down Results on

null 20 Jul 29, 2022
Official implementation of AAAI-21 paper "Label Confusion Learning to Enhance Text Classification Models"

Description: This is the official implementation of our AAAI-21 accepted paper Label Confusion Learning to Enhance Text Classification Models. The str

null 101 Nov 25, 2022
This is the code for the paper "Contrastive Clustering" (AAAI 2021)

Contrastive Clustering (CC) This is the code for the paper "Contrastive Clustering" (AAAI 2021) Dependency python>=3.7 pytorch>=1.6.0 torchvision>=0.8

Yunfan Li 210 Dec 30, 2022
git《Self-Attention Attribution: Interpreting Information Interactions Inside Transformer》(AAAI 2021) GitHub:

Self-Attention Attribution This repository contains the implementation for AAAI-2021 paper Self-Attention Attribution: Interpreting Information Intera

null 60 Dec 29, 2022
[AAAI 2021] MVFNet: Multi-View Fusion Network for Efficient Video Recognition

MVFNet: Multi-View Fusion Network for Efficient Video Recognition (AAAI 2021) Overview We release the code of the MVFNet (Multi-View Fusion Network).

Wenhao Wu 114 Nov 27, 2022
《LightXML: Transformer with dynamic negative sampling for High-Performance Extreme Multi-label Text Classification》(AAAI 2021) GitHub:

LightXML: Transformer with dynamic negative sampling for High-Performance Extreme Multi-label Text Classification

null 76 Dec 5, 2022
Code for "Graph-Evolving Meta-Learning for Low-Resource Medical Dialogue Generation". [AAAI 2021]

Graph Evolving Meta-Learning for Low-resource Medical Dialogue Generation Code to be further cleaned... This repo contains the code of the following p

Shuai Lin 29 Nov 1, 2022
:hot_pepper: R²SQL: "Dynamic Hybrid Relation Network for Cross-Domain Context-Dependent Semantic Parsing." (AAAI 2021)

R²SQL The PyTorch implementation of paper Dynamic Hybrid Relation Network for Cross-Domain Context-Dependent Semantic Parsing. (AAAI 2021) Requirement

huybery 60 Dec 31, 2022