LSTC: Boosting Atomic Action Detection with Long-Short-Term Context

Related tags

Deep Learning ActionDetection-LSTC
Overview

LSTC: Boosting Atomic Action Detection with Long-Short-Term Context

This Repository contains the code on AVA of our ACM MM 2021 paper: LSTC: Boosting Atomic Action Detection with Long-Short-Term Context

Installation

See INSTALL.md for details on installing the codebase, including requirement and environment settings

Data

For data preparation and setup, our LSTC strictly follows the processing of PySlowFast, See DATASET.md for details on preparing the data.

Run the code

We take SlowFast-ResNet50 as an example

  • train the model
python3 tools/run_net.py --cfg config/AVA/SLOWFAST_32x12_R50_LFB.yaml \
    AVA.FEATURE_BANK_PATH 'path/to/feature/bank/folder' \
    TRAIN.CHECKPOINT_FILE_PATH 'path/to/pretrained/backbone' \
    OUTPUT_DIR 'path/to/output/folder'
  • test the model
python3 tools/run_net.py --cfg config/AVA/SLOWFAST_32x12_R50_LFB.yaml \
    AVA.FEATURE_BANK_PATH 'path/to/feature/bank/folder' \
    OUTPUT_DIR 'path/to/output/folder' \
    TRAIN.ENABLE False \ 
    TEST.ENABLE True

If you want to start the DDP training from command line with torch.distributed.launch, please set start_method='cmd' in tools/run_net.py

Resource

The codebase provide following resources for fast training and validation

Pretrained backbone on Kinetics

backbone dataset model type link
ResNet50 Kinetics400 Caffe2 Google Drive/Baidu Disk (Code: y1wl)
ResNet101 Kinetics600 Caffe2 Google Drive/Baidu Disk (Code: slde)

Extracted long term feature bank

backbone feature bank (LMDB) dimension
ResNet50 Google Drive 1280
ResNet101 Google Drive 2304

Checkpoint file

backbone checkpoint model type
ResNet50 Google Drive/Baidu Disk (Code: fi0s) pytorch
ResNet101 Google Drive/Baidu Disk (Code: g63o) pytorch

Acknowledgement

This codebase is built upon PySlowFast.

Citation

If you find this repository helps your research, please refer following paper

@InProceedings{Yuxi_2021_ACM,
  author = {Li, Yuxi and Zhang, Boshen and Li, Jian and Wang, Yabiao and Wang, Chengjie and Li, Jilin and Huang, Feiyue and Lin, Weiyao},
  title = {LSTC: Boosting Atomic Action Detection with Long-Short-Term Context},
  booktitle = {ACM Conference on Multimedia},
  month = {October},
  year = {2021}
}
You might also like...
CVPR2021: Temporal Context Aggregation Network for Temporal Action Proposal Refinement
CVPR2021: Temporal Context Aggregation Network for Temporal Action Proposal Refinement

Temporal Context Aggregation Network - Pytorch This repo holds the pytorch-version codes of paper: "Temporal Context Aggregation Network for Temporal

Zhiwu Qing 63 Sep 27, 2022
Allows including an action inside another action (by preprocessing the Yaml file). This is how composite actions should have worked.

actions-includes Allows including an action inside another action (by preprocessing the Yaml file). Instead of using uses or run in your action step,

Tim Ansell 70 Nov 4, 2022
Official implementation of ACTION-Net: Multipath Excitation for Action Recognition (CVPR'21).

ACTION-Net Official implementation of ACTION-Net: Multipath Excitation for Action Recognition (CVPR'21). Getting Started EgoGesture data folder struct

V-Sense 171 Dec 26, 2022
Official Pytorch Implementation of 'Learning Action Completeness from Points for Weakly-supervised Temporal Action Localization' (ICCV-21 Oral)
Official Pytorch Implementation of 'Learning Action Completeness from Points for Weakly-supervised Temporal Action Localization' (ICCV-21 Oral)

Learning-Action-Completeness-from-Points Official Pytorch Implementation of 'Learning Action Completeness from Points for Weakly-supervised Temporal A

Pilhyeon Lee 67 Jan 3, 2023
Human Action Controller - A human action controller running on different platforms.
Human Action Controller - A human action controller running on different platforms.

Human Action Controller (HAC) Goal A human action controller running on different platforms. Fun Easy-to-use Accurate Anywhere Fun Examples Mouse Cont

null 27 Jul 20, 2022
This is 2nd term discrete maths project done by UCU students that uses backtracking to solve various problems.

Backtracking Project Sponsors This is a project made by UCU students: Olha Liuba - crossword solver implementation Hanna Yershova - sudoku solver impl

Dasha 4 Oct 17, 2021
Introduction to AI assignment 1 HCM University of Technology, term 211

Sokoban Bot Introduction to AI assignment 1 HCM University of Technology, term 211 Abstract This is basically a solver for Sokoban game using Breadth-

Quang Minh 4 Dec 12, 2022
Final term project for Bayesian Machine Learning Lecture (XAI-623)
Final term project for Bayesian Machine Learning Lecture (XAI-623)

Mixquality_AL Final Term Project For Bayesian Machine Learning Lecture (XAI-623) Youtube Link The presentation is given in YoutubeLink Problem Formula

JeongEun Park 3 Jan 18, 2022
H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.

H2O H2O is an in-memory platform for distributed, scalable machine learning. H2O uses familiar interfaces like R, Python, Scala, Java, JSON and the Fl

H2O.ai 6.1k Jan 5, 2023
Comments
  • extract the features of custom data sets

    extract the features of custom data sets

    How to extract the features of custom data sets? Can you give me some advice? I ran the script and added yaml configuration file, and there was an error that the dimensions could not be spliced. How did you extract the features at that time? Do you need to resize the images? Can the resized features still be used as feature bank?

    opened by yan-ctrl 0
  • extract the feature bank of the data set

    extract the feature bank of the data set

    Hello, when I want to extract the feature bank of the data set, I encountered a new problem. My yaml configuration file and the error I encountered are like this. Is the pre-training weight wrong? I hope you can give me an answer. SLOWFAST_32x2_BANK1.yaml: TRAIN: ENABLE: False DATASET: ava BATCH_SIZE: 64 EVAL_PERIOD: 20 CHECKPOINT_PERIOD: 1 AUTO_RESUME: True CHECKPOINT_FILE_PATH: "/home/tuxiangone/bang/Behavior_Model/LSTC/pretrained/lstc-resnet50.pyth" DATA: NUM_FRAMES: 32 SAMPLING_RATE: 2 TRAIN_JITTER_SCALES: [256, 320] TRAIN_CROP_SIZE: 224 TEST_CROP_SIZE: 224 INPUT_CHANNEL_NUM: [3, 3] DETECTION: ENABLE: True ALIGNED: True AVA: FEATURE_EXTRACTION: True FRAME_DIR: '/home/tuxiangone/bang/Behavior_Model/SlowFast/data/ava/frames' FRAME_LIST_DIR: '/home/tuxiangone/bang/Behavior_Model/SlowFast/data/ava/frame_lists' ANNOTATION_DIR: '/home/tuxiangone/bang/Behavior_Model/SlowFast/data/ava/annotations' DETECTION_SCORE_THRESH: 0.8 TRAIN_PREDICT_BOX_LISTS: [ "ava_train_v2.2.csv", "ava_train_predicted_boxes.csv", ] TEST_PREDICT_BOX_LISTS: ["ava_val_predicted_boxes.csv"] TEST_GT_BOX_LISTS: ["ava_val_v2.2.csv"] FEATURE_BANK_PATH: "output/feature_bank" SLIDING_WINDOW_SIZE: 15 GATHER_BANK: False SLOWFAST: ALPHA: 4 BETA_INV: 8 FUSION_CONV_CHANNEL_RATIO: 2 FUSION_KERNEL_SZ: 7 RESNET: ZERO_INIT_FINAL_BN: True WIDTH_PER_GROUP: 64 NUM_GROUPS: 1 DEPTH: 50 TRANS_FUNC: bottleneck_transform STRIDE_1X1: False NUM_BLOCK_TEMP_KERNEL: [[3, 3], [4, 4], [6, 6], [3, 3]] SPATIAL_DILATIONS: [[1, 1], [1, 1], [1, 1], [2, 2]] SPATIAL_STRIDES: [[1, 1], [2, 2], [2, 2], [1, 1]] NONLOCAL: LOCATION: [[[], []], [[], []], [[], []], [[], []]] GROUP: [[1, 1], [1, 1], [1, 1], [1, 1]] INSTANTIATION: dot_product POOL: [[[1, 2, 2], [1, 2, 2]], [[1, 2, 2], [1, 2, 2]], [[1, 2, 2], [1, 2, 2]], [[1, 2, 2], [1, 2, 2]]] BN: USE_PRECISE_STATS: False FREEZE: False NUM_BATCHES_PRECISE: 200 SOLVER: BASE_LR: 0.1 LR_POLICY: steps_with_relative_lrs STEPS: [0, 10, 15, 20] LRS: [1, 0.1, 0.01, 0.001] MAX_EPOCH: 20 MOMENTUM: 0.9 WEIGHT_DECAY: 1e-7 WARMUP_EPOCHS: 5.0 WARMUP_START_LR: 0.000125 OPTIMIZING_METHOD: sgd MODEL: NUM_CLASSES: 80 ARCH: slowfast MODEL_NAME: BankContext LOSS_FUNC: bce DROPOUT_RATE: 0.5 HEAD_ACT: sigmoid TEST: ENABLE: True DATASET: ava BATCH_SIZE: 8 CHECKPOINT_FILE_PATH: "/home/tuxiangone/bang/Behavior_Model/LSTC/pretrained/SLOWFAST_8x8_R50_KINETICS.pkl" DATA_LOADER: NUM_WORKERS: 2 PIN_MEMORY: True NUM_GPUS: 1 NUM_SHARDS: 1 RNG_SEED: 0 OUTPUT_DIR: "output/raw_bank"

    CACHE:

    ENABLE: True

    LOG_MODEL_INFO: False My error: [INFO: checkpoint.py: 401]: loading checkpoint from /home/tuxiangone/bang/Behavior_Model/LSTC/pretrained/SLOWFAST_8x8_R50_KINETICS.pkl [09/01 16:48:34][INFO] slowfast.utils.checkpoint: 401: loading checkpoint from /home/tuxiangone/bang/Behavior_Model/LSTC/pretrained/SLOWFAST_8x8_R50_KINETICS.pkl Traceback (most recent call last): File "tools/extract_feature.py", line 175, in launch_job( File "/home/tuxiangone/bang/Behavior_Model/LSTC/build/lib/slowfast/utils/misc.py", line 307, in launch_job func(cfg=cfg) File "tools/extract_feature.py", line 144, in extract_feature cu.load_test_checkpoint(cfg, model) File "/home/tuxiangone/bang/Behavior_Model/LSTC/build/lib/slowfast/utils/checkpoint.py", line 405, in load_test_checkpoint load_checkpoint( File "/home/tuxiangone/bang/Behavior_Model/LSTC/build/lib/slowfast/utils/checkpoint.py", line 268, in load_checkpoint checkpoint = torch.load(f, map_location="cpu") File "/home/tuxiangone/anaconda3/envs/pytorch/lib/python3.8/site-packages/torch/serialization.py", line 608, in load return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args) File "/home/tuxiangone/anaconda3/envs/pytorch/lib/python3.8/site-packages/torch/serialization.py", line 777, in _legacy_load magic_number = pickle_module.load(f, **pickle_load_args) UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd4 in position 1: invalid continuation byte

    opened by yan-ctrl 3
  • module name

    module name"lmdb "

    Hello, I reported an error during training, no module name"lmdb ".Could you please send me this script? I carefully read setup.py, which is basically the same as that of SlowFast, and there is no information about the generation of lmdb package, so it can't be trained normally. Can you update this library?

    opened by yan-ctrl 0
  • feature bank (LMDB)

    feature bank (LMDB)

    I like your paper very much. Can you tell me how to get the feature bank (LMDB)? I want to apply your work to my own data set. Do you run the script extract_feature.py directly or extract it like LFB?

    opened by yan-ctrl 9
Owner
Tencent YouTu Research
Tencent YouTu Research
GitHub
The official TensorFlow implementation of the paper Action Transformer: A Self-Attention Model for Short-Time Pose-Based Human Action Recognition

Action Transformer A Self-Attention Model for Short-Time Human Action Recognition This repository contains the official TensorFlow implementation of t

PIC4SeRCentre 20 Jan 3, 2023
[ECCVW2020] Robust Long-Term Object Tracking via Improved Discriminative Model Prediction (RLT-DiMP)

Feel free to visit my homepage Robust Long-Term Object Tracking via Improved Discriminative Model Prediction (RLT-DIMP) [ECCVW2020 paper] Presentation

Seokeon Choi 35 Oct 26, 2022
Synthesizing Long-Term 3D Human Motion and Interaction in 3D in CVPR2021

Long-term-Motion-in-3D-Scenes This is an implementation of the CVPR'21 paper "Synthesizing Long-Term 3D Human Motion and Interaction in 3D". Please ch

Jiashun Wang 76 Dec 13, 2022
LONG-TERM SERIES FORECASTING WITH QUERYSELECTOR – EFFICIENT MODEL OF SPARSEATTENTION

Query Selector Here you can find code and data loaders for the paper https://arxiv.org/pdf/2107.08687v1.pdf . Query Selector is a novel approach to sp

MORAI 62 Dec 17, 2022
Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting

Autoformer (NeurIPS 2021) Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting Time series forecasting is a c

THUML @ Tsinghua University 847 Jan 8, 2023
Multi-resolution SeqMatch based long-term Place Recognition

MRS-SLAM for long-term place recognition In this work, we imply an multi-resolution sambling based visual place recognition method. This work is based

METASLAM 6 Dec 6, 2022
Official implementation of Long-Short Transformer in PyTorch.

Long-Short Transformer (Transformer-LS) This repository hosts the code and models for the paper: Long-Short Transformer: Efficient Transformers for La

NVIDIA Corporation 198 Dec 29, 2022
PyTorch implementation of Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation.

ALiBi PyTorch implementation of Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. Quickstart Clone this reposit

Jake Tae 4 Jul 27, 2022
ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs

(Comet-) ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs Paper Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sa

AI2 152 Dec 27, 2022
Using pretrained GROVER to extract the atomic fingerprints from molecule

Extracting atomic fingerprints from molecules using pretrained Graph Neural Network models (GROVER).

Xuan Vu Nguyen 1 Jan 28, 2022