Two-Stream Adaptive Graph Convolutional Networks for Skeleton-Based Action Recognition in CVPR19

Overview

2s-AGCN

Two-Stream Adaptive Graph Convolutional Networks for Skeleton-Based Action Recognition in CVPR19

Note

PyTorch version should be 0.3! For PyTorch0.4 or higher, the codes need to be modified.
Now we have updated the code to >=Pytorch0.4.
A new model named AAGCN is added, which can achieve better performance.

Data Preparation

  • Download the raw data from NTU-RGB+D and Skeleton-Kinetics. Then put them under the data directory:

     -data\  
       -kinetics_raw\  
         -kinetics_train\
           ...
         -kinetics_val\
           ...
         -kinetics_train_label.json
         -keintics_val_label.json
       -nturgbd_raw\  
         -nturgb+d_skeletons\
           ...
         -samples_with_missing_skeletons.txt
    
  • Preprocess the data with

    python data_gen/ntu_gendata.py

    python data_gen/kinetics-gendata.py.

  • Generate the bone data with:

    python data_gen/gen_bone_data.py

Training & Testing

Change the config file depending on what you want.

`python main.py --config ./config/nturgbd-cross-view/train_joint.yaml`

`python main.py --config ./config/nturgbd-cross-view/train_bone.yaml`

To ensemble the results of joints and bones, run test firstly to generate the scores of the softmax layer.

`python main.py --config ./config/nturgbd-cross-view/test_joint.yaml`

`python main.py --config ./config/nturgbd-cross-view/test_bone.yaml`

Then combine the generated scores with:

`python ensemble.py` --datasets ntu/xview

Citation

Please cite the following paper if you use this repository in your reseach.

@inproceedings{2sagcn2019cvpr,  
      title     = {Two-Stream Adaptive Graph Convolutional Networks for Skeleton-Based Action Recognition},  
      author    = {Lei Shi and Yifan Zhang and Jian Cheng and Hanqing Lu},  
      booktitle = {CVPR},  
      year      = {2019},  
}

@article{shi_skeleton-based_2019,
    title = {Skeleton-{Based} {Action} {Recognition} with {Multi}-{Stream} {Adaptive} {Graph} {Convolutional} {Networks}},
    journal = {arXiv:1912.06971 [cs]},
    author = {Shi, Lei and Zhang, Yifan and Cheng, Jian and LU, Hanqing},
    month = dec,
    year = {2019},
}

Contact

For any questions, feel free to contact: [email protected]

Issues
  • Memory overloading issue

    Memory overloading issue

    First of all, thanks a lot for making your code public. I am trying to do the experiment on NTU RGB D 120 dataset and I have split the data into training and testing in CS as given in the NTU-RGB D 120 paper. I have 63026 training samples and 54702 testing samples. I am trying to train the model on a GPU cluster but after running for one epoch, my model exceeds the memory limit: image I try to clear the cache explicitly using gc.collect but the model still continues to grow in size. It will be great if you can help regarding this.

    opened by Anirudh257 46
  • I got some wrong when I was training the net

    I got some wrong when I was training the net

    首先我是得到了下面这个error, 1

    注释掉该参数后,got another error

    I got this error ,but I don't know how to solve. Could you give me some advice?

    Traceback (most recent call last): File "/home/sues/Desktop/2s-AGCN-master/main.py", line 550, in processor.start() File "/home/sues/Desktop/2s-AGCN-master/main.py", line 491, in start self.train(epoch, save_model=save_model) File "/home/sues/Desktop/2s-AGCN-master/main.py", line 372, in train loss.backward() File "/home/sues/anaconda3/envs/2sAGCN/lib/python3.5/site-pac[kages/torch/autograd/variable.py", line 167, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph, retain_variables) File "/home/sues/anaconda3/envs/2sAGCN/lib/python3.5/site-packages/torch/autograd/init.py", line 99, in backward variables, grad_variables, retain_graph) RuntimeError: cuda runtime error (59) : device-side assert triggered at /pytorch/torch/lib/THC/generic/THCTensorMath.cu:26 /pytorch/torch/lib/THCUNN/ClassNLLCriterion.cu:101: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [0,0,0] Assertion `t >= 0 && t < n_classes 2

    opened by Dongjiuqing 10
  • augmentation in feeder

    augmentation in feeder

    Hi, I want to know the data augmentation in the feeder has not improved? Does the length of the input have a big influence? Also, have you trained the model on the 120 dataset? How's the accuracy?

    opened by VSunN 7
  • 内存分配不够,Unable to allocate 29.0 GiB for an array with shape (7790126400,) and data type float32

    内存分配不够,Unable to allocate 29.0 GiB for an array with shape (7790126400,) and data type float32

    当运行python data_gen/gen_bone_data.py这据代码时,会在 File "data_gen/gen_bone_data.py", line 62, in data = np.load('./data/{}/{}_data.npy'.format(dataset, set)) 处遇到 MemoryError: Unable to allocate 29.0 GiB for an array with shape (7790126400,) and data type float32 这样的错误,请问该如何解决呢?

    opened by XieLinMofromsomewhere 7
  • Update Pytorch

    Update Pytorch

    dear, Can you update pytorch to 1.0? I can not install pytorch=0.3 cuda101.

    enhancement 
    opened by NguyenDangBinh 6
  • problem with gen_bone_data.py

    problem with gen_bone_data.py

    你好,请问一下我跑gen_bone_data.py时报错,好像是矩阵的维度有问题,该怎么解决,谢谢 [email protected]:~/2s-AGCN-master/data_gen$ python gen_bone_data.py ntu/xsub train 4%|█▋ | 1/25 [06:49<2:43:40, 409.20s/it]Traceback (most recent call last): File "gen_bone_data.py", line 50, in fp_sp[:, :, :, v1, :] = data[:, :, :, v1, :] - data[:, :, :, v2, :] IndexError: index 20 is out of bounds for axis 3 with size 18 4%|█▋ | 1/25 [06:49<2:43:45, 409.41s/it]

    opened by JaxferZ 5
  • 关于训练时日志打印的问题

    关于训练时日志打印的问题

    为什么训练打印日志的时候,Training Epoch的进度条一下子就结束了,而Eval Epoch的进度条正常进行,似乎是没有进行训练

    1111

    opened by XieLinMofromsomewhere 5
  • Accuracy of aagcn

    Accuracy of aagcn

    I ran your implemented code using J-AAGCN and NTU-RGBD CV dataset. But Accuracy is 94.64, not 95.1 in your paper. What is the difference? The batch size was 32, not 64 because of the resource limit. Are there any other things to be aware of? I use your implemented code.

    opened by ilikeokoge 4
  • 用released model做test的时候提示 Unexpected key(s) in state_dict:

    用released model做test的时候提示 Unexpected key(s) in state_dict:

    python main.py --config ./config/nturgbd-cross-view/test_joint.yaml 这段代码能得到论文的结果。 但是到了这段 python main.py --config ./config/nturgbd-cross-view/test_bone.yaml``,会提示RuntimeError: Error(s) in loading state_dict for Model:`

    Unexpected key(s) in state_dict: "l1.gcn1.conv_res.0.weight", "l1.gcn1.conv_res.0.bias", "l1.gcn1.conv_res.1.weigh t", "l1.gcn1.conv_res.1.bias", "l1.gcn1.conv_res.1.running_mean", "l1.gcn1.conv_res.1.running_var", "l5.gcn1.conv_res.0.we ight", "l5.gcn1.conv_res.0.bias", "l5.gcn1.conv_res.1.weight", "l5.gcn1.conv_res.1.bias", "l5.gcn1.conv_res.1.running_mean ", "l5.gcn1.conv_res.1.running_var", "l8.gcn1.conv_res.0.weight", "l8.gcn1.conv_res.0.bias", "l8.gcn1.conv_res.1.weight", "l8.gcn1.conv_res.1.bias", "l8.gcn1.conv_res.1.running_mean", "l8.gcn1.conv_res.1.running_var".

    看起来是这个pretrained模型与提供的代码不匹配,我怎么做才能得到结果呢! 期待回复!

    opened by tailin1009 3
  • trained model

    trained model

    dear, Can anyone give me the trained model because my pc is weak, no gpu? Thannk you so much!

    opened by NguyenDangBinh 3
  • Where is the code for visualization in Figure 8 and 9?

    Where is the code for visualization in Figure 8 and 9?

    Dear Authors,

    I have already read your "Two-Stream Adaptive Graph Convolutional Networks for Skeleton-Based Action Recognition". In that paper, you showed some experimental results in Figure 8 and 9. I would like to know which part of the code for that. Or, how to use layers to show these result's visualization? If you answer my question, I will really appreciate you. Thank you.

    opened by JasOlean 0
  • first-commit

    first-commit

    null

    opened by zlngan 0
  • the question about max sample.

    the question about max sample.

    From your paper we can see 'The max number of frames in each sample is 300. For samples with less than 300 frames, we repeat the samples until it reaches 300 frames.' How do your method deal with the spatial and temporal discontinuities at the end of a sequence and the start of the same sequence as those would essentially be adjacent frames in this padding scheme? What happens if the authors simply use zero-padding instead?

    opened by liqier 0
  • 训练完成后,执行 testXX.yaml 出错。

    训练完成后,执行 testXX.yaml 出错。

    作者您好,根据所给源码进行训练后,模型所保存的权重仅在训练阶段进行了保存。

    但在测试时,并没有对应的weights: ./pre_train/ki_agcn.pt。 https://github.com/lshiwjx/2s-AGCN/blob/master/config/kinetics-skeleton/test_joint.yaml#L23

    同时在测试时,模型加载阶段 https://github.com/lshiwjx/2s-AGCN/blob/master/main.py#L249,由于arg.weights也不对应所以出错。请问如何解决呢?

    opened by YuQi9797 0
  • How can I get the first layer weight of AGCN?

    How can I get the first layer weight of AGCN?

    How can I obtain the first layer weights of AGCN after training? And how to show it with histogram?

    opened by JasOlean 0
  • 请问self.mask是在哪里定义的?

    请问self.mask是在哪里定义的?

    在aagcn文件中,当我打算复现A-STGCN-STC时,报了self.mask没有定义的错误,我看了下代码,确实是没有定义这个变量,不知道这个变量的作用是什么,应该在哪里先定义或者传参尽量呢?

    else: A = self.A.cuda(x.get_device()) * self.mask for i inrange(self.num_subset):A1 = A[i] A2 = x.view(N, C * T, V) z = self.conv_d[i](torch.matmul(A2, A1).view(N, C, T, V)) y = z + y if y is not None else z

    opened by youyuanrsq 0
  • Where is spatial configuration partitioning?

    Where is spatial configuration partitioning?

    You stated that you adopted the Spatial configuration partitioning from STGCN in your paper. However, I could not find the correspondence in the codes. It will be appreciated if you do not mind pointing out the correspondence in the codes for me.

    opened by DonaldKam 0
Releases(v0.0)
Owner
LShi
Video Analysis, Action Recognition.
LShi
[ICCV2021] Official code for "Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition"

CTR-GCN This repo is the official implementation for Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition. The pap

Yuxin Chen 67 Nov 25, 2021
A pytorch reproduction of { Co-occurrence Feature Learning from Skeleton Data for Action Recognition and Detection with Hierarchical Aggregation }.

A PyTorch Reproduction of HCN Co-occurrence Feature Learning from Skeleton Data for Action Recognition and Detection with Hierarchical Aggregation. Ch

Guyue Hu 197 Dec 1, 2021
Code for Two-stage Identifier: "Locate and Label: A Two-stage Identifier for Nested Named Entity Recognition"

Code for Two-stage Identifier: "Locate and Label: A Two-stage Identifier for Nested Named Entity Recognition", accepted at ACL 2021. For details of the model and experiments, please see our paper.

tricktreat 43 Nov 25, 2021
Official implementation of ACTION-Net: Multipath Excitation for Action Recognition (CVPR'21).

ACTION-Net Official implementation of ACTION-Net: Multipath Excitation for Action Recognition (CVPR'21). Getting Started EgoGesture data folder struct

V-Sense 129 Nov 30, 2021
This is the repository for the AAAI 21 paper [Contrastive and Generative Graph Convolutional Networks for Graph-based Semi-Supervised Learning].

CG3 This is the repository for the AAAI 21 paper [Contrastive and Generative Graph Convolutional Networks for Graph-based Semi-Supervised Learning]. R

null 5 Nov 27, 2021
This repository implements and evaluates convolutional networks on the Möbius strip as toy model instantiations of Coordinate Independent Convolutional Networks.

Orientation independent Möbius CNNs This repository implements and evaluates convolutional networks on the Möbius strip as toy model instantiations of

Maurice Weiler 52 Nov 17, 2021
PyTorch implementation for Convolutional Networks with Adaptive Inference Graphs

Convolutional Networks with Adaptive Inference Graphs (ConvNet-AIG) This repository contains a PyTorch implementation of the paper Convolutional Netwo

Andreas Veit 170 Nov 26, 2021
Allows including an action inside another action (by preprocessing the Yaml file). This is how composite actions should have worked.

actions-includes Allows including an action inside another action (by preprocessing the Yaml file). Instead of using uses or run in your action step,

Tim Ansell 66 Nov 30, 2021
Official Pytorch Implementation of 'Learning Action Completeness from Points for Weakly-supervised Temporal Action Localization' (ICCV-21 Oral)

Learning-Action-Completeness-from-Points Official Pytorch Implementation of 'Learning Action Completeness from Points for Weakly-supervised Temporal A

Pilhyeon Lee 39 Nov 4, 2021
Human Action Controller - A human action controller running on different platforms.

Human Action Controller (HAC) Goal A human action controller running on different platforms. Fun Easy-to-use Accurate Anywhere Fun Examples Mouse Cont

null 29 Nov 8, 2021
TDN: Temporal Difference Networks for Efficient Action Recognition

TDN: Temporal Difference Networks for Efficient Action Recognition Overview We release the PyTorch code of the TDN(Temporal Difference Networks).

Multimedia Computing Group, Nanjing University 230 Nov 29, 2021
AEI: Actors-Environment Interaction with Adaptive Attention for Temporal Action Proposals Generation

AEI: Actors-Environment Interaction with Adaptive Attention for Temporal Action Proposals Generation A pytorch-version implementation codes of paper:

null 5 Nov 21, 2021
PyTorch implementation of the R2Plus1D convolution based ResNet architecture described in the paper "A Closer Look at Spatiotemporal Convolutions for Action Recognition"

R2Plus1D-PyTorch PyTorch implementation of the R2Plus1D convolution based ResNet architecture described in the paper "A Closer Look at Spatiotemporal

Irhum Shafkat 325 Nov 24, 2021
Web service for facial landmark detection, head pose estimation, facial action unit recognition, and eye-gaze estimation based on OpenFace 2.0

OpenGaze: Web Service for OpenFace Facial Behaviour Analysis Toolkit Overview OpenFace is a fantastic tool intended for computer vision and machine le

Sayom Shakib 1 Oct 13, 2021
Code repository for paper `Skeleton Merger: an Unsupervised Aligned Keypoint Detector`.

Skeleton Merger Skeleton Merger, an Unsupervised Aligned Keypoint Detector. The paper is available at https://arxiv.org/abs/2103.10814. A map of the r

北海若 34 Nov 2, 2021
Some useful blender add-ons for SMPL skeleton's poses and global translation.

Blender add-ons for SMPL skeleton's poses and trans There are two blender add-ons for SMPL skeleton's poses and trans.The first is for making an offli

犹在镜中 20 Nov 26, 2021
Spontaneous Facial Micro Expression Recognition using 3D Spatio-Temporal Convolutional Neural Networks

Spontaneous Facial Micro Expression Recognition using 3D Spatio-Temporal Convolutional Neural Networks Abstract Facial expression recognition in video

Bogireddy Sai Prasanna Teja Reddy 78 Nov 1, 2021
PyTorch implementation of "ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context" (INTERSPEECH 2020)

ContextNet ContextNet has CNN-RNN-transducer architecture and features a fully convolutional encoder that incorporates global context information into

Sangchun Ha 9 Oct 17, 2021