Official PyTorch implementation of "Contrastive Learning from Extremely Augmented Skeleton Sequences for Self-supervised Action Recognition" in AAAI2022.

Related tags

Deep Learning AimCLR
Overview

AimCLR

This is an official PyTorch implementation of "Contrastive Learning from Extremely Augmented Skeleton Sequences for Self-supervised Action Recognition" in AAAI2022.

Requirements

Python >=3.6 PyTorch >=1.6

Data Preparation

  • Download the raw data of NTU RGB+D and PKU-MMD.
  • For NTU RGB+D dataset, preprocess data with tools/ntu_gendata.py. For PKU-MMD dataset, preprocess data with tools/pku_part1_gendata.py.
  • Then downsample the data to 50 frames with feeder/preprocess_ntu.py and feeder/preprocess_pku.py.
  • If you don't want to process the original data, download the file folder action_dataset.

Installation

# Install torchlight
$ cd torchlight
$ python setup.py install
$ cd ..

# Install other python libraries
$ pip install -r requirements.txt

Unsupervised Pre-Training

Example for unsupervised pre-training of 3s-AimCLR. You can change some settings of .yaml files in config/ntu60/pretext folder.

# train on NTU RGB+D xview joint stream
$ python main.py pretrain_aimclr --config config/ntu60/pretext/pretext_aimclr_xview_joint.yaml

# train on NTU RGB+D xview motion stream
$ python main.py pretrain_aimclr --config config/ntu60/pretext/pretext_aimclr_xview_motion.yaml

# train on NTU RGB+D xview bone stream
$ python main.py pretrain_aimclr --config config/ntu60/pretext/pretext_aimclr_xview_bone.yaml

Linear Evaluation

Example for linear evaluation of 3s-AimCLR. You can change .yaml files in config/ntu60/linear_eval folder.

# Linear_eval on NTU RGB+D xview
$ python main.py linear_evaluation --config config/ntu60/linear_eval/linear_eval_aimclr_xview_joint.yaml

$ python main.py linear_evaluation --config config/ntu60/linear_eval/linear_eval_aimclr_xview_motion.yaml

$ python main.py linear_evaluation --config config/ntu60/linear_eval/linear_eval_aimclr_xview_bone.yaml

Trained models

We release several trained models in released_model. The performance is better than that reported in the paper. You can download them and test them with linear evaluation by changing weights in .yaml files.

Model NTU 60 xsub (%) NTU 60 xview (%) PKU-MMD Part I (%)
AimCLR-joint 74.34 79.68 83.43
AimCLR-motion 68.68 71.83 72.00
AimCLR-bone 71.87 77.02 82.03
3s-AimCLR 79.18 84.02 87.79

Visualization

The t-SNE visualization of the embeddings after AimCLR pre-training on NTU60-xsub.

Citation

Please cite our paper if you find this repository useful in your resesarch:

@inproceedings{guo2022aimclr,
  Title= {Contrastive Learning from Extremely Augmented Skeleton Sequences for Self-supervised Action Recognition},
  Author= {Tianyu, Guo and Hong, Liu and Zhan, Chen and Mengyuan, Liu and Tao, Wang  and Runwei, Ding},
  Booktitle= {AAAI},
  Year= {2022}
}

Acknowledgement

The framework of our code is extended from the following repositories. We sincerely thank the authors for releasing the codes.

  • The framework of our code is based on CrosSCLR.
  • The encoder is based on ST-GCN.

Licence

This project is licensed under the terms of the MIT license.

Comments
  • Request for sharing dataset

    Request for sharing dataset

    Thanks for your time, I encounter some problems using Google Drive to download PKUMMD dataset. It can help a lot if both parts of the datasets can be shared on BaiduYun if it is convenient for you, I appreciate it a lot for your kindness.

    By the way, is the preprocessing and experiment settings same for part1 and part2? Looking forward to your reply.

    opened by zhy0860 7
  •  How to solve it that when I verify

    How to solve it that when I verify

    [04.26.22|13:33:35] Can not find weights [encoder_q.fc.bias]. [04.26.22|13:33:35] Can not find weights [encoder_q.fc.weight]. Traceback (most recent call last): File "main.py", line 50, in p = Processor(sys.argv[2:]) File "/home/lailai/lailai_file/action/AimCLR/processor/processor.py", line 60, in init self.load_data() File "/home/lailai/lailai_file/action/AimCLR/processor/processor.py", line 92, in load_data dataset=train_feeder(**self.arg.train_feeder_args), File "/home/lailai/lailai_file/action/AimCLR/feeder/ntu_feeder.py", line 17, in init self.load_data(mmap) File "/home/lailai/lailai_file/action/AimCLR/feeder/ntu_feeder.py", line 22, in load_data self.sample_name, self.label = pickle.load(f) _pickle.UnpicklingError: STACK_GLOBAL requires str

    opened by shnulailai 3
  • What is the difference between KNN and Linear evaluation?

    What is the difference between KNN and Linear evaluation?

    Your paper has classified as "Notably, the KNN classifier does not require learning extra weights compared with the linear classifier". What is the "extra weights"? Can you explain it clearly?

    opened by yanghhx 3
  • PermissionError: [Errno 13] Permission denied: '/data/gty'

    PermissionError: [Errno 13] Permission denied: '/data/gty'

    when i run: $ python main.py linear_evaluation --config config/ntu60/linear_eval/linear_eval_aimclr_xview_joint.yaml out: Traceback (most recent call last): File "main.py", line 50, in p = Processor(sys.argv[2:]) File "/home/xxx/project1/AimCLR/processor/processor.py", line 43, in init self.init_environment() File "/home/xxx/project1/AimCLR/processor/processor.py", line 76, in init_environment super().init_environment() File "/home/xxx/project1/AimCLR/processor/io.py", line 55, in init_environment self.io.save_arg(self.arg) File "/home/xxx/anaconda3/envs/TEST2/lib/python3.7/site-packages/torchlight-1.0-py3.7.egg/torchlight/io.py", line 116, in save_arg File "/home/xxx/anaconda3/envs/TEST2/lib/python3.7/os.py", line 211, in makedirs makedirs(head, exist_ok=exist_ok) File "/home/xxx/anaconda3/envs/TEST2/lib/python3.7/os.py", line 211, in makedirs makedirs(head, exist_ok=exist_ok) File "/home/xxx/anaconda3/envs/TEST2/lib/python3.7/os.py", line 211, in makedirs makedirs(head, exist_ok=exist_ok) [Previous line repeated 1 more time] File "/home/xxx/anaconda3/envs/TEST2/lib/python3.7/os.py", line 221, in makedirs mkdir(name, mode) PermissionError: [Errno 13] Permission denied: '/data/gty'

    my .yaml: work_dir: /data/gty/AAAI_github/ntu60_cv/aimclr_joint/linear_eval

    weights: /data/gty/released_model/ntu60_xview_joint.pt

    weights: /data/gty/AAAI_github/ntu60_cv/aimclr_joint/pretext/epoch300_model.pt

    ignore_weights: [encoder_q.fc, encoder_k, queue]

    feeder

    train_feeder: feeder.ntu_feeder.Feeder_single train_feeder_args: data_path: /home/wuyushan/project1/data/NTU60_frame50/xview/train_position.npy label_path: /home/wuyushan/project1/data/NTU60_frame50/xview/train_label.pkl shear_amplitude: -1 temperal_padding_ratio: -1 mmap: True test_feeder: feeder.ntu_feeder.Feeder_single test_feeder_args: data_path: /home/xxx/project1/data/NTU60_frame50/xview/val_position.npy label_path: /home/xxx/project1/data/NTU60_frame50/xview/val_label.pkl shear_amplitude: -1 temperal_padding_ratio: -1 mmap: True

    model

    model: net.aimclr.AimCLR model_args: base_encoder: net.st_gcn.Model pretrain: False

    feature_dim: 128

    queue_size: 32768

    momentum: 0.999

    Temperature: 0.07

    mlp: True

    in_channels: 3 hidden_channels: 16 hidden_dim: 256 num_class: 60 dropout: 0.5 graph_args: layout: 'ntu-rgb+d' strategy: 'spatial' edge_importance_weighting: True

    optim

    nesterov: False weight_decay: 0.0 base_lr: 3. optimizer: SGD step: [80]

    training

    device: [1] # 3 batch_size: 64 # 128 test_batch_size: 64 # 128 num_epoch: 100 stream: 'joint'

    log

    save_interval: -1 eval_interval: 5

    i don't know why,can you help me,please!

    opened by wys2929 2
  • About KNN evaluation and finetune settings

    About KNN evaluation and finetune settings

    Hi, Thank you for sharing your work! You used KNN evaluation method in your paper, but I did not find the related implementation details in the article, nor did I find the corresponding code. Would please provide some more details of the KNN evaluation? And as to the finetune setting, did you adjust the weight decay when doing finetune? Looking forward to your reply

    opened by SLime1998 2
  • How to Finetune  and Semi-supervised evaluate AimCLR?

    How to Finetune and Semi-supervised evaluate AimCLR?

    I am very interested in your work and thanks for your sharing. I'd like to know how to finetune and semi-supervised evaluate AimCLR? Have the two evaluation been included in your code? If not , could you please share them for me? My email is :[email protected]

    opened by yanghhx 2
  • Preprocessing NTU dataset

    Preprocessing NTU dataset

    Hi, thank you for sharing your work with the community! I was intersted in training a model using 150 frames long sequences, but after preprocessing the NTU dataset I ended up with just .npy files, while .pkl are needed for the label. Could you please share the code to convert the .npy files into .pkl? Thank you, Paolo

    opened by paolomandica 1
  • How to calculate the comprehensive results of the three streams

    How to calculate the comprehensive results of the three streams

    Hello, for the papers,“For all the reported results of three streams, we use the weights of [0.6, 0.6, 0.4] for weighted fusion like other multi-stream GCN methods.” But I still don’t understand, can you teach me how to calculate the comprehensive results of the three streams? For example, how did the 83.8 in the last line of the fourth column in Table II calculated? image

    opened by wys2929 2
Owner
Gty
Gty
Repository for "Improving evidential deep learning via multi-task learning," published in AAAI2022

Improving evidential deep learning via multi task learning It is a repository of AAAI2022 paper, “Improving evidential deep learning via multi-task le

deargen 11 Nov 19, 2022
ALBERT-pytorch-implementation - ALBERT pytorch implementation

ALBERT-pytorch-implementation developing... 모델의 개념이해를 돕기 위한 구현물로 현재 변수명을 상세히 적었고

BG Kim 3 Oct 6, 2022
Official PyTorch implementation for paper Context Matters: Graph-based Self-supervised Representation Learning for Medical Images

Context Matters: Graph-based Self-supervised Representation Learning for Medical Images Official PyTorch implementation for paper Context Matters: Gra

null 49 Nov 23, 2022
StyleGAN2-ADA - Official PyTorch implementation

Abstract: Training generative adversarial networks (GAN) using too little data typically leads to discriminator overfitting, causing training to diverge. We propose an adaptive discriminator augmentation mechanism that significantly stabilizes training in limited data regimes.

NVIDIA Research Projects 3.2k Dec 30, 2022
Official PyTorch implementation of Joint Object Detection and Multi-Object Tracking with Graph Neural Networks

This is the official PyTorch implementation of our paper: "Joint Object Detection and Multi-Object Tracking with Graph Neural Networks". Our project website and video demos are here.

Richard Wang 443 Dec 6, 2022
Official pytorch implementation of paper "Image-to-image Translation via Hierarchical Style Disentanglement".

HiSD: Image-to-image Translation via Hierarchical Style Disentanglement Official pytorch implementation of paper "Image-to-image Translation

null 364 Dec 14, 2022
Official pytorch implementation of paper "Inception Convolution with Efficient Dilation Search" (CVPR 2021 Oral).

IC-Conv This repository is an official implementation of the paper Inception Convolution with Efficient Dilation Search. Getting Started Download Imag

Jie Liu 111 Dec 31, 2022
Official PyTorch Implementation of Unsupervised Learning of Scene Flow Estimation Fusing with Local Rigidity

UnRigidFlow This is the official PyTorch implementation of UnRigidFlow (IJCAI2019). Here are two sample results (~10MB gif for each) of our unsupervis

Liang Liu 28 Nov 16, 2022
Official implementation of our paper "LLA: Loss-aware Label Assignment for Dense Pedestrian Detection" in Pytorch.

LLA: Loss-aware Label Assignment for Dense Pedestrian Detection This project provides an implementation for "LLA: Loss-aware Label Assignment for Dens

null 35 Dec 6, 2022
An official implementation of "SFNet: Learning Object-aware Semantic Correspondence" (CVPR 2019, TPAMI 2020) in PyTorch.

PyTorch implementation of SFNet This is the implementation of the paper "SFNet: Learning Object-aware Semantic Correspondence". For more information,

CV Lab @ Yonsei University 87 Dec 30, 2022
Old Photo Restoration (Official PyTorch Implementation)

Bringing Old Photo Back to Life (CVPR 2020 oral)

Microsoft 11.3k Dec 30, 2022
Official PyTorch implementation of Spatial Dependency Networks.

Spatial Dependency Networks: Neural Layers for Improved Generative Image Modeling Đorđe Miladinović   Aleksandar Stanić   Stefan Bauer   Jürgen Schmid

Djordje Miladinovic 34 Jan 19, 2022
Official implementation of our CVPR2021 paper "OTA: Optimal Transport Assignment for Object Detection" in Pytorch.

OTA: Optimal Transport Assignment for Object Detection This project provides an implementation for our CVPR2021 paper "OTA: Optimal Transport Assignme

null 217 Jan 3, 2023
This is the official PyTorch implementation of the paper "TransFG: A Transformer Architecture for Fine-grained Recognition" (Ju He, Jie-Neng Chen, Shuai Liu, Adam Kortylewski, Cheng Yang, Yutong Bai, Changhu Wang, Alan Yuille).

TransFG: A Transformer Architecture for Fine-grained Recognition Official PyTorch code for the paper: TransFG: A Transformer Architecture for Fine-gra

Ju He 307 Jan 3, 2023
StyleGAN2-ADA - Official PyTorch implementation

Need Help? If you’re new to StyleGAN2-ADA and looking to get started, please check out this video series from a course Lia Coleman and I taught in Oct

Derrick Schultz 217 Jan 4, 2023
Official PyTorch implementation of "ArtFlow: Unbiased Image Style Transfer via Reversible Neural Flows"

ArtFlow Official PyTorch implementation of the paper: ArtFlow: Unbiased Image Style Transfer via Reversible Neural Flows Jie An*, Siyu Huang*, Yibing

null 123 Dec 27, 2022
Official PyTorch implementation of RobustNet (CVPR 2021 Oral)

RobustNet (CVPR 2021 Oral): Official Project Webpage Codes and pretrained models will be released soon. This repository provides the official PyTorch

Sungha Choi 173 Dec 21, 2022
Official PyTorch implementation for Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers, a novel method to visualize any Transformer-based network. Including examples for DETR, VQA.

PyTorch Implementation of Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers 1 Using Colab Please notic

Hila Chefer 489 Jan 7, 2023
[PyTorch] Official implementation of CVPR2021 paper "PointDSC: Robust Point Cloud Registration using Deep Spatial Consistency". https://arxiv.org/abs/2103.05465

PointDSC repository PyTorch implementation of PointDSC for CVPR'2021 paper "PointDSC: Robust Point Cloud Registration using Deep Spatial Consistency",

null 153 Dec 14, 2022