Official PyTorch Implementation of Learning Self-Similarity in Space and Time as Generalized Motion for Video Action Recognition, ICCV 2021

Related tags

Deep Learning SELFY
Overview

Learning Self-Similarity in Space and Time as Generalized Motion for Video Action Recognition

SELFY_block

This is the official implementation of the paper "Learning Self-Similarity in Space and Time as Generalized Motion for Video Action Recognition" by H.Kwon, M.Kim, S.Kwak, and M.Cho. For more information, checkout the project website and the paper on arXiv.

Environment:

Anaconda environment setting

git clone https://github.com/arunos728/SELFY.git
cd selfy
conda env create -f environment.yml
conda activate selfy

Installing Correlation sampler

cd Pytorch-Correlation-extension
python setup.py install

# check whether SpatialCorrelationSampler is installed correctly.
python check.py forward
python check.py backward
python checkCorrelationSampler.py

Please check this repo for the detailed instructions.

Dataset preparation

Please refer to TSM repo for the detailed data preparation instructions.

File lists (.txt files in ./data) specify configurations of each video clips (path, #frames, class). We upload our Something-Something-V1 & V2 video file lists in ./data. The path of the file lists should be added into the scripts for training (or testing).

Training & Testing

  • For training SELFYNet on Something-Something, use the following command:
    ./scripts/train_SELFY_Something.sh
  • For testing your trained model on Something-Something, use the following command:
    ./scripts/test_SELFY_Something.sh

Citation

If you use this code or ideas from the paper for your research, please cite our paper:

@inproceedings{kwon2021learning,
  title={Learning self-similarity in space and time as generalized motion for video action recognition},
  author={Kwon, Heeseung and Kim, Manjin and Kwak, Suha and Cho, Minsu},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={13065--13075},
  year={2021}
}

Contact

Heeseung Kwon([email protected]), Manjin Kim([email protected])

Questions can also be left as issues in the repository. We will be happy to answer them.

You might also like...
Official implementation of NeurIPS 2021 paper
Official implementation of NeurIPS 2021 paper "Contextual Similarity Aggregation with Self-attention for Visual Re-ranking"

CSA: Contextual Similarity Aggregation with Self-attention for Visual Re-ranking PyTorch training code for CSA (Contextual Similarity Aggregation). We

Hui Wu 19 Oct 21, 2022
TensorFlow Similarity is a python package focused on making similarity learning quick and easy.
TensorFlow Similarity is a python package focused on making similarity learning quick and easy.

TensorFlow Similarity is a python package focused on making similarity learning quick and easy.

null 912 Jan 8, 2023
The official pytorch implemention of the CVPR paper "Temporal Modulation Network for Controllable Space-Time Video Super-Resolution".

This is the official PyTorch implementation of TMNet in the CVPR 2021 paper "Temporal Modulation Network for Controllable Space-Time VideoSuper-Resolu

Gang Xu 95 Oct 24, 2022
code for ICCV 2021 paper 'Generalized Source-free Domain Adaptation'

G-SFDA Code (based on pytorch 1.3) for our ICCV 2021 paper 'Generalized Source-free Domain Adaptation'. [project] [paper]. Dataset preparing Download

Shiqi Yang 84 Dec 26, 2022
[ICCV'21] Official implementation for the paper Social NCE: Contrastive Learning of Socially-aware Motion Representations
[ICCV'21] Official implementation for the paper Social NCE: Contrastive Learning of Socially-aware Motion Representations

CrowdNav with Social-NCE This is an official implementation for the paper Social NCE: Contrastive Learning of Socially-aware Motion Representations by

VITA lab at EPFL 125 Dec 23, 2022
A clean and robust Pytorch implementation of PPO on continuous action space.
A clean and robust Pytorch implementation of PPO on continuous action space.

PPO-Continuous-Pytorch I found the current implementation of PPO on continuous action space is whether somewhat complicated or not stable. And this is

XinJingHao 56 Dec 16, 2022
PyTorch implementation of the WarpedGANSpace: Finding non-linear RBF paths in GAN latent space (ICCV 2021)
PyTorch implementation of the WarpedGANSpace: Finding non-linear RBF paths in GAN latent space (ICCV 2021)

Authors official PyTorch implementation of the "WarpedGANSpace: Finding non-linear RBF paths in GAN latent space" [ICCV 2021].

Christos Tzelepis 100 Dec 6, 2022
Official PyTorch implementation of N-ImageNet: Towards Robust, Fine-Grained Object Recognition with Event Cameras (ICCV 2021)
Official PyTorch implementation of N-ImageNet: Towards Robust, Fine-Grained Object Recognition with Event Cameras (ICCV 2021)

N-ImageNet: Towards Robust, Fine-Grained Object Recognition with Event Cameras Official PyTorch implementation of N-ImageNet: Towards Robust, Fine-Gra

null 32 Dec 26, 2022
official Pytorch implementation of ICCV 2021 paper FuseFormer: Fusing Fine-Grained Information in Transformers for Video Inpainting.
official Pytorch implementation of ICCV 2021 paper FuseFormer: Fusing Fine-Grained Information in Transformers for Video Inpainting.

FuseFormer: Fusing Fine-Grained Information in Transformers for Video Inpainting By Rui Liu, Hanming Deng, Yangyi Huang, Xiaoyu Shi, Lewei Lu, Wenxiu

null 77 Dec 27, 2022
Comments
  • The accuracy of TSM-SELFY-R18/R50 on sth-sth v1

    The accuracy of TSM-SELFY-R18/R50 on sth-sth v1

    Hi, I have downloaded your official codes to reproduce the results of TSM-SELFY-R18 and R50 on the something-v1 dataset. My top-1 accuracy of the R18 model is 42.6%, which is 48.4% in your paper. And the top-1 accuracy of the R50 model is 48.3%, which is 52.5% in your paper. Attached are my log files. Did I get something wrong? selfy-tsm50.log selfy-tsm18.log

    opened by whwu95 4
  • Error messages when trying to evaluate the model on SSV2

    Error messages when trying to evaluate the model on SSV2

    Hi, thanks for the great work. I'm trying to evaluate the model on the something-something V2 (SSV2) dataset, but I still get these error messages. Here's my log file. Hope someone can provide a solution, thanks. log.txt

    opened by mm840511m 0
  • The performance on STHV1 & V2

    The performance on STHV1 & V2

    Hello whwu95, It seems that the result of your selfy-TSM18 is similar to the result of TSM18 in our paper (Table 4b), which means SELFY block does not work in your result. I think that 'Spatial Correlation Sampler' does not work in your code, so could you try again after modifying line14 of selfy.py from 'use_corr_sampler=True' to 'use_corr_sampler=False'? This is a modification about computing self-similarity without using 'Spatial Correlation Sampler'.

    Thanks for your attention. Please let me know if you need any help.

    Yes, I have utilized the 'use_corr_sampler=False', but I get only 51.467% on sthv1, which is 52.5% on the paper. Also, I got 63.45% on sthv2, which is 64.5% on the paper. Attached is my log file. Did I get something wrong? SELFY-R50-TSM-log.txt

    Originally posted by @whwu95 in https://github.com/arunos728/SELFY/issues/1#issuecomment-981380619

    opened by whwu95 0
Owner
null
GitHub
Official Pytorch implementation of the paper "Action-Conditioned 3D Human Motion Synthesis with Transformer VAE", ICCV 2021

ACTOR Official Pytorch implementation of the paper "Action-Conditioned 3D Human Motion Synthesis with Transformer VAE", ICCV 2021. Please visit our we

Mathis Petrovich 248 Dec 23, 2022
Official Pytorch Implementation of 'Learning Action Completeness from Points for Weakly-supervised Temporal Action Localization' (ICCV-21 Oral)

Learning-Action-Completeness-from-Points Official Pytorch Implementation of 'Learning Action Completeness from Points for Weakly-supervised Temporal A

Pilhyeon Lee 67 Jan 3, 2023
H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.

H2O H2O is an in-memory platform for distributed, scalable machine learning. H2O uses familiar interfaces like R, Python, Scala, Java, JSON and the Fl

H2O.ai 6.1k Jan 5, 2023
Official implementation of ACTION-Net: Multipath Excitation for Action Recognition (CVPR'21).

ACTION-Net Official implementation of ACTION-Net: Multipath Excitation for Action Recognition (CVPR'21). Getting Started EgoGesture data folder struct

V-Sense 171 Dec 26, 2022
Official PyTorch implementation of "Contrastive Learning from Extremely Augmented Skeleton Sequences for Self-supervised Action Recognition" in AAAI2022.

AimCLR This is an official PyTorch implementation of "Contrastive Learning from Extremely Augmented Skeleton Sequences for Self-supervised Action Reco

Gty 44 Dec 17, 2022
The official pytorch implementation of our paper "Is Space-Time Attention All You Need for Video Understanding?"

TimeSformer This is an official pytorch implementation of Is Space-Time Attention All You Need for Video Understanding?. In this repository, we provid

Facebook Research 1k Dec 31, 2022
Official Pytorch implementation of the paper "MotionCLIP: Exposing Human Motion Generation to CLIP Space"

MotionCLIP Official Pytorch implementation of the paper "MotionCLIP: Exposing Human Motion Generation to CLIP Space". Please visit our webpage for mor

Guy Tevet 173 Dec 26, 2022
Siamese-nn-semantic-text-similarity - A repository containing comprehensive Neural Networks based PyTorch implementations for the semantic text similarity task

Siamese Deep Neural Networks for Semantic Text Similarity PyTorch A repository c

Shahrukh Khan 32 Dec 15, 2022
Sharpened cosine similarity torch - A Sharpened Cosine Similarity layer for PyTorch

Sharpened Cosine Similarity A layer implementation for PyTorch Install At your c

Brandon Rohrer 203 Nov 30, 2022
Official PyTorch implementation of "IntegralAction: Pose-driven Feature Integration for Robust Human Action Recognition in Videos", CVPRW 2021

IntegralAction: Pose-driven Feature Integration for Robust Human Action Recognition in Videos Introduction This repo is official PyTorch implementatio

Gyeongsik Moon 29 Sep 24, 2022