This is the official Pytorch implementation of the paper "Diverse Motion Stylization for Multiple Style Domains via Spatial-Temporal Graph-Based Generative Model"

Overview

Diverse Motion Stylization (Official)

This is the official Pytorch implementation of this paper.

teaser

Diverse Motion Stylization for Multiple Style Domains via Spatial-Temporal Graph-Based Generative Model
Soomin Park, Deok-Kyeong Jang, and Sung-Hee Lee
In The ACM SIGGRAPH / Eurographics Symposium on Computer Animation (SCA), 2021
Appeared in: PACM on Computer Graphics and Interactive Techniques (PACMCGIT)

Paper: https://dl.acm.org/doi/pdf/10.1145/3480145
Project: http://motionlab.kaist.ac.kr/?page_id=6301

Abstract: This paper presents a novel deep learning-based framework for translating a motion into various styles within multiple domains. Our framework is a single set of generative adversarial networks that learns stylistic features from a collection of unpaired motion clips with style labels to support mapping between multiple style domains. We construct a spatio-temporal graph to model a motion sequence and employ the spatial-temporal graph convolution networks (ST-GCN) to extract stylistic properties along spatial and temporal dimensions. Through spatial-temporal modeling, our framework shows improved style translation results between significantly different actions and on a long motion sequence containing multiple actions. In addition, we first develop a mapping network for motion stylization that maps a random noise to style, which allows for generating diverse stylization results without using reference motions. Through various experiments, we demonstrate the ability of our method to generate improved results in terms of visual quality, stylistic diversity, and content preservation.

Abstract video

Click the figure to watch the teaser video.
abstract

Requirements

  • matplotlib == 3.4.3
  • numpy == 1.21.3
  • scipy == 1.7.1
  • torch == 1.10.0+cu113

Installation

Clone this repository:

git clone https://github.com/soomean/Diverse-Motion-Stylization.git
cd Diverse-Motion-Stylization

Install the dependencies:

pip install -r requirements.txt

Prepare data

  1. Download the datasets from the following link: https://drive.google.com/drive/folders/1Anr9ouHSnZ80C9u2SB6X0f2Clzs4v7Dp?usp=sharing
  2. Put them in the datasets directory

Download pretrained model

  1. mkdir checkpoints
  2. Download the pretrained model from the following link: https://drive.google.com/drive/folders/1LBNddVo9A18FUz6y4LcA6vmIv3_Bm2QN?usp=sharing
  3. Put it in the checkpoints/[experiment_name] directory

Test pretrained model

python test.py --name [experiment_name] --mode test --load_iter 100000

Train from scratch

python train.py --name [experiment_name]

Supplementary video (full demo)

Click the figure to watch the supplementary video.
supp

Citation

If you find our work useful, please cite our paper as below:

@article{park2021diverse,
  title={Diverse Motion Stylization for Multiple Style Domains via Spatial-Temporal Graph-Based Generative Model},
  author={Park, Soomin and Jang, Deok-Kyeong and Lee, Sung-Hee},
  journal={Proceedings of the ACM on Computer Graphics and Interactive Techniques},
  volume={4},
  number={3},
  pages={1--17},
  year={2021},
  publisher={ACM New York, NY, USA}
}

Acknowledgements

This repository contains code snippets of the following projects:
https://theorangeduck.com/page/deep-learning-framework-character-motion-synthesis-and-editing https://github.com/yysijie/st-gcn
https://github.com/clovaai/stargan-v2
https://github.com/DeepMotionEditing/deep-motion-editing

License

This work is licensed under the terms of the MIT license.

Contact

If you have any question, please feel free to contact me ([email protected]).

Comments
  • Training ST-GCN for evaluation

    Training ST-GCN for evaluation

    Thanks for the great work. If I'm not mistaken, you've trained ST-GCN to compute FMD and recognition accuracy as written in the paper. Do you have any plans to upload ST-GCN model trained on the datasets used in the paper? As the data format is different from the on used in the original ST-GCN, I really appreciate if you can share the code for importing the data to ST-GCN. At least, any tips for reproducing it would be very helpful. Thank you for your consideration.

    opened by mkstmyk 7
  • Spatial Average Pooling has same implementation for pooling and unpooling?

    Spatial Average Pooling has same implementation for pooling and unpooling?

    Is this intended?

    class S_AvgPool(nn.Module):
        def __init__(self):
            super(S_AvgPool, self).__init__()
    
        def forward(self, x, M):
            x = torch.einsum('nctv,vw->nctw', x, M)
            return x.contiguous()
    
    
    class S_AvgUnpool(nn.Module):
        def __init__(self):
            super(S_AvgUnpool, self).__init__()
    
        def forward(self, x, M):
            x = torch.einsum('nctv,vw->nctw', x, M)
            return x.contiguous()
    
    opened by joverwey 2
  • How AdaIN works?

    How AdaIN works?

    Hi again! Please allow me to ask a newbie question. The way of calculating the mean and variance in AdaIN is a bit tricky for me. It seems to be just using nn.Linear to compute them at once and split it into gamma(variance) and beta(mean). Will it lean to compute mean and variance during the training? If so, where can I find the process to guarantee it? Thank you again in advance.

    class AdaIN(nn.Module):

    def __init__(self, style_dim, num_features):
        super(AdaIN, self).__init__()
        self.norm = nn.InstanceNorm2d(num_features, affine=False)
        self.fc = nn.Linear(style_dim, num_features * 2)
    
    def forward(self, x, s):
        h = self.fc(s)
        h = h.view(h.size(0), h.size(1), 1, 1)
        gamma, beta = torch.chunk(h, chunks=2, dim=1)
        return (1 + gamma) * self.norm(x) + beta
    
    opened by mkstmyk 0
  • Questions about dividing dataset and network architecture

    Questions about dividing dataset and network architecture

    Hi, @soomean I appreciate your great work.

    I have some questions about datasets and network.

    1. Are there any criteria to divide the datasets 'generate' and 'classify'? According to your paper, you divided the databases into homogeneous and heterogeneous datasets. Are the 'generate' datasets equivalent to homogeneous and the 'classify' dataset equivalent to heterogeneous? Then, do the 'generate' dataset mean Xia's datasets and the 'classify' dataset mean Aberman's datasets? I am wondering if all of Xia's datasets are used in training or are used in both training and testing.

    2. In the mapping network, the unshared layer is not used. Isn't there any difference in performance even if I add the unshared layer like Stargan v2?

    Thanks for your work again.

    opened by JimHeo 3
  • Force evaluation of iterator so that all joints are unloaded.

    Force evaluation of iterator so that all joints are unloaded.

    When loading an animation from Maya, only the first bone is loaded. The recursion doesn't happen because map is a lazy iterator in Python3. I force the evaluation by constructing a list. This should work in both Python2 and Python3.

    opened by joverwey 0
Owner
Soomin Park
Soomin Park
Official pytorch implementation of paper "Image-to-image Translation via Hierarchical Style Disentanglement".

HiSD: Image-to-image Translation via Hierarchical Style Disentanglement Official pytorch implementation of paper "Image-to-image Translation

null 364 Dec 14, 2022
Official pytorch implementation of paper "Inception Convolution with Efficient Dilation Search" (CVPR 2021 Oral).

IC-Conv This repository is an official implementation of the paper Inception Convolution with Efficient Dilation Search. Getting Started Download Imag

Jie Liu 111 Dec 31, 2022
Official implementation of our paper "LLA: Loss-aware Label Assignment for Dense Pedestrian Detection" in Pytorch.

LLA: Loss-aware Label Assignment for Dense Pedestrian Detection This project provides an implementation for "LLA: Loss-aware Label Assignment for Dens

null 35 Dec 6, 2022
Official implementation of our CVPR2021 paper "OTA: Optimal Transport Assignment for Object Detection" in Pytorch.

OTA: Optimal Transport Assignment for Object Detection This project provides an implementation for our CVPR2021 paper "OTA: Optimal Transport Assignme

null 217 Jan 3, 2023
This is the official PyTorch implementation of the paper "TransFG: A Transformer Architecture for Fine-grained Recognition" (Ju He, Jie-Neng Chen, Shuai Liu, Adam Kortylewski, Cheng Yang, Yutong Bai, Changhu Wang, Alan Yuille).

TransFG: A Transformer Architecture for Fine-grained Recognition Official PyTorch code for the paper: TransFG: A Transformer Architecture for Fine-gra

Ju He 307 Jan 3, 2023
[PyTorch] Official implementation of CVPR2021 paper "PointDSC: Robust Point Cloud Registration using Deep Spatial Consistency". https://arxiv.org/abs/2103.05465

PointDSC repository PyTorch implementation of PointDSC for CVPR'2021 paper "PointDSC: Robust Point Cloud Registration using Deep Spatial Consistency",

null 153 Dec 14, 2022
The official pytorch implementation of our paper "Is Space-Time Attention All You Need for Video Understanding?"

TimeSformer This is an official pytorch implementation of Is Space-Time Attention All You Need for Video Understanding?. In this repository, we provid

Facebook Research 1k Dec 31, 2022
Official Pytorch Implementation of: "ImageNet-21K Pretraining for the Masses"(2021) paper

ImageNet-21K Pretraining for the Masses Paper | Pretrained models Official PyTorch Implementation Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, Lihi Zelni

null 574 Jan 2, 2023
The official PyTorch implementation of the paper: *Xili Dai, Xiaojun Yuan, Haigang Gong, Yi Ma. "Fully Convolutional Line Parsing." *.

F-Clip — Fully Convolutional Line Parsing This repository contains the official PyTorch implementation of the paper: *Xili Dai, Xiaojun Yuan, Haigang

Xili Dai 115 Dec 28, 2022
The repository offers the official implementation of our paper in PyTorch.

Cloth Interactive Transformer (CIT) Cloth Interactive Transformer for Virtual Try-On Bin Ren1, Hao Tang1, Fanyang Meng2, Runwei Ding3, Ling Shao4, Phi

Bingoren 49 Dec 1, 2022
The official PyTorch implementation of recent paper - SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training

This repository is the official PyTorch implementation of SAINT. Find the paper on arxiv SAINT: Improved Neural Networks for Tabular Data via Row Atte

Gowthami Somepalli 284 Dec 21, 2022
Official PyTorch implementation and pretrained models of the paper Self-Supervised Classification Network

Self-Classifier: Self-Supervised Classification Network Official PyTorch implementation and pretrained models of the paper Self-Supervised Classificat

Elad Amrani 24 Dec 21, 2022
Official Pytorch implementation of paper "Reverse Engineering of Generative Models: Inferring Model Hyperparameters from Generated Images"

Reverse_Engineering_GMs Official Pytorch implementation of paper "Reverse Engineering of Generative Models: Inferring Model Hyperparameters from Gener

null 100 Dec 18, 2022
Official Pytorch Implementation of: "Semantic Diversity Learning for Zero-Shot Multi-label Classification"(2021) paper

Semantic Diversity Learning for Zero-Shot Multi-label Classification Paper Official PyTorch Implementation Avi Ben-Cohen, Nadav Zamir, Emanuel Ben Bar

null 28 Aug 29, 2022
Official PyTorch implementation of the preprint paper "Stylized Neural Painting", accepted to CVPR 2021.

Official PyTorch implementation of the preprint paper "Stylized Neural Painting", accepted to CVPR 2021.

Zhengxia Zou 1.5k Dec 28, 2022
Official Pytorch implementation of ICLR 2018 paper Deep Learning for Physical Processes: Integrating Prior Scientific Knowledge.

Deep Learning for Physical Processes: Integrating Prior Scientific Knowledge: Official Pytorch implementation of ICLR 2018 paper Deep Learning for Phy

emmanuel 47 Nov 6, 2022
This is the official pytorch implementation for the paper: Instance Similarity Learning for Unsupervised Feature Representation.

ISL This is the official pytorch implementation for the paper: Instance Similarity Learning for Unsupervised Feature Representation, which is accepted

null 19 May 4, 2022
Official PyTorch implementation of the paper "Recycling Discriminator: Towards Opinion-Unaware Image Quality Assessment Using Wasserstein GAN", accepted to ACM MM 2021 BNI Track.

RecycleD Official PyTorch implementation of the paper "Recycling Discriminator: Towards Opinion-Unaware Image Quality Assessment Using Wasserstein GAN

Yunan Zhu 23 Nov 5, 2022
Official PyTorch implementation of the paper: Improving Graph Neural Network Expressivity via Subgraph Isomorphism Counting.

Improving Graph Neural Network Expressivity via Subgraph Isomorphism Counting Official PyTorch implementation of the paper: Improving Graph Neural Net

Giorgos Bouritsas 58 Dec 31, 2022