This is the official Pytorch implementation of the paper "Diverse Motion Stylization for Multiple Style Domains via Spatial-Temporal Graph-Based Generative Model"

Soomin Park

Last update: Dec 16, 2022

Related tags

Overview

Diverse Motion Stylization (Official)

This is the official Pytorch implementation of this paper.

Diverse Motion Stylization for Multiple Style Domains via Spatial-Temporal Graph-Based Generative Model
Soomin Park, Deok-Kyeong Jang, and Sung-Hee Lee
In The ACM SIGGRAPH / Eurographics Symposium on Computer Animation (SCA), 2021
Appeared in: PACM on Computer Graphics and Interactive Techniques (PACMCGIT)

Paper: https://dl.acm.org/doi/pdf/10.1145/3480145
Project: http://motionlab.kaist.ac.kr/?page_id=6301

Abstract: This paper presents a novel deep learning-based framework for translating a motion into various styles within multiple domains. Our framework is a single set of generative adversarial networks that learns stylistic features from a collection of unpaired motion clips with style labels to support mapping between multiple style domains. We construct a spatio-temporal graph to model a motion sequence and employ the spatial-temporal graph convolution networks (ST-GCN) to extract stylistic properties along spatial and temporal dimensions. Through spatial-temporal modeling, our framework shows improved style translation results between significantly different actions and on a long motion sequence containing multiple actions. In addition, we first develop a mapping network for motion stylization that maps a random noise to style, which allows for generating diverse stylization results without using reference motions. Through various experiments, we demonstrate the ability of our method to generate improved results in terms of visual quality, stylistic diversity, and content preservation.

Abstract video

Click the figure to watch the teaser video.

Requirements

matplotlib == 3.4.3
numpy == 1.21.3
scipy == 1.7.1
torch == 1.10.0+cu113

Installation

Clone this repository:

git clone https://github.com/soomean/Diverse-Motion-Stylization.git
cd Diverse-Motion-Stylization

Install the dependencies:

pip install -r requirements.txt

Prepare data

Download the datasets from the following link: https://drive.google.com/drive/folders/1Anr9ouHSnZ80C9u2SB6X0f2Clzs4v7Dp?usp=sharing
Put them in the datasets directory

Download pretrained model

mkdir checkpoints
Download the pretrained model from the following link: https://drive.google.com/drive/folders/1LBNddVo9A18FUz6y4LcA6vmIv3_Bm2QN?usp=sharing
Put it in the checkpoints/[experiment_name] directory

Test pretrained model

python test.py --name [experiment_name] --mode test --load_iter 100000

Train from scratch

python train.py --name [experiment_name]

Supplementary video (full demo)

Click the figure to watch the supplementary video.

Citation

If you find our work useful, please cite our paper as below:

@article{park2021diverse,
  title={Diverse Motion Stylization for Multiple Style Domains via Spatial-Temporal Graph-Based Generative Model},
  author={Park, Soomin and Jang, Deok-Kyeong and Lee, Sung-Hee},
  journal={Proceedings of the ACM on Computer Graphics and Interactive Techniques},
  volume={4},
  number={3},
  pages={1--17},
  year={2021},
  publisher={ACM New York, NY, USA}
}

Acknowledgements

This repository contains code snippets of the following projects:
https://theorangeduck.com/page/deep-learning-framework-character-motion-synthesis-and-editing https://github.com/yysijie/st-gcn
https://github.com/clovaai/stargan-v2
https://github.com/DeepMotionEditing/deep-motion-editing

License

This work is licensed under the terms of the MIT license.

Contact

If you have any question, please feel free to contact me ([email protected]).

Comments

Training ST-GCN for evaluation

Thanks for the great work. If I'm not mistaken, you've trained ST-GCN to compute FMD and recognition accuracy as written in the paper. Do you have any plans to upload ST-GCN model trained on the datasets used in the paper? As the data format is different from the on used in the original ST-GCN, I really appreciate if you can share the code for importing the data to ST-GCN. At least, any tips for reproducing it would be very helpful. Thank you for your consideration.

opened by mkstmyk 7

Spatial Average Pooling has same implementation for pooling and unpooling?

Is this intended?

class S_AvgPool(nn.Module):
    def __init__(self):
        super(S_AvgPool, self).__init__()

    def forward(self, x, M):
        x = torch.einsum('nctv,vw->nctw', x, M)
        return x.contiguous()


class S_AvgUnpool(nn.Module):
    def __init__(self):
        super(S_AvgUnpool, self).__init__()

    def forward(self, x, M):
        x = torch.einsum('nctv,vw->nctw', x, M)
        return x.contiguous()

opened by joverwey 2

How AdaIN works?
Hi again! Please allow me to ask a newbie question. The way of calculating the mean and variance in AdaIN is a bit tricky for me. It seems to be just using nn.Linear to compute them at once and split it into gamma(variance) and beta(mean). Will it lean to compute mean and variance during the training? If so, where can I find the process to guarantee it? Thank you again in advance.

class AdaIN(nn.Module):

def __init__(self, style_dim, num_features): super(AdaIN, self).__init__() self.norm = nn.InstanceNorm2d(num_features, affine=False) self.fc = nn.Linear(style_dim, num_features * 2) def forward(self, x, s): h = self.fc(s) h = h.view(h.size(0), h.size(1), 1, 1) gamma, beta = torch.chunk(h, chunks=2, dim=1) return (1 + gamma) * self.norm(x) + beta
opened by mkstmyk 0
Questions about dividing dataset and network architecture
Hi, @soomean I appreciate your great work.

I have some questions about datasets and network.

Are there any criteria to divide the datasets 'generate' and 'classify'? According to your paper, you divided the databases into homogeneous and heterogeneous datasets. Are the 'generate' datasets equivalent to homogeneous and the 'classify' dataset equivalent to heterogeneous? Then, do the 'generate' dataset mean Xia's datasets and the 'classify' dataset mean Aberman's datasets? I am wondering if all of Xia's datasets are used in training or are used in both training and testing.

In the mapping network, the unshared layer is not used. Isn't there any difference in performance even if I add the unshared layer like Stargan v2?

Thanks for your work again.
opened by JimHeo 3
Force evaluation of iterator so that all joints are unloaded.

When loading an animation from Maya, only the first bone is loaded. The recursion doesn't happen because map is a lazy iterator in Python3. I force the evaluation by constructing a list. This should work in both Python2 and Python3.

opened by joverwey 0

This is the official Pytorch implementation of the paper "Diverse Motion Stylization for Multiple Style Domains via Spatial-Temporal Graph-Based Generative Model"

Related tags

Overview

Diverse Motion Stylization (Official)

Abstract video

Requirements

Installation

Prepare data

Download pretrained model

Test pretrained model

Train from scratch

Supplementary video (full demo)

Citation

Acknowledgements

License

Contact

Comments

Training ST-GCN for evaluation

Spatial Average Pooling has same implementation for pooling and unpooling?

How AdaIN works?

Questions about dividing dataset and network architecture

Force evaluation of iterator so that all joints are unloaded.

Owner

Soomin Park

Official pytorch implementation of paper "Image-to-image Translation via Hierarchical Style Disentanglement".

Official pytorch implementation of paper "Inception Convolution with Efficient Dilation Search" (CVPR 2021 Oral).

Official implementation of our paper "LLA: Loss-aware Label Assignment for Dense Pedestrian Detection" in Pytorch.

Official implementation of our CVPR2021 paper "OTA: Optimal Transport Assignment for Object Detection" in Pytorch.

This is the official PyTorch implementation of the paper "TransFG: A Transformer Architecture for Fine-grained Recognition" (Ju He, Jie-Neng Chen, Shuai Liu, Adam Kortylewski, Cheng Yang, Yutong Bai, Changhu Wang, Alan Yuille).

[PyTorch] Official implementation of CVPR2021 paper "PointDSC: Robust Point Cloud Registration using Deep Spatial Consistency". https://arxiv.org/abs/2103.05465

The official pytorch implementation of our paper "Is Space-Time Attention All You Need for Video Understanding?"

Official Pytorch Implementation of: "ImageNet-21K Pretraining for the Masses"(2021) paper

The official PyTorch implementation of the paper: *Xili Dai, Xiaojun Yuan, Haigang Gong, Yi Ma. "Fully Convolutional Line Parsing." *.

The repository offers the official implementation of our paper in PyTorch.

The official PyTorch implementation of recent paper - SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training

Official PyTorch implementation and pretrained models of the paper Self-Supervised Classification Network

Official Pytorch implementation of paper "Reverse Engineering of Generative Models: Inferring Model Hyperparameters from Generated Images"

Official Pytorch Implementation of: "Semantic Diversity Learning for Zero-Shot Multi-label Classification"(2021) paper

Official PyTorch implementation of the preprint paper "Stylized Neural Painting", accepted to CVPR 2021.

Official Pytorch implementation of ICLR 2018 paper Deep Learning for Physical Processes: Integrating Prior Scientific Knowledge.

This is the official pytorch implementation for the paper: Instance Similarity Learning for Unsupervised Feature Representation.

Official PyTorch implementation of the paper "Recycling Discriminator: Towards Opinion-Unaware Image Quality Assessment Using Wasserstein GAN", accepted to ACM MM 2021 BNI Track.

Official PyTorch implementation of the paper: Improving Graph Neural Network Expressivity via Subgraph Isomorphism Counting.

The official PyTorch implementation of the paper: Xili Dai, Xiaojun Yuan, Haigang Gong, Yi Ma. "Fully Convolutional Line Parsing." .