CharacterGAN: Few-Shot Keypoint Character Animation and Reposing

Overview

CharacterGAN

Implementation of the paper "CharacterGAN: Few-Shot Keypoint Character Animation and Reposing" by Tobias Hinz, Matthew Fisher, Oliver Wang, Eli Shechtman, and Stefan Wermter (open with Adobe Acrobat or similar to see visualizations).

Supplementary material can be found here.

Our model can be trained on only a few images (e.g. 10) of a given character labeled with user-chosen keypoints. The resulting model can be used to animate the character on which it was trained by interpolating between its poses specified by their keypoints. We can also repose characters by simply moving the keypoints into the desired positions. To train the model all we need are few images depicting the character in diverse poses from the same viewpoint, keypoints, a file that describes how the keypoints are connected (the characters skeleton) and which keypoints lie in the same layer.

Examples

Animation: For all examples the model was trained on 8-15 images (see first row) of the given character.

Training Images 12 15 9 12 15 15 8
Animation dog_animation maddy_animation ostrich_animation man_animation robot_animation man_animation cow_animation



Frame interpolation: Example of interpolations between two poses with the start and end keypoints highlighted.

man man man man man man man man man man man man man
dog dog dog dog dog dog dog dog dog dog dog dog dog



Reposing: You can use our interactive GUI to easily repose a given character based on keypoints.

Interactive dog_gui man_gui
Gui cow_gui man_gui

Installation

  • python 3.8
  • pytorch 1.7.1
pip install -r requirements.txt

Training

Training Data

All training data for a given character should be in a single folder. We used this website to label our images but there are of course other possibilities.

The folder should contain:

  • all training images (all in the same resolution),
  • a file called keypoints.csv (containing the keypoints for each image),
  • a file called keypoints_skeleton.csv (containing skeleton information, i.e. how keypoints are connected with each other), and
  • a file called keypoints_layers.csv (containing the information about which layer each keypoint resides in).

The structure of the keypoints.csv file is (no header): keypoint_label,x_coord,y_coord,file_name. The first column describes the keypoint label (e.g. head), the next two columns give the location of the keypoint, and the final column states which training image this keypoint belongs to.

The structure of the keypoints_skeleton.csv file is (no header): keypoint,connected_keypoint,connected_keypoint,.... The first column describes which keypoint we are describing in this line, the following columns describe which keypoints are connected to that keypoint (e.g. elbow, shoulder, hand would state that the elbow keypoint should be connected to the shoulder keypoint and the hand keypoint).

The structure of the keypoints_layers.csv file is (no header): keypoint,layer. "Keypoint" is the keypoint label (same as used in the previous two files) and "layer" is an integer value desribing which layer the keypoint resides in.

See our example training data in datasets for examples of both files.

We provide two examples (produced by Zuzana Studená) for training, located in datasets. Our other examples were trained on data from Adobe Stock or from Character Animator and I currently have no license to distribute them. You can purchase the Stock data here:

  • Man: we used all images
  • Dog: we used all images
  • Ostrich: we used the first nine images
  • Cow: we used the first eight images

There are also several websites where you can download Sprite sheets for free.

Train a Model

To train a model with the default parameters from our paper run:

python train.py --gpu_ids 0 --num_keypoints 14 --dataroot datasets/Watercolor-Man --fp16 --name Watercolor-Man

Training one model should take about 60 (FP16) to 90 (FP32) minutes on an NVIDIA GeForce GTX 2080Ti. You can usually use fewer iterations for training and still achieve good results (see next section).

Training Parameters

You can adjust several parameters at train time to possibly improve your results.

  • --name to change the name of the folder in which the results are stored (default is CharacterGAN-Timestamp)
  • --niter 4000 and --niter_decay 4000 to adjust the number of training steps (niter_decayis the number of training steps during which we reduce the learning rate linearly; default is 8000 for both, but you can get good results with fewer iterations)
  • --mask True --output_nc 4 to train with a mask
  • --skeleton False to train without skeleton information
  • --bkg_color 0 to set the background color of the training images to black (default is white, only important if you train with a mask)
  • --batch_size 10 to train with a different batch size (default is 5)

The file options/keypoints.py lets you modify/add/remove keypoints for your characters.

Results

The output is saved to checkpoints/ and we log the training process with Tensorboard. To monitor the progress go to the respective folder and run

 tensorboard --logdir .

Testing

At test time you can either use the model to animate the character or use our interactive GUI to change the position of individual keypoints.

Animate Character

To animate a character (or create interpolations between two images):

python animate_example.py --gpu_ids 0 --model_path checkpoints/Watercolor-Man-.../ --img_animation_list datasets/Watercolor-Man/animation_list.txt --dataroot datasets/Watercolor-Man

--img_animation_list points to a file that lists the images that should be used for animation. The file should contain one file name per line pointing to an image in dataroot. The model then generates an animation by interpolating between the images in the given order. See datasets/Watercolor-Man/animation_list.txt for an example.

You can add --draw_kps to visualize the keypoints in the animation. You can specifiy the gif parameters by setting --num_interpolations 10 and --fps 5. num_interpolations specifies how many images are generated between two real images (from img_animation_list), fps determines the frames per second of the generated gif.

Modify Individual Keypoints

To run the interactive GUI:

python visualizer.py --gpu_ids 0 --model_path checkpoints/Watercolor-Man-.../

Set --gpu_ids -1 to run the model on a CPU. You can also scale the images during visualization, e.g. use --scale 2.

Patch-based Refinement

We use this implementation to run the patch-based refinement step on our generated images. The easiest way to do this is to merge all your training images into a single large image file and use this image file as the style and source image.

Acknowledgements

Our implementation uses code from Pix2PixHD, the TPS augmentation from DeepSIM, and the patch-based refinement code from https://ebsynth.com/ (GitHub).

We would also like to thank Zuzana Studená who produced some of the artwork used in this work.

Citation

If you found this code useful please consider citing:

@article{hinz2021character,
    author    = {Hinz, Tobias and Fisher, Matthew and Wang, Oliver and Shechtman, Eli and Wermter, Stefan},
    title     = {CharacterGAN: Few-Shot Keypoint Character Animation and Reposing},
    journal = {arXiv preprint arXiv:2102.03141},
    year      = {2021}
}
Issues
  • pretrained model will be released?

    pretrained model will be released?

    opened by miaoYuanyuan 1
  • Question about the output of the model

    Question about the output of the model

    The output of the model is the animated gif or there is the possibility to get the bone o junction positioning? So for example if I use my image can I put the junction on it and after that animate the image?

    Thank you.

    opened by marcomameli1992 1
  • Generated Results

    Generated Results

    This is an excellent project. I am try to train the model on the "Waterman-color" dataset available in this project. But the results are not as expected. Please let me know what could be the problem. The generated gif is shown below:

    animation_interp_81_fps_5

    opened by Rayyan-Akhtar 1
Owner
Tobias Hinz
Research Associate at University of Hamburg
Tobias Hinz
a pytorch implementation of auto-punctuation learned character by character

Learning Auto-Punctuation by Reading Engadget Articles Link to Other of my work ?? Deep Learning Notes: A collection of my notes going from basic mult

Ge Yang 130 Nov 18, 2021
a pytorch implementation of auto-punctuation learned character by character

Learning Auto-Punctuation by Reading Engadget Articles Link to Other of my work ?? Deep Learning Notes: A collection of my notes going from basic mult

Ge Yang 130 Nov 18, 2021
Few-NERD: Not Only a Few-shot NER Dataset

Few-NERD: Not Only a Few-shot NER Dataset This is the source code of the ACL-IJCNLP 2021 paper: Few-NERD: A Few-shot Named Entity Recognition Dataset.

THUNLP 193 Nov 23, 2021
OpenPose: Real-time multi-person keypoint detection library for body, face, hands, and foot estimation

Build Type Linux MacOS Windows Build Status OpenPose has represented the first real-time multi-person system to jointly detect human body, hand, facia

null 22.7k Dec 1, 2021
This is an official implementation of our CVPR 2021 paper "Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression" (https://arxiv.org/abs/2104.02300)

Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression Introduction In this paper, we are interested in the bottom-up paradigm of estima

HRNet 214 Nov 25, 2021
Code repository for paper `Skeleton Merger: an Unsupervised Aligned Keypoint Detector`.

Skeleton Merger Skeleton Merger, an Unsupervised Aligned Keypoint Detector. The paper is available at https://arxiv.org/abs/2103.10814. A map of the r

北海若 34 Nov 2, 2021
KeypointDeformer: Unsupervised 3D Keypoint Discovery for Shape Control

KeypointDeformer: Unsupervised 3D Keypoint Discovery for Shape Control Tomas Jakab, Richard Tucker, Ameesh Makadia, Jiajun Wu, Noah Snavely, Angjoo Ka

Tomas Jakab 62 Nov 9, 2021
Implementation of "JOKR: Joint Keypoint Representation for Unsupervised Cross-Domain Motion Retargeting"

JOKR: Joint Keypoint Representation for Unsupervised Cross-Domain Motion Retargeting Pytorch implementation for the paper "JOKR: Joint Keypoint Repres

null 36 Nov 12, 2021
Single-stage Keypoint-based Category-level Object Pose Estimation from an RGB Image

CenterPose Overview This repository is the official implementation of the paper "Single-stage Keypoint-based Category-level Object Pose Estimation fro

NVIDIA Research Projects 67 Nov 29, 2021
68 keypoint annotations for COFW test data

68 keypoint annotations for COFW test data This repository contains manually annotated 68 keypoints for COFW test data (original annotation of CFOW da

null 29 Aug 19, 2021
Solving SMPL/MANO parameters from keypoint coordinates.

Minimal-IK A simple and naive inverse kinematics solver for MANO hand model, SMPL body model, and SMPL-H body+hand model. Briefly, given joint coordin

Yuxiao Zhou 189 Nov 24, 2021
Ready-to-use code and tutorial notebooks to boost your way into few-shot image classification.

Easy Few-Shot Learning Ready-to-use code and tutorial notebooks to boost your way into few-shot image classification. This repository is made for you

Sicara 133 Nov 26, 2021
Code and data of the ACL 2021 paper: Few-Shot Text Ranking with Meta Adapted Synthetic Weak Supervision

MetaAdaptRank This repository provides the implementation of meta-learning to reweight synthetic weak supervision data described in the paper Few-Shot

THUNLP 4 Sep 7, 2021
Adaptive Prototype Learning and Allocation for Few-Shot Segmentation (CVPR 2021)

ASGNet The code is for the paper "Adaptive Prototype Learning and Allocation for Few-Shot Segmentation" (accepted to CVPR 2021) [arxiv] Overview data/

Gen Li 53 Nov 29, 2021
Code for 'Self-Guided and Cross-Guided Learning for Few-shot segmentation. (CVPR' 2021)'

SCL Introduction Code for 'Self-Guided and Cross-Guided Learning for Few-shot segmentation. (CVPR' 2021)' We evaluated our approach using two baseline

null 22 Oct 25, 2021
Simple and Effective Few-Shot Named Entity Recognition with Structured Nearest Neighbor Learning

structshot Code and data for paper "Simple and Effective Few-Shot Named Entity Recognition with Structured Nearest Neighbor Learning", Yi Yang and Arz

ASAPP Research 25 Nov 17, 2021
Code for our method RePRI for Few-Shot Segmentation. Paper at http://arxiv.org/abs/2012.06166

Region Proportion Regularized Inference (RePRI) for Few-Shot Segmentation In this repo, we provide the code for our paper : "Few-Shot Segmentation Wit

Malik Boudiaf 101 Nov 29, 2021
Few-shot Learning of GPT-3

Few-shot Learning With Language Models This is a codebase to perform few-shot "in-context" learning using language models similar to the GPT-3 paper.

Tony Z. Zhao 138 Nov 19, 2021
git《FSCE: Few-Shot Object Detection via Contrastive Proposal Encoding》(CVPR 2021) GitHub: [fig8]

FSCE: Few-Shot Object Detection via Contrastive Proposal Encoding (CVPR 2021) This repo contains the implementation of our state-of-the-art fewshot ob

null 143 Dec 2, 2021