GANimation: Anatomically-aware Facial Animation from a Single Image (ECCV'18 Oral) [PyTorch]

Overview

GANimation: Anatomically-aware Facial Animation from a Single Image

[Project] [Paper]

Official implementation of GANimation. In this work we introduce a novel GAN conditioning scheme based on Action Units (AU) annotations, which describe in a continuous manifold the anatomical facial movements defining a human expression. Our approach permits controlling the magnitude of activation of each AU and combine several of them. For more information please refer to the paper.

This code was made public to share our research for the benefit of the scientific community. Do NOT use it for immoral purposes.

GANimation

Prerequisites

  • Install PyTorch (version 0.3.1), Torch Vision and dependencies from http://pytorch.org
  • Install requirements.txt (pip install -r requirements.txt)

Data Preparation

The code requires a directory containing the following files:

  • imgs/: folder with all image
  • aus_openface.pkl: dictionary containing the images action units.
  • train_ids.csv: file containing the images names to be used to train.
  • test_ids.csv: file containing the images names to be used to test.

An example of this directory is shown in sample_dataset/.

To generate the aus_openface.pkl extract each image Action Units with OpenFace and store each output in a csv file the same name as the image. Then run:

python data/prepare_au_annotations.py

Run

To train:

bash launch/run_train.sh

To test:

python test --input_path path/to/img

Citation

If you use this code or ideas from the paper for your research, please cite our paper:

@article{Pumarola_ijcv2019,
    title={GANimation: One-Shot Anatomically Consistent Facial Animation},
    author={A. Pumarola and A. Agudo and A.M. Martinez and A. Sanfeliu and F. Moreno-Noguer},
    booktitle={International Journal of Computer Vision (IJCV)},
    year={2019}
}
Comments
  • Training data

    Training data

    Hi, thank you very much for sharing your training and testing codes. Could you please also share the training data? I have applied for the training data on the website but still get no reply. Thank you very much.

    opened by staceycy 17
  • The test result is wrong

    The test result is wrong

    We use CK+ dataset, but when I use test dataset image to test ,the result doesn’t expect,The following is the process of my train. Can you give me some suggestions?

    1、Train GANimation image image image

    2、The GPU working status image

    3、The test result image image

    4、Train loss image image image

    5、The train image result image

    opened by shliang0603 13
  • Generation AU Problem

    Generation AU Problem

    First,I use the below command line to generation AU E:\MagicProject\code\OpenFace_2.0.4_win_x64>FeatureExtraction.exe -f E:\Dataset_processing\20180821_GANimation_CK+\imgs\S005_001_00000001.png -out_dir E:\out

    I generate CSV file of AU is from 679 column to 696,like below image

    Finally, I use this to train GANimation. But my train result is not expectation. I suspect that I generation is false.

    Today ,I use FeatureExtraction.exe to generate AU ,but I choice different parameter,the command line and generation CSV of Au in below

    1. generation AU from column 6 to column 25 E:\MagicProject\code\OpenFace_2.0.4_win_x64>FeatureExtraction.exe -f E:\Dataset_processing\20180821_GANimation_CK+\imgs\S005_001_00000001.png -aus -out_dir E:\

    image

    E:\MagicProject\code\OpenFace_2.0.4_win_x64>FeatureExtraction.exe -f E:\Dataset_processing\20180821_GANimation_CK+\imgs\S005_001_00000001.png -aus -au_static -out_dir E:\out

    image

    But ,today I use 1 and 2 openface command line to generate AU ,the result is different. (I used the same picture for the AU generation test)

    Can you tell me how to use the right openface command parameter to generate CSV of AU.

    opened by shliang0603 11
  • RuntimeError: cannot unsqueeze empty tensor

    RuntimeError: cannot unsqueeze empty tensor

    Hi, everyone!

    I came across this error when running the train process for several epochs. I know that it is a PyTorch error, however I want to know is there some problems with my dataset that raise this error, and whether this error can be resolved by this project.

    I tried to track the problem, this was caused by "src->nDimention <= 0" in PyTorch, but I don't know what was wrong with my data.

    Any reply will be appreciated!

    opened by XiangyuWu 11
  • Segfault during model training and bad results

    Segfault during model training and bad results

    I tried running the training on about 110k images. I used all the default parameters for the training. But after around 11 epochs, I get a segfault in the training. I don't know where this error is originating from.

    Also, since I had 11 epochs already saved, I just tried running a test on the sample images provided to see the performance. But the results seems very bad. Did you notice something similar? Is it because of the epoch that I used and it can be resolved with using a higher epoch?

    Along the same lines, what would you say as the optimum epoch for the training? The default is about 30 epochs.

    n_0000000437_00540 jpg_out

    Thank you for your time.

    opened by Morpheus3000 7
  • more problem.

    more problem.

    python train.py --data_dir mydata --name experiment_1 --batch_size 25 ------------ Options ------------- D_adam_b1: 0.5 D_adam_b2: 0.999 G_adam_b1: 0.5 G_adam_b2: 0.999 aus_file: aus_openface.pkl batch_size: 25 checkpoints_dir: ./checkpoints cond_nc: 17 data_dir: mydata dataset_mode: aus display_freq_s: 300 do_saturate_mask: False gpu_ids: [0] image_size: 128 images_folder: imgs is_train: True lambda_D_cond: 4000 lambda_D_gp: 10 lambda_D_prob: 1 lambda_cyc: 10 lambda_mask: 0.1 lambda_mask_smooth: 1e-05 load_epoch: 0 lr_D: 0.0001 lr_G: 0.0001 model: ganimation n_threads_test: 1 n_threads_train: 4 name: experiment_1 nepochs_decay: 10 nepochs_no_decay: 20 num_iters_validate: 1 poses_g_sigma: 0.06 print_freq_s: 60 save_latest_freq_s: 3600 serial_batches: False test_ids_file: test_ids.csv train_G_every_n_iterations: 5 train_ids_file: train_ids.csv -------------- End ---------------- ./checkpoints/experiment_1 Dataset AusDataset was created Dataset AusDataset was created #train images = 91 #test images = 91 Network generator_wasserstein_gan was created Network discriminator_wasserstein_gan was created Model GANimation was created Traceback (most recent call last): File "train.py", line 141, in Train() File "train.py", line 27, in init self._train() File "train.py", line 40, in _train self._train_epoch(i_epoch) File "train.py", line 69, in _train_epoch self._model.optimize_parameters(keep_data_for_visuals=do_visuals, train_generator=train_generator) File "/data/GANimation/models/ganimation.py", line 207, in optimize_parameters loss_D, fake_imgs_masked = self._forward_D() File "/data/GANimation/models/ganimation.py", line 278, in _forward_D d_real_img_prob, d_real_img_cond = self._D.forward(self._real_img) File "/data/GANimation/networks/discriminator_wasserstein_gan.py", line 29, in forward out_aux = self.conv2(h) File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 477, in call Traceback (most recent call last): File "/usr/lib/python2.7/multiprocessing/util.py", line 274, in _run_finalizers result = self.forward(*input, **kwargs) File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/conv.py", line 301, in forward finalizer() File "/usr/lib/python2.7/multiprocessing/util.py", line 207, in call res = self._callback(*self._args, **self._kwargs) File "/usr/lib/python2.7/shutil.py", line 252, in rmtree self.padding, self.dilation, self.groups) RuntimeError: sizes must be non-negative

    opened by jackylee1 7
  •  Zero-sized dimensions are not supported in PyTorch

    Zero-sized dimensions are not supported in PyTorch

    saving the latest model (epoch 3, total_steps 375825) saved net: ./checkpoints/experiment_1/net_epoch_3_id_G.pth saved net: ./checkpoints/experiment_1/net_epoch_3_id_D.pth [27/08/2018 18:08:54] (T, epoch: 3, it: 121/7516, t/smpl: 0.017s) g_fake:0.000 g_cond:0.000 g_mskd_fake:-0.242 g_mskd_cond:3.193 g_cyc:1.215 g_rgb:0.000 g_rgb_un:0.000 g_rgb_s:0.000 g_m1:0.067 g_m2:0.067 g_m1_s:0.134 g_m2_s:0.129 g_idt:0.000 d_real:-0.856 d_cond:2.661 d_fake:0.491 d_gp:0.107 Traceback (most recent call last): File "train.py", line 141, in Train() File "train.py", line 27, in init self._train() File "train.py", line 40, in _train self._train_epoch(i_epoch) File "train.py", line 59, in _train_epoch for i_train_batch, train_batch in enumerate(self._dataset_train): File "/appl/opt/python/mlpython-2.7.10/site-packages/torch/utils/data/dataloader.py", line 210, in next return self._process_next_batch(batch) File "/appl/opt/python/mlpython-2.7.10/site-packages/torch/utils/data/dataloader.py", line 230, in _process_next_batch raise batch.exc_type(batch.exc_msg) RuntimeError: Traceback (most recent call last): File "/appl/opt/python/mlpython-2.7.10/site-packages/torch/utils/data/dataloader.py", line 42, in _worker_loop samples = collate_fn([dataset[i] for i in batch_indices]) File "/appl/opt/python/mlpython-2.7.10/site-packages/torch/utils/data/dataloader.py", line 116, in default_collate return {key: default_collate([d[key] for d in batch]) for key in batch[0]} File "/appl/opt/python/mlpython-2.7.10/site-packages/torch/utils/data/dataloader.py", line 116, in return {key: default_collate([d[key] for d in batch]) for key in batch[0]} File "/appl/opt/python/mlpython-2.7.10/site-packages/torch/utils/data/dataloader.py", line 105, in default_collate return torch.stack([torch.from_numpy(b) for b in batch], 0) RuntimeError: the given numpy array has zero-sized dimensions. Zero-sized dimensions are not supported in PyTorch

    Hello, I get an error like this. Can you give me some suggestions? Maybe the pytorch version is not right?

    opened by xiaoiker 7
  • some odd bugs abuout my test

    some odd bugs abuout my test

    dear friend: thank you for sharing your code,however when i "import torchvision.transforms as transforms" and "from utils import face_utils",there was a "Segmentation fault (core dumped)".when i run only one of them it 's fine.i seems that there is a conflict .Would you mind take some suggest.

    opened by vimalkashore 7
  • How to add smile to the face with trained model?

    How to add smile to the face with trained model?

    I have trained the model but I can see that desired condition in test.py file is expression = np.random.uniform(0, 1, opt.cond_nc)

    I am running it as python test.py --input_path ds/imgcelebaprocessed/000009.jpg --name experiment_128v2

    How can I modify expression of generated output? Have you used different test file which is not shared?

    opened by AidasK 6
  • Trained model is not public?

    Trained model is not public?

    @albertpumarola Thank you for your contribution on this work ! When I run the test like this: python test.py --input_path ./sample_dataset/imgs/N_0000001507_00202.jpg It came : image

    And there is an image in output/ like this : image

    Is the model not public available? Thank you! Looking forward to your reply.

    opened by leeeeeeo 6
  •  input and target shapes do not match: input [25 x 17 x 2 x 2], target [25 x 17]

    input and target shapes do not match: input [25 x 17 x 2 x 2], target [25 x 17]

    Any idea why this error occurs? I am using 224x224 celeba aligned database.

    Traceback (most recent call last):
      File "train.py", line 141, in <module>
        Train()
      File "train.py", line 27, in __init__
        self._train()
      File "train.py", line 40, in _train
        self._train_epoch(i_epoch)
      File "train.py", line 69, in _train_epoch
        self._model.optimize_parameters(keep_data_for_visuals=do_visuals, train_generator=train_generator)
      File "/home/aidas/GANimation/models/ganimation.py", line 208, in optimize_parameters
        loss_D, fake_imgs_masked = self._forward_D()
      File "/home/aidas/GANimation/models/ganimation.py", line 281, in _forward_D
        self._loss_d_cond = self._criterion_D_cond(d_real_img_cond, self._real_cond) / self._B * self._opt.lambda_D_cond
      File "/root/miniconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 477, in __call__
        result = self.forward(*input, **kwargs)
      File "/root/miniconda2/lib/python2.7/site-packages/torch/nn/modules/loss.py", line 421, in forward
        return F.mse_loss(input, target, reduction=self.reduction)
      File "/root/miniconda2/lib/python2.7/site-packages/torch/nn/functional.py", line 1716, in mse_loss
        return _pointwise_loss(lambda a, b: (a - b) ** 2, torch._C._nn.mse_loss, input, target, reduction)
      File "/root/miniconda2/lib/python2.7/site-packages/torch/nn/functional.py", line 1674, in _pointwise_loss
        return lambd_optimized(input, target, reduction)
    RuntimeError: input and target shapes do not match: input [25 x 17 x 2 x 2], target [25 x 17] at /pytorch/aten/src/THCUNN/generic/MSECriterion.cu:12
    
    opened by AidasK 5
  • IndexError: During training 0-dim tensor error

    IndexError: During training 0-dim tensor error

    While executing the train.py file,

    IndexError: invalid index of a 0-dim tensor. Use tensor.item() in Python or tensor.item<T>() in C++ to convert a 0-dim tensor to a number

    The complete log has been attached below:

    D_adam_b1: 0.5
    D_adam_b2: 0.999
    G_adam_b1: 0.5
    G_adam_b2: 0.999
    aus_file: aus_openface.pkl
    batch_size: 4
    checkpoints_dir: ./checkpoints
    cond_nc: 17
    data_dir: GANimation-master/sample_dataset1
    dataset_mode: aus
    display_freq_s: 300
    do_saturate_mask: False
    gpu_ids: [0]
    image_size: 128
    images_folder: imgs
    is_train: True
    lambda_D_cond: 4000
    lambda_D_gp: 10
    lambda_D_prob: 1
    lambda_cyc: 10
    lambda_mask: 0.1
    lambda_mask_smooth: 1e-05
    load_epoch: 0
    lr_D: 0.0001
    lr_G: 0.0001
    model: ganimation
    n_threads_test: 1
    n_threads_train: 4
    name: experiment_1
    nepochs_decay: 10
    nepochs_no_decay: 20
    num_iters_validate: 1
    poses_g_sigma: 0.06
    print_freq_s: 60
    save_latest_freq_s: 3600
    serial_batches: False
    test_ids_file: test_ids.csv
    train_G_every_n_iterations: 5
    train_ids_file: train_ids.csv
    -------------- End ----------------
    ./checkpoints/experiment_1
    Dataset AusDataset was created
    Dataset AusDataset was created
    #train images = 15
    #test images = 15
    Network generator_wasserstein_gan was created
    Network discriminator_wasserstein_gan was created
    Model GANimation was created
    ---------------------------------------------------------------------------
    IndexError                                Traceback (most recent call last)
    /content/GANimation-master/train.py in <module>()
        139 
        140 if __name__ == "__main__":
    --> 141     Train()
    
    4 frames
    /content/GANimation-master/models/ganimation.py in get_current_errors(self)
        318         loss_dict = OrderedDict([('g_fake', self._loss_g_fake.data[0]),
        319                                  ('g_cond', self._loss_g_cond.data[0]),
    --> 320                                  ('g_mskd_fake', self._loss_g_masked_fake.data[0]),
        321                                  ('g_mskd_cond', self._loss_g_masked_cond.data[0]),
        322                                  ('g_cyc', self._loss_g_cyc.data[0]),
    
    IndexError: invalid index of a 0-dim tensor. Use tensor.item() in Python or tensor.item<T>() in C++ to convert a 0-dim tensor to a number
    
    
    

    I have run all the previous commands, including the requirements.txt file. While looking for this over the net, I found that it is due to the update in the Pytorch version which does not allow accessing the last element as [0], we need to use .item(). But, even after changing it, I am getting the same error.

    Anyone facing the same issues?

    opened by divyanshujhawar 2
  • Generator is different from that in paper

    Generator is different from that in paper

    Fake image generation in ganimation.py is like this: # generate fake images fake_imgs, fake_img_mask = self._G.forward(real_img_a, real_cond_b) fake_img_mask = self._do_if_necessary_saturate_mask(fake_img_mask, saturate=self._opt.do_saturate_mask) fake_imgs_masked = fake_img_mask * real_img_a + (1 - fake_img_mask) * fake_imgs The last line seems to be different from Eq. (1) in the paper. Did I miss something or both work?

    opened by sunshine2004 1
  • when training  “the loss_d_real is negative value ” is OK? and why

    when training “the loss_d_real is negative value ” is OK? and why

    The result is amazing! I have two questions about loss.

    1. But I have a question that the three loss below will influence each other?some positive some negative.

    “self._loss_d_real + self._loss_d_cond + self._loss_d_fake”

    1. when I am training “loss_d_real is negative value” is right" and why
    self._loss_d_real = self._compute_loss_D(d_real_img_prob, True) * self._opt.lambda_D_prob
    
    def _compute_loss_D(self, estim, is_real):
            return -torch.mean(estim) if is_real else torch.mean(estim)
    

    批注 2020-05-31 114808 image

    image

    opened by talengu 0
  • TypeError: Cannot handle this data type

    TypeError: Cannot handle this data type

    print(target.shape,type(target))
    (40, 40, 9) <class 'numpy.ndarray'>
    m = Image.fromarray(target.astype('uint8'))
    
    
    <ipython-input-31-b037b3c8b026> in make_dataset()
         26         print(target.shape,type(target))
         27         #rescaled = (255.0/feature.max() * (feature - feature.min())).astype(np.uint8)
    ---> 28         im = Image.fromarray(target.astype('uint8'))
         29         #im.save(train_path)
         30 
    
    ~/venv/lib/python3.7/site-packages/PIL/Image.py in fromarray(obj, mode)
       2645             mode, rawmode = _fromarray_typemap[typekey]
       2646         except KeyError:
    -> 2647             raise TypeError("Cannot handle this data type")
       2648     else:
       2649         rawmode = mode
    
    TypeError: Cannot handle this data type
    
    opened by SlowMonk 1
Owner
Albert Pumarola
Computer Vision Researcher at Facebook Reality Labs
Albert Pumarola
OpenFace – a state-of-the art tool intended for facial landmark detection, head pose estimation, facial action unit recognition, and eye-gaze estimation.

OpenFace 2.2.0: a facial behavior analysis toolkit Over the past few years, there has been an increased interest in automatic facial behavior analysis

Tadas Baltrusaitis 5.8k Dec 31, 2022
Automatically measure the facial Width-To-Height ratio and get facial analysis results provided by Microsoft Azure

fwhr-calc-website This project is to automatically measure the facial Width-To-Height ratio and get facial analysis results provided by Microsoft Azur

SoohyunPark 1 Feb 7, 2022
Official Pytorch Implementation of 3DV2021 paper: SAFA: Structure Aware Face Animation.

SAFA: Structure Aware Face Animation (3DV2021) Official Pytorch Implementation of 3DV2021 paper: SAFA: Structure Aware Face Animation. Getting Started

QiulinW 122 Dec 23, 2022
Official PyTorch implementation of "Camera Distance-aware Top-down Approach for 3D Multi-person Pose Estimation from a Single RGB Image", ICCV 2019

PoseNet of "Camera Distance-aware Top-down Approach for 3D Multi-person Pose Estimation from a Single RGB Image" Introduction This repo is official Py

Gyeongsik Moon 677 Dec 25, 2022
Official pytorch implementation of "Feature Stylization and Domain-aware Contrastive Loss for Domain Generalization" ACMMM 2021 (Oral)

Feature Stylization and Domain-aware Contrastive Loss for Domain Generalization This is an official implementation of "Feature Stylization and Domain-

null 22 Sep 22, 2022
Image Segmentation Animation using Quadtree concepts.

QuadTree Image Segmentation Animation using QuadTree concepts. Usage usage: quad.py [-h] [-fps FPS] [-i ITERATIONS] [-ws WRITESTART] [-b] [-img] [-s S

Alex Eidt 29 Dec 25, 2022
Based on the paper "Geometry-aware Instance-reweighted Adversarial Training" ICLR 2021 oral

Geometry-aware Instance-reweighted Adversarial Training This repository provides codes for Geometry-aware Instance-reweighted Adversarial Training (ht

Jingfeng 47 Dec 22, 2022
This Repo is the official CUDA implementation of ICCV 2019 Oral paper for CARAFE: Content-Aware ReAssembly of FEatures

Introduction This Repo is the official CUDA implementation of ICCV 2019 Oral paper for CARAFE: Content-Aware ReAssembly of FEatures. @inproceedings{Wa

Jiaqi Wang 42 Jan 7, 2023
TAP: Text-Aware Pre-training for Text-VQA and Text-Caption, CVPR 2021 (Oral)

TAP: Text-Aware Pre-training TAP: Text-Aware Pre-training for Text-VQA and Text-Caption by Zhengyuan Yang, Yijuan Lu, Jianfeng Wang, Xi Yin, Dinei Flo

Microsoft 61 Nov 14, 2022
Implementation of ICCV2021(Oral) paper - VMNet: Voxel-Mesh Network for Geodesic-aware 3D Semantic Segmentation

VMNet: Voxel-Mesh Network for Geodesic-Aware 3D Semantic Segmentation Created by Zeyu HU Introduction This work is based on our paper VMNet: Voxel-Mes

HU Zeyu 82 Dec 27, 2022
[ICCV 2021 Oral] PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformers

PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformers Created by Xumin Yu*, Yongming Rao*, Ziyi Wang, Zuyan Liu, Jiwen Lu, Jie Zhou

Xumin Yu 317 Dec 26, 2022
Code for "Single-view robot pose and joint angle estimation via render & compare", CVPR 2021 (Oral).

Single-view robot pose and joint angle estimation via render & compare Yann Labbé, Justin Carpentier, Mathieu Aubry, Josef Sivic CVPR: Conference on C

Yann Labbé 51 Oct 14, 2022
(CVPR 2022 - oral) Multi-View Depth Estimation by Fusing Single-View Depth Probability with Multi-View Geometry

Multi-View Depth Estimation by Fusing Single-View Depth Probability with Multi-View Geometry Official implementation of the paper Multi-View Depth Est

Bae, Gwangbin 138 Dec 28, 2022
Facial Image Inpainting with Semantic Control

Facial Image Inpainting with Semantic Control In this repo, we provide a model for the controllable facial image inpainting task. This model enables u

Ren Yurui 8 Nov 22, 2021
Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set (CVPRW 2019). A PyTorch implementation.

Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set —— PyTorch implementation This is an unofficial offici

Sicheng Xu 833 Dec 28, 2022
Official PyTorch implementation of Retrieve in Style: Unsupervised Facial Feature Transfer and Retrieval.

Retrieve in Style: Unsupervised Facial Feature Transfer and Retrieval PyTorch This is the PyTorch implementation of Retrieve in Style: Unsupervised Fa

null 60 Oct 12, 2022
Official pytorch implementation for Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion (CVPR 2022)

Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion This repository contains a pytorch implementation of "Learning to Listen: Modeling

null 50 Dec 17, 2022
style mixing for animation face

An implementation of StyleGAN on Animation dataset. Install git clone https://github.com/MorvanZhou/anime-StyleGAN cd anime-StyleGAN pip install -r re

Morvan 46 Nov 30, 2022
CharacterGAN: Few-Shot Keypoint Character Animation and Reposing

CharacterGAN Implementation of the paper "CharacterGAN: Few-Shot Keypoint Character Animation and Reposing" by Tobias Hinz, Matthew Fisher, Oliver Wan

Tobias Hinz 181 Dec 27, 2022