[CVPR 2022] Thin-Plate Spline Motion Model for Image Animation.

Overview

[CVPR2022] Thin-Plate Spline Motion Model for Image Animation

License: MIT stars GitHub repo size

Source code of the CVPR'2022 paper "Thin-Plate Spline Motion Model for Image Animation"

Paper | Supp

Example animation

vox ted

PS: The paper trains the model for 100 epochs for a fair comparison. You can use more data and train for more epochs to get better performance.

Web demo for animation

  • Try the web demo for animation here: Replicate
  • Google Colab: Open In Colab

Pre-trained models

Installation

We support python3.(Recommended version is Python 3.9). To install the dependencies run:

pip install -r requirements.txt

YAML configs

There are several configuration files one for each dataset in the config folder named as config/dataset_name.yaml.

See description of the parameters in the config/taichi-256.yaml.

Datasets

  1. MGif. Follow Monkey-Net.

  2. TaiChiHD and VoxCeleb. Follow instructions from video-preprocessing.

  3. TED-talks. Follow instructions from MRAA.

Training

To train a model on specific dataset run:

CUDA_VISIBLE_DEVICES=0,1 python run.py --config config/dataset_name.yaml --device_ids 0,1

A log folder named after the timestamp will be created. Checkpoints, loss values, reconstruction results will be saved to this folder.

Training AVD network

To train a model on specific dataset run:

CUDA_VISIBLE_DEVICES=0 python run.py --mode train_avd --checkpoint '{checkpoint_folder}/checkpoint.pth.tar' --config config/dataset_name.yaml

Checkpoints, loss values, reconstruction results will be saved to {checkpoint_folder}.

Evaluation on video reconstruction

To evaluate the reconstruction performance run:

CUDA_VISIBLE_DEVICES=0 python run.py --mode reconstruction --config config/dataset_name.yaml --checkpoint '{checkpoint_folder}/checkpoint.pth.tar'

The reconstruction subfolder will be created in {checkpoint_folder}. The generated video will be stored to this folder, also generated videos will be stored in png subfolder in loss-less '.png' format for evaluation. To compute metrics, follow instructions from pose-evaluation.

Image animation demo

  • notebook: demo.ipynb, edit the config cell and run for image animation.
  • python:
CUDA_VISIBLE_DEVICES=0 python demo.py --config config/vox-256.yaml --checkpoint checkpoints/vox.pth.tar --source_image ./source.jpg --driving_video ./driving.mp4

Acknowledgments

The main code is based upon FOMM and MRAA

Thanks for the excellent works!

Thanks iperov, this work has been integrated in DeepFaceLive

Issues
  • output of dense_motion

    output of dense_motion

    Maybe stupid question.

    Is it possible to get the deformed/'animated' source_ image before using the inpaint module?

    What is the output of the dense motion module.?....

    out_dict['contribution_maps'] = contribution_maps out_dict['deformation'] = deformation # Optical Flow out_dict['occlusion_map'] = occlusion_map # Multi-resolution Occlusion Masks

    out_dict['deformed_source'] = deformed_source ?

    Thanks in advance

    opened by instant-high 6
  • Add Web Demo & Docker environment

    Add Web Demo & Docker environment

    Hey @yoyo-nb ! 👋

    This pull request makes it possible to run your model inside a Docker environment, which makes it easier for other people to run it. We're using an open source tool called Cog to make this process easier.

    This also means we can make a web page where other people can try out your model! View it here: https://replicate.com/yoyo-nb/thin-plate-spline-motion-model. The docker file can be found under the tab ‘run model with docker’.

    Do claim the page so you can own the page, customise the Example gallery as you like, push any future update to the web demo, and we'll feature it on our website and tweet about it too. You can find the 'Claim this model' button on the top of the page ~ When the page is claimed, it will be automatically linked to the arXiv website as well (under “Demos”).

    In case you're wondering who I am, I'm from Replicate, where we're trying to make machine learning reproducible. We got frustrated that we couldn't run all the really interesting ML work being done. So, we're going round implementing models we like. 😊

    opened by chenxwh 2
  • Can eyes always face forward?

    Can eyes always face forward?

    First off, amazing work on this project!

    Is it possible to make the eyes always face forward so that it appears that they are talking straight into the camera at all times?

    opened by andrewkuo 1
  • hope you make a colab notebook

    hope you make a colab notebook

    great work guys!

    would be great if you can a colab notebook for this project with the pretrained model , such that user can upload a video with the action and an image so that you can transfer the motion.

    opened by GeorvityLabs 1
  • Can you provide colab cready code?

    Can you provide colab cready code?

    Downloading the code, resolving dependency issues, etc would eat a lot of time. A Google Colab notebook would be really helpful to quickly check the code and implement it on the cloud.

    opened by snehitvaddi 0
  • imageio.mimread() has read over 256000000B of image data

    imageio.mimread() has read over 256000000B of image data

    Thanks for your nice work! I met a problem while I'm training on the TED dataset (Two 32G GPUs).

      File "Thin-Plate-Spline-Motion-Model/train.py", line 55, in train
        for x in dataloader:
      File "anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 530, in __next__
        data = self._next_data()
      File "anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1224, in _next_data
        return self._process_data(data)
      File "anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1250, in _process_data
        data.reraise()
      File "anaconda3/lib/python3.7/site-packages/torch/_utils.py", line 457, in reraise
        raise exception
    RuntimeError: Caught RuntimeError in DataLoader worker process 0.
    Original Traceback (most recent call last):
      File "anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
        data = fetcher.fetch(index)
      File "anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
        data = [self.dataset[idx] for idx in possibly_batched_index]
      File "anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
        data = [self.dataset[idx] for idx in possibly_batched_index]
      File "Thin-Plate-Spline-Motion-Model/frames_dataset.py", line 172, in __getitem__
        return self.dataset[idx % self.dataset.__len__()]
      File "Thin-Plate-Spline-Motion-Model/frames_dataset.py", line 133, in __getitem__
        video_array = read_video(path, frame_shape=self.frame_shape)
      File "Thin-Plate-Spline-Motion-Model/frames_dataset.py", line 43, in read_video
        video = mimread(name)
      File "anaconda3/lib/python3.7/site-packages/imageio/core/functions.py", line 369, in mimread
        int(nbyte_limit)
    
    RuntimeError: imageio.mimread() has read over 256000000B of image data.
    Stopped to avoid memory problems. Use imageio.get_reader(), increase threshold, or memtest=False
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "run.py", line 83, in <module>
        train(config, inpainting, kp_detector, bg_predictor, dense_motion_network, opt.checkpoint, log_dir, dataset)
      File "Thin-Plate-Spline-Motion-Model/train.py", line 93, in train
        logger.log_epoch(epoch, model_save, inp=x, out=generated)
    TypeError: __exit__() takes 1 positional argument but 4 were given
    

    Thanks for your replay!

    opened by aishoot 2
  • It seems that there are some side effects?

    It seems that there are some side effects?

    Hi, I have try your demo with my own video and image, but there seems some green or gray outliers in the generated video, just like the mouth showed here,any suggestions? Uploading image.jpg…

    opened by howardgriffin 4
  • Plans to modify it to perform partswap/faceswap?

    Plans to modify it to perform partswap/faceswap?

    Thank you for this great repo.

    I'm using the original FOMM for about 2 years. But your modification gives much better results. Are you planning to modify it so we can use it for face swap / part swap like the motion co-segmentation branch of FOMM?

    I've made major modifications to the motion co-segmentation repo, so that now it performs 'automatic' face tracking / cropping / face swap /colour correction and putting all together again.

    It would be nice to do that with a version that gives even better results.

    opened by instant-high 2
  • Could you please release the testset of VoxCeleb?

    Could you please release the testset of VoxCeleb?

    Hi @yoyo-nb ,

    Thanks for your excellent work. The video-processing provides a large test set for VoxCeleb. Do you use all samples in that test set? If not, could you please share your own test set sampled from the large test set? Or share your test samples recorded file like .csv?

    opened by harlanhong 0
Owner
yoyo-nb
nmd,why?
yoyo-nb
[CVPR 2022] CoTTA Code for our CVPR 2022 paper Continual Test-Time Domain Adaptation

CoTTA Code for our CVPR 2022 paper Continual Test-Time Domain Adaptation Prerequisite Please create and activate the following conda envrionment. To r

Qin Wang 42 Jun 24, 2022
Code for Motion Representations for Articulated Animation paper

Motion Representations for Articulated Animation This repository contains the source code for the CVPR'2021 paper Motion Representations for Articulat

Snap Research 714 Jun 28, 2022
Neural Nano-Optics for High-quality Thin Lens Imaging

Neural Nano-Optics for High-quality Thin Lens Imaging Project Page | Paper | Data Ethan Tseng, Shane Colburn, James Whitehead, Luocheng Huang, Seung-H

Ethan Tseng 32 May 24, 2022
Official pytorch implementation for Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion (CVPR 2022)

Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion This repository contains a pytorch implementation of "Learning to Listen: Modeling

null 34 Jun 23, 2022
The 7th edition of NTIRE: New Trends in Image Restoration and Enhancement workshop will be held on June 2022 in conjunction with CVPR 2022.

NTIRE 2022 - Image Inpainting Challenge Important dates 2022.02.01: Release of train data (input and output images) and validation data (only input) 2

Andrés Romero 25 Jun 22, 2022
[SIGGRAPH 2022 Journal Track] AvatarCLIP: Zero-Shot Text-Driven Generation and Animation of 3D Avatars

AvatarCLIP: Zero-Shot Text-Driven Generation and Animation of 3D Avatars Fangzhou Hong1*  Mingyuan Zhang1*  Liang Pan1  Zhongang Cai1,2,3  Lei Yang2 

Fangzhou Hong 451 Jun 26, 2022
CCPD: a diverse and well-annotated dataset for license plate detection and recognition

CCPD (Chinese City Parking Dataset, ECCV) UPdate on 10/03/2019. CCPD Dataset is now updated. We are confident that images in subsets of CCPD is much m

detectRecog 1.7k Jun 22, 2022
Indonesian Car License Plate Character Recognition using Tensorflow, Keras and OpenCV.

Monopol Indonesian Car License Plate (Indonesia Mobil Nomor Polisi) Character Recognition using Tensorflow, Keras and OpenCV. Background This applicat

Jayaku Briliantio 3 Apr 7, 2022
Automatic number plate recognition using tech: Yolo, OCR, Scene text detection, scene text recognation, flask, torch

Automatic Number Plate Recognition Automatic Number Plate Recognition (ANPR) is the process of reading the characters on the plate with various optica

Meftun AKARSU 40 Jun 11, 2022
This repository contains the code for the paper "Hierarchical Motion Understanding via Motion Programs"

Hierarchical Motion Understanding via Motion Programs (CVPR 2021) This repository contains the official implementation of: Hierarchical Motion Underst

Sumith Kulal 31 Jun 20, 2022
Exploring Versatile Prior for Human Motion via Motion Frequency Guidance (3DV2021)

Exploring Versatile Prior for Human Motion via Motion Frequency Guidance This is the codebase for video-based human motion reconstruction in human-mot

Jiachen Xu 4 Jan 18, 2022
Imposter-detector-2022 - HackED 2022 Team 3IQ - 2022 Imposter Detector

HackED 2022 Team 3IQ - 2022 Imposter Detector By Aneeljyot Alagh, Curtis Kan, Jo

Joshua Ji 4 Jan 27, 2022
GANimation: Anatomically-aware Facial Animation from a Single Image (ECCV'18 Oral) [PyTorch]

GANimation: Anatomically-aware Facial Animation from a Single Image [Project] [Paper] Official implementation of GANimation. In this work we introduce

Albert Pumarola 1.8k Jun 19, 2022
Image Segmentation Animation using Quadtree concepts.

QuadTree Image Segmentation Animation using QuadTree concepts. Usage usage: quad.py [-h] [-fps FPS] [-i ITERATIONS] [-ws WRITESTART] [-b] [-img] [-s S

Alex Eidt 25 Jun 9, 2022
This is a model to classify Vietnamese sign language using Motion history image (MHI) algorithm and CNN.

Vietnamese sign lagnuage recognition using MHI and CNN This is a model to classify Vietnamese sign language using Motion history image (MHI) algorithm

Phat Pham 3 Feb 24, 2022
This repository contains a pytorch implementation of "HeadNeRF: A Real-time NeRF-based Parametric Head Model (CVPR 2022)".

HeadNeRF: A Real-time NeRF-based Parametric Head Model This repository contains a pytorch implementation of "HeadNeRF: A Real-time NeRF-based Parametr

null 182 Jun 24, 2022
Official repository for "Restormer: Efficient Transformer for High-Resolution Image Restoration". SOTA for motion deblurring, image deraining, denoising (Gaussian/real data), and defocus deblurring.

Restormer: Efficient Transformer for High-Resolution Image Restoration Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan,

Syed Waqas Zamir 650 Jun 21, 2022
Self-Supervised Pillar Motion Learning for Autonomous Driving (CVPR 2021)

Self-Supervised Pillar Motion Learning for Autonomous Driving Chenxu Luo, Xiaodong Yang, Alan Yuille Self-Supervised Pillar Motion Learning for Autono

QCraft 87 May 23, 2022
A variational Bayesian method for similarity learning in non-rigid image registration (CVPR 2022)

A variational Bayesian method for similarity learning in non-rigid image registration We provide the source code and the trained models used in the re

daniel grzech 11 Jun 19, 2022