A warping based image translation model focusing on upper body synthesis.

Related tags

Image Processing TalkingAnchor
Overview

Pose2Img

Upper body image synthesis from skeleton(Keypoints). Sub module in the ICCV-2021 paper "Speech Drives Templates: Co-Speech Gesture Synthesis with Learned Templates". [arxiv / github]

This is a modified implementation of Synthesizing Images of Humans in Unseen Poses.

Setup

To install dependencies, run

pip install -r requirements.txt

To run this module, you need two NVIDIA gpus with at least 11 GB respectively. Our code is tested on Ubuntu 18.04LTS with Python3.6.

Train on the custom dataset

For your own dataset, you need to modify custom config.yaml.

python main.py \
    --name test \
    --config_path {config_path}.yaml \
    --batch_size 1 \
  • The raw keypoints for each frame is of shape (3, 137)

Demo

Generate a realistic video for Oliver from {keypoints}.npz. You need to do some minor modifications to $configs/yaml/oliver.yaml$.

python inference.py \
    --cfg_path configs/yaml/oliver.yaml \
    --name test \
    --npz_path target_pose/oliver_pose.npz \
    --wav_path target_pose/oliver_audio.wav
  • In the result directory, you can find jpg files which correspond to the npz.

Demo Dataset and Checkpoint

  • Coming soon

Citation

If you find this code useful for your research, please use the following BibTeX entry.

@inproceedings{qian2021speech,
  title={Speech Drives Templates: Co-Speech Gesture Synthesis with Learned Templates},
  author={Qian, Shenhan and Tu, Zhi and Zhi, YiHao and Liu, Wen and Gao, Shenghua},
  journal={International Conference on Computer Vision (ICCV)},
  year={2021}
}
You might also like...
Quickly 'anonymize' all people in an image. This script will put a black bar over all eye-pairs in an image

Face-Detacher Quickly 'anonymize' all people in an image. This script will put a black bar over all eye-pairs in an image This is a small python scrip

Don Cato 1 Oct 29, 2021
Fast Image Retrieval is an open source image retrieval framework

Fast Image Retrieval is an open source image retrieval framework release by Center of Image and Signal Processing Lab (CISiP Lab), Universiti Malaya. This framework implements most of the major binary hashing methods, together with both popular backbone networks and public datasets.

CISiP Lab 39 Nov 25, 2022
Fast Image Retrieval (FIRe) is an open source image retrieval project

Fast Image Retrieval (FIRe) is an open source image retrieval project release by Center of Image and Signal Processing Lab (CISiP Lab), Universiti Malaya. This project implements most of the major binary hashing methods to date, together with different popular backbone networks and public datasets.

CISiP Lab 39 Nov 25, 2022
A Python Script to convert Normal PNG Image to Apple iDOT PNG Image.

idot-png-encoder A Python Script to convert Normal PNG Image to Apple iDOT PNG Image (Multi-threaded Decoding PNG). Usage idotpngencoder.py -i inputf

Lrdcq 2 Feb 17, 2022
Anaglyph 3D Converter - A python script that adds a 3D anaglyph style effect to an image using the Pillow image processing package.
Anaglyph 3D Converter - A python script that adds a 3D anaglyph style effect to an image using the Pillow image processing package.

Anaglyph 3D Converter - A python script that adds a 3D anaglyph style effect to an image using the Pillow image processing package.

Kizdude 2 Jan 22, 2022
Pyconvert is a python script that you can use to convert image files to another image format! (eg. PNG to ICO)
Pyconvert is a python script that you can use to convert image files to another image format! (eg. PNG to ICO)

Pyconvert is a python script that you can use to convert image files to another image format! (eg. PNG to ICO)

null 1 Jan 16, 2022
Simple Python package to convert an image into a quantized image using a customizable palette
Simple Python package to convert an image into a quantized image using a customizable palette

Simple Python package to convert an image into a quantized image using a customizable palette. Resulting image can be displayed by ePaper displays such as Waveshare displays.

Luis Obis 3 Apr 13, 2022
An esoteric visual language that takes image files as input based on a multi-tape turing machine, designed for compatibility with C.
An esoteric visual language that takes image files as input based on a multi-tape turing machine, designed for compatibility with C.

vizh An esoteric visual language that takes image files as input based on a multi-tape turing machine, designed for compatibility with C. Overview Her

Sy Brand 228 Dec 17, 2022
impy is an all-in-one image analysis library, equipped with parallel processing, GPU support, GUI based tools and so on.
impy is an all-in-one image analysis library, equipped with parallel processing, GPU support, GUI based tools and so on.

impy is All You Need in Image Analysis impy is an all-in-one image analysis library, equipped with parallel processing, GPU support, GUI based tools a

null 24 Dec 20, 2022
Comments
  • Replace new photos in the demo

    Replace new photos in the demo

    Thank you for your work. Can I generate a video corresponding to any photo according to the pre training model you provided? For example, use Oliver's model and a photo of Luo Xiang to generate Luo Xiang's video.

    I have tried to repalce Oliver's img to Luoxiang's img in inference.py（def process_source）, def process_source(self):

        kp_path = self.cfg.INFER.src_kp_path
    kp = np.load(kp_path)
    kp = pose137_to_pose122(kp).transpose(1,0)
    path = kp_path.split("/")[-1]
    filename, _ = os.path.splitext(path)  
    img_path = os.path.join(self.img_base,filename+self.img_extension)  #img_path is a Luoxiang's img now.

    ` but it still generate Oliver's video. I feel very confused .

    opened by liuweie 5
  • Error in vis_hyperparams.py

    Error in vis_hyperparams.py

    hello,when I use vis_hyperparams.py to compute kp_var_root and mask_sigma_perp ,the following error ocurred:

    image

    I've been debugging for a long time and still can't solve it. Can you help me to see what the problem is? .

    opened by liuweie 9
Owner
zhiyh
zhiyh
GitHub
Image enhancing model for making a blurred image to be somehow clearer than before

This is a very small prject which helps in enhancing the images by taking a Input images. This project has many features like detcting the faces and enhaning the faces itself and also a feature which enhances the whole image

null 3 Dec 3, 2021
Nanosensor Image Processor (NanoImgPro), a python-based image analysis tool for dopamine nanosensors

NanoImgPro Nanosensor Image Processor (NanoImgPro), a python-based image analysis tool for dopamine nanosensors NanoImgPro.py contains the main class

null 1 Mar 2, 2022
Image-Viewer is a Windows image viewer based on Python 3.

Image-Viewer Hi! Image-Viewer is a Windows image viewer based on Python 3. Using You must download Image-Viewer.exe from the root of the repository. T

null 2 Apr 18, 2022
Seaborn-image is a Python image visualization library based on matplotlib and provides a high-level API to draw attractive and informative images quickly and effectively.

seaborn-image: image data visualization Description Seaborn-image is a Python image visualization library based on matplotlib and provides a high-leve

null 48 Jan 5, 2023
New program to export a Blender model to the LBA2 model format.

LBA2 Blender to Model 2 This is a new program to export a Blender model to the LBA2 model format. This is also the first publicly released version of

null 2 Nov 30, 2022
An open source image editor which can manipulate an image in many ways!

Image Editor - An open source image editor which can manipulate an image in many ways! If you need any more modes in repo or I

TroJanzHEX 44 Nov 17, 2022
Easily turn large sets of image urls to an image dataset. Can download, resize and package 100M urls in 20h on one machine.

img2dataset Easily turn large sets of image urls to an image dataset. Can download, resize and package 100M urls in 20h on one machine. Also supports

Romain Beaumont 1.4k Jan 1, 2023
A pure python implementation of the GIMP XCF image format. Use this to interact with GIMP image formats

Pure Python implementation of the GIMP image formats (.xcf projects as well as brushes, patterns, etc)

FHPyhtonUtils 8 Dec 30, 2022
Image Reading, Metadata Conversion, and Image Writing for Microscopy Images in Python

AICSImageIO Image Reading, Metadata Conversion, and Image Writing for Microscopy Images in Pure Python Features Supports reading metadata and imaging

Allen Institute for Cell Science - Modeling 137 Dec 14, 2022
This app finds duplicate to near duplicate images by generating a hash value for each image stored with a specialized data structure called VP-Tree which makes searching an image on a dataset of 100Ks almost instantanious

Offline Reverse Image Search Overview This app finds duplicate to near duplicate images by generating a hash value for each image stored with a specia

null 53 Nov 15, 2022