Implementation for HFGI: High-Fidelity GAN Inversion for Image Attribute Editing

Related tags

Deep Learning computer-vision deep-learning image-editing gan image-manipulation video-editing gan-inversion
Overview

HFGI: High-Fidelity GAN Inversion for Image Attribute Editing

High-Fidelity GAN Inversion for Image Attribute Editing

Update: We released the inference code and the pre-trained model on Oct. 31. The training code is coming soon.

paper | project website | demo video

Introduction

We present a novel high-fidelity GAN inversion framework that enables attribute editing with image-specific details well-preserved (e.g., background, appearance and illumination).

To Do

  • Release the inference code
  • Release the pretrained model
  • Release the training code (upon approval)

Set up

Installation

git clone https://github.com/Tengfei-Wang/HFGI.git
cd HFGI

Environment

The environment can be simply set up by Anaconda (only tested for inference):

conda create -n HFGI python=3.7
conda activate HFGI
pip install torch==1.6.0+cu101 torchvision==0.7.0+cu101 -f https://download.pytorch.org/whl/torch_stable.html
pip install matplotlib
conda install ninja
conda install -c 3dhubs gcc-5

Or, you can also set up the environment from the provided environment.yml:

conda env create -f environment.yml

Quick Start

Pretrained Models

Please download our pre-trained model and put it in ./checkpoint.

Model Description
Face Editing Trained on FFHQ.

Prepare Images

We put some images from CelebA-HQ in ./test_imgs, and you can quickly try them (and other images from CelebA-HQ or FFHQ).
For customized images, it is encouraged to first pre-process (align & crop) them, and then edit with our model. See FFHQ for alignment details.

Inference

Modify inference.sh according to the follwing instructions, and run:
(It is possibly slow for the first-time running.)

bash inference.sh
Args Description
--images_dir the path of images.
--n_sample number of images that you want to infer.
--edit_attribute We provide options of 'inversion', 'age', 'smile', 'eyes', 'lip' and 'beard' in the script.
--edit_degree control the degree of editing (works for 'age' and 'smile').

Training

Coming soon

Video Editing

The source videos and edited results in our paper can be found in this link.
For video editing, we first pre-process (align & crop) each frame, and then perform editing with the pre-trained model.

More Results

Citation

If you find this work useful for your research, please cite:

@article{wang2021HFGI,
      author = {Tengfei Wang and Yong Zhang and Yanbo Fan and Jue Wang and Qifeng Chen},
      title = {High-Fidelity GAN Inversion for Image Attribute Editing}, 
      journal = {arxiv:2109.06590},  
      year = {2021}
}
Comments
  • Problem about the results of pose editing

    Problem about the results of pose editing

    Thank you for the great work! I have tried the inference code with the pretrained checkpoint for pose editing, but there are obvious artifacts in the edited images. Could you please double check that the checkpoint is correct? BTW, why the pose editing is not included in the inference code or playground notebook?

    opened by huiqu18 2
  • Add Docker environment & web demo

    Add Docker environment & web demo

    Hey @Tengfei-Wang! 👋

    This pull request makes it possible to run your model inside a Docker environment, which makes it easier for other people to run it. We're using an open source tool called Cog to make this process easier.

    This also means we can make a web page where other people can try out your model (we have enabled web cam input as well, so it is very easy to try facial image editing models like HGFI 😊 ~) View it here: https://replicate.ai/tengfei-wang/hfgi.

    Do claim your page here so you can own the page, customise the Example gallery as you like, and we'll make it public, feature it on our website and tweet about it too.

    In case you're wondering who I am, I'm from Replicate, where we're trying to make machine learning reproducible. We got frustrated that we couldn't run all the really interesting ML work being done. So, we're going round implementing models we like. 😊

    opened by chenxwh 0
  • Upgrade to Cog version 0.1

    Upgrade to Cog version 0.1

    The new version of Cog improves the Python API, along with several other changes. Particularly pydantic is now used for Predictor and the previous version will be deprecated.

    This PR upgrades the Replicate demo and API to Cog version >= 0.1. I have already pushed this to Replicate, so you don't need to do anything for the demo to keep working :) https://replicate.com/tengfei-wang/hfgi

    opened by chenxwh 0
  • About Editing Hair

    About Editing Hair

    Really thanks for your great works! However, when I implement it on styleCLIP for hair change, after the step of adding conditions to the generator, it not only fine-tune the face but also add back the original hair on it. Could you give me some suggestions on that? Really thanks!

    opened by duzixsiansheng 0
  • question about generating edited codes

    question about generating edited codes

    What if I want to use this model to put a mask on a person instead of modifying age and smile? How can i generate masked face attribution edited codes? Thanks!

    opened by wrainbow0705 0
  • Usage of discriminator for adversarial loss

    Usage of discriminator for adversarial loss

    In your code, you do not use a discriminator and an additional adversarial loss for better reconstruction. This is different from what is written in the paper. Is there another version of code that leverages a well-trained discriminator, or are the checkpoint results based on the official code without discriminator???

    opened by chaewonklleon 0
  • The resolution of consultation branch

    The resolution of consultation branch

    Hi, thank for sharing code! I have a question about the resolution of consultation branch. As the default resolution is 64x64 in layer 7. Have you test other higher resolution, like 11 for 256, 9 for 128 as shown below: https://github.com/Tengfei-Wang/HFGI/blob/e30f33cbdf37fc57b9e8354b11a528a62d476049/models/stylegan2/model.py#L530 That usually higher resolutin and later layer might imporve the details. Hope for your reply~

    opened by zhongtao93 2
Owner
Tengfei Wang
Ph.D. candidate @ HKUST / Computer Vision
Tengfei Wang
GitHub https://tengfei-wang.github.io/HFGI/
HiFi-GAN: High Fidelity Denoising and Dereverberation Based on Speech Deep Features in Adversarial Networks

HiFiGAN Denoiser This is a Unofficial Pytorch implementation of the paper HiFi-GAN: High Fidelity Denoising and Dereverberation Based on Speech Deep F

Rishikesh (ऋषिकेश) 134 Dec 27, 2022
HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis

HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis Jungil Kong, Jaehyeon Kim, Jaekyoung Bae In our paper, we p

Rishikesh (ऋषिकेश) 31 Dec 8, 2022
A collection of resources on GAN Inversion.

This repo is a collection of resources on GAN inversion, as a supplement for our survey

null 857 Dec 29, 2022
[CVPR 2021] Unsupervised 3D Shape Completion through GAN Inversion

ShapeInversion Paper Junzhe Zhang, Xinyi Chen, Zhongang Cai, Liang Pan, Haiyu Zhao, Shuai Yi, Chai Kiat Yeo, Bo Dai, Chen Change Loy "Unsupervised 3D

null 100 Dec 22, 2022
A Simplied Framework of GAN Inversion

Framework of GAN Inversion Introcuction You can implement your own inversion idea using our repo. We offer a full range of tuning settings (in hparams

Kangneng Zhou 13 Sep 27, 2022
Style-based Neural Drum Synthesis with GAN inversion

Style-based Drum Synthesis with GAN Inversion Demo TensorFlow implementation of a style-based version of the adversarial drum synth (ADS) from the pap

Sound and Music Analysis (SoMA) Group 29 Nov 19, 2022
Disentangled Face Attribute Editing via Instance-Aware Latent Space Search, accepted by IJCAI 2021.

Instance-Aware Latent-Space Search This is a PyTorch implementation of the following paper: Disentangled Face Attribute Editing via Instance-Aware Lat

null 67 Dec 21, 2022
DyStyle: Dynamic Neural Network for Multi-Attribute-Conditioned Style Editing

DyStyle: Dynamic Neural Network for Multi-Attribute-Conditioned Style Editing Figure: Joint multi-attribute edits using DyStyle model. Great diversity

null 74 Dec 3, 2022
A fast poisson image editing implementation that can utilize multi-core CPU or GPU to handle a high-resolution image input.

Poisson Image Editing - A Parallel Implementation Jiayi Weng (jiayiwen), Zixu Chen (zixuc) Poisson Image Editing is a technique that can fuse two imag

Jiayi Weng 110 Dec 27, 2022
SCI-AIDE : High-fidelity Few-shot Histopathology Image Synthesis for Rare Cancer Diagnosis

SCI-AIDE : High-fidelity Few-shot Histopathology Image Synthesis for Rare Cancer Diagnosis Pretrained Models In this work, we created synthetic tissue

Emirhan KurtuluÅŸ 1 Feb 7, 2022
Tensorflow python implementation of "Learning High Fidelity Depths of Dressed Humans by Watching Social Media Dance Videos"

Learning High Fidelity Depths of Dressed Humans by Watching Social Media Dance Videos This repository is the official tensorflow python implementation

Yasamin Jafarian 287 Jan 6, 2023
Unofficial PyTorch Implementation of UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation

UnivNet UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation This is an unofficial PyTorch

MINDs Lab 170 Jan 4, 2023
Unofficial PyTorch Implementation of UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation

UnivNet UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation This is an unofficial PyTorch

MINDs Lab 54 Aug 30, 2021
PyTorch Implementation of DiffGAN-TTS: High-Fidelity and Efficient Text-to-Speech with Denoising Diffusion GANs

DiffGAN-TTS - PyTorch Implementation PyTorch implementation of DiffGAN-TTS: High

Keon Lee 157 Jan 1, 2023
FuseDream: Training-Free Text-to-Image Generationwith Improved CLIP+GAN Space OptimizationFuseDream: Training-Free Text-to-Image Generationwith Improved CLIP+GAN Space Optimization

FuseDream This repo contains code for our paper (paper link): FuseDream: Training-Free Text-to-Image Generation with Improved CLIP+GAN Space Optimizat

XCL 191 Dec 31, 2022
Deep generative modeling for time-stamped heterogeneous data, enabling high-fidelity models for a large variety of spatio-temporal domains.

Neural Spatio-Temporal Point Processes [arxiv] Ricky T. Q. Chen, Brandon Amos, Maximilian Nickel Abstract. We propose a new class of parameterizations

Facebook Research 75 Dec 19, 2022
《Towards High Fidelity Face Relighting with Realistic Shadows》(CVPR 2021)

Towards High Fidelity Face-Relighting with Realistic Shadows Andrew Hou, Ze Zhang, Michel Sarkis, Ning Bi, Yiying Tong, Xiaoming Liu. In CVPR, 2021. T

null 114 Dec 10, 2022
UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation

UnivNet UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation. Training python train.py --c

Rishikesh (ऋषिकेश) 55 Dec 26, 2022
This repository contains the code for using the H3DS dataset introduced in H3D-Net: Few-Shot High-Fidelity 3D Head Reconstruction

H3DS Dataset This repository contains the code for using the H3DS dataset introduced in H3D-Net: Few-Shot High-Fidelity 3D Head Reconstruction Access

Crisalix 72 Dec 10, 2022