Official PyTorch implementation of "BlendGAN: Implicitly GAN Blending for Arbitrary Stylized Face Generation" (NeurIPS 2021)

Last update: Dec 29, 2022

Overview

BlendGAN: Implicitly GAN Blending for Arbitrary Stylized Face Generation
_{Official PyTorch implementation of the NeurIPS 2021 paper}

Mingcong Liu, Qiang Li, Zekui Qin, Guoxin Zhang, Pengfei Wan, Wen Zheng

Y-tech, Kuaishou Technology

Project page | Paper

Abstract: Generative Adversarial Networks (GANs) have made a dramatic leap in high-fidelity image synthesis and stylized face generation. Recently, a layer-swapping mechanism has been developed to improve the stylization performance. However, this method is incapable of fitting arbitrary styles in a single model and requires hundreds of style-consistent training images for each style. To address the above issues, we propose BlendGAN for arbitrary stylized face generation by leveraging a flexible blending strategy and a generic artistic dataset. Specifically, we first train a self-supervised style encoder on the generic artistic dataset to extract the representations of arbitrary styles. In addition, a weighted blending module (WBM) is proposed to blend face and style representations implicitly and control the arbitrary stylization effect. By doing so, BlendGAN can gracefully fit arbitrary styles in a unified model while avoiding case-by-case preparation of style-consistent training images. To this end, we also present a novel large-scale artistic face dataset AAHQ. Extensive experiments demonstrate that BlendGAN outperforms state-of-the-art methods in terms of visual quality and style diversity for both latent-guided and reference-guided stylized face synthesis.

Updates

✔️ (2021-11-19) Inference code and pretrained models have been released!

Pre-trained Models

You can download the following pretrained models to ./pretrained_models:

Model	Discription
blendgan	BlendGAN model (together with style_encoder)
psp_encoder	PSP Encoder model
style_encoder	Individual Style Encoder model (optional)

Inference

1. Generate image pairs with random face codes

for latent-guided generation, run:

python generate_image_pairs.py --size 1024 --pics N_PICS --ckpt ./pretrained_models/blendgan.pt --outdir results/generated_pairs/latent_guided/

for reference-guided generation, run:

python generate_image_pairs.py --size 1024 --pics N_PICS --ckpt ./pretrained_models/blendgan.pt --style_img ./test_imgs/style_imgs/100036.png --outdir results/generated_pairs/reference_guided/

2. Style tranfer with given face images

python style_transfer_folder.py --size 1024 --ckpt ./pretrained_models/blendgan.pt --psp_encoder_ckpt ./pretrained_models/psp_encoder.pt --style_img_path ./test_imgs/style_imgs/ --input_img_path ./test_imgs/face_imgs/ --outdir results/style_transfer/

3. Generate interpolation videos

python gen_video.py --size 1024 --ckpt ./pretrained_models/blendgan.pt --psp_encoder_ckpt ./pretrained_models/psp_encoder.pt --style_img_path ./test_imgs/style_imgs/ --input_img_path ./test_imgs/face_imgs/ --outdir results/inter_videos/

Bibtex

If you use this code for your research, please cite our paper:

@inproceedings{liu2021blendgan,
    title = {BlendGAN: Implicitly GAN Blending for Arbitrary Stylized Face Generation},
    author = {Liu, Mingcong and Li, Qiang and Qin, Zekui and Zhang, Guoxin and Wan, Pengfei and Zheng, Wen},
    booktitle = {Advances in Neural Information Processing Systems},
    year = {2021}
}

Credits

StyleGAN2 model and implementation:
https://github.com/rosinality/stylegan2-pytorch
Copyright (c) 2019 Kim Seonghyeon
License (MIT) https://github.com/rosinality/stylegan2-pytorch/blob/master/LICENSE

IR-SE50 model and implementations:
https://github.com/TreB1eN/InsightFace_Pytorch
Copyright (c) 2018 TreB1eN
License (MIT) https://github.com/TreB1eN/InsightFace_Pytorch/blob/master/LICENSE

pSp model and implementation:
https://github.com/eladrich/pixel2style2pixel
Copyright (c) 2020 Elad Richardson, Yuval Alaluf
License (MIT) https://github.com/eladrich/pixel2style2pixel/blob/master/LICENSE

Please Note:

The CUDA files under the StyleGAN2 ops directory are made available under the Nvidia Source Code License-NC
The face images under the test_imgs directory are selected from the FFHQ dataset, which is made available under Creative Commons BY-NC-SA 4.0 license by NVIDIA Corporation.
The artistic images under the test_imgs directory are collected from Artstation, and the copyright remains with the original owners.

Acknowledgements

We sincerely thank all the reviewers for their comments. We also thank Zhenyu Guo for help in preparing the comparison to StarGANv2. This code borrows heavily from the pytorch re-implementation of StyleGAN2 by rosinality.

Comments

Add Weights & Biases Image Logging

Hi @onion-liu

I am an ML Engineer at Weights & Biases and I wanted to know if you were actively reviewing PRs at the moment? We would love to make a PR to add Weights & Biases' image logging of images if you had time to review it?

We have built integrations into transformers, YOLOv5, PyTorch Lightning etc so we should be able to make a quick and clean PR that shouldn't take too much time for you to review :)

opened by morganmcg1 4
The implementation of WBM

I really like your paper!! Thank you for sharing your work. I am a beginner in GAN and image editing and interested in your implementation of WBM module which is used to blend latent codes in two domain. I want to ask would you like to release your code about WBM?☺ Thanks.

opened by zhanjiahui 2
google colab workbook
Here is a collab workbook for you or others :

https://colab.research.google.com/github/3dsf/BlendGAN/blob/main/BlendGAN_3dsf.ipynb

It won't display the results in the book, but should download results when cell(s) have completed running.

It is written in a way, that users should be able to change folders/files with minimal effort.

note: video example is not currently working due to system ram overflow #3
opened by 3dsf 2
Video Example overflowing ram

I've tried locally and on colab. The ram over flows on writing of the video -- locally I have 32 GB.

I might try writing a different video implementation for you in a week or two, but maybe you'll find a fix sooner.

opened by 3dsf 2
About the style-latent D

Hi, thanks for your awesome work!

I read the BlendGAN paper, and I'm confused about the role the style-latent discriminator plays in the network. I wonder that will the results be negatively affected a lot if I remove this module away from the network.

Do you have any intuition or idea behind the design of style-latent D?

Looking forward to hearing from you! Thanks again.

opened by silence-tang 0
Not good result

Hi, I use your model to run reference-guided generation. However, I give a strong 3D style reference, while the result seems to be an oil panting just like this:

Is there anything wrong with me?

opened by rainsoulsrx 0
Add Docker environment & web demo

Hey @onion-liu! 👋

This pull request makes it possible to run your model inside a Docker environment, which makes it easier for other people to run it. We're using an open source tool called Cog to make this process easier.

This also means we can make a web page where other people can try out your model! We have included the face alignment to the input images. View it here: https://replicate.ai/onion-liu/blendgan

Do claim your page here so you can own the page, customise the Example gallery as you like, and push any future update to the web demo. We'll feature it on our website and tweet about it too.

In case you're wondering who I am, I'm from Replicate, where we're trying to make machine learning reproducible. We got frustrated that we couldn't run all the really interesting ML work being done. So, we're going round implementing models we like. 😊

opened by chenxwh 0

Owner

onion

GAN, Style Transfer, Image Enhancement, Infrared Image, HDR

GitHub

Official Pytorch implementation for Deep Contextual Video Compression, NeurIPS 2021

Introduction Official Pytorch implementation for Deep Contextual Video Compression, NeurIPS 2021 Prerequisites Python 3.8 and conda, get Conda CUDA 11

51 Dec 3, 2022

Official implementation of NeurIPS 2021 paper "One Loss for All: Deep Hashing with a Single Cosine Similarity based Learning Objective"

71 Dec 22, 2022

Official implementation of NeurIPS 2021 paper "Contextual Similarity Aggregation with Self-attention for Visual Re-ranking"

CSA: Contextual Similarity Aggregation with Self-attention for Visual Re-ranking PyTorch training code for CSA (Contextual Similarity Aggregation). We

19 Oct 21, 2022

Official implementation of "Open-set Label Noise Can Improve Robustness Against Inherent Label Noise" (NeurIPS 2021)

Open-set Label Noise Can Improve Robustness Against Inherent Label Noise NeurIPS 2021: This repository is the official implementation of ODNL. Require

12 Dec 7, 2022

Official implementation of Generalized Data Weighting via Class-level Gradient Manipulation (NeurIPS 2021).

Generalized Data Weighting via Class-level Gradient Manipulation This repository is the official implementation of Generalized Data Weighting via Clas

9 Nov 3, 2021

The official implementation of NeurIPS 2021 paper: Finding Optimal Tangent Points for Reducing Distortions of Hard-label Attacks

11 Nov 27, 2022

Official implementation of Neural Bellman-Ford Networks (NeurIPS 2021)

NBFNet: Neural Bellman-Ford Networks This is the official codebase of the paper Neural Bellman-Ford Networks: A General Graph Neural Network Framework

136 Dec 21, 2022

Official implementation of NeurIPS'2021 paper TransformerFusion

TransformerFusion: Monocular RGB Scene Reconstruction using Transformers Project Page | Paper | Video TransformerFusion: Monocular RGB Scene Reconstru

118 Dec 25, 2022

Official Pytorch implementation of 'GOCor: Bringing Globally Optimized Correspondence Volumes into Your Neural Network' (NeurIPS 2020)

Official implementation of GOCor This is the official implementation of our paper : GOCor: Bringing Globally Optimized Correspondence Volumes into You

71 Nov 18, 2022

Pytorch implementation of RED-SDS (NeurIPS 2021).

Recurrent Explicit Duration Switching Dynamical Systems (RED-SDS) This repository contains a reference implementation of RED-SDS, a non-linear state s

10 Dec 2, 2022

The PyTorch implementation of Directed Graph Contrastive Learning (DiGCL), NeurIPS-2021

Directed Graph Contrastive Learning The PyTorch implementation of Directed Graph Contrastive Learning (DiGCL). In this paper, we present the first con

28 Jan 8, 2023

PyTorch implementation of NeurIPS 2021 paper: "CoFiNet: Reliable Coarse-to-fine Correspondences for Robust Point Cloud Registration"

76 Jan 3, 2023

PyTorch implementation for our NeurIPS 2021 Spotlight paper "Long Short-Term Transformer for Online Action Detection".

Long Short-Term Transformer for Online Action Detection Introduction This is a PyTorch implementation for our NeurIPS 2021 Spotlight paper "Long Short

77 Dec 16, 2022

Official code for On Path Integration of Grid Cells: Group Representation and Isotropic Scaling (NeurIPS 2021)

On Path Integration of Grid Cells: Group Representation and Isotropic Scaling This repo contains the official implementation for the paper On Path Int

39 Nov 10, 2022

Official implementation of "GS-WGAN: A Gradient-Sanitized Approach for Learning Differentially Private Generators" (NeurIPS 2020)

GS-WGAN This repository contains the implementation for GS-WGAN: A Gradient-Sanitized Approach for Learning Differentially Private Generators (NeurIPS

46 Nov 9, 2022

Official implementation for Likelihood Regret: An Out-of-Distribution Detection Score For Variational Auto-encoder at NeurIPS 2020

Likelihood-Regret Official implementation of Likelihood Regret: An Out-of-Distribution Detection Score For Variational Auto-encoder at NeurIPS 2020. T

33 Oct 12, 2022