Official implementation of "StyleCariGAN: Caricature Generation via StyleGAN Feature Map Modulation" (SIGGRAPH 2021)

Overview

StyleCariGAN: Caricature Generation via StyleGAN Feature Map Modulation

License CC BY_NC

This repository contains the official PyTorch implementation of the following paper:

StyleCariGAN: Caricature Generation via StyleGAN Feature Map Modulation
Wonjong Jang, Gwangjin Ju, Yucheol Jung, Jiaolong Yang, Xin Tong, Seungyong Lee, SIGGRAPH 2021

Requirements

  • PyTorch 1.3.1
  • torchvision 0.4.2
  • CUDA 10.1/10.2
  • dlib 19.22.0
  • requests 2.23.0
  • tqdm 4.46.2

If you are using Anaconda environment and get errors regarding compiler version mismatch, check issue #1.

Usage

First download pre-trained model weights:

bash ./download.sh

Train

python -m torch.distributed.launch --nproc_per_node=N_GPU train.py --name EXPERIMENT_NAME --freeze_D

Test

Test on user's input images:

python test.py --ckpt CHECKPOINT_PATH --input_dir INPUT_IMAGE_PATH --output_dir OUTPUT_CARICATURE_PATH --invert_images

We provide some sample images. Test on sample images:

python test.py --ckpt CHECKPOINT_PATH --input_dir examples/samples --output_dir examples/results --invert_images

It inverts latent codes from input photos and generates caricatures from latent codes.

Input image Output caricature
img1 cari1
img2 cari2
img3 cari3

Citation

If you find this code useful, please consider citing:

@article{Jang2021StyleCari,
  author    = {Wonjong Jang and Gwangjin Ju and Yucheol Jung and Jiaolong Yang and Xin Tong and Seungyong Lee},
  title     = {StyleCariGAN: Caricature Generation via StyleGAN Feature Map Modulation},
  booktitle = {ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH)},
  publisher = {ACM},
  volume = {40},
  number = {4},
  year = {2021}
}

Contact

You can have contact with [email protected] or [email protected]

License

This software is being made available under the terms in the LICENSE file.

Any exemptions to these terms requrie a licens from the Pohang University of Science and Technology.

Comments
  • Update need and add Windows compatibility

    Update need and add Windows compatibility

    Hello, Your code need updat here:

    • in model.py, line 19: old code: self.resnet = resnet18(pretrained=True) new code: self.resnet = resnet18(weights='ResNet18_Weights.DEFAULT')

    • in invert.py, line 19: old code: perceptual = torchvision.models.vgg16(pretrained=True) new code: perceptual = torchvision.models.vgg16(weights='VGG16_Weights.DEFAULT')

    And your code not Windows compatible. Need install Visual Studio Community edition and add PATH this line (where have cl.exe):

    • for Windows x64: C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.33.31629\bin\Hostx64\x64
    • for Windows x86: C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.33.31629\bin\Hostx64\x86

    I then created a testimages folder with an in and an out subfolder. I put in a JPG image and ran the test. This is what I got: J:\StyleCariGAN-master>test.py --ckpt checkpoint\StyleCariGAN\001000.pt --input_dir testimages\in --output_dir testimages\out --invert_images optimizing w optimizing w: loss_pixel: 0.4483; loss_feature: 4.0835: 100%|█| 250/250 [00:23< optimizing wp optimizing wp: loss_pixel: 0.2393; loss_feature: 2.2942: 100%|█| 2000/2000 [04: Traceback (most recent call last): File "J:\StyleCariGAN-master\test.py", line 120, in <module> invert(g_ema.photo_generator, perceptual, photo, device, args) File "J:\StyleCariGAN-master\invert.py", line 341, in invert torch.save(result, args.result_dir + f'/{args.image_name}.pt') File "C:\Users\Mykee\miniconda3\lib\site-packages\torch\serialization.py", line 376, in save with _open_file_like(f, 'wb') as opened_file: File "C:\Users\Mykee\miniconda3\lib\site-packages\torch\serialization.py", line 230, in _open_file_like return _open_file(name_or_buffer, mode) File "C:\Users\Mykee\miniconda3\lib\site-packages\torch\serialization.py", line 211, in __init__ super(_open_file, self).__init__(open(name, mode)) FileNotFoundError: [Errno 2] No such file or directory: 'testimages\\in/testimages\\in\\20200913_124752.pt'

    The .pt file(s) has not been generated or saved, so it cannot be found. How can I fix this?

    opened by mykeehu 1
  • Is there a way to generate clothes style mapping the cari avatar?

    Is there a way to generate clothes style mapping the cari avatar?

    Hi this work is fantastic!

    I wonder know, is there a way to generate clothes style mapping the cari avatar?

    For example, the man wears a suit, our model generates the full body portrait in cari style.

    Thanks first!

    opened by lizekui 1
  • bash download.sh is not working

    bash download.sh is not working

    Hello, when I executive "bash download.sh", it appears that www.dropbox.com is not connected. Could you upload the pretrained weights? Thank you very much!

    opened by OumRuk 1
  • Is InterFaceGAN;s face editing is prior than styleCariGAN in paper?

    Is InterFaceGAN;s face editing is prior than styleCariGAN in paper?

    Firstly, this work is damn fascinating! Admiring you all! Thanks!

    I am a beginner of GAN, and I wonder know a naive question, Is InterFaceGAN's face editing is prior than styleCariGAN in paper?

    For example, if I understand correctly, I want to draw an open-mouth face caricature from a close-mouth photo, I need to edit face firstly using some GAN like InterFaceGAN, and then use the prior output image, send as input to styleCariGAN?

    I test styleCariGAN's xx.pt invert codes seems not compatible with InterFaceGAN's xx.npy format, couldn't edit after style-cari processing directly. haha

    opened by lizekui 1
  • Pretrained model Caricature StyleGan2

    Pretrained model Caricature StyleGan2

    hi @wonjongg, your work is interesting! Can you share with me the pre-trained model Caricature StyleGan2 fine-tuned on the caricature dataset (WebCaricature)? Thank you!

    opened by TorRient 1
  • Number of images generated

    Number of images generated

    Hi I have tried the model, really cool! Just wondering whether it always generate 64 images? is there a way to output the 'best one'/ 'best ones'? Or do the 64 output from the style palette have style names if I just want to have one of a few output?

    Thank you

    opened by chenxwh 1
  • The code fails to run under Anaconda environment

    The code fails to run under Anaconda environment

    ! WARNING !!

    Environment

    Anaconda environment under Arch Linux. Python = 3.7 with other requirements with matching versions specified in the README. (Pytorch=1.3.1)

    Steps I took

    Run the test.py with

    python test.py --ckpt checkpoint/StyleCariGAN/001000.pt --input_dir examples/samples --output_dir examples/results --invert_images
    

    Expectation

    The code should run without error

    What I get

    The code fails to run and I get this error message.

    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Your compiler (c++) is not compatible with the compiler Pytorch was built with for this platform, which is g++ on linux. Please use g++ to to compile your extension. Alternatively, you may compile PyTorch from source using c++, and then you can also use c++ to compile your extension.

    See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help with compiling PyTorch from source. !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

                              !! WARNING !!
    

    platform=sys.platform)) Traceback (most recent call last): File "test.py", line 10, in from invert import * File "/home/ycjung/Downloads/git/StyleCariGAN/invert.py", line 12, in from exaggeration_model import StyleCariGAN File "/home/ycjung/Downloads/git/StyleCariGAN/exaggeration_model.py", line 7, in from model import PixelNorm, EqualLinear, ConstantInput, StyledConv, ToRGB, ExaggerationLayer File "/home/ycjung/Downloads/git/StyleCariGAN/model.py", line 13, in from op import FusedLeakyReLU, fused_leaky_relu, upfirdn2d File "/home/ycjung/Downloads/git/StyleCariGAN/op/init.py", line 1, in from .fused_act import FusedLeakyReLU, fused_leaky_relu File "/home/ycjung/Downloads/git/StyleCariGAN/op/fused_act.py", line 15, in os.path.join(module_path, "fused_bias_act_kernel.cu"), File "/home/ycjung/.conda/envs/stylecarigan/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 661, in load is_python_module) File "/home/ycjung/.conda/envs/stylecarigan/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 841, in _jit_compile return _import_module_from_library(name, build_directory, is_python_module) File "/home/ycjung/.conda/envs/stylecarigan/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1052, in _import_module_from_library return imp.load_module(module_name, file, path, description) File "/home/ycjung/.conda/envs/stylecarigan/lib/python3.7/imp.py", line 242, in load_module return load_dynamic(name, filename, file) File "/home/ycjung/.conda/envs/stylecarigan/lib/python3.7/imp.py", line 342, in load_dynamic return _load(spec) ImportError: /home/ycjung/.conda/envs/stylecarigan/lib/python3.7/site-packages/torch/../../../libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /tmp/torch_extensions/fused/fused.so)

    Analysis

    The testing code internally invokes g++ and compiles external modules in op/, which contains several cpp and cu files. The invoked g++ is installed on your system and is probably new (In my case it was 11.1.0). The g++ used for compiling PyTorch 1.3.1 in anaconda environment is old and does not produce binaries compatible with the external module compiled with the new g++.

    Troubleshooting

    It seems libtorch.so in pytorch/pytorch package is compiled with GCC 7.3.1. So install GCC 7.3.0 in your anaconda environment. (7.3.1 is not in anaconda)

    conda install -c anaconda gxx_linux-64=7.3.0
    

    Then run the code with environment variables specifying GCC compilers installed in the conda environment

    CC=x86_64-conda_cos6-linux-gnu-gcc CXX=x86_64-conda_cos6-linux-gnu-g++ python test.py --ckpt checkpoint/StyleCariGAN/001000.pt --input_dir examples/custom --output_dir examples/results --invert_images
    

    This solution got the code running fine. I hope it does to your environment!

    opened by ycjungSubhuman 1
  • Add Docker environment & web demo

    Add Docker environment & web demo

    Hey @wonjongg ! 👋

    This pull request makes it possible to run your model inside a Docker environment, which makes it easier for other people to run it. We're using an open source tool called Cog to make this process easier.

    This also means we can make a web page where other people can try out your model! View it here: https://replicate.ai/wonjongg/stylecarigan

    We added some examples provided in the repository in the Example tag, but do Claim your page here, so you can own the page! That means you can edit it, add or delete examples in the gallery and we'll feature it on our website and tweet about it too.

    In case you're wondering who I am, I'm from Replicate, where we're trying to make machine learning reproducible. We got frustrated that we couldn't run all the really interesting ML work being done. So, we're going round implementing models we like. 😊

    opened by chenxwh 0
Owner
Wonjong Jang
Ph.D. candidate at POSTECH
Wonjong Jang
Implementation for "Seamless Manga Inpainting with Semantics Awareness" (SIGGRAPH 2021 issue)

Seamless Manga Inpainting with Semantics Awareness [SIGGRAPH 2021](To appear) | Project Website | BibTex Introduction: Manga inpainting fills up the d

null 101 Jan 1, 2023
General Virtual Sketching Framework for Vector Line Art (SIGGRAPH 2021)

General Virtual Sketching Framework for Vector Line Art - SIGGRAPH 2021 Paper | Project Page Outline Dependencies Testing with Trained Weights Trainin

Haoran MO 118 Dec 27, 2022
Supplementary code for SIGGRAPH 2021 paper: Discovering Diverse Athletic Jumping Strategies

SIGGRAPH 2021: Discovering Diverse Athletic Jumping Strategies project page paper demo video Prerequisites Important Notes We suspect there are bugs i

null 54 Dec 6, 2022
Code for HodgeNet: Learning Spectral Geometry on Triangle Meshes, in SIGGRAPH 2021.

HodgeNet | Webpage | Paper | Video HodgeNet: Learning Spectral Geometry on Triangle Meshes Dmitriy Smirnov, Justin Solomon SIGGRAPH 2021 Set-up To ins

Dima Smirnov 61 Nov 27, 2022
Tracing Versus Freehand for Evaluating Computer-Generated Drawings (SIGGRAPH 2021)

Tracing Versus Freehand for Evaluating Computer-Generated Drawings (SIGGRAPH 2021) Zeyu Wang, Sherry Qiu, Nicole Feng, Holly Rushmeier, Leonard McMill

Zach Zeyu Wang 23 Dec 9, 2022
PyTorch implementations for our SIGGRAPH 2021 paper: Editable Free-viewpoint Video Using a Layered Neural Representation.

st-nerf We provide PyTorch implementations for our paper: Editable Free-viewpoint Video Using a Layered Neural Representation SIGGRAPH 2021 Jiakai Zha

Diplodocus 258 Jan 2, 2023
Code for the SIGGRAPH 2021 paper "Consistent Depth of Moving Objects in Video".

Consistent Depth of Moving Objects in Video This repository contains training code for the SIGGRAPH 2021 paper "Consistent Depth of Moving Objects in

Google 203 Jan 5, 2023
ManipNet: Neural Manipulation Synthesis with a Hand-Object Spatial Representation - SIGGRAPH 2021

ManipNet: Neural Manipulation Synthesis with a Hand-Object Spatial Representation - SIGGRAPH 2021 Dataset Code Demos Authors: He Zhang, Yuting Ye, Tak

HE ZHANG 194 Dec 6, 2022
[SIGGRAPH Asia 2021] DeepVecFont: Synthesizing High-quality Vector Fonts via Dual-modality Learning.

DeepVecFont This is the homepage for "DeepVecFont: Synthesizing High-quality Vector Fonts via Dual-modality Learning". Yizhi Wang and Zhouhui Lian. WI

Yizhi Wang 17 Dec 22, 2022
[SIGGRAPH Asia 2021] DeepVecFont: Synthesizing High-quality Vector Fonts via Dual-modality Learning.

DeepVecFont This is the homepage for "DeepVecFont: Synthesizing High-quality Vector Fonts via Dual-modality Learning". Yizhi Wang and Zhouhui Lian. WI

Yizhi Wang 5 Oct 22, 2021
The implemetation of Dynamic Nerual Garments proposed in Siggraph Asia 2021

DynamicNeuralGarments Introduction This repository contains the implemetation of Dynamic Nerual Garments proposed in Siggraph Asia 2021. ./GarmentMoti

null 42 Dec 27, 2022
Implementation for "Manga Filling Style Conversion with Screentone Variational Autoencoder" (SIGGRAPH ASIA 2020 issue)

Manga Filling with ScreenVAE SIGGRAPH ASIA 2020 | Project Website | BibTex This repository is for ScreenVAE introduced in the following paper "Manga F

null 30 Dec 24, 2022
Face Identity Disentanglement via Latent Space Mapping [SIGGRAPH ASIA 2020]

Face Identity Disentanglement via Latent Space Mapping Description Official Implementation of the paper Face Identity Disentanglement via Latent Space

null 150 Dec 7, 2022
A code repository associated with the paper A Benchmark for Rough Sketch Cleanup by Chuan Yan, David Vanderhaeghe, and Yotam Gingold from SIGGRAPH Asia 2020.

A Benchmark for Rough Sketch Cleanup This is the code repository associated with the paper A Benchmark for Rough Sketch Cleanup by Chuan Yan, David Va

null 33 Dec 18, 2022
TilinGNN: Learning to Tile with Self-Supervised Graph Neural Network (SIGGRAPH 2020)

TilinGNN: Learning to Tile with Self-Supervised Graph Neural Network (SIGGRAPH 2020) About The goal of our research problem is illustrated below: give

null 59 Dec 9, 2022
Code release for Local Light Field Fusion at SIGGRAPH 2019

Local Light Field Fusion Project | Video | Paper Tensorflow implementation for novel view synthesis from sparse input images. Local Light Field Fusion

null 1.1k Dec 27, 2022
Temporally Coherent GAN SIGGRAPH project.

TecoGAN This repository contains source code and materials for the TecoGAN project, i.e. code for a TEmporally COherent GAN for video super-resolution

Duc Linh Nguyen 2 Jan 18, 2022
Code release for Local Light Field Fusion at SIGGRAPH 2019

Local Light Field Fusion Project | Video | Paper Tensorflow implementation for novel view synthesis from sparse input images. Local Light Field Fusion

null 748 Nov 27, 2021