Official code release for: EditGAN: High-Precision Semantic Image Editing

Overview

EditGAN

Official code release for:

EditGAN: High-Precision Semantic Image Editing

Huan Ling*, Karsten Kreis*, Daiqing Li, Seung Wook Kim, Antonio Torralba, Sanja Fidler

(* authors contributed equally)

NeurIPS 2021

[project page] [paper] [supplementary material]

Demos and results

Left: The video showcases EditGAN in an interacitve demo tool. Right: The video demonstrates EditGAN where we apply multiple edits and exploit pre-defined editing vectors. Note that the demo is accelerated. See paper for run times.

Left: The video shows interpolations and combinations of multiple editing vectors. Right: The video presents the results of applying EditGAN editing vectors on out-of-domain images.

Requirements

  • Python 3.8 is supported.

  • Pytorch >= 1.4.0.

  • The code is tested with CUDA 10.1 toolkit with Pytorch==1.4.0 and CUDA 11.4 with Pytorch==1.10.0.

  • All results in our paper are based on NVIDIA Tesla V100 GPUs with 32GB memory.

  • Set up python environment:

virtualenv env
source env/bin/activate
pip install -r requirements.txt
  • Add the project to PYTHONPATH:
export PYTHONPATH=$PWD

Use of pre-trained model

We released a pre-trained model for the car class. Follow these steps to set up our interactive WebAPP:

  • Download all checkpoints from checkpoints and put them into a ./checkpoint folder:

    • ./checkpoint/stylegan_pretrain: Download the pre-trained checkpoint from StyleGAN2 and convert the tensorflow checkpoint to pytorch. We also released the converted checkpoint for your convenience.
    • ./checkpoint/encoder_pretrain: Pre-trained encoder.
    • ./checkpoint/encoder_pretrain/testing_embedding: Test image embeddings.
    • ./checkpoint/encoder_pretrain/training_embedding: Training image embeddings.
    • ./checkpoint/datasetgan_pretrain: Pre-trained DatasetGAN (segmentation branch).
  • Run the app using python run_app.py.

  • The app is then deployed on the web browser at locolhost:8888.

Training your own model

Here, we provide step-by-step instructions to create a new EditGAN model. We use our fully released car class as an example.

  • Step 0: Train StyleGAN.

    • Download StyleGAN training images from LSUN.

    • Train your own StyleGAN model using the official StyleGAN2 code and convert the tensorflow checkpoint to pytorch. Note the specific "stylegan_checkpoint" fields in experiments/datasetgan_car.json ; experiments/encoder_car.json ; experiments/tool_car.json.

  • Step 1: Train StyleGAN Encoder.

    • Specify location of StyleGAN checkpoint in the "stylegan_checkpoint" field in experiments/encoder_car.json.

    • Specify path with training images downloaded in Step 0 in the "training_data_path" field in experiments/encoder_car.json.

    • Run python train_encoder.py --exp experiments/encoder_car.json.

  • Step 2: Train DatasetGAN.

    • Specify "stylegan_checkpoint" field in experiments/datasetgan_car.json.

    • Download DatasetGAN training images and annotations from drive and fill in "annotation_mask_path" in experiments/datasetgan_car.json.

    • Embed DatasetGAN training images in latent space using

      python train_encoder.py --exp experiments/encoder_car.json --resume *encoder checkppoint* --testing_path data/annotation_car_32_clean --latent_sv_folder model_encoder/car_batch_8_loss_sampling_train_stylegan2/training_embedding --test True
      

      and complete "optimized_latent_path" in experiments/datasetgan_car.json.

    • Train DatasetGAN (interpreter branch for segmentation) via

      python train_interpreter.py --exp experiments/datasetgan_car.json
      
  • Step 3: Run the app.

    • Download DatasetGAN test images and annotations from drive.

    • Embed DatasetGAN test images in latent space via

      python train_encoder.py --exp experiments/encoder_car.json --resume *encoder checkppoint* --testing_path *testing image path* --latent_sv_folder model_encoder/car_batch_8_loss_sampling_train_stylegan2/training_embedding --test True
      
    • Specify the "stylegan_checkpoint", "encoder_checkpoint", "classfier_checkpoint", "datasetgan_testimage_embedding_path" fields in experiments/tool_car.json.

    • Run the app via python run_app.py.

Citations

Please use the following citation if you use our data or code:

@inproceedings{ling2021editgan,
  title = {EditGAN: High-Precision Semantic Image Editing}, 
  author = {Huan Ling and Karsten Kreis and Daiqing Li and Seung Wook Kim and Antonio Torralba and Sanja Fidler},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year = {2021}
}

License

Copyright © 2022, NVIDIA Corporation. All rights reserved.

This work is made available under the Nvidia Source Code License-NC. Please see our main LICENSE file.

License Dependencies

For any code dependencies related to StyleGAN2, the license is the Nvidia Source Code License-NC by NVIDIA Corporation, see StyleGAN2 LICENSE.

For any code dependencies related to DatasetGAN, the license is the MIT License, see DatasetGAN LICENSE.

The dataset of DatasetGAN is released under the Creative Commons BY-NC 4.0 license by NVIDIA Corporation.

For any code dependencies related to the frontend tool (including html, css and Javascript), the license is the Nvidia Source Code License-NC. To view a copy of this license, visit ./static/LICENSE.md. To view a copy of terms of usage, visit ./static/term.txt.

Issues
  • RuntimeError: CUDA out of memory

    RuntimeError: CUDA out of memory

    When I run run_app.py, I got a exception: image I am not run run_app.py on Tesla v100. I only have 16G GPU memory, how can I make it run successfully.

    In addition I have adjust PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:21, it still cannot finish.

    opened by o98k-ok 1
  • FileNotFoundError: [Errno 2] No such file or directory: './static/results/current_editing_latent_cache.pickle'

    FileNotFoundError: [Errno 2] No such file or directory: './static/results/current_editing_latent_cache.pickle'

    Excuse me, where can I download this file?

    FileNotFoundError: [Errno 2] No such file or directory: './static/results/current_editing_latent_cache.pickle'

    opened by su161021 0
  •  how to convert the tensorflow checkpoint to pytorch?

    how to convert the tensorflow checkpoint to pytorch?

    i want to use the 256x256 pre-trained stylegan2 model. there are some problems to convert the tensorflow checkpoint to pytorch. can you provide a convert demo?

    opened by 506143026 0
  • fail to run the app

    fail to run the app

    Hi ! thanks for the great contribution. I've got some trouble when I tried to run the app.Similar to #8, I managed to setup the app but it seemed the UI is not working. All the buttons are not functional. The window is just like this:

    2022-05-09 14-31-38

    And my log info is as follow: python run_app.py

    • Serving Flask app 'run_app' (lazy loading)
    • Environment: production WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
    • Debug mode: off
    • Running on all addresses (0.0.0.0) WARNING: This is a development server. Do not use it in a production deployment.
    • Running on http://127.0.0.1:8888
    • Running on http://192.168.1.111:8888 (Press CTRL+C to quit) Load stylegan from, ./checkpoint/stylegan_pretrain/stylegan2_networks_stylegan2-car-config-f.pt at res, 512 make_mean_latent Load Classifier path, ./checkpoint/datasetgan_pretrain/classifier Setting up Perceptual loss... Loading model from: /home/hp/editGAN/lpips/weights/v0.1/vgg.pth ...[net-lin [vgg]] initialized ...Done 0%| | 0/10 [00:00<?, ?it/s]/home/hp/.conda/envs/editGAN/lib/python3.8/site-packages/torch/nn/functional.py:3631: UserWarning: Default upsampling behavior when mode=bilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details. warnings.warn( 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:08<00:00, 1.24it/s] TOOL init!! 192.168.1.111 - - [09/May/2022 14:22:31] "GET / HTTP/1.1" 200 - 192.168.1.111 - - [09/May/2022 14:22:31] "GET /static/demo.css HTTP/1.1" 200 - 192.168.1.111 - - [09/May/2022 14:22:31] "GET /static/demo_origin.js HTTP/1.1" 200 - 192.168.1.111 - - [09/May/2022 14:22:31] "GET /static/nvidia.png HTTP/1.1" 200 - 192.168.1.111 - - [09/May/2022 14:22:31] "GET /static/loading.gif HTTP/1.1" 404 - 192.168.1.111 - - [09/May/2022 14:24:41] "GET /static/brush_circle.png HTTP/1.1" 200 - 192.168.1.111 - - [09/May/2022 14:24:41] "GET /static/brush_square.png HTTP/1.1" 200 - 192.168.1.111 - - [09/May/2022 14:24:41] "GET /static/brush_diamond.png HTTP/1.1" 200 - 192.168.1.111 - - [09/May/2022 14:24:41] "GET /static/paint-brush.png HTTP/1.1" 200 - 192.168.1.111 - - [09/May/2022 14:24:41] "GET /static/paint-can.png HTTP/1.1" 200 - 192.168.1.111 - - [09/May/2022 14:24:41] "GET /static/eyedropper.png HTTP/1.1" 200 - 192.168.1.111 - - [09/May/2022 14:24:41] "GET /static/undo.png HTTP/1.1" 200 - 192.168.1.111 - - [09/May/2022 14:24:41] "GET /static/save.png HTTP/1.1" 200 - 192.168.1.111 - - [09/May/2022 14:24:41] "GET /static/run.png HTTP/1.1" 200 - 192.168.1.111 - - [09/May/2022 14:24:41] "GET /static/random.png HTTP/1.1" 200 - 192.168.1.111 - - [09/May/2022 14:24:41] "GET /static/images/car_real/0.jpg HTTP/1.1" 200 - 192.168.1.111 - - [09/May/2022 14:24:41] "GET /static/images/car_real/1.jpg HTTP/1.1" 200 - 192.168.1.111 - - [09/May/2022 14:24:41] "GET /static/images/car_real/2.jpg HTTP/1.1" 200 - 192.168.1.111 - - [09/May/2022 14:24:41] "GET /static/images/car_real/3.jpg HTTP/1.1" 200 - 192.168.1.111 - - [09/May/2022 14:24:41] "GET /static/images/car_real/5.jpg HTTP/1.1" 200 - 192.168.1.111 - - [09/May/2022 14:24:41] "GET /static/images/car_real/4.jpg HTTP/1.1" 200 - 192.168.1.111 - - [09/May/2022 14:24:41] "GET /static/images/car_real/6.jpg HTTP/1.1" 200 - 192.168.1.111 - - [09/May/2022 14:24:41] "GET /static/images/car_real/7.jpg HTTP/1.1" 200 - 192.168.1.111 - - [09/May/2022 14:24:41] "GET /static/images/car_real/8.jpg HTTP/1.1" 200 - 192.168.1.111 - - [09/May/2022 14:24:41] "GET /static/images/car_real/10.jpg HTTP/1.1" 404 - 192.168.1.111 - - [09/May/2022 14:24:41] "GET /static/images/car_real/9.jpg HTTP/1.1" 200 - 192.168.1.111 - - [09/May/2022 14:24:41] "GET /static/info.png HTTP/1.1" 200 - 192.168.1.111 - - [09/May/2022 14:24:42] "GET /favicon.ico HTTP/1.1" 404 -

    I have no idea what's wrong with it. Any idea? Thank you!

    opened by cannotseeme 7
  • How to edit custom image of car

    How to edit custom image of car

    Hi, I run the app and find the "upload image" button. It seems to support editing custom images. However, when I upload a custom image of the car, the semantic image on the left does not change simultaneously so that I can not edit it. Can you offer me a tutorial on how to edit custom images?

    opened by 1612190130 4
  • add web demo/model to Huggingface

    add web demo/model to Huggingface

    Hi, would you be interested in adding editGAN to Hugging Face? The Hub offers free hosting, and it would make your work more accessible and visible to the rest of the ML community. Models/datasets/spaces(web demos) can be added to a user account or organization similar to github.

    Example from other organizations: Keras: https://huggingface.co/keras-io Microsoft: https://huggingface.co/microsoft Facebook: https://huggingface.co/facebook

    Example spaces with repos: github: https://github.com/salesforce/BLIP Spaces: https://huggingface.co/spaces/salesforce/BLIP

    github: https://github.com/facebookresearch/omnivore Spaces: https://huggingface.co/spaces/akhaliq/omnivore

    and here are guides for adding spaces/models/datasets to your org

    How to add a Space: https://huggingface.co/blog/gradio-spaces how to add models: https://huggingface.co/docs/hub/adding-a-model uploading a dataset: https://huggingface.co/docs/datasets/upload_dataset.html

    Please let us know if you would be interested and if you have any questions, we can also help with the technical implementation.

    opened by AK391 0
  • Failure getting EditGAN to run in Colab

    Failure getting EditGAN to run in Colab

    Thanks for sharing your code! I tried getting this to load up in Google Colab. After some hassle and experimenting, I got it to the state where it's loading up at least.

    If I try to click something like checking the box, or uploading a file, I see the following reports: image

    image

    The UI seems unresponsive. It is not perfectly clear whether that is due to a problem, or because I don't know how to use the software. Any hints?

    Thank you 🙏

    opened by dubtor 11
Quantization library for PyTorch. Support low-precision and mixed-precision quantization, with hardware implementation through TVM.

HAWQ: Hessian AWare Quantization HAWQ is an advanced quantization library written for PyTorch. HAWQ enables low-precision and mixed-precision uniform

Zhen Dong 237 Jun 19, 2022
code for paper"A High-precision Semantic Segmentation Method Combining Adversarial Learning and Attention Mechanism"

PyTorch implementation of UAGAN(U-net Attention Generative Adversarial Networks) This repository contains the source code for the paper "A High-precis

Tong 8 Apr 25, 2022
A fast poisson image editing implementation that can utilize multi-core CPU or GPU to handle a high-resolution image input.

Poisson Image Editing - A Parallel Implementation Jiayi Weng (jiayiwen), Zixu Chen (zixuc) Poisson Image Editing is a technique that can fuse two imag

Jiayi Weng 22 Jun 22, 2022
HandTailor: Towards High-Precision Monocular 3D Hand Recovery

HandTailor This repository is the implementation code and model of the paper "HandTailor: Towards High-Precision Monocular 3D Hand Recovery" (arXiv) G

Lv Jun 102 Jun 27, 2022
Implementation for HFGI: High-Fidelity GAN Inversion for Image Attribute Editing

HFGI: High-Fidelity GAN Inversion for Image Attribute Editing High-Fidelity GAN Inversion for Image Attribute Editing Update: We released the inferenc

Tengfei Wang 290 Jun 21, 2022
(ICCV 2021) Official code of "Dressing in Order: Recurrent Person Image Generation for Pose Transfer, Virtual Try-on and Outfit Editing."

Dressing in Order (DiOr) ?? [Paper] ?? [Webpage] ?? [Running this code] The official implementation of "Dressing in Order: Recurrent Person Image Gene

Aiyu Cui 221 Jun 23, 2022
Official code release for "GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis"

GRAF This repository contains official code for the paper GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis. You can find detailed usage i

null 296 Jun 27, 2022
Code release for "Transferable Semantic Augmentation for Domain Adaptation" (CVPR 2021)

Transferable Semantic Augmentation for Domain Adaptation Code release for "Transferable Semantic Augmentation for Domain Adaptation" (CVPR 2021) Paper

null 59 Jun 24, 2022
Official codebase for running the small, filtered-data GLIDE model from GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models.

GLIDE This is the official codebase for running the small, filtered-data GLIDE model from GLIDE: Towards Photorealistic Image Generation and Editing w

OpenAI 2.5k Jun 29, 2022
Official implementation for "Style Transformer for Image Inversion and Editing" (CVPR 2022)

Style Transformer for Image Inversion and Editing (CVPR2022) https://arxiv.org/abs/2203.07932 Existing GAN inversion methods fail to provide latent co

Xueqi Hu 109 Jun 28, 2022
VGGFace2-HQ - A high resolution face dataset for face editing purpose

The first open source high resolution dataset for face swapping!!! A high resolution version of VGGFace2 for academic face editing purpose

Naiyuan Liu 177 Jun 23, 2022
[CVPR 2020] Interpreting the Latent Space of GANs for Semantic Face Editing

InterFaceGAN - Interpreting the Latent Space of GANs for Semantic Face Editing Figure: High-quality facial attributes editing results with InterFaceGA

GenForce: May Generative Force Be with You 1.2k Jun 28, 2022
InterFaceGAN - Interpreting the Latent Space of GANs for Semantic Face Editing

InterFaceGAN - Interpreting the Latent Space of GANs for Semantic Face Editing Figure: High-quality facial attributes editing results with InterFaceGA

GenForce: May Generative Force Be with You 1.2k Jun 23, 2022
Source code for Acorn, the precision farming rover by Twisted Fields

Acorn precision farming rover This is the software repository for Acorn, the precision farming rover by Twisted Fields. For more information see twist

Twisted Fields 156 Jun 17, 2022
Release of SPLASH: Dataset for semantic parse correction with natural language feedback in the context of text-to-SQL parsing

SPLASH: Semantic Parsing with Language Assistance from Humans SPLASH is dataset for the task of semantic parse correction with natural language feedba

Microsoft Research - Language and Information Technologies (MSR LIT) 30 Jun 18, 2022
This is the official code release for the paper Shape and Material Capture at Home

This is the official code release for the paper Shape and Material Capture at Home. The code enables you to reconstruct a 3D mesh and Cook-Torrance BRDF from one or more images captured with a flashlight or camera with flash.

null 83 Jun 9, 2022
Official code release for ICCV 2021 paper SNARF: Differentiable Forward Skinning for Animating Non-rigid Neural Implicit Shapes.

Official code release for ICCV 2021 paper SNARF: Differentiable Forward Skinning for Animating Non-rigid Neural Implicit Shapes.

null 192 Jun 14, 2022
Official code release for "Learned Spatial Representations for Few-shot Talking-Head Synthesis" ICCV 2021

Official code release for "Learned Spatial Representations for Few-shot Talking-Head Synthesis" ICCV 2021

Moustafa Meshry 13 Jun 20, 2022