Official code release for: EditGAN: High-Precision Semantic Image Editing

Overview

EditGAN

Official code release for:

EditGAN: High-Precision Semantic Image Editing

Huan Ling*, Karsten Kreis*, Daiqing Li, Seung Wook Kim, Antonio Torralba, Sanja Fidler

(* authors contributed equally)

NeurIPS 2021

[project page] [paper] [supplementary material]

Demos and results

Left: The video showcases EditGAN in an interacitve demo tool. Right: The video demonstrates EditGAN where we apply multiple edits and exploit pre-defined editing vectors. Note that the demo is accelerated. See paper for run times.

Left: The video shows interpolations and combinations of multiple editing vectors. Right: The video presents the results of applying EditGAN editing vectors on out-of-domain images.

Requirements

  • Python 3.8 is supported.

  • Pytorch >= 1.4.0.

  • The code is tested with CUDA 10.1 toolkit with Pytorch==1.4.0 and CUDA 11.4 with Pytorch==1.10.0.

  • All results in our paper are based on NVIDIA Tesla V100 GPUs with 32GB memory.

  • Set up python environment:

virtualenv env
source env/bin/activate
pip install -r requirements.txt
  • Add the project to PYTHONPATH:
export PYTHONPATH=$PWD

Use of pre-trained model

We released a pre-trained model for the car class. Follow these steps to set up our interactive WebAPP:

  • Download all checkpoints from checkpoints and put them into a ./checkpoint folder:

    • ./checkpoint/stylegan_pretrain: Download the pre-trained checkpoint from StyleGAN2 and convert the tensorflow checkpoint to pytorch. We also released the converted checkpoint for your convenience.
    • ./checkpoint/encoder_pretrain: Pre-trained encoder.
    • ./checkpoint/encoder_pretrain/testing_embedding: Test image embeddings.
    • ./checkpoint/encoder_pretrain/training_embedding: Training image embeddings.
    • ./checkpoint/datasetgan_pretrain: Pre-trained DatasetGAN (segmentation branch).
  • Run the app using python run_app.py.

  • The app is then deployed on the web browser at locolhost:8888.

Training your own model

Here, we provide step-by-step instructions to create a new EditGAN model. We use our fully released car class as an example.

  • Step 0: Train StyleGAN.

    • Download StyleGAN training images from LSUN.

    • Train your own StyleGAN model using the official StyleGAN2 code and convert the tensorflow checkpoint to pytorch. Note the specific "stylegan_checkpoint" fields in experiments/datasetgan_car.json ; experiments/encoder_car.json ; experiments/tool_car.json.

  • Step 1: Train StyleGAN Encoder.

    • Specify location of StyleGAN checkpoint in the "stylegan_checkpoint" field in experiments/encoder_car.json.

    • Specify path with training images downloaded in Step 0 in the "training_data_path" field in experiments/encoder_car.json.

    • Run python train_encoder.py --exp experiments/encoder_car.json.

  • Step 2: Train DatasetGAN.

    • Specify "stylegan_checkpoint" field in experiments/datasetgan_car.json.

    • Download DatasetGAN training images and annotations from drive and fill in "annotation_mask_path" in experiments/datasetgan_car.json.

    • Embed DatasetGAN training images in latent space using

      python train_encoder.py --exp experiments/encoder_car.json --resume *encoder checkppoint* --testing_path data/annotation_car_32_clean --latent_sv_folder model_encoder/car_batch_8_loss_sampling_train_stylegan2/training_embedding --test True
      

      and complete "optimized_latent_path" in experiments/datasetgan_car.json.

    • Train DatasetGAN (interpreter branch for segmentation) via

      python train_interpreter.py --exp experiments/datasetgan_car.json
      
  • Step 3: Run the app.

    • Download DatasetGAN test images and annotations from drive.

    • Embed DatasetGAN test images in latent space via

      python train_encoder.py --exp experiments/encoder_car.json --resume *encoder checkppoint* --testing_path *testing image path* --latent_sv_folder model_encoder/car_batch_8_loss_sampling_train_stylegan2/training_embedding --test True
      
    • Specify the "stylegan_checkpoint", "encoder_checkpoint", "classfier_checkpoint", "datasetgan_testimage_embedding_path" fields in experiments/tool_car.json.

    • Run the app via python run_app.py.

Citations

Please use the following citation if you use our data or code:

@inproceedings{ling2021editgan,
  title = {EditGAN: High-Precision Semantic Image Editing}, 
  author = {Huan Ling and Karsten Kreis and Daiqing Li and Seung Wook Kim and Antonio Torralba and Sanja Fidler},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year = {2021}
}

License

Copyright © 2022, NVIDIA Corporation. All rights reserved.

This work is made available under the Nvidia Source Code License-NC. Please see our main LICENSE file.

License Dependencies

For any code dependencies related to StyleGAN2, the license is the Nvidia Source Code License-NC by NVIDIA Corporation, see StyleGAN2 LICENSE.

For any code dependencies related to DatasetGAN, the license is the MIT License, see DatasetGAN LICENSE.

The dataset of DatasetGAN is released under the Creative Commons BY-NC 4.0 license by NVIDIA Corporation.

For any code dependencies related to the frontend tool (including html, css and Javascript), the license is the Nvidia Source Code License-NC. To view a copy of this license, visit ./static/LICENSE.md. To view a copy of terms of usage, visit ./static/term.txt.

Comments
  • How to edit custom image of car

    How to edit custom image of car

    Hi, I run the app and find the "upload image" button. It seems to support editing custom images. However, when I upload a custom image of the car, the semantic image on the left does not change simultaneously so that I can not edit it. Can you offer me a tutorial on how to edit custom images?

    opened by 1612190130 4
  • fail to run the app CUDA out of memory

    fail to run the app CUDA out of memory

    Hi! I've got this trouble when I tried to run the app. return torch._C._nn.upsample_bilinear2d(input, output_size, align_corners, scale_factors) torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 512.00 MiB (GPU 0; 6.00 GiB total capacity; 5.01 GiB already allocated; 0 bytes free; 5.20 GiB reserved in total by PyTorch) If reserved memory is >> allocated memor y try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF 127.0.0.1 - - [18/Nov/2022 14:52:07] "GET / HTTP/1.1" 500 - 127.0.0.1 - - [18/Nov/2022 14:52:07] "GET /favicon.ico HTTP/1.1" 404 -

    i'm new to this stuff how can i solve this problem?

    opened by talhaortas 2
  • Hi! encoder training data!

    Hi! encoder training data!

    Hello. I really impressed your paper. I have one question. What dataset did you use to train the encoder in your paper? Did you learn encoder(car model) using all the LSUN-cars used in styleganv2? thank you!

    opened by scj0709 1
  • RuntimeError: CUDA out of memory

    RuntimeError: CUDA out of memory

    When I run run_app.py, I got a exception: image I am not run run_app.py on Tesla v100. I only have 16G GPU memory, how can I make it run successfully.

    In addition I have adjust PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:21, it still cannot finish.

    opened by o98k-ok 1
  • ROI  don't initialize

    ROI don't initialize

    The roi region will accumulation everytime when i edit the mask. the region i editted before dont change even when i choose another mask. Here r some pic bellow. I wonder how i can initialize the roi when i try new edit to the mask

    first edit and the visual of generated roi image image

    second edit and the visual of generated roi image image

    as u can see, the first one track still on the second one even i didn't draw it

    opened by Ley-lele 0
  • FileNotFoundError: [Errno 2] No such file or directory: './static/results/current_editing_latent_cache.pickle'

    FileNotFoundError: [Errno 2] No such file or directory: './static/results/current_editing_latent_cache.pickle'

    Excuse me, where can I download this file?

    FileNotFoundError: [Errno 2] No such file or directory: './static/results/current_editing_latent_cache.pickle'

    opened by su161021 0
  • Question about editing of traffic flow heatmap

    Question about editing of traffic flow heatmap

    Traffic flow heat map is affected by many potential vectors, but it can not obtain obvious semantic segmentation like face or car images. Is this model still applicable in this case? 15_1010 15_62

    opened by czwangry123 0
  • Question about style interpreter

    Question about style interpreter

    What are the benefit or reason of training 10 pixel classifiers with same architecture(fully connected)?

    Because I think it will take a lot of time to train theses models.

    opened by Eric07110904 0
  • The editing vector can not work well on different viewpoints

    The editing vector can not work well on different viewpoints

    Hi, I have great fun with the excellent work! However, I find a problem that, when I create a custom editing vector on view0 ( add a row of seats) and add the editing vector to the style code of view1, the view1 does not produce a car with a new row of seats. view0: 54_out view0+editing vector: upload_54_editinglatent_to_54_latent_out_00 view1: 48_out view1+editing vector: upload_54_editinglatent_to_48_latent_out_00

    opened by 1612190130 2
  • Update train_encoder.py

    Update train_encoder.py

    without torch.no_grad() in line 62, the reconstruction image will be wrong in step 2 "Embed DatasetGAN training images in latent space using...."

    opened by jwzxgy2007 1
  •  how to convert the tensorflow checkpoint to pytorch?

    how to convert the tensorflow checkpoint to pytorch?

    i want to use the 256x256 pre-trained stylegan2 model. there are some problems to convert the tensorflow checkpoint to pytorch. can you provide a convert demo?

    opened by 506143026 3
Owner
null
Quantization library for PyTorch. Support low-precision and mixed-precision quantization, with hardware implementation through TVM.

HAWQ: Hessian AWare Quantization HAWQ is an advanced quantization library written for PyTorch. HAWQ enables low-precision and mixed-precision uniform

Zhen Dong 276 Nov 25, 2022
code for paper"A High-precision Semantic Segmentation Method Combining Adversarial Learning and Attention Mechanism"

PyTorch implementation of UAGAN(U-net Attention Generative Adversarial Networks) This repository contains the source code for the paper "A High-precis

Tong 8 Apr 25, 2022
A fast poisson image editing implementation that can utilize multi-core CPU or GPU to handle a high-resolution image input.

Poisson Image Editing - A Parallel Implementation Jiayi Weng (jiayiwen), Zixu Chen (zixuc) Poisson Image Editing is a technique that can fuse two imag

Jiayi Weng 106 Dec 4, 2022
HandTailor: Towards High-Precision Monocular 3D Hand Recovery

HandTailor This repository is the implementation code and model of the paper "HandTailor: Towards High-Precision Monocular 3D Hand Recovery" (arXiv) G

Lv Jun 108 Oct 24, 2022
Implementation for HFGI: High-Fidelity GAN Inversion for Image Attribute Editing

HFGI: High-Fidelity GAN Inversion for Image Attribute Editing High-Fidelity GAN Inversion for Image Attribute Editing Update: We released the inferenc

Tengfei Wang 362 Nov 29, 2022
(ICCV 2021) Official code of "Dressing in Order: Recurrent Person Image Generation for Pose Transfer, Virtual Try-on and Outfit Editing."

Dressing in Order (DiOr) ?? [Paper] ?? [Webpage] ?? [Running this code] The official implementation of "Dressing in Order: Recurrent Person Image Gene

Aiyu Cui 273 Dec 8, 2022
Official code release for "GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis"

GRAF This repository contains official code for the paper GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis. You can find detailed usage i

null 343 Nov 30, 2022
Code release for "Transferable Semantic Augmentation for Domain Adaptation" (CVPR 2021)

Transferable Semantic Augmentation for Domain Adaptation Code release for "Transferable Semantic Augmentation for Domain Adaptation" (CVPR 2021) Paper

null 64 Dec 7, 2022
Official codebase for running the small, filtered-data GLIDE model from GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models.

GLIDE This is the official codebase for running the small, filtered-data GLIDE model from GLIDE: Towards Photorealistic Image Generation and Editing w

OpenAI 2.9k Dec 4, 2022
Official implementation for "Style Transformer for Image Inversion and Editing" (CVPR 2022)

Style Transformer for Image Inversion and Editing (CVPR2022) https://arxiv.org/abs/2203.07932 Existing GAN inversion methods fail to provide latent co

Xueqi Hu 152 Dec 1, 2022
Source code for Acorn, the precision farming rover by Twisted Fields

Acorn precision farming rover This is the software repository for Acorn, the precision farming rover by Twisted Fields. For more information see twist

Twisted Fields 164 Nov 12, 2022
VGGFace2-HQ - A high resolution face dataset for face editing purpose

The first open source high resolution dataset for face swapping!!! A high resolution version of VGGFace2 for academic face editing purpose

Naiyuan Liu 222 Dec 2, 2022
[CVPR 2020] Interpreting the Latent Space of GANs for Semantic Face Editing

InterFaceGAN - Interpreting the Latent Space of GANs for Semantic Face Editing Figure: High-quality facial attributes editing results with InterFaceGA

GenForce: May Generative Force Be with You 1.3k Dec 1, 2022
InterFaceGAN - Interpreting the Latent Space of GANs for Semantic Face Editing

InterFaceGAN - Interpreting the Latent Space of GANs for Semantic Face Editing Figure: High-quality facial attributes editing results with InterFaceGA

GenForce: May Generative Force Be with You 1.3k Dec 3, 2022
Release of SPLASH: Dataset for semantic parse correction with natural language feedback in the context of text-to-SQL parsing

SPLASH: Semantic Parsing with Language Assistance from Humans SPLASH is dataset for the task of semantic parse correction with natural language feedba

Microsoft Research - Language and Information Technologies (MSR LIT) 35 Oct 31, 2022
This is the official code release for the paper Shape and Material Capture at Home

This is the official code release for the paper Shape and Material Capture at Home. The code enables you to reconstruct a 3D mesh and Cook-Torrance BRDF from one or more images captured with a flashlight or camera with flash.

null 88 Nov 11, 2022
Official code release for ICCV 2021 paper SNARF: Differentiable Forward Skinning for Animating Non-rigid Neural Implicit Shapes.

Official code release for ICCV 2021 paper SNARF: Differentiable Forward Skinning for Animating Non-rigid Neural Implicit Shapes.

null 229 Nov 27, 2022
Official code release for "Learned Spatial Representations for Few-shot Talking-Head Synthesis" ICCV 2021

Official code release for "Learned Spatial Representations for Few-shot Talking-Head Synthesis" ICCV 2021

Moustafa Meshry 16 Oct 5, 2022