PyTorch implementation of the WarpedGANSpace: Finding non-linear RBF paths in GAN latent space (ICCV 2021)

Overview

WarpedGANSpace: Finding non-linear RBF paths in GAN latent space

Authors official PyTorch implementation of the WarpedGANSpace: Finding non-linear RBF paths in GAN latent space (ICCV 2021). If you use this code for your research, please cite our paper.

Overview

In this work, we try to discover non-linear interpretable paths in GAN latent space. For doing so, we model non-linear paths using RBF-based warping functions, which by warping the latent space, endow it with vector fields (their gradients). We use the latter to traverse the latent space across the paths determined by the aforementioned vector fields for any given latent code.

WarpedGANSpace Overview

Each warping function is defined by a set of N support vectors (a "support set") and its gradient is given analytically as shown above. For a given warping function fk and a given latent code z, we traverse the latent space as illustrated below:

Non-linear interpretable path

Each warping function gives rise to a family of non-linear paths. We learn a set of such warping functions (implemented by the *Warping Network*), i.e., a set of such non-linear path families, so as they are distinguishable to each other; that is, the image transformations that they produce should be easily distinguishable be a discriminator network (the *Reconstructor*). An overview of the method is given below.

WarpedGANSpace Overview

Installation

We recommend installing the required packages using python's native virtual environment. For Python 3.4+, this can be done as follows:

$ python -m venv warped-gan-space
$ source warped-gan-space/bin/activate
(warped-gan-space) $ pip install --upgrade pip
(warped-gan-space) $ pip install -r requirements.txt

Prerequisite pretrained models

Download the prerequisite pretrained models (i.e., GAN generators, face detector, pose estimator, etc.) as follows:

$ python download.py	

This will create a directory models/pretrained with the following sub-directories (~3.2GiB):

./models/pretrained/
├── generators/
├── arcface/
├── fairface/
├── hopenet/
└── sfd/

Training

For training a WarpedGANSpace model you need to use train.py (check its basic usage by running python train.py -h).

For example, in order to train a WarpedGANSpace model on the ProgGAN pre-trained (on CelebA) generator for discovering K=128 interpretable paths (latent warping functions) with N=32 support dipoles each (i.e., 32 pairs of bipolar RBFs) run the following command:

python train.py -v --gan-type=ProgGAN --reconstructor-type=ResNet --learn-gammas --num-support-sets=128 --num-support-dipoles=32 --min-shift-magnitude=0.15 --max-shift-magnitude=0.25 --batch-size=8 --max-iter=200000

In the example above, batch size is set to 8 and the training will be conducted for 200000 iterations. Minimum and maximum shift magnitudes are set to 0.15 and 0.25, respectively (please see Sect. 3.2 in the paper for more details). A set of auxiliary training scripts (for all available GAN generators) can be found under scripts/train/.

The training script will create a directory with the following name format:


   
    (-
    
     )-
     
      -K
      
       -N
       
        (-LearnAlphas)(-LearnGammas)-eps
        
         _
          
         
        
       
      
     
    
   

E.g., ProgGAN-ResNet-K128-N128-LearnGammas-eps0.15_0.25, under experiments/wip/ while training is in progress, which after training completion, will be copied under experiments/complete/. This directory has the following structure:

├── models/
├── tensorboard/
├── args.json
├── stats.json
└── command.sh

where models/ contains the weights for the reconstructor (reconstructor.pt) and the support sets (support_sets.pt). While training is in progress (i.e., while this directory is found under experiments/wip/), the corresponding models/ directory contains a checkpoint file (checkpoint.pt) containing the last iteration, and the weights for the reconstructor and the support sets, so as to resume training. Re-run the same command, and if the last iteration is less than the given maximum number of iterations, training will resume from the last iteration. This directory will be referred to as EXP_DIR for the rest of this document.

Evaluation

After a WarpedGANSpace is trained, the corresponding experiment's directory (i.e., EXP_DIR) can be found under experiments/complete/. The evaluation of the model includes the following steps:

  • Latent space traversals For a given set of latent codes, we first generate images for all K paths (warping functions) and save the traversals (path latent codes and generated image sequences).
  • Attribute space traversals In the case of facial images (i.e., ProgGAN and StyleGAN2), for the latent traversals above, we calculate the corresponding attribute paths (i.e., facial expressions, pose, etc.).
  • Interpretable paths discovery and ranking [To Appear Soon]

Before calculating latent space traversals, you need to create a pool of latent codes/images for the corresponding GAN type. This can be done using sample_gan.py. The name of the pool can be passed using --pool; if left empty will be used instead. The pool of latent codes/images will be stored under experiments/latent_codes/ / . We will be referring to it as a POOL for the rest of this document.

For example, the following command will create a pool named ProgGAN_4 under experiments/latent_codes/ProgGAN/:

python sample_gan.py -v --gan-type=ProgGAN --num-samples=4

Latent space traversals

Latent space traversals can be calculated using the script traverse_latent_space.py (please check its basic usage by running traverse_latent_space.py -h) for a given model and a given POOL.

Attribute space traversals

[To Appear Soon]

Interpretable paths discovery and ranking

[To Appear Soon]

Citation

[1] Christos Tzelepis, Georgios Tzimiropoulos, and Ioannis Patras. WarpedGANSpace: Finding non-linear rbf paths in gan latent space. IEEE International Conference on Computer Vision (ICCV), 2021.

Bibtex entry:

@inproceedings{warpedganspace,
  title={{WarpedGANSpace}: Finding non-linear {RBF} paths in {GAN} latent space},
  author={Tzelepis, Christos and Tzimiropoulos, Georgios and Patras, Ioannis},
  booktitle={IEEE International Conference on Computer Vision (ICCV)},
  year={2021}
}

Acknowledgment

This research was supported by the EU's Horizon 2020 programme H2020-951911 AI4Media project.

You might also like...
With this package, you can generate mixed-integer linear programming (MIP) models of trained artificial neural networks (ANNs) using the rectified linear unit (ReLU) activation function

With this package, you can generate mixed-integer linear programming (MIP) models of trained artificial neural networks (ANNs) using the rectified linear unit (ReLU) activation function. At the moment, only TensorFlow sequential models are supported. Interfaces to either the Pyomo or Gurobi modeling environments are offered.

Simple Linear 2nd ODE Solver GUI - A 2nd constant coefficient linear ODE solver with simple GUI using euler's method
Simple Linear 2nd ODE Solver GUI - A 2nd constant coefficient linear ODE solver with simple GUI using euler's method

Simple_Linear_2nd_ODE_Solver_GUI Description It is a 2nd constant coefficient li

Hitters Linear Regression - Hitters Linear Regression With Python
Hitters Linear Regression - Hitters Linear Regression With Python

Hitters_Linear_Regression Kullanacağımız veri seti Carnegie Mellon Üniversitesi'

Official pytorch code for SSC-GAN: Semi-Supervised Single-Stage Controllable GANs for Conditional Fine-Grained Image Generation(ICCV 2021)

SSC-GAN_repo Pytorch implementation for 'Semi-Supervised Single-Stage Controllable GANs for Conditional Fine-Grained Image Generation'.PDF SSC-GAN:Sem

source code for 'Finding Valid Adjustments under Non-ignorability with Minimal DAG Knowledge' by A. Shah, K. Shanmugam, K. Ahuja

Source code for "Finding Valid Adjustments under Non-ignorability with Minimal DAG Knowledge" Reference: Abhin Shah, Karthikeyan Shanmugam, Kartik Ahu

Generating images from caption and vice versa via CLIP-Guided Generative Latent Space Search

CLIP-GLaSS Repository for the paper Generating images from caption and vice versa via CLIP-Guided Generative Latent Space Search An in-browser demo is

Face Identity Disentanglement via Latent Space Mapping [SIGGRAPH ASIA 2020]
Face Identity Disentanglement via Latent Space Mapping [SIGGRAPH ASIA 2020]

Face Identity Disentanglement via Latent Space Mapping Description Official Implementation of the paper Face Identity Disentanglement via Latent Space

Navigating StyleGAN2 w latent space using CLIP
Navigating StyleGAN2 w latent space using CLIP

Navigating StyleGAN2 w latent space using CLIP an attempt to build sth with the official SG2-ADA Pytorch impl kinda inspired by Generating Images from

[CVPR 2020] Interpreting the Latent Space of GANs for Semantic Face Editing
[CVPR 2020] Interpreting the Latent Space of GANs for Semantic Face Editing

InterFaceGAN - Interpreting the Latent Space of GANs for Semantic Face Editing Figure: High-quality facial attributes editing results with InterFaceGA

Comments
Owner
Christos Tzelepis
Postdoctoral research associate at Queen Mary University of London | MultiMedia & Vision Research Group (MMV Group).
Christos Tzelepis
FuseDream: Training-Free Text-to-Image Generationwith Improved CLIP+GAN Space OptimizationFuseDream: Training-Free Text-to-Image Generationwith Improved CLIP+GAN Space Optimization

FuseDream This repo contains code for our paper (paper link): FuseDream: Training-Free Text-to-Image Generation with Improved CLIP+GAN Space Optimizat

XCL 191 Dec 31, 2022
Non-Official Pytorch implementation of "Face Identity Disentanglement via Latent Space Mapping" https://arxiv.org/abs/2005.07728 Using StyleGAN2 instead of StyleGAN

Face Identity Disentanglement via Latent Space Mapping - Implement in pytorch with StyleGAN 2 Description Pytorch implementation of the paper Face Ide

Daniel Roich 58 Dec 24, 2022
Linear algebra python - Number of operations and problems in Linear Algebra and Numerical Linear Algebra

Linear algebra in python Number of operations and problems in Linear Algebra and

Alireza 5 Oct 9, 2022
A non-linear, non-parametric Machine Learning method capable of modeling complex datasets

Fast Symbolic Regression Symbolic Regression is a non-linear, non-parametric Machine Learning method capable of modeling complex data sets. fastsr aim

VAMSHI CHOWDARY 3 Jun 22, 2022
Disentangled Face Attribute Editing via Instance-Aware Latent Space Search, accepted by IJCAI 2021.

Instance-Aware Latent-Space Search This is a PyTorch implementation of the following paper: Disentangled Face Attribute Editing via Instance-Aware Lat

null 67 Dec 21, 2022
Implementation of TransGanFormer, an all-attention GAN that combines the finding from the recent GanFormer and TransGan paper

TransGanFormer (wip) Implementation of TransGanFormer, an all-attention GAN that combines the finding from the recent GansFormer and TransGan paper. I

Phil Wang 146 Dec 6, 2022
An official implementation of "Exploiting a Joint Embedding Space for Generalized Zero-Shot Semantic Segmentation" (ICCV 2021) in PyTorch.

Exploiting a Joint Embedding Space for Generalized Zero-Shot Semantic Segmentation This is an official implementation of the paper "Exploiting a Joint

CV Lab @ Yonsei University 35 Oct 26, 2022
Official Implementation of LARGE: Latent-Based Regression through GAN Semantics

LARGE: Latent-Based Regression through GAN Semantics [Project Website] [Google Colab] [Paper] LARGE: Latent-Based Regression through GAN Semantics Yot

null 83 Dec 6, 2022
Implementation based on Paper - Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

Implementation based on Paper - Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

HamasKhan 3 Jul 8, 2022