Diverse Image Generation via Self-Conditioned GANs
Project | Paper
Diverse Image Generation via Self-Conditioned GANs
Steven Liu, Tongzhou Wang, David Bau, Jun-Yan Zhu, Antonio Torralba
MIT, Adobe Research
in CVPR 2020.
Our proposed self-conditioned GAN model learns to perform clustering and image synthesis simultaneously. The model training requires no manual annotation of object classes. Here, we visualize several discovered clusters for both Places365 (top) and ImageNet (bottom). For each cluster, we show both real images and the generated samples conditioned on the cluster index.
Getting Started
Installation
- Clone this repo:
git clone https://github.com/stevliu/self-conditioned-gan.git
cd self-conditioned-gan
- Install the dependencies
conda create --name selfcondgan python=3.6
conda activate selfcondgan
conda install --file requirements.txt
conda install -c conda-forge tensorboardx
Training and Evaluation
- Train a model on CIFAR:
python train.py configs/cifar/selfcondgan.yaml
- Visualize samples and inferred clusters:
python visualize_clusters.py configs/cifar/selfcondgan.yaml --show_clusters
The samples and clusters will be saved to output/cifar/selfcondgan/clusters
. If this directory lies on an Apache server, you can open the URL to output/cifar/selfcondgan/clusters/+lightbox.html
in the browser and visualize all samples and clusters in one webpage.
- Evaluate the model's FID: You will need to first gather a set of ground truth train set images to compute metrics against.
python utils/get_gt_imgs.py --cifar
python metrics.py configs/cifar/selfcondgan.yaml --fid --every -1
You can also evaluate with other metrics by appending additional flags, such as Inception Score (--inception
), the number of covered modes + reverse-KL divergence (--modes
), and cluster metrics (--cluster_metrics
).
Pretrained Models
You can load and evaluate pretrained models on ImageNet and Places. If you have access to ImageNet or Places directories, first fill in paths to your ImageNet and/or Places dataset directories in configs/imagenet/default.yaml
and configs/places/default.yaml
respectively. You can use the following config files with the evaluation scripts, and the code will automatically download the appropriate models.
configs/pretrained/imagenet/selfcondgan.yaml
configs/pretrained/places/selfcondgan.yaml
configs/pretrained/imagenet/conditional.yaml
configs/pretrained/places/conditional.yaml
configs/pretrained/imagenet/baseline.yaml
configs/pretrained/places/baseline.yaml
Evaluation
Visualizations
To visualize generated samples and inferred clusters, run
python visualize_clusters.py config-file
You can set the flag --show_clusters
to also visualize the real inferred clusters, but this requires that you have a path to training set images.
Metrics
To obtain generation metrics, fill in paths to your ImageNet or Places dataset directories in utils/get_gt_imgs.py
and then run
python utils/get_gt_imgs.py --imagenet --places
to precompute batches of GT images for FID/FSD evaluation.
Then, you can use
python metrics.py config-file
with the appropriate flags compute the FID (--fid
), FSD (--fsd
), IS (--inception
), number of modes covered/ reverse-KL divergence (--modes
) and clustering metrics (--cluster_metrics
) for each of the checkpoints.
Training models
To train a model, set up a configuration file (examples in /configs
), and run
python train.py config-file
An example config of self-conditioned GAN on ImageNet is config/imagenet/selfcondgan.yaml
and on Places is config/places/selfcondgan.yaml
.
Some models may be too large to fit on one GPU, so you may want to add --devices DEVICE_NUMBERS
as an additional flag to do multi GPU training.
2D-experiments
For synthetic dataset experiments, first go into the 2d_mix
directory.
To train a self-conditioned GAN on the 2D-ring and 2D-grid dataset, run
python train.py --clusterer selfcondgan --data_type ring
python train.py --clusterer selfcondgan --data_type grid
You can test several other configurations via the command line arguments.
Acknowledgments
This code is heavily based on the GAN-stability code base. Our FSD code is taken from the GANseeing work. To compute inception score, we use the code provided from Shichang Tang. To compute FID, we use the code provided from TTUR. We also use pretrained classifiers given by the pytorch-playground.
We thank all the authors for their useful code.
Citation
If you use this code for your research, please cite the following work.
@inproceedings{liu2020selfconditioned,
title={Diverse Image Generation via Self-Conditioned GANs},
author={Liu, Steven and Wang, Tongzhou and Bau, David and Zhu, Jun-Yan and Torralba, Antonio},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2020}
}