StackGAN
Tensorflow implementation for reproducing main results in the paper StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks by Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, Dimitris Metaxas.
Dependencies
python 2.7
[Optional] Torch is needed, if use the pre-trained char-CNN-RNN text encoder.
[Optional] skip-thought is needed, if use the skip-thought text encoder.
In addition, please add the project folder to PYTHONPATH and pip install the following packages:
prettytensorprogressbarpython-dateutileasydictpandastorchfile
Data
- Download our preprocessed char-CNN-RNN text embeddings for birds and flowers and save them to
Data/.
- [Optional] Follow the instructions reedscot/icml2016 to download the pretrained char-CNN-RNN text encoders and extract text embeddings.
- Download the birds and flowers image data. Extract them to
Data/birds/andData/flowers/, respectively. - Preprocess images.
- For birds:
python misc/preprocess_birds.py - For flowers:
python misc/preprocess_flowers.py
Training
- The steps to train a StackGAN model on the CUB dataset using our preprocessed data for birds.
- Step 1: train Stage-I GAN (e.g., for 600 epochs)
python stageI/run_exp.py --cfg stageI/cfg/birds.yml --gpu 0 - Step 2: train Stage-II GAN (e.g., for another 600 epochs)
python stageII/run_exp.py --cfg stageII/cfg/birds.yml --gpu 1
- Step 1: train Stage-I GAN (e.g., for 600 epochs)
- Change
birds.ymltoflowers.ymlto train a StackGAN model on Oxford-102 dataset using our preprocessed data for flowers. *.ymlfiles are example configuration files for training/testing our models.- If you want to try your own datasets, here are some good tips about how to train GAN. Also, we encourage to try different hyper-parameters and architectures, especially for more complex datasets.
Pretrained Model
- StackGAN for birds trained from char-CNN-RNN text embeddings. Download and save it to
models/. - StackGAN for flowers trained from char-CNN-RNN text embeddings. Download and save it to
models/. - StackGAN for birds trained from skip-thought text embeddings. Download and save it to
models/(Just used the same setting as the char-CNN-RNN. We assume better results can be achieved by playing with the hyper-parameters).
Run Demos
- Run
sh demo/flowers_demo.shto generate flower samples from sentences. The results will be saved toData/flowers/example_captions/. (Need to download the char-CNN-RNN text encoder for flowers tomodels/text_encoder/. Note: this text encoder is provided by reedscot/icml2016). - Run
sh demo/birds_demo.shto generate bird samples from sentences. The results will be saved toData/birds/example_captions/.(Need to download the char-CNN-RNN text encoder for birds tomodels/text_encoder/. Note: this text encoder is provided by reedscot/icml2016). - Run
python demo/birds_skip_thought_demo.py --cfg demo/cfg/birds-skip-thought-demo.yml --gpu 2to generate bird samples from sentences. The results will be saved toData/birds/example_captions-skip-thought/. (Need to download vocabulary for skip-thought vectors toData/skipthoughts/).
Examples for birds (char-CNN-RNN embeddings), more on youtube: 
Examples for flowers (char-CNN-RNN embeddings), more on youtube: 
Save your favorite pictures generated by our models since the randomness from noise z and conditioning augmentation makes them creative enough to generate objects with different poses and viewpoints from the same discription
Citing StackGAN
If you find StackGAN useful in your research, please consider citing:
@inproceedings{han2017stackgan,
Author = {Han Zhang and Tao Xu and Hongsheng Li and Shaoting Zhang and Xiaogang Wang and Xiaolei Huang and Dimitris Metaxas},
Title = {StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks},
Year = {2017},
booktitle = {{ICCV}},
}
Our follow-up work
- StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks
- AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks [supplementary] [code]
References
375 Dec 31, 2022
78 Dec 10, 2022
31 Dec 8, 2022
7k Dec 30, 2022
3k Jan 8, 2023
373 Jan 2, 2023
2.2k Jan 1, 2023
2.9k Dec 28, 2022
2 Nov 19, 2021
117 Nov 27, 2022
349 Dec 29, 2022
7 Apr 4, 2022
1 Jan 6, 2022
150 Dec 26, 2022
148 Nov 18, 2022
59 Dec 11, 2022
23 Jul 21, 2022
502 Jan 3, 2023
13.4k Jan 8, 2023