T2I_CL
This is the official Pytorch implementation of the paper Improving Text-to-Image Synthesis Using Contrastive Learning
Requirements
-
Linux
-
Python ≥ 3.6
-
PyTorch ≥ 1.4.0
Prepare Data
Download the preprocessed datasets from AttnGAN
Alternatively, another site is from DM-GAN
Training
-
Pretrain DAMSM+CL:
- For bird dataset:
python pretrain_DAMSM.py --cfg cfg/DAMSM/bird.yml --gpu 0 - For coco dataset:
python pretrain_DAMSM.py --cfg cfg/DAMSM/coco.yml --gpu 0
- For bird dataset:
-
Train AttnGAN+CL:
- For bird dataset:
python main.py --cfg cfg/bird_attn2.yml --gpu 0 - For coco dataset:
python main.py --cfg cfg/coco_attn2.yml --gpu 0
- For bird dataset:
-
Train DM-GAN+CL:
- For bird dataset:
python main.py --cfg cfg/bird_DMGAN.yml --gpu 0 - For coco dataset:
python main.py --cfg cfg/coco_DMGAN.yml --gpu 0
- For bird dataset:
Pretrained Models
- DAMSM+CL for bird. Download and save it to
DAMSMencoders/ - DAMSM+CL for coco. Download and save it to
DAMSMencoders/ - AttnGAN+CL for bird. Download and save it to
models/ - AttnGAN+CL for coco. Download and save it to
models/ - DM-GAN+CL for bird. Download and save it to
models/ - DM-GAN+CL for coco. Download and save it to
models/
Evaluation
-
Sampling and get the R-precision:
python main.py --cfg cfg/eval_bird.yml --gpu 0python main.py --cfg cfg/eval_coco.yml --gpu 0
-
Inception score:
python inception_score_bird.py --image_folder fake_images_birdpython inception_score_coco.py fake_images_coco
-
FID:
python fid_score.py --gpu 0 --batch-size 50 --path1 real_images_bird --path2 fake_images_birdpython fid_score.py --gpu 0 --batch-size 50 --path1 real_images_coco --path2 fake_images_coco
Citation
If you find this work useful in your research, please consider citing:
@article{ye2021improving,
title={Improving Text-to-Image Synthesis Using Contrastive Learning},
author={Ye, Hui and Yang, Xiulong and Takac, Martin and Sunderraman, Rajshekhar and Ji, Shihao},
journal={arXiv preprint arXiv:2107.02423},
year={2021}
}
Acknowledge
Our work is based on the following works:
28 Dec 12, 2022
42 Nov 24, 2022
146 Nov 25, 2022
83 Dec 15, 2022
24 Dec 17, 2022
44 Dec 14, 2022
565 Dec 30, 2022
183 Jan 3, 2023
809 Dec 16, 2022
8 Oct 7, 2022
2.2k Jan 1, 2023
94 Nov 12, 2022
59 Dec 6, 2022
1.8k Jan 8, 2023
58 Dec 31, 2022
46 Dec 15, 2022
49 Dec 21, 2022