StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks

Related tags

Deep Learning StackGAN
Overview

StackGAN

Tensorflow implementation for reproducing main results in the paper StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks by Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, Dimitris Metaxas.

Dependencies

python 2.7

TensorFlow 0.12

[Optional] Torch is needed, if use the pre-trained char-CNN-RNN text encoder.

[Optional] skip-thought is needed, if use the skip-thought text encoder.

In addition, please add the project folder to PYTHONPATH and pip install the following packages:

  • prettytensor
  • progressbar
  • python-dateutil
  • easydict
  • pandas
  • torchfile

Data

  1. Download our preprocessed char-CNN-RNN text embeddings for birds and flowers and save them to Data/.
  • [Optional] Follow the instructions reedscot/icml2016 to download the pretrained char-CNN-RNN text encoders and extract text embeddings.
  1. Download the birds and flowers image data. Extract them to Data/birds/ and Data/flowers/, respectively.
  2. Preprocess images.
  • For birds: python misc/preprocess_birds.py
  • For flowers: python misc/preprocess_flowers.py

Training

  • The steps to train a StackGAN model on the CUB dataset using our preprocessed data for birds.
    • Step 1: train Stage-I GAN (e.g., for 600 epochs) python stageI/run_exp.py --cfg stageI/cfg/birds.yml --gpu 0
    • Step 2: train Stage-II GAN (e.g., for another 600 epochs) python stageII/run_exp.py --cfg stageII/cfg/birds.yml --gpu 1
  • Change birds.yml to flowers.yml to train a StackGAN model on Oxford-102 dataset using our preprocessed data for flowers.
  • *.yml files are example configuration files for training/testing our models.
  • If you want to try your own datasets, here are some good tips about how to train GAN. Also, we encourage to try different hyper-parameters and architectures, especially for more complex datasets.

Pretrained Model

  • StackGAN for birds trained from char-CNN-RNN text embeddings. Download and save it to models/.
  • StackGAN for flowers trained from char-CNN-RNN text embeddings. Download and save it to models/.
  • StackGAN for birds trained from skip-thought text embeddings. Download and save it to models/ (Just used the same setting as the char-CNN-RNN. We assume better results can be achieved by playing with the hyper-parameters).

Run Demos

  • Run sh demo/flowers_demo.sh to generate flower samples from sentences. The results will be saved to Data/flowers/example_captions/. (Need to download the char-CNN-RNN text encoder for flowers to models/text_encoder/. Note: this text encoder is provided by reedscot/icml2016).
  • Run sh demo/birds_demo.sh to generate bird samples from sentences. The results will be saved to Data/birds/example_captions/.(Need to download the char-CNN-RNN text encoder for birds to models/text_encoder/. Note: this text encoder is provided by reedscot/icml2016).
  • Run python demo/birds_skip_thought_demo.py --cfg demo/cfg/birds-skip-thought-demo.yml --gpu 2 to generate bird samples from sentences. The results will be saved to Data/birds/example_captions-skip-thought/. (Need to download vocabulary for skip-thought vectors to Data/skipthoughts/).

Examples for birds (char-CNN-RNN embeddings), more on youtube:

Examples for flowers (char-CNN-RNN embeddings), more on youtube:

Save your favorite pictures generated by our models since the randomness from noise z and conditioning augmentation makes them creative enough to generate objects with different poses and viewpoints from the same discription 😃

Citing StackGAN

If you find StackGAN useful in your research, please consider citing:

@inproceedings{han2017stackgan,
Author = {Han Zhang and Tao Xu and Hongsheng Li and Shaoting Zhang and Xiaogang Wang and Xiaolei Huang and Dimitris Metaxas},
Title = {StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks},
Year = {2017},
booktitle = {{ICCV}},
}

Our follow-up work

References

  • Generative Adversarial Text-to-Image Synthesis Paper Code
  • Learning Deep Representations of Fine-grained Visual Descriptions Paper Code
Comments
  • Where to find example_captions.t7

    Where to find example_captions.t7

    Hi I've followed the install instructions in the README, but I can't for the life of me figure out where to find Data/flowers/example_captions.t7? Same with birds. Are they simply the files like lm_sje_nc4_cub_hybrid_gru18_a1_c512_ ... 70_1_10_trainvalids.txt_iter30000.t7 files that have been renamed?

    opened by brannondorsey 13

  • "Too many values to unpack" (stageI training)

    Ubuntu 16.04 Python 2.7 Tensorflow 0.12.0 Two GE Force GTX 1080 TI Core I7 64 GB RAM

    When running stageI training with GPU enabled:

    python stageI/run_exp.py --cfg stageI/cfg/birds.yml --gpu 1

    Stack Trace

    Free memory: 10.75GiB 2017-10-17 21:02:08.926351: I tensorflow/core/common_runtime/gpu/gpu_device.cc:961] DMA: 0 1 2017-10-17 21:02:08.926360: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0: Y Y 2017-10-17 21:02:08.926363: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 1: Y Y 2017-10-17 21:02:08.926374: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0) 2017-10-17 21:02:08.926379: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Creating TensorFlow device (/gpu:1) -> (device: 1, name: GeForce GTX 1080 Ti, pci bus id: 0000:02:00.0) Traceback (most recent call last): File "stageI/run_exp.py", line 68, in algo.train() File "/home/kyle/git/StackGAN/stageI/trainer.py", line 306, in train counter = self.build_model(sess) File "/home/kyle/git/StackGAN/stageI/trainer.py", line 280, in build_model self.init_opt() File "/home/kyle/git/StackGAN/stageI/trainer.py", line 108, in init_opt self.embeddings) File "/home/kyle/git/StackGAN/stageI/trainer.py", line 134, in compute_losses real_logit = self.model.get_discriminator(images, embeddings) File "/home/kyle/git/StackGAN/stageI/model.py", line 217, in get_discriminator x_code = self.d_encode_img_template.construct(input=x_var) File "/home/kyle/git/StackGAN/.env/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1248, in construct return self._construct(context) File "/home/kyle/git/StackGAN/.env/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1173, in _construct method_args = self._replace_deferred(self._method_args, context) File "/home/kyle/git/StackGAN/.env/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1145, in _replace_deferred return [self._replace_deferred(x, context) for x in arg] File "/home/kyle/git/StackGAN/.env/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1140, in _replace_deferred return arg._construct(context) File "/home/kyle/git/StackGAN/.env/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1173, in _construct method_args = self._replace_deferred(self._method_args, context) File "/home/kyle/git/StackGAN/.env/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1145, in _replace_deferred return [self._replace_deferred(x, context) for x in arg] File "/home/kyle/git/StackGAN/.env/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1140, in _replace_deferred return arg._construct(context) File "/home/kyle/git/StackGAN/.env/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1173, in _construct method_args = self._replace_deferred(self._method_args, context) File "/home/kyle/git/StackGAN/.env/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1145, in _replace_deferred return [self._replace_deferred(x, context) for x in arg] File "/home/kyle/git/StackGAN/.env/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1140, in _replace_deferred return arg._construct(context) File "/home/kyle/git/StackGAN/.env/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1173, in _construct method_args = self._replace_deferred(self._method_args, context) File "/home/kyle/git/StackGAN/.env/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1145, in _replace_deferred return [self._replace_deferred(x, context) for x in arg] File "/home/kyle/git/StackGAN/.env/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1140, in _replace_deferred return arg._construct(context) File "/home/kyle/git/StackGAN/.env/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1173, in _construct method_args = self._replace_deferred(self._method_args, context) File "/home/kyle/git/StackGAN/.env/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1145, in _replace_deferred return [self._replace_deferred(x, context) for x in arg] File "/home/kyle/git/StackGAN/.env/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1140, in _replace_deferred return arg._construct(context) File "/home/kyle/git/StackGAN/.env/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1173, in _construct method_args = self._replace_deferred(self._method_args, context) File "/home/kyle/git/StackGAN/.env/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1145, in _replace_deferred return [self._replace_deferred(x, context) for x in arg] File "/home/kyle/git/StackGAN/.env/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1140, in _replace_deferred return arg._construct(context) File "/home/kyle/git/StackGAN/.env/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1173, in _construct method_args = self._replace_deferred(self._method_args, context) File "/home/kyle/git/StackGAN/.env/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1145, in _replace_deferred return [self._replace_deferred(x, context) for x in arg] File "/home/kyle/git/StackGAN/.env/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1140, in _replace_deferred return arg._construct(context) File "/home/kyle/git/StackGAN/.env/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1173, in _construct method_args = self._replace_deferred(self._method_args, context) File "/home/kyle/git/StackGAN/.env/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1145, in _replace_deferred return [self._replace_deferred(x, context) for x in arg] File "/home/kyle/git/StackGAN/.env/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1140, in _replace_deferred return arg._construct(context) File "/home/kyle/git/StackGAN/.env/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1173, in _construct method_args = self._replace_deferred(self._method_args, context) File "/home/kyle/git/StackGAN/.env/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1145, in _replace_deferred return [self._replace_deferred(x, context) for x in arg] File "/home/kyle/git/StackGAN/.env/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1140, in _replace_deferred return arg._construct(context) File "/home/kyle/git/StackGAN/.env/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1173, in _construct method_args = self._replace_deferred(self._method_args, context) File "/home/kyle/git/StackGAN/.env/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1145, in _replace_deferred return [self._replace_deferred(x, context) for x in arg] File "/home/kyle/git/StackGAN/.env/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1140, in _replace_deferred return arg._construct(context) File "/home/kyle/git/StackGAN/.env/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1173, in _construct method_args = self._replace_deferred(self._method_args, context) File "/home/kyle/git/StackGAN/.env/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1145, in _replace_deferred return [self._replace_deferred(x, context) for x in arg] File "/home/kyle/git/StackGAN/.env/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1140, in _replace_deferred return arg._construct(context) File "/home/kyle/git/StackGAN/.env/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1176, in _construct _strip_unnecessary_contents_from_stack(result, set()) File "/home/kyle/git/StackGAN/.env/local/lib/python2.7/site-packages/prettytensor/pretty_tensor_class.py", line 1335, in _strip_unnecessary_contents_from_stack for f, line_no, method, _ in result._traceback: ValueError: too many values to unpack

    ========================== Contents of my Requirements.txt

    The following requirements were added by pip freeze:

    backports.weakref==1.0.post1 bleach==1.5.0 easydict==1.7 enum34==1.1.6 funcsigs==1.0.2 html5lib==0.9999999 Markdown==2.6.9 mock==2.0.0 numpy==1.13.3 pandas==0.20.3 pbr==3.1.1 pkg-resources==0.0.0 prettytensor==0.7.4 progressbar==2.3 protobuf==3.4.0 python-dateutil==2.6.1 pytz==2017.2 six==1.11.0 tensorflow-gpu==1.3.0 tensorflow-tensorboard==0.1.8 torchfile==0.1.0 Werkzeug==0.12.2

    The following requirements were added by pip freeze:

    backports.weakref==1.0.post1 bleach==1.5.0 easydict==1.7 enum34==1.1.6 funcsigs==1.0.2 html5lib==0.9999999 Markdown==2.6.9 mock==2.0.0 numpy==1.13.3 pandas==0.20.3 pbr==3.1.1 pkg-resources==0.0.0 prettytensor==0.7.4 progressbar==2.3 protobuf==3.4.0 python-dateutil==2.6.1 pytz==2017.2 six==1.11.0 tensorflow-gpu==1.3.0 tensorflow-tensorboard==0.1.8 torchfile==0.1.0 Werkzeug==0.12.2

    The following requirements were added by pip freeze:

    scikit-learn==0.19.0

    The following requirements were added by pip freeze:

    backports.weakref==1.0rc1 bleach==1.5.0 easydict==1.7 enum34==1.1.6 funcsigs==1.0.2 html5lib==0.9999999 Markdown==2.2.0 mock==2.0.0 numpy==1.13.3 pandas==0.20.3 pbr==3.1.1 pkg-resources==0.0.0 prettytensor==0.7.4 progressbar==2.3 protobuf==3.4.0 python-dateutil==2.6.1 pytz==2017.2 six==1.11.0 tensorflow-gpu==1.2.0 tensorflow-tensorboard==0.1.8 torchfile==0.1.0 Werkzeug==0.12.2 scikit-learn==0.19.0

    The following requirements were added by pip freeze:

    backports.functools-lru-cache==1.4 cycler==0.10.0 decorator==4.1.2 matplotlib==2.1.0 networkx==2.0 olefile==0.44 Pillow==4.3.0 pyparsing==2.2.0 PyWavelets==0.5.2 PyYAML==3.12 scikit-image==0.13.1 scipy==0.19.1 subprocess32==3.2.7

    opened by kyle-pena-nlp 6
  • error on StageI _VARSCOPE_KEY

    error on StageI _VARSCOPE_KEY 

     python stageI/run_exp.py --cfg stageI/cfg/birds.yml --gpu 0
Using config:
{'CONFIG_NAME': 'stageI',
 'DATASET_NAME': 'birds',
 'EMBEDDING_TYPE': 'cnn-rnn',
 'GAN': {'DF_DIM': 64,
         'EMBEDDING_DIM': 128,
         'GF_DIM': 128,
         'NETWORK_TYPE': 'default'},
 'GPU_ID': 0,
 'TEST': {'BATCH_SIZE': 64,
          'CAPTION_PATH': '',
          'HR_IMSIZE': 256,
          'LR_IMSIZE': 64,
          'NUM_COPY': 16,
          'PRETRAINED_MODEL': ''},
 'TRAIN': {'BATCH_SIZE': 64,
           'B_WRONG': True,
           'COEFF': {'KL': 2.0},
           'COND_AUGMENTATION': True,
           'DISCRIMINATOR_LR': 0.0002,
           'FINETUNE_LR': False,
           'FLAG': True,
           'FT_LR_RETIO': 0.1,
           'GENERATOR_LR': 0.0002,
           'LR_DECAY_EPOCH': 50,
           'MAX_EPOCH': 600,
           'NUM_COPY': 4,
           'NUM_EMBEDDING': 4,
           'PRETRAINED_EPOCH': 600,
           'PRETRAINED_MODEL': '',
           'SNAPSHOT_INTERVAL': 2000},
 'Z_DIM': 100}
images:  (2933, 76, 76, 3)
embeddings:  (2933, 10, 1024)
list_filenames:  2933 001.Black_footed_Albatross/Black_Footed_Albatross_0046_18
images:  (8855, 76, 76, 3)
embeddings:  (8855, 10, 1024)
list_filenames:  8855 002.Laysan_Albatross/Laysan_Albatross_0002_1027
Traceback (most recent call last):
  File "stageI/run_exp.py", line 61, in <module>
    image_shape=dataset.image_shape
  File "/home/adminis/StackGAN/stageI/model.py", line 31, in __init__
    self.d_encode_img_template = self.d_encode_image()
  File "/home/adminis/StackGAN/stageI/model.py", line 161, in d_encode_image
    custom_conv2d(self.df_dim, k_h=4, k_w=4).
  File "/usr/local/lib/python2.7/dist-packages/prettytensor/pretty_tensor_class.py", line 1965, in method
    with _method_scope(input_layer, scope_name) as (scope, _):
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/usr/local/lib/python2.7/dist-packages/prettytensor/pretty_tensor_class.py", line 1776, in _method_scope
    scopes.var_and_name_scope((name, None)) as (scope, var_scope):
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/usr/local/lib/python2.7/dist-packages/prettytensor/scopes.py", line 55, in var_and_name_scope
    vs_key = tf.get_collection_ref(variable_scope._VARSCOPE_KEY)
AttributeError: 'module' object has no attribute '_VARSCOPE_KEY'

    Anyone who know how to fix this issue?

    opened by gamagamagama 3
  • T7ReaderException: unknown object type / typeidx: 1936287828

    T7ReaderException: unknown object type / typeidx: 1936287828

    The error log I am getting is :- T7ReaderException Traceback (most recent call last) in () 6 cap_path ="/home/oslab/Data/flowers/example_captions" 7 #torch.save(cap_path,'ascii')# ///// 'output.t7', ----> 8 t_file = torchfile.load(cap_path) 9 10 captions_list = t_file.raw_txt

    ~/anaconda3/envs/tensorflow/lib/python3.6/site-packages/torchfile.py in load(filename, **kwargs) 422 with open(filename, 'rb') as f: 423 reader = T7Reader(f, **kwargs) --> 424 return reader.read_obj()

    ~/anaconda3/envs/tensorflow/lib/python3.6/site-packages/torchfile.py in read_obj(self) 412 else: 413 raise T7ReaderException( --> 414 "unknown object type / typeidx: {}".format(typeidx)) 415 416

    T7ReaderException: unknown object type / typeidx: 1936287828

    opened by Shubham-kale 3
  • module 'cunn' not found:No LuaRocks module found for cunn

    module 'cunn' not found:No LuaRocks module found for cunn

    (tensorflow) oslab how_to_convert_text_to_images-master $ sh demo/flowers_demo.sh /home/oslab/torch-cl/install/bin/luajit: /home/oslab/torch-cl/install/share/lua/5.1/trepl/init.lua:384: module 'cunn' not found:No LuaRocks module found for cunn no field package.preload['cunn'] no file '/home/oslab/.luarocks/share/lua/5.1/cunn.lua' no file '/home/oslab/.luarocks/share/lua/5.1/cunn/init.lua' no file '/home/oslab/torch-cl/install/share/lua/5.1/cunn.lua' no file '/home/oslab/torch-cl/install/share/lua/5.1/cunn/init.lua' no file '/home/oslab/torch/install/share/lua/5.1/cunn.lua' no file '/home/oslab/torch/install/share/lua/5.1/cunn/init.lua' no file './cunn.lua' no file '/home/oslab/torch/install/share/luajit-2.1.0-beta1/cunn.lua' no file '/usr/local/share/lua/5.1/cunn.lua' no file '/usr/local/share/lua/5.1/cunn/init.lua' no file '/home/oslab/.luarocks/lib/lua/5.1/cunn.so' no file '/home/oslab/torch-cl/install/lib/lua/5.1/cunn.so' no file '/home/oslab/torch-cl/install/lib/cunn.so' no file '/home/oslab/torch/install/lib/cunn.so' no file '/home/oslab/torch/install/lib/lua/5.1/cunn.so' no file './cunn.so' no file '/usr/local/lib/lua/5.1/cunn.so' no file '/usr/local/lib/lua/5.1/loadall.so' stack traceback: [C]: in function 'error' /home/oslab/torch-cl/install/share/lua/5.1/trepl/init.lua:384: in function 'require' demo/get_embedding.lua:7: in main chunk [C]: in function 'dofile' ...b/torch-cl/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk [C]: at 0x00405e90

    opened by Shubham-kale 2
  • Stage II Training Error

    Stage II Training Error

    Traceback (most recent call last): File "stageII/run_exp.py", line 60, in algo.train() File "E:\StackGAN\StackGAN3\stageII\trainer.py", line 374, in train counter = self.build_model(sess) File "E:\StackGAN\StackGAN3\stageII\trainer.py", line 314, in build_model saver.restore(sess, self.model_path) File "C:\Users\Arun\Anaconda3\envs\GAN\lib\site-packages\tensorflow_core\python\training\saver.py", line 1280, in restore if not checkpoint_management.checkpoint_exists_internal(checkpoint_prefix): File "C:\Users\Arun\Anaconda3\envs\GAN\lib\site-packages\tensorflow_core\python\training\checkpoint_management.py", line 366, in checkpoint_exists_internal if file_io.get_matching_files(pathname): File "C:\Users\Arun\Anaconda3\envs\GAN\lib\site-packages\tensorflow_core\python\lib\io\file_io.py", line 363, in get_matching_files return get_matching_files_v2(filename) File "C:\Users\Arun\Anaconda3\envs\GAN\lib\site-packages\tensorflow_core\python\lib\io\file_io.py", line 384, in get_matching_files_v2 compat.as_bytes(pattern)) tensorflow.python.framework.errors_impl.NotFoundError: FindFirstFile failed for: ./ckt_logs/flowers/stageI : The system cannot find the path specified. ; No such process

    @akhilvasvani

    opened by ArunSachdev 1
  • What to do after finish training StageII.

    What to do after finish training StageII.

    After I had finish training StageII, how can I use the model. What to do and what is the command? I copied model_164000.ckpt.index and model_164000.ckpt.meta to /models. I'm not sure how to run the model itself for testing my own descriptions.

    opened by KevinToh0329 1
  • UnicodeDecodeError while stage - 1 training

    UnicodeDecodeError while stage - 1 training

    This is the code I am trying to run on anaconda IDE :-

    #Training Step 1: train Stage-I GAN #python stageI/run_exp.py --cfg stageI/cfg/birds.yml --gpu 0 from future import division from future import print_function

    import dateutil import dateutil.tz import datetime import argparse import pprint

    from Project.misc.datasets import TextDataset from Project.stageI.model import CondGAN from Project.stageI.trainer import CondGANTrainer from Project.misc.utils import mkdir_p from Project.misc.config import cfg, cfg_from_file

    import codecs #module and function addition by shubham kale : start def replace_line(file_name, line_num, text): f = codecs.open(file_name, 'r', encoding='utf-8') lines = f.readlines() lines[line_num] = text f.close() w = codecs.open(file_name, 'w', encoding='utf-8') w.writelines(lines) w.close() #End of additions import sys #change by shubham kale line -1 from importlib import reload #line - 2 reload(sys) #line - 3 #sys.setdefaultencoding('utf8') #meaningless in python 3

    print('Using config:') pprint.pprint(cfg)

    now = datetime.datetime.now(dateutil.tz.tzlocal()) timestamp = now.strftime('%Y_%m_%d_%H_%M_%S')

    #datadir = 'Data/%s' % cfg.DATASET_NAME datadir = 'Data/flowers' dataset = TextDataset(datadir, cfg.EMBEDDING_TYPE, 1) filename_test = '%s/test' % (datadir) dataset.test = dataset.get_data(filename_test) if cfg.TRAIN.FLAG: filename_train = '%s/train' % (datadir) dataset.train = dataset.get_data(filename_train) ckt_logs_dir = "ckt_logs/%s/%s_%s" %(cfg.DATASET_NAME, cfg.CONFIG_NAME, timestamp) mkdir_p(ckt_logs_dir) else: s_tmp = cfg.TRAIN.PRETRAINED_MODEL ckt_logs_dir = s_tmp[:s_tmp.find('.ckpt')]

    model = CondGAN( image_shape=dataset.image_shape )

    algo = CondGANTrainer( model=model, dataset=dataset, ckt_logs_dir=ckt_logs_dir ) if cfg.TRAIN.FLAG: algo.train() else: ''' For every input text embedding/sentence in the training and test datasets, generate cfg.TRAIN.NUM_COPY images with randomness from noise z and conditioning augmentation.''' algo.evaluate()

    The Output error log is :-

    Using config: {'CONFIG_NAME': '', 'DATASET_NAME': 'birds', 'EMBEDDING_TYPE': 'cnn-rnn', 'GAN': {'DF_DIM': 64, 'EMBEDDING_DIM': 128, 'GF_DIM': 128, 'NETWORK_TYPE': 'default'}, 'GPU_ID': 0, 'TEST': {'BATCH_SIZE': 64, 'CAPTION_PATH': '', 'HR_IMSIZE': 256, 'LR_IMSIZE': 64, 'NUM_COPY': 16, 'PRETRAINED_MODEL': ''}, 'TRAIN': {'BATCH_SIZE': 64, 'B_WRONG': True, 'COEFF': {'KL': 2.0}, 'COND_AUGMENTATION': True, 'DISCRIMINATOR_LR': 0.0002, 'FINETUNE_LR': False, 'FLAG': True, 'FT_LR_RETIO': 0.1, 'GENERATOR_LR': 0.0002, 'LR_DECAY_EPOCH': 50, 'MAX_EPOCH': 600, 'NUM_COPY': 4, 'NUM_EMBEDDING': 4, 'PRETRAINED_EPOCH': 600, 'PRETRAINED_MODEL': '', 'SNAPSHOT_INTERVAL': 2000}, 'Z_DIM': 100} images: (1155, 76, 76, 3)

    UnicodeDecodeError Traceback (most recent call last) in () 42 dataset = TextDataset(datadir, cfg.EMBEDDING_TYPE, 1) 43 filename_test = '%s/test' % (datadir) ---> 44 dataset.test = dataset.get_data(filename_test) 45 if cfg.TRAIN.FLAG: 46 filename_train = '%s/train' % (datadir)

    ~/Project/misc/datasets.py in get_data(self, pickle_path, aug_flag) 227 images = pickle.load(f) 228 images = np.array(images) --> 229 print('images: ', images.shape) 230 231 with open(pickle_path + self.embedding_filename, 'rb') as f:

    UnicodeDecodeError: 'ascii' codec can't decode byte 0xc4 in position 0: ordinal not in range(128)

    opened by Shubham-kale 1
  • CUDA driver version is insufficient for CUDA runtime version

    CUDA driver version is insufficient for CUDA runtime version

    The Error Log :-

    $ sh demo/birds_demo.sh /home/oslab/torch-cl/install/bin/luajit: /home/oslab/torch-cl/install/share/lua/5.1/trepl/init.lua:384: /home/oslab/torch-cl/install/share/lua/5.1/trepl/init.lua:384: cuda runtime error (35) : CUDA driver version is insufficient for CUDA runtime version at /tmp/luarocks_cutorch-scm-1-5883/cutorch/lib/THC/THCGeneral.c:16 stack traceback: [C]: in function 'error' /home/oslab/torch-cl/install/share/lua/5.1/trepl/init.lua:384: in function 'require' demo/get_embedding.lua:7: in main chunk [C]: in function 'dofile' ...b/torch-cl/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk [C]: at 0x00405e90

    opened by Shubham-kale 1
  • IOError during StageI training

    IOError during StageI training

    Here's the traceback for python run_exp.py --cfg cfg/birds.yml --gpu 0

    Using config: {'CONFIG_NAME': 'stageI', 'DATASET_NAME': 'birds', 'EMBEDDING_TYPE': 'cnn-rnn', 'GAN': {'DF_DIM': 64, 'EMBEDDING_DIM': 128, 'GF_DIM': 128, 'NETWORK_TYPE': 'default'}, 'GPU_ID': 0, 'TEST': {'BATCH_SIZE': 64, 'CAPTION_PATH': '', 'HR_IMSIZE': 256, 'LR_IMSIZE': 64, 'NUM_COPY': 16, 'PRETRAINED_MODEL': ''}, 'TRAIN': {'BATCH_SIZE': 64, 'B_WRONG': True, 'COEFF': {'KL': 2.0}, 'COND_AUGMENTATION': True, 'DISCRIMINATOR_LR': 0.0002, 'FINETUNE_LR': False, 'FLAG': True, 'FT_LR_RETIO': 0.1, 'GENERATOR_LR': 0.0002, 'LR_DECAY_EPOCH': 50, 'MAX_EPOCH': 600, 'NUM_COPY': 4, 'NUM_EMBEDDING': 4, 'PRETRAINED_EPOCH': 600, 'PRETRAINED_MODEL': '', 'SNAPSHOT_INTERVAL': 2000}, 'Z_DIM': 100} Traceback (most recent call last): File "run_exp.py", line 47, in dataset.test = dataset.get_data(filename_test) File "/home/rahul/StackGAN_Rahul/misc/datasets.py", line 223, in get_data with open(pickle_path + self.image_filename, 'rb') as f: IOError: [Errno 2] No such file or directory: 'Data/birds/test/76images.pickle'

    Any idea about a step I may have missed?

    opened by EruditeStranger 1
  • How about the results if i don't use the text-image function?

    How about the results if i don't use the text-image function?

    Hi, guys. I'm really curious about whether this network can be applied to generate images merely based on image dataset.

    And i have deleted the text-encoding module from the code, train the model intimately.The result of stageII is not so satisfactory.

    Can anyone give me some advices? Thanks

    opened by jianglai3303 1
  • torch的问题

    torch的问题

    /home/ml/torch/install/bin/luajit: /home/ml/torch/install/share/lua/5.1/trepl/init.lua:389: module 'cunn' not found:No LuaRocks module found for cunn 请问这个该怎么解决呢

    opened by xiao222-max 0
  • what could be the reason for this issue

    what could be the reason for this issue

    WARNING:tensorflow:From C:\Users\HP\Anaconda3\lib\site-packages\tensorflow_core\contrib\layers\python\layers\layers.py:650: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version. Instructions for updating: Please use layer.__call__ method instead. WARNING:tensorflow:From C:\Users\HP\Anaconda3\lib\site-packages\tensorflow_core\python\ops\nn_impl.py:183: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.where in 2.0, which has the same broadcast rule as np.where WARNING:tensorflow:From C:\stageI\trainer.py:102: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

    WARNING:tensorflow:From C:\stageI\trainer.py:197: The name tf.summary.merge is deprecated. Please use tf.compat.v1.summary.merge instead.

    success 2020-04-02 16:26:34.762406: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [ /job:localhost/replica:0/task:0/device:CPU:0]. See below for details of this colocation group: Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[] Assign: CPU Add: CPU RandomStandardNormal: CPU Const: CPU VariableV2: CPU Mul: CPU Identity: CPU ApplyAdam: CPU Fill: CPU

    Colocation members, user-requested devices, and framework assigned devices, if any: g_net/gen_cond/fc/weights/Initializer/random_normal/shape (Const) g_net/gen_cond/fc/weights/Initializer/random_normal/mean (Const) g_net/gen_cond/fc/weights/Initializer/random_normal/stddev (Const) g_net/gen_cond/fc/weights/Initializer/random_normal/RandomStandardNormal (RandomStandardNormal) g_net/gen_cond/fc/weights/Initializer/random_normal/mul (Mul) g_net/gen_cond/fc/weights/Initializer/random_normal (Add) g_net/gen_cond/fc/weights (VariableV2) /device:GPU:0 g_net/gen_cond/fc/weights/Assign (Assign) /device:GPU:0 g_net/gen_cond/fc/weights/read (Identity) /device:GPU:0 d_net/g_net/gen_cond/fc/weights/Adam/Initializer/zeros/shape_as_tensor (Const) /device:GPU:0 d_net/g_net/gen_cond/fc/weights/Adam/Initializer/zeros/Const (Const) /device:GPU:0 d_net/g_net/gen_cond/fc/weights/Adam/Initializer/zeros (Fill) /device:GPU:0 d_net/g_net/gen_cond/fc/weights/Adam (VariableV2) /device:GPU:0 d_net/g_net/gen_cond/fc/weights/Adam/Assign (Assign) /device:GPU:0 d_net/g_net/gen_cond/fc/weights/Adam/read (Identity) /device:GPU:0 d_net/g_net/gen_cond/fc/weights/Adam_1/Initializer/zeros/shape_as_tensor (Const) /device:GPU:0 d_net/g_net/gen_cond/fc/weights/Adam_1/Initializer/zeros/Const (Const) /device:GPU:0 d_net/g_net/gen_cond/fc/weights/Adam_1/Initializer/zeros (Fill) /device:GPU:0 d_net/g_net/gen_cond/fc/weights/Adam_1 (VariableV2) /device:GPU:0 d_net/g_net/gen_cond/fc/weights/Adam_1/Assign (Assign) /device:GPU:0 d_net/g_net/gen_cond/fc/weights/Adam_1/read (Identity) /device:GPU:0 d_net/Adam/update_g_net/gen_cond/fc/weights/ApplyAdam (ApplyAdam) /device:GPU:0

    opened by Sivaaaa 0
  • python 2.7 with tensorflow

    python 2.7 with tensorflow

    I am getting some errors . Just the heads up ,

    1. I think so code is not updated to python 3 .
    2. Because of that , am getting some errors related to tensorflow .
    3. Code is in python 2.7 version . In python 2.7 version , tensorflow is not supported. What can be done ? Please check .. and Thank you :)
    opened by Sivaaaa 2
  • TypeError: 'NoneType' object is not subscriptable

    TypeError: 'NoneType' object is not subscriptable

    Traceback (most recent call last): File "run_exp.py", line 68, in algo.train() File "C:\stageI\trainer.py", line 306, in train counter = self.build_model(sess) File "C:\stageI\trainer.py", line 280, in build_model self.init_opt() File "C:\stageI\trainer.py", line 97, in init_opt c, kl_loss = self.sample_encoded_context(self.embeddings) File "C:\stageI\trainer.py", line 76, in sample_encoded_context c_mean_logsigma = self.model.generate_condition(embeddings) File "C:\stageI\model.py", line 47, in generate_condition custom_fully_connected(self.ef_dim * 2). File "C:\Users\HP\Anaconda3\lib\site-packages\prettytensor\pretty_tensor_class.py", line 1973, in method return input_layer._method_complete(result) File "C:\Users\HP\Anaconda3\lib\site-packages\prettytensor\pretty_tensor_class.py", line 691, in _method_complete return self.with_tensor(result) File "C:\Users\HP\Anaconda3\lib\site-packages\prettytensor\pretty_tensor_class.py", line 978, in with_tensor layer = Layer(copy=self, tensor=unwrap(tensor), sequence=None) File "C:\Users\HP\Anaconda3\lib\site-packages\prettytensor\pretty_tensor_class.py", line 931, in init self._name = self._sequence[0].op.name TypeError: 'NoneType' object is not subscriptable

    opened by Sivaaaa 0
Owner
Han Zhang
Han Zhang
GitHub
Ï€-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis

Ï€-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis Project Page | Paper | Data Eric Ryan Chan*, Marco Monteiro*, Pe

null 375 Dec 31, 2022
Multi-View Consistent Generative Adversarial Networks for 3D-aware Image Synthesis (CVPR2022)

Multi-View Consistent Generative Adversarial Networks for 3D-aware Image Synthesis Multi-View Consistent Generative Adversarial Networks for 3D-aware

Xuanmeng Zhang 78 Dec 10, 2022
HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis

HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis Jungil Kong, Jaehyeon Kim, Jaekyoung Bae In our paper, we p

Rishikesh (ऋषिकेश) 31 Dec 8, 2022
PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models

PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models Code accompanying CVPR'20 paper of the same title. Paper lin

Alex Damian 7k Dec 30, 2022
StudioGAN is a Pytorch library providing implementations of representative Generative Adversarial Networks (GANs) for conditional/unconditional image generation.

StudioGAN is a Pytorch library providing implementations of representative Generative Adversarial Networks (GANs) for conditional/unconditional image generation.

null 3k Jan 8, 2023
[ICLR 2021, Spotlight] Large Scale Image Completion via Co-Modulated Generative Adversarial Networks

Large Scale Image Completion via Co-Modulated Generative Adversarial Networks, ICLR 2021 (Spotlight) Demo | Paper [NEW!] Time to play with our interac

Shengyu Zhao 373 Jan 2, 2023
Image Deblurring using Generative Adversarial Networks

DeblurGAN arXiv Paper Version Pytorch implementation of the paper DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks. Our netwo

Orest Kupyn 2.2k Jan 1, 2023
Flickr-Faces-HQ (FFHQ) is a high-quality image dataset of human faces, originally created as a benchmark for generative adversarial networks (GAN)

Flickr-Faces-HQ Dataset (FFHQ) Flickr-Faces-HQ (FFHQ) is a high-quality image dataset of human faces, originally created as a benchmark for generative

NVIDIA Research Projects 2.9k Dec 28, 2022
Semi-supervised Representation Learning for Remote Sensing Image Classification Based on Generative Adversarial Networks

SSRL-for-image-classification Semi-supervised Representation Learning for Remote Sensing Image Classification Based on Generative Adversarial Networks

Feng 2 Nov 19, 2021
Minimal PyTorch implementation of Generative Latent Optimization from the paper "Optimizing the Latent Space of Generative Networks"

Minimal PyTorch implementation of Generative Latent Optimization This is a reimplementation of the paper Piotr Bojanowski, Armand Joulin, David Lopez-

Thomas Neumann 117 Nov 27, 2022
Official code release for "GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis"

GRAF This repository contains official code for the paper GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis. You can find detailed usage i

null 349 Dec 29, 2022
A Multi-attribute Controllable Generative Model for Histopathology Image Synthesis

A Multi-attribute Controllable Generative Model for Histopathology Image Synthesis This is the pytorch implementation for our MICCAI 2021 paper. A Mul

Jiarong Ye 7 Apr 4, 2022
This Artificial Intelligence program can take a black and white/grayscale image and generate a realistic or plausible colorized version of the same picture.

Colorizer The point of this project is to write a program capable of taking a black and white / grayscale image, and generating a realistic or plausib

Maitri Shah 1 Jan 6, 2022
Official implementation of the paper 'Details or Artifacts: A Locally Discriminative Learning Approach to Realistic Image Super-Resolution' in CVPR 2022

LDL Paper | Supplementary Material Details or Artifacts: A Locally Discriminative Learning Approach to Realistic Image Super-Resolution Jie Liang*, Hu

null 150 Dec 26, 2022
Regularizing Generative Adversarial Networks under Limited Data (CVPR 2021)

Regularizing Generative Adversarial Networks under Limited Data [Project Page][Paper] Implementation for our GAN regularization method. The proposed r

Google 148 Nov 18, 2022
NR-GAN: Noise Robust Generative Adversarial Networks

NR-GAN: Noise Robust Generative Adversarial Networks (CVPR 2020) This repository provides PyTorch implementation for noise robust GAN (NR-GAN). NR-GAN

Takuhiro Kaneko 59 Dec 11, 2022
Generating Anime Images by Implementing Deep Convolutional Generative Adversarial Networks paper

AnimeGAN - Deep Convolutional Generative Adverserial Network PyTorch implementation of DCGAN introduced in the paper: Unsupervised Representation Lear

Rohit Kukreja 23 Jul 21, 2022
Unofficial implementation of Alias-Free Generative Adversarial Networks. (https://arxiv.org/abs/2106.12423) in PyTorch

alias-free-gan-pytorch Unofficial implementation of Alias-Free Generative Adversarial Networks. (https://arxiv.org/abs/2106.12423) This implementation

Kim Seonghyeon 502 Jan 3, 2023
PyTorch implementations of Generative Adversarial Networks.

This repository has gone stale as I unfortunately do not have the time to maintain it anymore. If you would like to continue the development of it as

Erik Linder-Norén 13.4k Jan 8, 2023