Tensorflow Implementation of the paper "Spectral Normalization for Generative Adversarial Networks" (ICML 2017 workshop)

Nhat M. Nguyen

Last update: Nov 25, 2022

Related tags

Overview

tf-SNDCGAN

Tensorflow implementation of the paper "Spectral Normalization for Generative Adversarial Networks" (https://www.researchgate.net/publication/318572189_Spectral_Normalization_for_Generative_Adversarial_Networks, ICML 2017)

The implementation is based on the author's original code at: https://github.com/pfnet-research/chainer-gan-lib

This implementation works for tensorflow default data format "NHWC"

Spectral Normalization for Generative Adversarial Networks:

This method enforces Lipschitz-1 condition on the Discrminator of Wasserstein-GAN by normalizing its weight matrices with their own respective maximum singular value. This can be used together with Gradient Penalty in the paper "Improved Training of Wasserstein GAN".

The author uses a fast approximation method to compute the maximum singular value of weight matrices.

Quick run:

Keras is required for loading Cifar10 data set

python3 train.py

How to use spectral normalization:

# Import spectral norm wrapper
from libs.sn import spectral_normed_weight
# Create weight variable
W = tf.Variable(np.random.normal(size=[784, 10], scale=0.02), name='W', dtype=tf.float32)
# name of tf collection used for storing the update ops (u)
SPECTRAL_NORM_UPDATE_OPS = "spectral_norm_update_ops"
# call wrapping function, W_bar will be the spectral normed weight matrix
W_bar = spectral_normed_weight(W, num_iters=1, update_collection=SPECTRAL_NORM_UPDATE_OPS)
# Get the update ops
spectral_norm_update_ops = tf.get_collection(SPECTRAL_NORM_UPDATE_OPS)
...
# During training, run the update ops at the end of the iteration
for iter in range(max_iters):
    # Training goes here
    ...
    # Update ops at the end
    for update_op in spectral_norm_update_ops:
        sess.run(update_op)

For an example, see the file test_sn_implementation.py

Training curve:

Generated image samples on Cifar10:

Inception score:

After using in place batch norm update and use the optimal training parameters from the paper, I was able to match their claimed Inception score at 100k iteration: 7.4055686 +/- 0.087728456

The official github repostiory has an inception score of 7.41

Issues:

GPU under-utilization: The original implementation of the author in chainer uses 80%+ GPU most of the time. On an NVIDIA GTX 1080TI, their implementation run at nearly 3 iterations/s. This implementation use less than 50% GPU and run at less than 2 iterations/s. Solved. It was the global_step assignment that makes tensorflow create new assign node for graph each iteration, slow down the execution. This also made the graph become very large over time leading to gigantic event files. GPU utilization is now around 85+%
No Fréchet Inception Distance (https://arxiv.org/abs/1706.08500) evaluation yet.

Comments

How do I generate a lot of images?
How do I generate a lot of images?

python train.py

if max_iter = 100000, then only 64 images are generated

I want to generate a lot of features, but also a lot of images.

how ?
opened by taki0112 2

I want to apply to tf.layers.con2d

If I want to apply to tf.layers.con2d, what should I do? I want to put it in the kernel initializer.

for example

    x = tf.layers.conv2d(inputs=x, filters=channels,
                                      kernel_size=kernel, 
                                      kernel_initializer=spectral_normed_weight(),
                                      strides=stride)

Should I just use tf.nn.conv2d... ?

opened by taki0112 1

Data format?

While reading https://github.com/minhnhat93/tf-SNDCGAN/blob/6dce0688bcf7657d2a7f4c392682a5fc414d7084/libs/sn.py#L15-L17, I noticed you use W_shape[-1]. Did you assume data format is NHWC? So in the case I use NCHW, it should be changed to W_shape[1]?

opened by wchen342 1
Why do you use `update collection`?
While skimming through your code, I don't understand why do you need update_collection variable here, https://github.com/minhnhat93/tf-SNDCGAN/blob/master/train.py#L39

If i don't add update collection in the code. just write like below

with tf.control_dependencies([u.assign(u_hat)]): w_norm = tf.reshape(w_norm, w_shape)

d_real and d_loss both use the above procedure. Is there anything wrong? thx
opened by IPNUISTlegal 0
warning in `spectral_normed_weight`

Hi! I've been using your implementation of SNGAN and I have some questions. If I follow the training procedure intrain.py, then the warning in the spectral_normed_weight function will print

Setting update_collection to None will make u being updated every W execution. This maybe undesirable. Please consider using a update collection instead.

I wonder if the warning was meaningful. I also found that the way you used the collection api is different in train.py and test_sn_implementation.py. Which way should I follow?

Thanks!

opened by unilight 0
Input Error for Evaluation

There comes a error when doing evaluation: Cannot feed value of shape (100, 32, 32, 3) for Tensor 'ExpandDims:0' at libs/inception_score/model.py line 45.

Thanks for sharing.

opened by OswinGuai 1

Tensorflow Implementation of the paper "Spectral Normalization for Generative Adversarial Networks" (ICML 2017 workshop)

Related tags

Overview

tf-SNDCGAN

Spectral Normalization for Generative Adversarial Networks:

Quick run:

How to use spectral normalization:

Training curve:

Generated image samples on Cifar10:

Inception score:

Issues:

Comments

How do I generate a lot of images?

I want to apply to tf.layers.con2d

Data format?

Why do you use `update collection`?

warning in `spectral_normed_weight`

Input Error for Evaluation

Owner

Nhat M. Nguyen

PyTorch implementation of NIPS 2017 paper Dynamic Routing Between Capsules

Implementation of EMNLP 2017 Paper "Natural Language Does Not Emerge 'Naturally' in Multi-Agent Dialog" using PyTorch and ParlAI

A PyTorch implementation of the paper "Semantic Image Synthesis via Adversarial Learning" in ICCV 2017

Implementation supporting the ICCV 2017 paper "GANs for Biological Image Synthesis"

Implementation of EMNLP 2017 Paper "Natural Language Does Not Emerge 'Naturally' in Multi-Agent Dialog" using PyTorch and ParlAI

The PyTorch improved version of TPAMI 2017 paper: Face Alignment in Full Pose Range: A 3D Total Solution.

PyTorch version of the paper 'Enhanced Deep Residual Networks for Single Image Super-Resolution' (CVPRW 2017)

Code for the USENIX 2017 paper: kAFL: Hardware-Assisted Feedback Fuzzing for OS Kernels

The implementation of the algorithm in the paper "Safe Deep Semi-Supervised Learning for Unseen-Class Unlabeled Data" published in ICML 2020.

An efficient PyTorch implementation of the winning entry of the 2017 VQA Challenge.

Code for the ICML 2021 paper: "ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision"

Code for the ICML 2021 paper: "ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision"

Code for ICML 2021 paper: How could Neural Networks understand Programs?

An interpreter for RASP as described in the ICML 2021 paper "Thinking Like Transformers"

Official Code for ICML 2021 paper "Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline"

Code for Fold2Seq paper from ICML 2021

Code for the ICML 2021 paper "Bridging Multi-Task Learning and Meta-Learning: Towards Efficient Training and Effective Adaptation", Haoxiang Wang, Han Zhao, Bo Li.

Code for the paper "Adversarially Regularized Autoencoders (ICML 2018)" by Zhao, Kim, Zhang, Rush and LeCun