I noticed in the code for inputs.py that there appears to be instance noise applied universally to all training examples (along with other data augmentation). Was this and the other data augmentation used for all experimental results in the paper? I don't remember it being mentioned. I was surprised to see this as instance noise was one of the regularization approaches you were comparing eg the R1 approaches to - I didn't realize both were being used simultaneously. Or is the scale of the noise injected here (uniform on [0-1/128)) much lower than required for use as a regularizer?

Hello @LMescheder , thanks for the nice repo. I'd like to create a minimal Dirac-GAN example. Ideally, I'd want a side-by-side comparison with and without the proposed regularization. This is what I have so far:

import torch as th
from torch import nn
import numpy as np
from tqdm import tqdm
import os
from matplotlib import pyplot as plt


class Generator(nn.Module):
  def __init__(self, z_dim=1, gen_dim=1):
    super(Generator, self).__init__()
    self.z_dim = 1
    self.h1 = nn.Linear(z_dim, gen_dim, bias=False)

  def forward(self, x):
    x = self.h1(x)
    return x


class Discriminator(nn.Module):
  def __init__(self, gen_dim=1):
    super(Discriminator, self).__init__()
    self.h1 = nn.Linear(gen_dim, 1, bias=False)
    self.sigmoid = nn.Sigmoid()

  def forward(self, x):
    x = self.sigmoid(self.h1(x))
    return x


def binary_crossentropy(y_pred, y_true):
  y_pred = th.clamp(y_pred, 1e-7, 1. - 1e-7)
  bce = -y_true * th.log(y_pred) - (1. - y_true) * th.log(1. - y_pred)
  return th.mean(bce)


def generator_real(batch_size):
  while True:
    yield th.randn(batch_size, 1, requires_grad=True, dtype=th.float32)


def plot_figure(theta, iteration):
  delta = 1. / 300
  x = np.arange(-1., 1., delta)
  y = np.zeros_like(x)
  y[int(theta * 300) + 300] = 1
  plt.cla()
  plt.plot(x, y)
  plt.yticks([])
  plt.xlabel(r'$\theta$')
  plt.grid('off')
  # plt.savefig('/tmp/gif/tmp_%08d.jpg' % (iteration))
  plt.pause(0.01)


def main():
  if not os.path.exists('/tmp/gif'):
    os.makedirs('/tmp/gif')
  use_reg = False
  gamma = 1.0
  real_label = 1
  fake_label = 0
  epochs = 5
  steps_per_epoch = 100
  batch_size = 128
  generator = Generator()
  # set the one generator weight
  generator.h1.weight = th.nn.Parameter(th.tensor([[0.7]], dtype=th.float32))
  discriminator = Discriminator()
  # set the one discriminator weight
  discriminator.h1.weight = th.nn.Parameter(th.tensor([[0.7]], dtype=th.float32))
  real_data_gen = generator_real(batch_size)

  g_optimizer = th.optim.SGD(generator.parameters(), lr=0.1, momentum=0.0)
  d_optimizer = th.optim.SGD(discriminator.parameters(), lr=0.1, momentum=0.0)

  for epoch in range(1, epochs + 1):
    for step in tqdm(range(steps_per_epoch)):
      # Save figure
      plot_figure(discriminator.h1.weight.item(), iteration=epoch * steps_per_epoch + step)
      # 1.) Update the discriminator

      # a.) real data
      d_optimizer.zero_grad()
      X_real = next(real_data_gen)
      y_pred = discriminator(X_real)
      real_loss = binary_crossentropy(y_pred, real_label * th.ones_like(y_pred))
      if use_reg:
        real_loss += 0.5 * gamma * discriminator.h1.weight.pow(2).item()
      real_loss.backward()

      # b.) fake data
      noise = th.randn(batch_size, 1)
      X_fake = generator(noise)
      y_pred = discriminator(X_fake.detach())
      fake_loss = binary_crossentropy(y_pred, fake_label * th.ones_like(y_pred))
      fake_loss.backward()
      d_optimizer.step()

      # 2.) Update the generator

      g_optimizer.zero_grad()
      y_pred = discriminator(X_fake)
      # use real label in the loss
      fake_loss_if_real = binary_crossentropy(y_pred, real_label * th.ones_like(y_pred))
      fake_loss_if_real.backward()
      g_optimizer.step()


if __name__ == '__main__':
  main()

Output:

Your animations looked much better (i.e more oscillation without regularization). Is this due to a bug (I think so) or hyperparameters?

Hello,

First, thank you very much for your paper, it was very interesting and it helped me a lot! In your paper, in figure 8 and figure 9 in the appendices, you do some testing with 2D data distributions. Is it possible to reproduce these distributions with the code?

Thank you in advance!

Hi, I've recently read your paper and I think it is great. I have a question about Eq.1 in the article. I found that the equation could not be led to the original GAN's objective by replacing f with -log(1+exp(-t)). I think the negation of discriminator output in both terms on the right-hand side is switched. Am I correct?

Hi, thank you for sharing the code. It seems like that you used a large model for ImageNet, and a ralatively small one for the other dataset, e.g. celeba, chruch. Is it true that applying the much complex model (used for ImageNet) on celeba will produce a much better results? I am very curious about your selection of the model for different datasets, can you share it with me? Thank you very much.

Hi, I'm using gradient penalty with LSGAN loss but it did not work ,so I replace LSGAN with Standard GAN loss it work fine !!

except WGAN loss which type gan loss it support ??

RaGAN ?

I noticed the moving average in your training, but I think its influence to BN is quite small. I cannot understand the reason why adding BN cannot converge. Looking forward to your reply!