StudioGAN is a Pytorch library providing implementations of representative Generative Adversarial Networks (GANs) for conditional/unconditional image generation.

Training always stops after this step:

Visualize (num_rows x 8) fake image canvans. Save image canvas to output/figures/CUSTOM-ReACGAN-train-2022_03_23_14_40_18/generated_canvas_4000.png Start Evaluation (4000 Step): CUSTOM-ReACGAN-train-2022_03_23_14_40_18 generate images and stack features (36808 images).

I'm training ReACGAN with my own dataset. However, the training stops always after this step and there are no logs to know why!

Any help or idea would be highly appreciated!

Hi， thank you for your sharing the good repo for reimplement for GAN.

I run the command : python main.py --eval -t -c "./configs/Table2/biggan32_cifar_hinge_no.json" the result is

model=G-step=80000-Inception_mean=9.465-Inception_std=0.00551-FID=10.344.pth

but when I use contraGAN with the same dataset

python main.py --eval -t -c "./configs/Table2/contragan32_cifar_hinge_no.json"

the result is:

Current best model (FID) is ..... step=78000-Inception_mean=8.89-Inception_std=0.1667-FID=12.648

when contraGAN is not better than biggan?

I encountered this problem when running the following command

python ./src/main.py -t -metrics is fid prdc -cfg ./src/configs/CIFAR10/LGAN.yaml -data ./data/ -save ./result/

Traceback (most recent call last): File "/root/gan/PyTorch-StudioGAN-master/src/main.py", line 19, in import config File "/root/gan/PyTorch-StudioGAN-master/src/config.py", line 16, in import utils.misc as misc File "/root/gan/PyTorch-StudioGAN-master/src/utils/misc.py", line 33, in import utils.sample as sample File "/root/gan/PyTorch-StudioGAN-master/src/utils/sample.py", line 21, in import utils.losses as losses File "/root/gan/PyTorch-StudioGAN-master/src/utils/losses.py", line 36, in import utils.misc as misc AttributeError: module 'utils' has no attribute 'misc'

Dear author,

I am reading your implementation on latent sampling from sample.py (function: sample_latents). For a gaussian sampling as implemented by latents = torch.randn(batch_size, dim, device=device)/truncated_factor.

I notice that the above implementation is not a standard truncation trick, which is defined by

The Truncation Trick is a latent sampling procedure for generative adversarial networks, where we sample from a truncated normal (where values which fall outside a range are resampled to fall inside that range)

I am not getting any improvement from contraGAN at all. For the main contribution, the FID improvement is not even 0.1. If i rerun FID score, I get worse score many times. Same for IS. other metrics not improving at all, as your table show. Including other method like diffaug is not fair as biggan can benefit from those method too.

fid score can change a lot more than 0.1, sometimes more than 1 point if you run it many times. how many times do you train the model or get the fid score?

for imagenet 128x128, ContraGAN score is even worse than bigGAN baseline. Since contragan is build on biggan, the result is no improvement or worse performance.

I want to use your work but it is hard to believe the score now, can you help explain? thanks

Hi, I am testing DDP code with 4 V100 GPU as below,

export MASTER_ADDR="localhost" export MASTER_PORT=2222 CUDA_VISIBLE_DEVICES=0,1,2,3 python3 src/main.py -t -metrics none -cfg CONFIG_PATH -data DATA_PATH -save SAVE_PATH -DDP -sync_bn -mpc

but im having less than 48% gpu utilization. Could I get some help to improve this problem?

below is what I have trained with 1* RTX 3090ti for a comparison.

Hi,

Thank you for your great work in the GAN field. I would like to know what should std_max and std_step be set to reproduce the results in the ImageNet experiment of the ReACGAN paper.

Hi Minguk,

Any plans for implementing the StyleGAN2 generator architecture in the library? Are there some issues which are making it hard to implement in this framework?

Thanks for the great library.

Hi, thank you for a great repository!

I'm trying to train SNGAN on my custom dataset (128x128) in 4 GPU, but found that the GPU utils are ~90%. I also increased num_workers=16 and applied -rm_API but it is still not addressed. Do you have any suspicion on this issue?

By the way, even BigGAN_256 raised out-of-memory (OOM) error for 4 GPUs of 12G memory. Do I need 8 GPU server or distributed training for BigGAN_256?

Thank you for your help!

p.s. typo: load_frameowrk -> load_framework

Hi,

I run the ContraGAN on CIFAR10 using the config file you provided and without any modification. However, I got a worse result with 10.435 FID while your result is 8.065. I wonder what's the different between us. In addition, you get a much better result than the figure reported in the paper(10.597±0.273), how did you do that?

If anyone faces the same issue, please tell me. Thank you!

Hi Mniguk,

Recently, I want to reproduce results of ProjGAN on ImageNet with config file src/configs/ILSVRC2012/BigGAN256.json. I use the command

python src/main.py -t -e -sync_bn -c src/configs/ILSVRC2012/BigGAN256.json --eval_type "valid"

From logging file logs/IMAGENET/BigGAN256-train-2021_01_24_03_52_15.log the FID at iteration 44000 is 46.32, however, I get 43.87 when I run it. The difference is not negligible. I train BigGAN256 on 4 A100 gpus. I am wondering is this normal behavior? or is there anything I potentially did wrong?

The only differences of my experiment setting from using the command in README file are:

1. I use CenterCrop during training, but I think that would only make FID score worse.
2. I did not use -l option to save loading time.
3. -stat_otf flag is also off since it contradicts sync_bn.
4. I wrapped valid folder in another one since ILSVRC2012's valid folder is not organized in subfolders and causes problem in ImageFolder.

Thanks a lot in advance!

• missing optimizer.zero_grad()
• validation accuracy was computed on the training set instead of the validation set
• fix log

Also, I believe would be better to generate a fake dataset only once and use it to train the classifier for N epochs. So, the correct pipeline would be:

1. Generate a fake dataset of the same length (and same number of images per class, preferably) of the original training dataset (e.g: CIFAR10 -> 5000 images per class)
2. Shuffle the dataset
3. Train the classifier for N epochs and at the end of each epoch compute the accuracy on the original validation dataset

Hi,

This is 'cal_grad_penalty' function in /src/utils/losses.py

def cal_grad_penalty(real_images, real_labels, fake_images, discriminator, device):
    batch_size, c, h, w = real_images.shape
    alpha = torch.rand(batch_size, 1)
    alpha = alpha.expand(batch_size, real_images.nelement() // batch_size).contiguous().view(batch_size, c, h, w)
    alpha = alpha.to(device)

    real_images = real_images.to(device)
    interpolates = alpha * real_images + ((1 - alpha) * fake_images)
    interpolates = interpolates.to(device)
    interpolates = autograd.Variable(interpolates, requires_grad=True)
    fake_dict = discriminator(interpolates, real_labels, eval=False)
    grads = cal_deriv(inputs=interpolates, outputs=fake_dict["adv_output"], device=device)
    grads = grads.view(grads.size(0), -1)

    grad_penalty = ((grads.norm(2, dim=1) - 1)**2).mean() + interpolates[:,0,0,0].mean()*0
    return grad_penalty

In the last line, grad_penalty = ((grads.norm(2, dim=1) - 1)**2).mean() + interpolates[:,0,0,0].mean()*0, I wanted to know what additive term + interpolates[:,0,0,0].mean()*0 means. Since it's zero-multiplicated, I think it has actually no effect for code.

I'll be waiting for your answer

Thank you!

Hi,

I discovered that, in WGAN-GP for CIFAR10, Discriminator and Generator loss always display NaN (Not A Number) after some time elapsed. The problem existed when I ran src/configs/CIFAR10/WGAN-GP.yaml file but I did not check if this also happens in other training data.

+) The one thing I consider interesting is that this always happens (I think) exactly at 1500 steps since the start of the training.

==================== This is the code I ran =======================

!python src/main.py -t -hdf5 -l -metrics is fid prdc -ref test --num_workers 2 --save_freq 5000 \
    -cfg src/configs/CIFAR10/WGAN-GP.yaml \
    -data ./cifar10 -save . -mpc --post_resizer friendly --eval_backbone InceptionV3_tf

==================== This is the log I got ====================

 [INFO] 2022-11-02 23:50:13 > Step:    900 Progress: 0.9% Elapsed: 0:07:59 Gen_loss: -2.982 Dis_loss: -1.749 
 [INFO] 2022-11-02 23:51:06 > Step:   1000 Progress: 1.0% Elapsed: 0:08:52 Gen_loss: 12.56 Dis_loss: -1.571 
 [INFO] 2022-11-02 23:52:00 > Step:   1100 Progress: 1.1% Elapsed: 0:09:46 Gen_loss: 6.941 Dis_loss: -1.659 
 [INFO] 2022-11-02 23:52:53 > Step:   1200 Progress: 1.2% Elapsed: 0:10:38 Gen_loss: 6.988 Dis_loss: -3.02 
 [INFO] 2022-11-02 23:53:46 > Step:   1300 Progress: 1.3% Elapsed: 0:11:32 Gen_loss: 9.656 Dis_loss: -1.676 
 [INFO] 2022-11-02 23:54:38 > Step:   1400 Progress: 1.4% Elapsed: 0:12:24 Gen_loss: 4.777 Dis_loss: -1.606 
 [INFO] 2022-11-02 23:55:29 > Step:   1500 Progress: 1.5% Elapsed: 0:13:14 Gen_loss: nan Dis_loss: nan 
 [INFO] 2022-11-02 23:56:17 > Step:   1600 Progress: 1.6% Elapsed: 0:14:03 Gen_loss: nan Dis_loss: nan 
 [INFO] 2022-11-02 23:57:05 > Step:   1700 Progress: 1.7% Elapsed: 0:14:51 Gen_loss: nan Dis_loss: nan 
 [INFO] 2022-11-02 23:57:54 > Step:   1800 Progress: 1.8% Elapsed: 0:15:39 Gen_loss: nan Dis_loss: nan 
 [INFO] 2022-11-02 23:58:42 > Step:   1900 Progress: 1.9% Elapsed: 0:16:28 Gen_loss: nan Dis_loss: nan 
 [INFO] 2022-11-02 23:59:31 > Step:   2000 Progress: 2.0% Elapsed: 0:17:17 Gen_loss: nan Dis_loss: nan

Hello,

thanks for this repo and all of the implementations. I have been trying to get the Implementation to work on some custom data, however, the loss curves always indicate that something is going wrong. The data I am training on is 256 x 256 and when training on it the discriminator keeps going to values very close to 0 and eventually either goes to zero or comes back for a few training steps and then repeats the same procedure.

Do you have any tips or suggestions on which parameters to change so the training runs more stable with a custom dataset?

Thanks

hi,

I found glob. glob (in utils.ckpt. load_StudioGAN_ckpts())returns an empty list, causing list index out of range to appear.

The reason is that the folder name contains special character '=', so you need to run glob.escape() to get the file in the path normally.

I used

x = join(ckpt_dir, "model=G-{when}-weights-step=".format(when=when))
Gen_ckpt_path = glob.glob(glob.escape(x) + '*.pth')[0]
    
y = join(ckpt_dir, "model=D-{when}-weights-step=".format(when=when))
Dis_ckpt_path = glob.glob(glob.escape(y) + '*.pth')[0]

instead of

Gen_ckpt_path = glob.glob(join(ckpt_dir, "model=G-{when}-weights-step*.pth".format(when=when)))[0]

Dis_ckpt_path = glob.glob(join(ckpt_dir, "model=D-{when}-weights-step*.pth".format(when=when)))[0]