[SIGGRAPH'22] StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets

After following the installation instructions, I get the following error running Cuda 11.6 on an RTX 2080ti

Traceback (most recent call last):
  File "/home/alex/Spring-2022/CV/GAN/resources/stylegan_xl/train.py", line 332, in <module>
    main()  # pylint: disable=no-value-for-parameter
  File "/home/alex/miniconda3/envs/sgxl/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/home/alex/miniconda3/envs/sgxl/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/home/alex/miniconda3/envs/sgxl/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/alex/miniconda3/envs/sgxl/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/alex/Spring-2022/CV/GAN/resources/stylegan_xl/train.py", line 317, in main
    launch_training(c=c, desc=desc, outdir=opts.outdir, dry_run=opts.dry_run)
  File "/home/alex/Spring-2022/CV/DogeGAN/resources/stylegan_xl/train.py", line 104, in launch_training
    subprocess_fn(rank=0, c=c, temp_dir=temp_dir)
  File "/home/alex/Spring-2022/CV/GAN/resources/stylegan_xl/train.py", line 49, in subprocess_fn
    training_loop.training_loop(rank=rank, **c)
  File "/home/alex/Spring-2022/CV/GAN/resources/stylegan_xl/training/training_loop.py", line 339, in training_loop
    loss.accumulate_gradients(phase=phase.name, real_img=real_img, real_c=real_c, gen_z=gen_z, gen_c=gen_c, gain=phase.interval, cur_nimg=cur_nimg)
  File "/home/alex/Spring-2022/CV/GAN/resources/stylegan_xl/training/loss.py", line 121, in accumulate_gradients
    loss_Gmain.backward()
  File "/home/alex/miniconda3/envs/sgxl/lib/python3.9/site-packages/torch/_tensor.py", line 363, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/home/alex/miniconda3/envs/sgxl/lib/python3.9/site-packages/torch/autograd/__init__.py", line 173, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
  File "/home/alex/miniconda3/envs/sgxl/lib/python3.9/site-packages/torch/autograd/function.py", line 253, in apply
    return user_fn(self, *args)
  File "/home/alex/Spring-2022/CV/GAN/resources/stylegan_xl/torch_utils/ops/conv2d_gradfix.py", line 144, in backward
    grad_weight = Conv2dGradWeight.apply(grad_output, input)
  File "/home/alex/Spring-2022/CV/GAN/resources/stylegan_xl/torch_utils/ops/conv2d_gradfix.py", line 173, in forward
    return torch._C._jit_get_operation(name)(weight_shape, grad_output, input, padding, stride, dilation, groups, *flags)
RuntimeError: No such operator aten::cudnn_convolution_transpose_backward_weight

When I train super-resolution stages, a message was thrown out: Traceback (most recent call last): File "train.py", line 338, in main() # pylint: disable=no-value-for-parameter File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1137, in call return self.main(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1062, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 763, in invoke return __callback(*args, **kwargs) File "train.py", line 323, in main launch_training(c=c, desc=desc, outdir=opts.outdir, dry_run=opts.dry_run) File "train.py", line 106, in launch_training torch.multiprocessing.spawn(fn=subprocess_fn, args=(c, temp_dir), nprocs=c.num_gpus) File "/home/ubuntu/MyFiles/.local/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 230, in spawn return start_processes(fn, args, nprocs, join, daemon, start_method='spawn') File "/home/ubuntu/MyFiles/.local/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 188, in start_processes while not context.join(): File "/home/ubuntu/MyFiles/.local/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 150, in join raise ProcessRaisedException(msg, error_index, failed_process.pid) torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 0 terminated with the following error: Traceback (most recent call last): File "/home/ubuntu/MyFiles/.local/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 59, in wrap fn(i, *args) File "/home/ubuntu/user_space/stylegan_xl/train.py", line 49, in subprocess_fn training_loop.training_loop(rank=rank, **c) File "/home/ubuntu/user_space/stylegan_xl/training/training_loop.py", line 170, in training_loop G = dnnlib.util.construct_class_by_name(**G_kwargs, **common_kwargs).train().requires_grad(False).to(device) # subclass of torch.nn.Module File "/home/ubuntu/user_space/stylegan_xl/dnnlib/util.py", line 303, in construct_class_by_name return call_func_by_name(*args, func_name=class_name, **kwargs) File "/home/ubuntu/user_space/stylegan_xl/dnnlib/util.py", line 298, in call_func_by_name return func_obj(*args, **kwargs) File "/home/ubuntu/user_space/stylegan_xl/torch_utils/persistence.py", line 104, in init super().init(*args, **kwargs) File "/home/ubuntu/user_space/stylegan_xl/training/networks_stylegan3_resetting.py", line 612, in init G_stem = legacy.load_network_pkl(f)['G_ema'] File "/home/ubuntu/user_space/stylegan_xl/legacy.py", line 25, in load_network_pkl data = _LegacyUnpickler(f).load() File "/home/ubuntu/MyFiles/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1177, in getattr raise AttributeError("'{}' object has no attribute '{}'".format( AttributeError: 'VisionTransformer' object has no attribute 'forward_flex'

I use stylegan2, torch ==1.10.1.

Hello, may I apply to you for sharing the data set of POKEMON? I've been working on a Pokemon image generation project recently, but the quality of the generated Pokemon is so poor (FID93) that they are like Cthulhu growling

Hello, your project mentions the use of imagenet dataset, but I have some problems in reproducing it, because there is no usage method for imagenet dataset in datatool. Also, I would like to know how to effectively train the sota results you mentioned in paperswithcode, do you have your training plan? For example, what settings are used for training 16, 32, 64 stages, which are usually written in the yaml file, and how much time it takes to train Imagenet with these settings, if v100/day is used It may be a bit long.

Dear StyleGAN_xl team,

Thank you for your great work. The results are amazing.

Do you have plan to release the code and pretrained models? When you will release them?

Thank you for your help.

Best Wishes,

Zongze

When I try to generated videos I get the following error:

    raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'DummyMapping' object has no attribute 'w_avg

I appreciate your help!

Does anyone know what this error is about? KeyError Traceback (most recent call last) Input In [19], in <cell line: 16>() 14 fetch_model(network_url[Model]) 16 with dnnlib.util.open_url(network_name) as f: ---> 17 G = legacy.load_network_pkl(f)['G_ema'].to(device) # type: ignore 20 zs = torch.randn([10000, G.mapping.z_dim], device=device) 21 cs = torch.zeros([10000, G.mapping.c_dim], device=device)

File /workspace/./stylegan_xl/legacy.py:25, in load_network_pkl(f, force_fp16) 24 def load_network_pkl(f, force_fp16=False): ---> 25 data = _LegacyUnpickler(f).load() 27 # Legacy TensorFlow pickle => convert. 28 if isinstance(data, tuple) and len(data) == 3 and all(isinstance(net, _TFNetworkStub) for net in data):

File /workspace/./stylegan_xl/torch_utils/persistence.py:193, in _reconstruct_persistent_obj(meta) 190 module = _src_to_module(meta.module_src) 192 assert meta.type == 'class' --> 193 orig_class = module.dict[meta.class_name] 194 decorator_class = persistent_class(orig_class) 195 obj = decorator_class.new(decorator_class)

KeyError: 'FullyConnectedLayer'

when i try to train your great project, something wrong. do you know the reason for RuntimeError: No such operator aten::cudnn_convolution_backward_weight??? thank you very much~

After resuming from a stem network, the fakes will not change. The training continues and snapshots get produced, but they are visually identical to previous ones. (e.g., every snapshot will mimic fakes_init)

I am trying to use the discriminator in imagenet512.pkl This is how I load it:

network_pkl = config['network_pkl']
print(network_pkl)
with dnnlib.util.open_url(network_pkl) as f:
    module = legacy.load_network_pkl(f)
    G = module['G_ema']
    discriminator = module['D']
    G = G.eval().requires_grad_(False).to(device)

then:

class_indices = torch.full((real_img.shape[0],), config['image_class_idx']).cuda()
c = F.one_hot(class_indices, G.c_dim)
recon_pred = discriminator(recon_img, c)

However, I got

File "/apdcephfs/private_v_boooowang/liif/pg_modules/discriminator.py", line 203, in forward
    x_n = Normalize(feat.normstats['mean'], feat.normstats['std'])(x)
TypeError: 'NoneType' object is not subscriptable

What is the problem? Please help me with it. Thank you so much.

Hi, I notice that the normalization of input images is different from traditional normalization way ( x = (x-mean)/std. ):

def norm_with_stats(x, stats):
    x_ch0 = torch.unsqueeze(x[:, 0], 1) * (0.5 / stats['mean'][0]) + (0.5 - stats['std'][0]) / stats['mean'][0]
    x_ch1 = torch.unsqueeze(x[:, 1], 1) * (0.5 / stats['mean'][1]) + (0.5 - stats['std'][1]) / stats['mean'][1]
    x_ch2 = torch.unsqueeze(x[:, 2], 1) * (0.5 / stats['mean'][2]) + (0.5 - stats['std'][2]) / stats['mean'][2]
    x = torch.cat((x_ch0, x_ch1, x_ch2), 1)
    return x

Is there any paper or previous work introduces this kind of normalization method?

As the code shows, no matter the resolution is, the project_type is always 2. So what is the point of this line of code here?

https://github.com/autonomousvision/stylegan_xl/blob/4241ff9cfeb69d617427107a75d69e9d1c2d92f2/train.py#L285

When I use this command to train stylegan2:python3 train.py --outdir=./out --cfg=stylegan2 --cond=True --data=./dataset/cifar10.zip --gpus=4 --batch=128 --mirror=True --batch-gpu 32 --cbase 16734 --cmax 256 --kimg 25000 --snap 200 --syn_layers 7 --head_layers 4 It takes errors: TypeError: init() got an unexpected keyword argument 'num_layers'

But I can successfully train stylegan3-r and fastgan.

It is so weird. How to solve these problem/ Has anyone successfully run these two models?

Hi! I found that GPU memory consumption is highly unbalanced between GPU0 and the rest of GPUs. Here's the command I used to train on imagenet with resolution 128.

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
python train.py
--outdir=/storage/guangrun/qijia_3d_model/stylegan-xl/finetune128/
--cfg=stylegan3-t
--data=/datasets/guangrun/qijia_3d_model/imagenet/stylegan_xl/imagenet_sub_seg128.zip
--gpus=8
--batch=32
--mirror=1
--snap 10
--batch-gpu 4
--kimg 10000
--cond True
--superres
--up_factor 2
--head_layers 7
--path_stem /scratch/local/ssd/guangrun/qijia_3d_model/stylegan_xl/imagenet64.pkl
--resume /scratch/local/ssd/guangrun/qijia_3d_model/stylegan_xl/imagenet128.pkl

As you can see, the GPU0 only consumes much less memory than rest of the GPUs. May I ask what caused such imbalance and what's the normal memory consumption is when training at 128 resolution with the settings above?

Hi, thank you so much for sharing this great work. I'm thinking of doing some finetune one the 128^2 imagenet pretrained pkl model. However, I'm not sure how could I do that. For example, what syn_layers, head_layers,... should be and what arguments should I pass in to the train.py? Is it possible for you to provide an example on that?

on SG2 and SG3 given you use them on a modified fork you can resume training on a completely different batch size and still keep your Tick / Nkimg progress , by specifying it with the kwarg ""--nkimg"" example --nkimg=2500 resumes training with an assumed 2500kimg progress.

SGXL resets kimg to 0 if you change the batch size

I found its extremely useful to start with a very low batch size such as --batch=2 --glr=0.0008 --dlr=0.0006 to improve diversity and then swittch to a batch size of 32 / 64 / 128 for better FID when im starting to bottom out FID with batch=2 ,

however because the Augmentation , the G epoch and kimg resets in SGXL when doing this , I am having a really bad time.