I fixed it by adding .state_dict(): torch.save(ema.state_dict(), os.path.join(opt.output_dir, now + 'ema.pth')) torch.save(ema2.state_dict(), os.path.join(opt.output_dir, now + 'ema2.pth'))

Hey, thank you for your splendid work! After some tests and diving into your example codes, I am shocked by your model's performance on generating consistent and truthful faces on different views quickly. A Neural Radiance Field representing human faces of diverse situations is reached. Thus, I want to write an encoder to project real faces into w space quickly and faithfully. However, during training, I found that the discriminator in pi-GAN may help the encoder perform better since it not only outputs whether faces are real or fake, which can be used as an additional adversarial loss term, but also outputs pitch and yaw, which should be used for parameters when rendering. Tilted faces from real dataset seem fail to be encoded well into w space, given h_mean and v_mean are set to pi / 2. Of course, I could train a discriminator of pi-GAN by myself based on pretrained generators and I am currently doing it, but because of the potential instability of initialization and the cost of time, I will really appreciate it if you could share the pretrained model for the discriminator. Thank you!

Hi,

First of all, thanks for sharing this great work. I am currently trying to train a model on images of full-bodies of humans and is using a curriculum based on the CARLA curriculum. While training on multiple GPUS, I encountered this error:

RuntimeError: Expected to mark a variable ready only once. This error is caused by one of the following reasons: 1) Use of a module pa
rameter outside the `forward` function. Please make sure model parameters are not shared across multiple concurrent forward-backward p
asses. or try to use _set_static_graph() as a workaround if this module graph does not change during training loop.2) Reused parameter
s in multiple reentrant backward passes. For example, if you use multiple `checkpoint` functions to wrap the same part of your model,
it would result in the same set of parameters been used by different reentrant backward passes multiple times, and hence marking a var
iable ready multiple times. DDP does not support such use cases in default. You can try to use _set_static_graph() as a workaround if
your module graph does not change over iterations.
Parameter at index 52 with name fromRGB.5.model.0.weight has been marked as ready twice. This means that multiple autograd engine  hoo
ks have fired for this particular parameter during this iteration.

I noticed that fromRGB.5.model.0.weight is a variable in the ProgressiveEncoderDiscriminator. Since I have been able to reproduce results with the Celeba curriculum, and that the CARLA curriculum uses a different Discriminator class, I think the problem can be narrowed down to the ProgressiveEncoderDiscriminator class. But currently I do not have other clues. Could you please have a look at this?

Many thanks and best!

I use the command python render_video.py ckpt/CelebA/generator.pth --curriculum CelebA --seeds 0 1 2 3 in readme, it seems the pipe is broken?

I tried to re-install scikit-video, it doesn't work.

Is there anything I am doing wrong ? Please help me out.

Hello, thanks for sharing this interesting work! At lines 360 and 361 in train.py, there exist

                torch.save(generator_losses, os.path.join(opt.output_dir, 'generator.losses'))
                torch.save(discriminator_losses, os.path.join(opt.output_dir, 'discriminator.losses'))

Could you please tell me how to load the saved generator.losses file and discriminator.losses file? I have tried torch.load, pickle.load, but none of them work.

Hi,

I tried to train the code on the CARLA dataset. But I am getting Cuda out of memory error. These are the things I have tried so far:

1. I have tried running on a single as well multiple 2080 Ti GpuS (specified using CUDA_VISIBLE_DEVICES), each with 11GB memory, but it still generates OOM error.
2. I tried on 3090 GPU, but the code generates errors on 3090 GPU (that are not related to Cuda OOM error)
3. I have also tried to reduce the batch size for the Carla dataset in curriculum.py from 30 to 10 as shown below. But I still get the OOM error when I run on a single or multiple 2080Ti GPUs.

CARLA = { 0: {'batch_size': 10, 'num_steps': 48, 'img_size': 32, 'batch_split': 1, 'gen_lr': 4e-5, 'disc_lr': 4e-4}, int(10e3): {'batch_size': 14, 'num_steps': 48, 'img_size': 64, 'batch_split': 2, 'gen_lr': 2e-5, 'disc_lr': 2e-4}, int(55e3): {'batch_size': 10, 'num_steps': 48, 'img_size': 128, 'batch_split': 5, 'gen_lr': 10e-6, 'disc_lr': 10e-5}, int(200e3): {},

Is there anything else I can do to fix the OOM error?

thanks

marco, thanks for the work

i had to fix a few things to get this to run on a single GPU setting - if you care, i added a new file train_local.py that makes single gpu training functional on a fork https://github.com/xvdp/pi-GAN, I did not combine the two to avoid excess if statements. you could want to refactor everything to call the local and distributed functions from single code but I dont know if it is worth it.

I noticed double definitions, so i linted the whole project; I saw some missing arguments in some functions and so on. I also noticed that you have your local dataset_path for cats and carla reversed. If you want i can pull request.

xvdp

Hi, @ericryanchan,

I am really curious about how you solve the head position problem. I see that real images are not paired with ground-truth head positions, thus, the network learns the head position in an unsupervised way.

After checking your code, I find that in every turn, you sample the head position, and let the discriminator to predict the head position of the rendered faces. The output of discriminator is corrected by the sampled head position in a self-supervised way.

I am really puzzled by this mechanism, and fail to figure out why this works. Can you help me?

Hi, Thanks for sharing your work! I got this problem when training CelebA with pi-GAN. And I don't know how to solve it. It was runned in one GPU V100-32GB with pytorch 1.7.0 and cuda 10.1.

Exception:
-- Process 1 terminated with the following error:
Traceback (most recent call last):
File "/mnt/lustre/gaosicheng/anaconda3/envs/pigan/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
fn(i, *args)
File "/mnt/lustre/gaosicheng/codes/pi-GAN-master/train.py", line 249, in train
scaler.scale(d_loss).backward()
File "/mnt/lustre/gaosicheng/anaconda3/envs/pigan/lib/python3.7/site-packages/torch/tensor.py", line 221, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/mnt/lustre/gaosicheng/anaconda3/envs/pigan/lib/python3.7/site-packages/torch/autograd/init.py", line 132, in allow_unreachable=True) # allow_unreachable flag
RuntimeError: [/opt/conda/conda-bld/pytorch_1603729006826/work/third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:84] Timed out waiting 1800000ms for recv operation to complete

Hi,

Thank you very much for the well-organized repository and inspiring work! I find the codebase very clean and convenient since it has all the necessary visualization code ready.

I'm confused about one thing though: In line 278 of train.py, there is an identity penalty loss for the generator. But as far as I understand, the gradient for identity penalty does not flow through the generator because the sampling of z and pitch yaw values is not learnable. Is that right?

Cheers, Yufeng

Hi, According to the Readme, the command for shape extraction is:

python3 shape_extraction.py path/to/generator.pth --curriculum CelebA --seed 0

However, there is no shape_extraction.py file. Instead, please correct this as follows:

python extract_shapes.py --seeds 0 --output_dir path/to/output path/to/generator.pth

Also, the generator extracts the shape as a voxel in .mrc format. Could you please provide some guidance on how we can extract the 3D shape as a mesh in .obj format?

thanks

Dear friends: I am wondering that why the lr for G and D are 5e-5 and 4e-4, respectively. Do you have any idea? I came up with this queation because I saw that in GRAF and GIRAFFE, the lr for G and D are 5e-4 and 1e-4, respectively. Does it mean that in pi-GAN the G is "more powerful" than the D? Looking forward to hear from you, thanks!

This is a question about the method/paper, not so much the implementation.

During training, do you provide the corresponding camera pose (denoted as ξ in the paper) to the discriminator? It appears the answer is no. If this is the case, why doesn't the generator just ignore the camera pose altogether, and just learn to generate images from a random angle each time? In my mind, the discriminator wouldn't be able to tell? Perhaps you train on multiple samples with the same z per batch, enforcing that different ξ give reasonable results for the same z?

In other GAN papers, we usually train D for k steps and train G for only one step in each GAN training iteration In this case, the z used for training G and D are obviously different. But in this paper, we train G and D simultaneously, i.e., training G and D for both one step in a training iteration. I'm wondering if I could use the same z to train both two networks in each iteration to reduce the computational cost? Just as shown in Pytorch's official GAN tutorial (https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html#training):

# Training Loop

# Lists to keep track of progress
img_list = []
G_losses = []
D_losses = []
iters = 0

print("Starting Training Loop...")
# For each epoch
for epoch in range(num_epochs):
    # For each batch in the dataloader
    for i, data in enumerate(dataloader, 0):

        ############################
        # (1) Update D network: maximize log(D(x)) + log(1 - D(G(z)))
        ###########################
        ## Train with all-real batch
        netD.zero_grad()
        # Format batch
        real_cpu = data[0].to(device)
        b_size = real_cpu.size(0)
        label = torch.full((b_size,), real_label, dtype=torch.float, device=device)
        # Forward pass real batch through D
        output = netD(real_cpu).view(-1)
        # Calculate loss on all-real batch
        errD_real = criterion(output, label)
        # Calculate gradients for D in backward pass
        errD_real.backward()
        D_x = output.mean().item()

        ## Train with all-fake batch
        # Generate batch of latent vectors
        noise = torch.randn(b_size, nz, 1, 1, device=device)
        # Generate fake image batch with G
        fake = netG(noise)
        label.fill_(fake_label)
        # Classify all fake batch with D
        output = netD(fake.detach()).view(-1)
        # Calculate D's loss on the all-fake batch
        errD_fake = criterion(output, label)
        # Calculate the gradients for this batch, accumulated (summed) with previous gradients
        errD_fake.backward()
        D_G_z1 = output.mean().item()
        # Compute error of D as sum over the fake and the real batches
        errD = errD_real + errD_fake
        # Update D
        optimizerD.step()

        ############################
        # (2) Update G network: maximize log(D(G(z)))
        ###########################
        netG.zero_grad()
        label.fill_(real_label)  # fake labels are real for generator cost
        # Since we just updated D, perform another forward pass of all-fake batch through D
        output = netD(fake).view(-1)
        # Calculate G's loss based on this output
        errG = criterion(output, label)
        # Calculate gradients for G
        errG.backward()
        D_G_z2 = output.mean().item()
        # Update G
        optimizerG.step()

        # Output training stats
        if i % 50 == 0:
            print('[%d/%d][%d/%d]\tLoss_D: %.4f\tLoss_G: %.4f\tD(x): %.4f\tD(G(z)): %.4f / %.4f'
                  % (epoch, num_epochs, i, len(dataloader),
                     errD.item(), errG.item(), D_x, D_G_z1, D_G_z2))

        # Save Losses for plotting later
        G_losses.append(errG.item())
        D_losses.append(errD.item())

        # Check how the generator is doing by saving G's output on fixed_noise
        if (iters % 500 == 0) or ((epoch == num_epochs-1) and (i == len(dataloader)-1)):
            with torch.no_grad():
                fake = netG(fixed_noise).detach().cpu()
            img_list.append(vutils.make_grid(fake, padding=2, normalize=True))

        iters += 1

Such that we only need to call G.forward once in each iteration. Will it affect the model performance?

Hi,

I just run the inverse_render.py with Biden's portrait as shown in your demo, but I fail to get the same reconstruct video like yours as shown below:

Is there anything I can do to refine the result? If anyone knows how to solve this problem, please leave your comment! Thanks.

Hi,

On the project website I found a pretty good 3d shape rendered from the CARLA dataset, so I am trying to extract similar mesh using the file provided in this repository. However, I noticed that no matter how I change the parameters, I am not able to generate a 3d mesh with similar quality using the checkpoint provided by the authors. Here is an example using a slightly larger cube length and 512 resolution: The car itself seems fine, but I am not sure why the big mass below the car is generated. I believe this is partially due to the fact that CARLA dataset does not include images view from the bottom or lower angle? Can you provide any hint on how to generate the fine 3D mesh like the one on the project website?

Thanks