Thank you for open-sourcing this project!

I was trying to train semanticGAN on our dataset (3 labels), by running:

python -m torch.distributed.launch --nproc_per_node=N_GPU --master_port=AVAILABLE_PORT train_seg_gan.py --img_dataset=IMG_DATASET_LOC  --seg_dataset=SEG_DATASET_LOC --inception=INCEPTION_FILE

(Note: I already updated seg_dim to 3 in our case in argparse and reduced the color_map to only 3 colors in dataset.py)

However, after the code initializates, I run into this error:

RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior

Here is the entire output:

/home/user/anaconda3/envs/semanticGAN/lib/python3.7/site-packages/torch/distributed/launch.py:186: FutureWarning: The module torch.distributed.launch is deprecated
and will be removed in future. Use torchrun.
Note that --use_env is set by default in torchrun.
If your script expects `--local_rank` argument to be set, please
change it to read from `os.environ['LOCAL_RANK']` instead. See 
https://pytorch.org/docs/stable/distributed.html#launch-utility for 
further instructions

  FutureWarning,
WARNING:torch.distributed.run:
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************
Loading unlabel dataloader with size  5
Loading unlabel dataloader with size  5
Loading train dataloader with size  4
Loading train dataloader with size  4
Loading val dataloader with size  2
Loading val dataloader with size  2
Loading unlabel dataloader with size Loading unlabel dataloader with size   55

Loading train dataloader with size  4
Loading train dataloader with size  4
Loading val dataloader with size  2
Loading val dataloader with size  2
[W reducer.cpp:1303] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
[W reducer.cpp:1303] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
[W reducer.cpp:1303] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
[W reducer.cpp:1303] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
Traceback (most recent call last):
  File "train_seg_gan.py", line 737, in <module>
    g_optim, d_img_optim, d_seg_optim, g_ema, device, writer)
  File "train_seg_gan.py", line 386, in train
    fake_img, latents, mean_path_length
  File "train_seg_gan.py", line 111, in g_path_regularize
    outputs=(fake_img * noise).sum(), inputs=latents, create_graph=True
  File "/home/user/anaconda3/envs/semanticGAN/lib/python3.7/site-packages/torch/autograd/__init__.py", line 236, in grad
    inputs, allow_unused, accumulate_grad=False)
RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.
Traceback (most recent call last):
  File "train_seg_gan.py", line 737, in <module>
    g_optim, d_img_optim, d_seg_optim, g_ema, device, writer)
  File "train_seg_gan.py", line 386, in train
    fake_img, latents, mean_path_length
  File "train_seg_gan.py", line 111, in g_path_regularize
    outputs=(fake_img * noise).sum(), inputs=latents, create_graph=True
  File "/home/user/anaconda3/envs/semanticGAN/lib/python3.7/site-packages/torch/autograd/__init__.py", line 236, in grad
    inputs, allow_unused, accumulate_grad=False)
RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.
Traceback (most recent call last):
  File "train_seg_gan.py", line 737, in <module>
    g_optim, d_img_optim, d_seg_optim, g_ema, device, writer)
  File "train_seg_gan.py", line 386, in train
    fake_img, latents, mean_path_length
  File "train_seg_gan.py", line 111, in g_path_regularize
    outputs=(fake_img * noise).sum(), inputs=latents, create_graph=True
  File "/home/user/anaconda3/envs/semanticGAN/lib/python3.7/site-packages/torch/autograd/__init__.py", line 236, in grad
    inputs, allow_unused, accumulate_grad=False)
RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.
Traceback (most recent call last):
  File "train_seg_gan.py", line 737, in <module>
    g_optim, d_img_optim, d_seg_optim, g_ema, device, writer)
  File "train_seg_gan.py", line 386, in train
    fake_img, latents, mean_path_length
  File "train_seg_gan.py", line 111, in g_path_regularize
    outputs=(fake_img * noise).sum(), inputs=latents, create_graph=True
  File "/home/user/anaconda3/envs/semanticGAN/lib/python3.7/site-packages/torch/autograd/__init__.py", line 236, in grad
    inputs, allow_unused, accumulate_grad=False)
RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 113892) of binary: /home/user/anaconda3/envs/semanticGAN/bin/python
Traceback (most recent call last):
  File "/home/user/anaconda3/envs/semanticGAN/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/user/anaconda3/envs/semanticGAN/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/user/anaconda3/envs/semanticGAN/lib/python3.7/site-packages/torch/distributed/launch.py", line 193, in <module>
    main()
  File "/home/user/anaconda3/envs/semanticGAN/lib/python3.7/site-packages/torch/distributed/launch.py", line 189, in main
    launch(args)
  File "/home/user/anaconda3/envs/semanticGAN/lib/python3.7/site-packages/torch/distributed/launch.py", line 174, in launch
    run(args)
  File "/home/user/anaconda3/envs/semanticGAN/lib/python3.7/site-packages/torch/distributed/run.py", line 713, in run
    )(*cmd_args)
  File "/home/user/anaconda3/envs/semanticGAN/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 131, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/user/anaconda3/envs/semanticGAN/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 261, in launch_agent
    failures=result.failures,
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
train_seg_gan.py FAILED
------------------------------------------------------------

The error seems to be in this function:

def g_path_regularize(fake_img, latents, mean_path_length, decay=0.01):
    noise = torch.randn_like(fake_img) / math.sqrt(
        fake_img.shape[2] * fake_img.shape[3]
    )
    grad, = autograd.grad(
        outputs=(fake_img * noise).sum(), inputs=latents, create_graph=True
    )
    path_lengths = torch.sqrt(grad.pow(2).sum(2).mean(1))

    path_mean = mean_path_length + decay * (path_lengths.mean() - mean_path_length)

    path_penalty = (path_lengths - path_mean).pow(2).mean()

    return path_penalty, path_mean.detach(), path_lengths

Specifically, in line 110-111 where the grad function is being called, on the output comprising of fake_img and noise, with the input being latents.

Please let me know if there I'm making an error while executing the command or if I need to make some more changes to the code, considering that we're trying to train on our own dataset.

Thank you for your time!

Hi, thanks for releasing the source code. I am wondering if there is any plan to make the annotated MetFaces public? As in the paper, 40 images are selected from MetFaces and manually annotated for out-of-domain evaluation. I want to do a similar evaluation for our methods. It would be a huge help if this dataset were made public. Thanks a lot!

When I set mean_init == False to load the encoder checkpoints it returns a KeyError for not finding the checkpoints for the FPNEncoder. Isn't the encoder weights already part of the saved checkpoint during training?

Hi, Thanks for releasing the code! Just a silly question: I have no experience with GANs and the loss functions employed so I was wondering if the optimization process works better when binary masks are given in two channels or just with one (0:background, 1:Region of interest). For example for the X-Ray problem, how do you feed the model?

Thank you!

I would like to ask about the stopping criteria for train_seg_gan.py. So after every 2000 iterations, I get the values of FID, IS, d_img val, and d_seg val. I'm not quite sure what the last two are for? It seems like it's the averaged prediction of the segmentation discriminator, and image discriminator.

Usually, I stop training when the images look appealing, and the FID score plateaus. Do the other scores play any role? How to evaluate the segmentation branch during training? I'm in a situation where FID is low, but segmentation visual inspection is bad.

Any insight would be highly appreciated :)

Thanks for sharing codes of your very interesting work. While I have a question about the work: According to the Motivation-Sec. training GAN can avoid overfitting limited labeled training subset, but I wonder whether the GAN will also overfit or struggle from model collapse with only limited labeled training data (limited real images), or the GAN will be hard to converge? Many thanks!

Hello,

I am facing this issue while running train_seg_gan.py (both on single and multiple GPUs) where the training will get to the first iteration and then get stuck there. My GPU utilization remains constant and there is no further logging.

This is the output I am getting:

==================Start calculating validation scores==================
d_img val scores: -1.5312, d_seg val scores: 0.1412
==================Start calculating FID==================
Gathering activations...
Calculating Inception Score...
Calculating means and covariances...
Covariances calculated, getting FID...
iteration 00000000: FID: 332.5834, IS_mean: 2.0889, IS_std: 0.0289

Once it reaches this point, nothing further happens. I ended up canceling the run after an hour of being stuck here. I am using the CelebAMask dataset for training.

Pytorch 1.4.0, CUDA Version: 11.0, Python 3.6.13

I appreciate any help you're able to provide me with!

Hello.

CelebAMask-HQ provides 19 classes of annotation on face according to original paper and their github repo.

| Label list | | | | ----------- | ----------- | ---------- | 0: 'background' | 1: 'skin' | 2: 'nose' 3: 'eye_g' | 4: 'l_eye' | 5: 'r_eye' 6: 'l_brow' | 7: 'r_brow' | 8: 'l_ear' 9: 'r_ear' | 10: 'mouth' | 11: 'u_lip' 12: 'l_lip' | 13: 'hair' | 14: 'hat' 15: 'ear_r' | 16: 'neck_l' | 17: 'neck' 18: 'cloth'| | |

However, in your work only 8 classes are classified. How do you define all those classes?

Moreover, DatasetGAN, which is work of your lab as well, seems also adopting 8-classes protocol. Do you share the same definition?

BTW, let me make a guess, your mask labels are defined as follow, which could be directly merged by original 19 classes masks. | 8 classes | ID that corresponds to in original 19 classes |
| ----------- | ----------- | 0: 'background' | 0, 3, 14, 15, 16, 17, 18 | 1: 'skin' | 1 | 2: 'nose' | 2 | 3: 'eye' | 4, 5 | 4: 'brow' | 6, 7 | 5: 'ear' | 8, 9 | 6: 'mouth' | 10, 11, 12 | 7: 'hair' | 13 |

Please reply if I have made it correct. Also, if I am correct about this, how do you deal with eye_g (i.e. glasses ) in original settings, since annotating glasses to background will lead to aliases in label image.

Here is a snapshot from original celebAMask-HQ with glasses.

Hi, thanks for sharing the code. Amazing work.

I have a question about the losses used to train the GAN, specifically the generator. I understand the adversarial losses coming from the discriminator_img and discriminator_seg. But I don't understand the L1 loss between the intermediate features of the fake and real predictions of the joint discriminator.

https://github.com/nv-tlabs/semanticGAN_code/blob/342889ebbe817695c0e64133100ede8f9877f3de/semanticGAN/train_seg_gan.py#L360-L365

What section of the paper mentions this L1 feature loss between fake and real features of discriminator_seg?

Thanks in advance.

When trying to run the prepare_inception.py script, I am getting the following error:

OSError: ./semanticGAN_code/CelebAMask-HQ/CelebA-HQ-img/unlabel_data/unlabel_list.txt not found.

It seems like the unlabel_data directory does not exist within the CelebA Mask dataset. How can I find the missing file and organize the directory?

Somehow I am getting the input batch size multiplied by 4 compared to the target batch size. Btw, I am using my own custom dataset instead of CelebaDataset

Hi, your project semanticGAN_code(commit id: 342889ebbe817695c0e64133100ede8f9877f3de) requires "albumentations==0.5.2" in its dependency. After analyzing the source code, we found that the following versions of albumentations can also be suitable, i.e., albumentations 0.5.1, since all functions that you directly (3 APIs: albumentations.augmentations.transforms.ShiftScaleRotate.init, albumentations.augmentations.transforms.HorizontalFlip.init, albumentations.core.composition.Compose.init) or indirectly (propagate to 12 albumentations's internal APIs and 0 outsider APIs) used from the package have not been changed in these versions, thus not affecting your usage.

Therefore, we believe that it is quite safe to loose your dependency on albumentations from "albumentations==0.5.2" to "albumentations>=0.5.1,<=0.5.2". This will improve the applicability of semanticGAN_code and reduce the possibility of any further dependency conflict with other projects.

May I pull a request to further loosen the dependency on albumentations?

By the way, could you please tell us whether such an automatic tool for dependency analysis may be potentially helpful for maintaining dependencies easier during your development?

Could you please upload these files "unlabel_list.txt","train_full_list.txt", "val_full_list.txt", "unlabel_list.txt'' "? I want to know their details,thanks.

Hello

I am working with datasets that contain more labeled images than unlabeled. In my experiments with default settings, it seems that the generator struggles to learn the input, and images can be quite noisy. Is there a reason for this? From what I can tell, more labels should not harm the generator but I may be wrong.

This is for single class images with very little background, as they are cropped to the height and width of the desired object.

Hi there,

Thank you so much for your code and paper. Very impressive work.

I just want to reproduce your result, and so far so good. But when I want to resume my ckpt from the previous training, it throws the following error. load model: /research/cbim/vast/qc58/work/projects/semanticGAN/ckpt/run-Feb03_16-23-14/ckpt/060000.pt Traceback (most recent call last): File "train_seg_gan.py", line 679, in <module> g_optim.load_state_dict(ckpt['g_optim']) KeyError: 'g_optim'

I think there is no code save g_optim, d_img_optim and d_seg_optim when training, how can you load these setting from the checkpoint? Please let me know how you resume your training and if I missed anything?

Thanks again.

Hello,

Thank you for this fantastic work and this repository. I am working on modifying the code to work with 1024x1024 patches from whole slide images to generate patches and corresponding annotation masks. I was successful at implementing a dataloader for this dataset and tweaking the code to work with it so far, but I am trying to troubleshoot why the training gets stuck on the g_regularize step for this dataset. The code is able to calculate the path loss, but it gets stuck trying to calculate the weighted_path_loss.backward(). I am running this on 4 a100 GPUs with 32 cores on the cloud with torch.distributed.launch with a batch=8 (assuming this is 8 per GPU).

I forked the repo and you can look at the changes I made to it here. https://github.com/crobbins327/semanticGAN_WSI.git It's mainly just a new dataset for WSIs in dataset.py and some tweaks to train_seg_gan.py to load the dataset. When I disable the path length regularization by setting the g_regularize if statement to False, the code works fine.

Do you have any advice on how to troubleshoot this problem or why the code freezes for the g_regularize step for this dataset at 1024x1024 patch resolution?

In train_seg_gan.py, when train dseg

requires_grad(generator, False)
requires_grad(discriminator_img, False)
requires_grad(discriminator_seg, True)

but in regulation part, dont open the toggle, dose this infection the dimg regularize?

if d_regularize:
    real_img.requires_grad = True
    real_pred = discriminator_img(real_img)
    r1_img_loss = d_r1_loss(real_pred, real_img)

    discriminator_img.zero_grad()
    (args.r1 / 2 * r1_img_loss * args.d_reg_every + 0 * real_pred[0]).backward()

    d_img_optim.step()