Code for the paper: Sketch Your Own GAN

When I was running bash scripts/train_teaser_cat.sh in my Linux server, I got a TypeError. My virtual environment in conda already installed: Python 3.7.12, pytorch 1.7.1, cudatoolkit 10.1.243, torchvision 0.8.2, and ninja 1.10.2. I already tried higher version of pytorch 1.8.1, but it has another problem of subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status. So I returned to the pytorch 1.7.1 environment. I have been troubled by the TypeError problem for several days. Can someone help me solve it? Thanks.

Logs in detail:

(test3) yuanWang@server-TiTan:~/GANtest1$ bash scripts/train_teaser_cat.sh
Traceback (most recent call last):
  File "train.py", line 6, in <module>
    from eval import Evaluator
  File "/data4/yuanWang/GANtest1/eval/__init__.py", line 1, in <module>
    from .evaluation import Evaluator
  File "/data4/yuanWang/GANtest1/eval/evaluation.py", line 7, in <module>
    from run_metrics import get_vgg_features, make_eval_images
  File "/data4/yuanWang/GANtest1/run_metrics.py", line 13, in <module>
    from training.networks.stylegan2 import Generator
  File "/data4/yuanWang/GANtest1/training/networks/__init__.py", line 2, in <module>
    from .misc import *
  File "/data4/yuanWang/GANtest1/training/networks/misc.py", line 1, in <module>
    from . import stylegan2
  File "/data4/yuanWang/GANtest1/training/networks/stylegan2.py", line 8, in <module>
    from .op import FusedLeakyReLU, fused_leaky_relu, upfirdn2d
  File "/data4/yuanWang/GANtest1/training/networks/op/__init__.py", line 1, in <module>
    from .fused_act import FusedLeakyReLU, fused_leaky_relu
  File "/data4/yuanWang/GANtest1/training/networks/op/fused_act.py", line 15, in <module>
    os.path.join(module_path, "fused_bias_act_kernel.cu"),
  File "/data4/yuanWang/anaconda3/envs/test3/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1091, in load
    keep_intermediates=keep_intermediates)
  File "/data4/yuanWang/anaconda3/envs/test3/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1269, in _jit_compile
    is_standalone=is_standalone,
TypeError: bump_version_if_changed() got an unexpected keyword argument 'is_python_module'

How can I use GANspace to adjust the image generated by the net_G? For example, the original generator can generate cats pictures indoor but I want the background to be outdoor grassland. Is such an operation possible? If possible, how can I operate through GANspace? Thank you very much for your help!

Hello, thank you for your valuable advice. Now, I have solved the compilation problem about the fused module under the environment of pytorch **1.7.1**. Now I've got the required .so file:

(test4torch) yuanWang@server-TiTan:~/.cache/torch_extensions/fused$ ls
build.ninja  fused_bias_act_kernel.cuda.o  fused_bias_act.o  fused.so

I encountered a new problem when I tried to use _import_module_from_libraryto load the compiled module fused, I changed the beginning code in fused_act.py to:

try:
    user_home_path = os.path.expanduser('~')
    fused = _import_module_from_library('fused', user_home_path+'/.cache/torch_extensions/fused', True)
    print(f'Load fused from {user_home_path}/.cache/torch_extensions/fused')
    print("Load success!")
except:
    module_path = os.path.dirname(__file__)
    fused = load(
        name='fused',
        sources=[
            os.path.join(module_path, 'fused_bias_act.cpp'),
            os.path.join(module_path, 'fused_bias_act_kernel.cu'),
        ],
        verbose=True
    )
    print(f'Load function used. Build fused from cpp & cu files')

Executing the code step by step will not report an error, but when I type quit (), the command line will report an error Segmentation fault (Core dumped). My gdb traceback content shows:

(gdb) run fused_act.py
Starting program: /data4/yuanWang/anaconda3/envs/test4torch/bin/python fused_act.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fff7d446700 (LWP 4153)]
Load fused from /data4/yuanWang/.cache/torch_extensions/fused
Load success!

Thread 1 "python" received signal SIGSEGV, Segmentation fault.
0x00007fffdeafa9b8 in ?? ()
   from /data4/yuanWang/anaconda3/envs/test4torch/lib/python3.7/site-packages/torch/lib/../../../../libcudart.so.10.1
(gdb) backtrace
#0  0x00007fffdeafa9b8 in ?? ()
   from /data4/yuanWang/anaconda3/envs/test4torch/lib/python3.7/site-packages/torch/lib/../../../../libcudart.so.10.1
#1  0x00007fffdeafb1a3 in ?? ()
   from /data4/yuanWang/anaconda3/envs/test4torch/lib/python3.7/site-packages/torch/lib/../../../../libcudart.so.10.1
#2  0x00007fffdeafb8a5 in ?? ()
   from /data4/yuanWang/anaconda3/envs/test4torch/lib/python3.7/site-packages/torch/lib/../../../../libcudart.so.10.1
#3  0x00007ffff7806031 in __run_exit_handlers (status=0, 
    listp=0x7ffff7bae718 <__exit_funcs>, 
    run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true)
    at exit.c:108
#4  0x00007ffff780612a in __GI_exit (status=<optimized out>) at exit.c:139
#5  0x00007ffff77e4c8e in __libc_start_main (main=0x555555645ab0 <main>, argc=2, 
    argv=0x7fffffffe158, init=<optimized out>, fini=<optimized out>, 
    rtld_fini=<optimized out>, stack_end=0x7fffffffe148)
    at ../csu/libc-start.c:344
#6  0x000055555572b73d in _start () at ../sysdeps/x86_64/elf/start.S:103

Do you know how to solve this problem? By the way, do you choose to use Windows as the operating system to run code? I strongly suspect that this problem is related to the Ubuntu operating system. Thank you for your time. Best wishes.

Hello, thank you for the code available,

I executed the training script:

bash scripts/train_photosketch_horse_riders.sh

and then I compared the metrics of the trained model with those of the file:

weights/photosketch_horse_riders_aug.pth

using the evaluation script:

run_metrics.py

with the command:

python run_metrics.py --models_list weights/eval_list --output metric_results.csv

I added the trained weights into the weights folder and into the eval_list file so that I could check its FID too.

I very different FIDs for the photosketch_horse_riders_aug.pth (FID ~20.13) and the trained one (FID ~37.07). I assumed that the training script would generate a model similar to the one stored at photosketch_horse_riders_aug.pth.

There is some other different procedure to get a trained model with the same FID as photosketch_horse_riders_aug.pth, which is very close but not quite the same as in the paper (FID ~19.94)?

I found that there are codes in train.py that automatically saves the latest model after a certain number of iterations. When I stop training manually, how can I continue the unfinished training with the saved model? Thanks for your help！

Not exactly an issue.

I created a Google Colab Notebook that sets up the repo and calls the example commands in the readme. I did this because of issues I was having on my local setup. Hopefully this could be added to give an easier quick start for those interested in the project.

Colab Link

When I was runningfused_act.py, I got a bug about TypeError. I am using a Linux server without root and sudo permission, and my virtual environment in conda is: Python 3.7, pytorch 1.7.1, torchvision 0.8.2, cudatoolkit V10.1.243, gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04) and ninja 1.10.2. I used conda install -c conda-forge ninja to install ninja but not sure about ninja's working status. I already changed line 1631 in myName/anaconda3/envs/test1/lib/python3.7/site-packages/torch/utils/cpp_extension.py, from ['ninja', '-v'] to ['ninja', '-version'] to avoid subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status. After that, I got the TypeError problem. Could anybody help me please? Logs in detail: Traceback (most recent call last): (in virtual environment pytorch 1.7.1) File "fused_act.py", line 15, in os.path.join(module_path, "fused_bias_act_kernel.cu"), File "/data4/yuanWang/anaconda3/envs/test3/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1091, in load keep_intermediates=keep_intermediates) File "/data4/yuanWang/anaconda3/envs/test3/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1269, in _jit_compile is_standalone=is_standalone, TypeError: bump_version_if_changed() got an unexpected keyword argument 'is_python_module'

I was trying to train for the first time using the example:

"python train.py --name cat_train_5 --batch 16 --dataroot_sketch ./data/sketch/by_author/cat --dataroot_image ./data/image/cat --l_image 0.7 - -g_pretrained ./pretrained/stylegan2-cat/netG.pth --d_pretrained ./pretrained/stylegan2-cat/netD.pth --max_iter 150000 --disable_eval --diffaug_policy translation --no_wandb"

And everything seems to be fine:

Using pretrained weight for D1... Using pretrained weight for D_image... ----------------- Options --------------- batch: 16 [default: 4] beta1: 0.0 beta2: 0.99 channel_multiplier: 2 checkpoints_dir: checkpoint d_pretrained: ./pretrained/stylegan2-cat/netD.pth [default: ] d_reg_every: 16 dataroot_image: ./data/image/cat [default: None] dataroot_sketch: ./data/sketch/by_author/cat [default: None] diffaug_policy: translation [default: ] disable_eval: True [default: False] display_freq: 2500 display_winsize: 400 dsketch_no_pretrain: False eval_batch: 50 eval_dir: None eval_freq: 5000 g_pretrained: ./pretrained/stylegan2-cat/netG.pth [default: ] gan_mode: softplus isTrain: True [default: None] l_image: 0.7 [default: 0] l_weight: 0 latent_avg_samples: 8192 lr: 0.002 lr_mlp: 0.01 max_epoch: 1000000 max_iter: 150000 [default: 75001] mixing: 0.9 n_mlp: 8 name: cat_train_5 [default: None] no_d_regularize: False no_html: False no_wandb: True [default: False] optim_param_g: style photosketch_path: ./pretrained/photosketch.pth print_freq: 100 r1: 10 reduce_visuals: False resume_iter: None save_freq: 2500 size: 256 sketch_channel: 1 transform_fake: toSketch,to3ch transform_real: to3ch use_cpu: False z_dim: 512 ----------------- End -------------------

-------------- Trainables --------------- (G trainable parameters) style.1.weight style.1.bias style.2.weight style.2.bias style.3.weight style.3.bias style.4.weight style.4.bias style.5.weight style.5.bias style.6.weight style.6.bias style.7.weight style.7.bias style.8.weight style.8.bias ----------------- End ------------------- create web directory checkpoint\cat_train_5\web... Training was successfully finished.

I expected that the weights would be generated in the checkpoint folder and then used to generate the images, but I only found two files "ops" and "log_loss" which is empty.

I am running this on Windows 10.

The process is correct? have I forgotten something? Could it be that the problem is OS?

I'm new working with gans, so any help will be appreciated.

Hi Wang,

Thanks for sharing your code.

I have installed all the prerequisite packages. When I try to compile the module 'fused', errors below occur. I removed all cache file in torch_extensions folder by using rm -rf before every building tests but still got the same error. By the way I already changed '['ninja', '-v']' to '['ninja', '--version']' in cpp_extension.py under anaconda3/envs/CTtest1/lib/python3.7/site-packages/torch/utils/ to avoid another error. Could you please tell me how to fix it?

Detected CUDA files, patching ldflags
Emitting ninja build file /home/slhua/.cache/torch_extensions/fused/build.ninja...
Building extension module fused...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
1.7.2
Loading extension module fused...
Traceback (most recent call last):
  File "fused_act.py", line 25, in <module>
    is_python_module = True)
  File "/home/slhua/anaconda3/envs/CTtest1/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 997, in load
    keep_intermediates=keep_intermediates)
  File "/home/slhua/anaconda3/envs/CTtest1/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1213, in _jit_compile
    return _import_module_from_library(name, build_directory, is_python_module)
  File "/home/slhua/anaconda3/envs/CTtest1/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1560, in _import_module_from_library
    file, path, description = imp.find_module(module_name, [path])
  File "/home/slhua/anaconda3/envs/CTtest1/lib/python3.7/imp.py", line 296, in find_module
    raise ImportError(_ERR_MSG.format(name), name=name)
ImportError: No module named 'fused'

My environment: cudatoolkit 10.2.89, cudnn 7.6.5, python 3.7.6, pytorch 1.7.1, torchvision 0.8.2, ninja 1.7.2

Hi! Glad to see your work! But I have a question, as follows.

Consider practical usage: step 1: I make a cat sketch image by hand. step 2: transform the sketch image to latent_z. step 3: feed latent_z to netG network to get a cat image.

I am surprised how to realize step2 ? Do you mean that I need netG, photo2Sketch network, and use pix2latent method? Or only need netG and use pix2latent method? If I just use netG to get z, it will still generate the cat sketch, but not cat image. Is not it ?

Thanks!

Is it possible to generate cats by providing my own sketches? Looking at the code it seems that all sketches are hardcoded, but maybe I'm missing something.