Official implementation of the paper Image Generators with Conditionally-Independent Pixel Synthesis https://arxiv.org/abs/2011.13775

Multimodal Lab @ Samsung AI Center Moscow

Last update: Dec 21, 2022

Overview

CIPS -- Official Pytorch Implementation

of the paper Image Generators with Conditionally-Independent Pixel Synthesis

Requirements

pip install -r requirements.txt

Usage

First create lmdb datasets:

python prepare_data.py images --out LMDB_PATH --n_worker N_WORKER --size SIZE1,SIZE2,SIZE3,... DATASET_PATH

This will convert images to jpeg and pre-resizes it.

To train on FFHQ-256 or churches please run:

python3 -m torch.distributed.launch --nproc_per_node=8 --master_port=1234 train.py --n_sample=8 --batch=4 --fid_batch=8 --Generator=CIPSskip --output_dir=skip-[ffhq/churches] --img2dis --num_workers=16 DATASET_PATH

To train on patches add --crop=PATCH_SIZE. PATCH_SIZE has to be a power of 2.

Pretrained Checkpoints

Generate samples

To play with the models please download checkpoints and check out a notebook.ipynb

Progressive training

We also tried to train progressively on FFHQ starting from 256×256 initialization and got FID 10.07. We will update the paper with the training details soon. Checkpoint name is ffhq1024.pt. Samples are below.

Citation

If you found our work useful, please don't forget to cite

@article{anokhin2020image,
  title={Image Generators with Conditionally-Independent Pixel Synthesis},
  author={Anokhin, Ivan and Demochkin, Kirill and Khakhulin, Taras and Sterkin, Gleb and Lempitsky, Victor and Korzhenkov, Denis},
  journal={arXiv preprint arXiv:2011.13775},
  year={2020}
}

The code is heavely based on the styleganv2 pytorch implementation

Nvidia-licensed CUDA kernels (fused_bias_act_kernel.cu, upfirdn2d_kernel.cu) is for non-commercial use only.

Comments

Quick question about training time

Hello authors,

Thank you for your paper -- it looks very interesting! I had a quick question about training time: how long (and with how many GPUs) does it take you to train the pretrained models provided in the repo? Do you find that the method is slow or fast relative to StyleGAN2?

I look forward to playing around with the repo over the next week!

Best, greeneggsandyaml

opened by greeneggsandyaml 2
About the blur kernel that affect pixel indenpence

I see that in the code for CIPS block, you use a Blur layer, which use 3x3 blur kernel so I think one pixel is affect by 8 surrouding pixel, is it correct ?

opened by thuanz123 1

Timed out when calculating FID

Hi. First of all, thanks for your great work.

I have a problem when calculating FID during training. Every args.save_checkpoint_frequency iterations, the model is evaluated by calculating FID score. However, at this phase, I have a timed out problem. Here is the error log.

[E ProcessGroupNCCL.cpp:566] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1802801 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:325] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. To avoid this inconsistency, we are taking the entire process down.
terminate called after throwing an instance of 'std::runtime_error'
  what():  [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1802801 milliseconds before timing out.
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -6) local_rank: 1 (pid: 4414) of binary: /opt/conda/bin/python
ERROR:torch.distributed.elastic.agent.server.local_elastic_agent:[default] Worker group failed

Before this log came out, there was no change in the prompt but GPUs were still working. As I'm not that familiar with distributed training, I want to ask how to fix this problem. I've tried to lower the number of args.fid_samples, but this doesn't help. Thank you.

Best Wishes, Lee

opened by shlee625 1

train model

Why do you have an generator and g_ema in your training code？And Your optimizer optimizes generator directly，why not optimizes g_ema? And why accumulate(g_ema, g_module, accum) ，not And why accumulate(g_ema, generator, accum) ？

opened by dengshuhai-cmd 0
Checkpoint for Ablation Study?

Hi,

Really nice work! I wonder if you have checkpoints for models in the ablation study? For example, models trained without the coordinate embedding? THanks!

opened by stevenygd 0
I tried to train with your code,but it didn't work.The main progress seems to stay waiting for something to load.

/home/hello/miniconda3/envs/deepsort/bin/python3 /home/hello/桌面/CIPS-main/train.py LMDB_data1024*1024_anime --nproc_per_node=8 --master_port=1234 --n_sample=8 --batch=4 --fid_batch=8 --Generator=CIPSskip --output_dir=skip-[ffhq/churches] --img2dis --num_workers=16 Traceback (most recent call last): File "/home/hello/桌面/CIPS-main/train.py", line 16, in import model File "/home/hello/桌面/CIPS-main/model/init.py", line 1, in from .Discriminators import * File "/home/hello/桌面/CIPS-main/model/Discriminators.py", line 8, in from .blocks import ConvLayer, ResBlock, EqualLinear File "/home/hello/桌面/CIPS-main/model/blocks.py", line 8, in from op import FusedLeakyReLU, fused_leaky_relu, upfirdn2d File "/home/hello/桌面/CIPS-main/op/init.py", line 2, in from .upfirdn2d import upfirdn2d File "/home/hello/桌面/CIPS-main/op/upfirdn2d.py", line 13, in os.path.join(module_path, 'upfirdn2d_kernel.cu'), File "/home/hello/miniconda3/envs/deepsort/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1092, in load keep_intermediates=keep_intermediates) File "/home/hello/miniconda3/envs/deepsort/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1307, in _jit_compile baton.wait() File "/home/hello/miniconda3/envs/deepsort/lib/python3.7/site-packages/torch/utils/file_baton.py", line 42, in wait time.sleep(self.wait_seconds) KeyboardInterrupt

Process finished with exit code 1

opened by xiaoxiaoxiaogu 2
Would you please share discriminator's weight also?

Dear authors:

Hello, thank you for sharing the great project.

I would like to use your project for the baseline of my future research.

Would you please share your discriminator's weight also?

Thank you.

Best, chkimmmmm

opened by chkimmmmm 1
Would you share progressive training code?

Dear authors:

Hello.

First of all, thank you for the sharing this great work.

In the Read.me, you mentioned that you are going to share the progressive training details.

When will you share it?

Thank you.

Best. chkimmmmm

opened by chkimmmmm 0

Official implementation of the paper Image Generators with Conditionally-Independent Pixel Synthesis https://arxiv.org/abs/2011.13775

Related tags

Overview

CIPS -- Official Pytorch Implementation

Requirements

Usage

Pretrained Checkpoints

Generate samples

Progressive training

Citation

Comments

Quick question about training time

About the blur kernel that affect pixel indenpence

Timed out when calculating FID

train model

Checkpoint for Ablation Study?

I tried to train with your code,but it didn't work.The main progress seems to stay waiting for something to load.

Would you please share discriminator's weight also?

Would you share progressive training code?

Owner

Multimodal Lab @ Samsung AI Center Moscow

[PyTorch] Official implementation of CVPR2021 paper "PointDSC: Robust Point Cloud Registration using Deep Spatial Consistency". https://arxiv.org/abs/2103.05465

This is an official implementation of our CVPR 2021 paper "Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression" (https://arxiv.org/abs/2104.02300)

Official repository with code and data accompanying the NAACL 2021 paper "Hurdles to Progress in Long-form Question Answering" (https://arxiv.org/abs/2103.06332).

Source code for models described in the paper "AudioCLIP: Extending CLIP to Image, Text and Audio" (https://arxiv.org/abs/2106.13043)

Official Implementation for "ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement" https://arxiv.org/abs/2104.02699

Non-Official Pytorch implementation of "Face Identity Disentanglement via Latent Space Mapping" https://arxiv.org/abs/2005.07728 Using StyleGAN2 instead of StyleGAN

Unofficial Tensorflow-Keras implementation of Fastformer based on paper [Fastformer: Additive Attention Can Be All You Need](https://arxiv.org/abs/2108.09084).

Supplementary code for the paper "Meta-Solver for Neural Ordinary Differential Equations" https://arxiv.org/abs/2103.08561

Code for paper "A Critical Assessment of State-of-the-Art in Entity Alignment" (https://arxiv.org/abs/2010.16314)

Code for the paper: Learning Adversarially Robust Representations via Worst-Case Mutual Information Maximization (https://arxiv.org/abs/2002.11798)

Pytorch implementation of Each Part Matters: Local Patterns Facilitate Cross-view Geo-localization https://arxiv.org/abs/2008.11646

Minimal implementation of PAWS (https://arxiv.org/abs/2104.13963) in TensorFlow.

A PyTorch implementation of EventProp [https://arxiv.org/abs/2009.08378], a method to train Spiking Neural Networks

Unofficial implementation of "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" (https://arxiv.org/abs/2103.14030)

Unofficial implementation of Alias-Free Generative Adversarial Networks. (https://arxiv.org/abs/2106.12423) in PyTorch

Pytorch implementation of Distributed Proximal Policy Optimization: https://arxiv.org/abs/1707.02286

Tensorflow implementation of Semi-supervised Sequence Learning (https://arxiv.org/abs/1511.01432)

PyTorch implementation of Asymmetric Siamese (https://arxiv.org/abs/2204.00613)

This repository contains the code used for Predicting Patient Outcomes with Graph Representation Learning (https://arxiv.org/abs/2101.03940).