Official implementation of the paper Image Generators with Conditionally-Independent Pixel Synthesis https://arxiv.org/abs/2011.13775

Related tags

Deep Learning deep-learning generative-adversarial-network gan implicit-functions foveated-rendering
Overview

CIPS -- Official Pytorch Implementation

of the paper Image Generators with Conditionally-Independent Pixel Synthesis

PWC

Teaser

Requirements

pip install -r requirements.txt

Usage

First create lmdb datasets:

python prepare_data.py images --out LMDB_PATH --n_worker N_WORKER --size SIZE1,SIZE2,SIZE3,... DATASET_PATH

This will convert images to jpeg and pre-resizes it.

To train on FFHQ-256 or churches please run:

python3 -m torch.distributed.launch --nproc_per_node=8 --master_port=1234 train.py --n_sample=8 --batch=4 --fid_batch=8 --Generator=CIPSskip --output_dir=skip-[ffhq/churches] --img2dis --num_workers=16 DATASET_PATH

To train on patches add --crop=PATCH_SIZE. PATCH_SIZE has to be a power of 2.

Pretrained Checkpoints

churches

ffhq256

ffhq1024

landscapes

Generate samples

To play with the models please download checkpoints and check out a notebook.ipynb

Progressive training

We also tried to train progressively on FFHQ starting from 256×256 initialization and got FID 10.07. We will update the paper with the training details soon. Checkpoint name is ffhq1024.pt. Samples are below.

Sample from FFHQ trained progressively

Citation

If you found our work useful, please don't forget to cite

@article{anokhin2020image,
  title={Image Generators with Conditionally-Independent Pixel Synthesis},
  author={Anokhin, Ivan and Demochkin, Kirill and Khakhulin, Taras and Sterkin, Gleb and Lempitsky, Victor and Korzhenkov, Denis},
  journal={arXiv preprint arXiv:2011.13775},
  year={2020}
}

The code is heavely based on the styleganv2 pytorch implementation

Nvidia-licensed CUDA kernels (fused_bias_act_kernel.cu, upfirdn2d_kernel.cu) is for non-commercial use only.

Comments
  • Quick question about training time

    Quick question about training time

    Hello authors,

    Thank you for your paper -- it looks very interesting! I had a quick question about training time: how long (and with how many GPUs) does it take you to train the pretrained models provided in the repo? Do you find that the method is slow or fast relative to StyleGAN2?

    I look forward to playing around with the repo over the next week!

    Best, greeneggsandyaml

    opened by greeneggsandyaml 2
  • About the blur kernel that affect pixel indenpence

    About the blur kernel that affect pixel indenpence

    I see that in the code for CIPS block, you use a Blur layer, which use 3x3 blur kernel so I think one pixel is affect by 8 surrouding pixel, is it correct ?

    opened by thuanz123 1
  • Timed out when calculating FID

    Timed out when calculating FID

    Hi. First of all, thanks for your great work.

    I have a problem when calculating FID during training. Every args.save_checkpoint_frequency iterations, the model is evaluated by calculating FID score. However, at this phase, I have a timed out problem. Here is the error log.

    [E ProcessGroupNCCL.cpp:566] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1802801 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:325] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. To avoid this inconsistency, we are taking the entire process down.
terminate called after throwing an instance of 'std::runtime_error'
  what():  [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1802801 milliseconds before timing out.
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -6) local_rank: 1 (pid: 4414) of binary: /opt/conda/bin/python
ERROR:torch.distributed.elastic.agent.server.local_elastic_agent:[default] Worker group failed

    Before this log came out, there was no change in the prompt but GPUs were still working. As I'm not that familiar with distributed training, I want to ask how to fix this problem. I've tried to lower the number of args.fid_samples, but this doesn't help. Thank you.

    Best Wishes, Lee

    opened by shlee625 1
  • train model

    train model

    Why do you have an generator and g_ema in your training code？And Your optimizer optimizes generator directly，why not optimizes g_ema? And why accumulate(g_ema, g_module, accum) ，not And why accumulate(g_ema, generator, accum) ？

    opened by dengshuhai-cmd 0
  • Checkpoint for Ablation Study?

    Checkpoint for Ablation Study?

    Hi,

    Really nice work! I wonder if you have checkpoints for models in the ablation study? For example, models trained without the coordinate embedding? THanks!

    opened by stevenygd 0
  • I tried to train with your code,but it didn't work.The main progress seems to stay waiting for something to load.

    I tried to train with your code,but it didn't work.The main progress seems to stay waiting for something to load.

    /home/hello/miniconda3/envs/deepsort/bin/python3 /home/hello/桌面/CIPS-main/train.py LMDB_data1024*1024_anime --nproc_per_node=8 --master_port=1234 --n_sample=8 --batch=4 --fid_batch=8 --Generator=CIPSskip --output_dir=skip-[ffhq/churches] --img2dis --num_workers=16 Traceback (most recent call last): File "/home/hello/桌面/CIPS-main/train.py", line 16, in import model File "/home/hello/桌面/CIPS-main/model/init.py", line 1, in from .Discriminators import * File "/home/hello/桌面/CIPS-main/model/Discriminators.py", line 8, in from .blocks import ConvLayer, ResBlock, EqualLinear File "/home/hello/桌面/CIPS-main/model/blocks.py", line 8, in from op import FusedLeakyReLU, fused_leaky_relu, upfirdn2d File "/home/hello/桌面/CIPS-main/op/init.py", line 2, in from .upfirdn2d import upfirdn2d File "/home/hello/桌面/CIPS-main/op/upfirdn2d.py", line 13, in os.path.join(module_path, 'upfirdn2d_kernel.cu'), File "/home/hello/miniconda3/envs/deepsort/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1092, in load keep_intermediates=keep_intermediates) File "/home/hello/miniconda3/envs/deepsort/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1307, in _jit_compile baton.wait() File "/home/hello/miniconda3/envs/deepsort/lib/python3.7/site-packages/torch/utils/file_baton.py", line 42, in wait time.sleep(self.wait_seconds) KeyboardInterrupt

    Process finished with exit code 1

    opened by xiaoxiaoxiaogu 2
  • Would you please share discriminator's weight also?

    Would you please share discriminator's weight also?

    Dear authors:

    Hello, thank you for sharing the great project.

    I would like to use your project for the baseline of my future research.

    Would you please share your discriminator's weight also?

    Thank you.

    Best, chkimmmmm

    opened by chkimmmmm 1
  • Would you share progressive training code?

    Would you share progressive training code?

    Dear authors:

    Hello.

    First of all, thank you for the sharing this great work.

    In the Read.me, you mentioned that you are going to share the progressive training details.

    When will you share it?

    Thank you.

    Best. chkimmmmm

    opened by chkimmmmm 0
Owner
Multimodal Lab @ Samsung AI Center Moscow
Multimodal Lab @ Samsung AI Center Moscow
GitHub
[PyTorch] Official implementation of CVPR2021 paper "PointDSC: Robust Point Cloud Registration using Deep Spatial Consistency". https://arxiv.org/abs/2103.05465

PointDSC repository PyTorch implementation of PointDSC for CVPR'2021 paper "PointDSC: Robust Point Cloud Registration using Deep Spatial Consistency",

null 153 Dec 14, 2022
This is an official implementation of our CVPR 2021 paper "Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression" (https://arxiv.org/abs/2104.02300)

Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression Introduction In this paper, we are interested in the bottom-up paradigm of estima

HRNet 367 Dec 27, 2022
Official repository with code and data accompanying the NAACL 2021 paper "Hurdles to Progress in Long-form Question Answering" (https://arxiv.org/abs/2103.06332).

Hurdles to Progress in Long-form Question Answering This repository contains the official scripts and datasets accompanying our NAACL 2021 paper, "Hur

Kalpesh Krishna 41 Nov 8, 2022
Source code for models described in the paper "AudioCLIP: Extending CLIP to Image, Text and Audio" (https://arxiv.org/abs/2106.13043)

AudioCLIP Extending CLIP to Image, Text and Audio This repository contains implementation of the models described in the paper arXiv:2106.13043. This

null 458 Jan 2, 2023
Official Implementation for "ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement" https://arxiv.org/abs/2104.02699

ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement Recently, the power of unconditional image synthesis has significantly advanced th

null 967 Jan 4, 2023
Non-Official Pytorch implementation of "Face Identity Disentanglement via Latent Space Mapping" https://arxiv.org/abs/2005.07728 Using StyleGAN2 instead of StyleGAN

Face Identity Disentanglement via Latent Space Mapping - Implement in pytorch with StyleGAN 2 Description Pytorch implementation of the paper Face Ide

Daniel Roich 58 Dec 24, 2022
Unofficial Tensorflow-Keras implementation of Fastformer based on paper [Fastformer: Additive Attention Can Be All You Need](https://arxiv.org/abs/2108.09084).

Fastformer-Keras Unofficial Tensorflow-Keras implementation of Fastformer based on paper Fastformer: Additive Attention Can Be All You Need. Tensorflo

Yam Peleg 10 Jan 30, 2022
Supplementary code for the paper "Meta-Solver for Neural Ordinary Differential Equations" https://arxiv.org/abs/2103.08561

Meta-Solver for Neural Ordinary Differential Equations Towards robust neural ODEs using parametrized solvers. Main idea Each Runge-Kutta (RK) solver w

Julia Gusak 25 Aug 12, 2021
Code for paper "A Critical Assessment of State-of-the-Art in Entity Alignment" (https://arxiv.org/abs/2010.16314)

A Critical Assessment of State-of-the-Art in Entity Alignment This repository contains the source code for the paper A Critical Assessment of State-of

Max Berrendorf 16 Oct 14, 2022
Code for the paper: Learning Adversarially Robust Representations via Worst-Case Mutual Information Maximization (https://arxiv.org/abs/2002.11798)

Representation Robustness Evaluations Our implementation is based on code from MadryLab's robustness package and Devon Hjelm's Deep InfoMax. For all t

Sicheng 19 Dec 7, 2022
Pytorch implementation of Each Part Matters: Local Patterns Facilitate Cross-view Geo-localization https://arxiv.org/abs/2008.11646

[TCSVT] Each Part Matters: Local Patterns Facilitate Cross-view Geo-localization LPN [Paper] NEWs Prerequisites Python 3.6 GPU Memory >= 8G Numpy > 1.

null 46 Dec 14, 2022
Minimal implementation of PAWS (https://arxiv.org/abs/2104.13963) in TensorFlow.

PAWS-TF ?? Implementation of Semi-Supervised Learning of Visual Features by Non-Parametrically Predicting View Assignments with Support Samples (PAWS)

Sayak Paul 43 Jan 8, 2023
A PyTorch implementation of EventProp [https://arxiv.org/abs/2009.08378], a method to train Spiking Neural Networks

Spiking Neural Network training with EventProp This is an unofficial PyTorch implemenation of EventProp, a method to compute exact gradients for Spiki

Pedro Savarese 35 Jul 29, 2022
Unofficial implementation of "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" (https://arxiv.org/abs/2103.14030)

Swin-Transformer-Tensorflow A direct translation of the official PyTorch implementation of "Swin Transformer: Hierarchical Vision Transformer using Sh

null 52 Dec 29, 2022
Unofficial implementation of Alias-Free Generative Adversarial Networks. (https://arxiv.org/abs/2106.12423) in PyTorch

alias-free-gan-pytorch Unofficial implementation of Alias-Free Generative Adversarial Networks. (https://arxiv.org/abs/2106.12423) This implementation

Kim Seonghyeon 502 Jan 3, 2023
Pytorch implementation of Distributed Proximal Policy Optimization: https://arxiv.org/abs/1707.02286

Pytorch-DPPO Pytorch implementation of Distributed Proximal Policy Optimization: https://arxiv.org/abs/1707.02286 Using PPO with clip loss (from https

Alexis David Jacq 163 Dec 26, 2022
Tensorflow implementation of Semi-supervised Sequence Learning (https://arxiv.org/abs/1511.01432)

Transfer Learning for Text Classification with Tensorflow Tensorflow implementation of Semi-supervised Sequence Learning(https://arxiv.org/abs/1511.01

DONGJUN LEE 82 Oct 22, 2022
PyTorch implementation of Asymmetric Siamese (https://arxiv.org/abs/2204.00613)

Asym-Siam: On the Importance of Asymmetry for Siamese Representation Learning This is a PyTorch implementation of the Asym-Siam paper, CVPR 2022: @inp

Meta Research 89 Dec 18, 2022
This repository contains the code used for Predicting Patient Outcomes with Graph Representation Learning (https://arxiv.org/abs/2101.03940).

Predicting Patient Outcomes with Graph Representation Learning This repository contains the code used for Predicting Patient Outcomes with Graph Repre

Emma Rocheteau 76 Dec 22, 2022