a morph transfer UGATIT for image translation.

Last update: Nov 14, 2022

Related tags

Deep Learning Morph-UGATIT

Overview

Morph-UGATIT

a morph transfer UGATIT for image translation.

Introduction

中文技术文档

This is Pytorch implementation of UGATIT, paper "U-GAT-IT: Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translation".

Additionally, I DIY the model by adding two modules, a MLP module to learn a latent zone and an identity preserving loss. These two factors make UGATIT to achieve a progressive domain transfer for image translation. I call this method Morph UGATIT.

My work has two aspects:

Firstly, according to official TensorFlow code of UGATIT, I use PyTorch to reimplement it, very close to original TF model including network, training hyper parameters.
I add a MLP module, introducing a latent code for generator. And an identity preserving loss is used to learn more common feature for different domains.

I train model on two datasets, "adult2child" and "selfie2anime".

Requirements

python3.7
Pytorch >= 1.6
dlib. Before installing dlib, you should install Cmake and Boost

pip install Cmake
pip install Boost
pip install dlib

other common-used libraries.

How to Use

There are many models in my repo, but you just need two models and corresponding python script files.

UGATIT: "configs/cfgs_ugatit.py", "models/ugatit.py", "tool/train_ugatit.py", "tool/demo_ugatit.py"
Morph UGATIT: "configs/cfgs_s_ugatit_plus.py", "models/s_ugatit_plus.py", "tool/train_s_ugatit_plus.py", "tool/demo_morph_ugatit.py"

train step

getting dataset. The "adult2child" dataset comes from G-Lab, which is generated by StyleGAN. You can download here

The "selfie2anime" dataset comes from official UGATIT repo.

set configurations. configuration files can be found "configs" dir. You just focus on "cfgs_ugatit.py" and "cfgs_s_ugatit_plus.py". Please change:

dirA: domain A dataset path.
dirB: domain B dataset path.
anime: whether dataset is "selfie2anime".
tensorboard: tensorboard log path.
saved_dir: save model weight into "saved_dir".

start to train.

cd tool
python train_ugatit.py   # ugatit
python train_s_ugatit_plus.py   #  morph ugatit

you can also use tensorboard to check loss curves and some visualizations.

evaluation step

Since dlib is necessary, you should download dlib model weight here. change "alignment_loc" at "tool/demo_xxxx.py". "xxx" means "ugatit" or "morph_ugatit" to your dlib model weight path. Then put a test image into a dir.

cd tool
python demo_ugatit.py --type ugatit --resume ${ckpt path}$ --input ${image dir}$ --saved-dir ${result location}$ --align
python demo_morph_ugatit.py --resume ${ckpt path}$ --input ${image dir}$ --saved-dir ${result location}$ --align

Note: if you want to try "selfie2anime", please add a extra term "--anime".

Here I provide my pretrained model weights.

for "adult2child" dataset

ugatit

morph ugatit

for "selfie2anime" dataset

ugatit

More results can be seen here

References

official UGATIT repo
official CycleGAN repo
GLab, http://www.seeprettyface.com/
paper "Lifespan age transformation synthesis" and its' official code.

Comments

Adding resume for source code

Dear sir!

I wrote a script to resume training phase for your repo but I do not know is it true or not (I am just new to Pytorch). I need your feedback and advice, thank you so much.

opened by duongquangvinh 9
About pre-model provided by the author's "selfie2anime" of config-A.

Thanks for the author's work, I tried the pre-model provided by the author's "selfie2anime" of config-A. The test set test and the author's experimental results of config-A are quite different. Did the author use the model provided by himself for testing?

opened by XGuider 7
Resume problem

Dear sir,

Based on your resume for ugatit, I wrote the resume for Morph-UGATIT similarly.

But then I got this error:

I also modified other things similar to what you did. The error is from this load_state_dict. Please help me sir. Thank you.

opened by duongquangvinh 3
CUDA memory for training Morph_UGATIT

Dear sir,

Could you tell me how many GPU memory used for default training setting in your configs? I only have 12 GB for GPU so which one should I reduce in the file cfgs_s_ugatit_plus.py? Thank you, sir.

opened by duongquangvinh 2
About resume from a checkpoint

Dear sir,

Could I resume to train from a checkpoint? I do not see this option in source code. I also tried to rewrite the load_state_dict but I cannot know where I should put those loads into training method?

opened by duongquangvinh 1
CUDA out of memory

Hello ,About the question of CUDA out of memory. My computer is 10018MIB and I changed the batch_size = 1,worker=1,etc but It still break,how should I do?

opened by 2289175662 2
About using average weights

Thanks for your good work!

In the s_ugatit_plus.py, you added a using_avg_w_forward function but didn't use it. I'm curious if you have any ablation studies or insight about this? Will generating w via averaging z from multi-layer MLP leads to a better or more stable result? Thanks!

opened by xyimaging 1
Is it "light" version? I've noticied it uses less memory than original implementation, thus wondering.

In the original repository https://github.com/taki0112/UGATIT there is remark at almost bottom of page: "If the memory of gpu is not sufficient, set --light to True But it may not perform well paper version is --light to False " So I am wondering which version is implemented in there, light or not? It uses as much memory as light version; I hope it is not "light".

opened by Kitty-sunray 0

Owner

GitHub

This repository contains several image-to-image translation models, whcih were tested for RGB to NIR image generation. The models are Pix2Pix, Pix2PixHD, CycleGAN and PointWise.

RGB2NIR_Experimental This repository contains several image-to-image translation models, whcih were tested for RGB to NIR image generation. The models

5 Jan 4, 2023

code for our paper "Source Data-absent Unsupervised Domain Adaptation through Hypothesis Transfer and Labeling Transfer"

SHOT++ Code for our TPAMI submission "Source Data-absent Unsupervised Domain Adaptation through Hypothesis Transfer and Labeling Transfer" that is ext

75 Dec 16, 2022

Transfer-Learn is an open-source and well-documented library for Transfer Learning.

Transfer-Learn is an open-source and well-documented library for Transfer Learning. It is based on pure PyTorch with high performance and friendly API. Our code is pythonic, and the design is consistent with torchvision. You can easily develop new algorithms, or readily apply existing algorithms.

2.2k Jan 3, 2023

Image-to-Image Translation with Conditional Adversarial Networks (Pix2pix) implementation in keras

pix2pix-keras Pix2pix implementation in keras. Original paper: Image-to-Image Translation with Conditional Adversarial Networks (pix2pix) Paper Author

141 Dec 30, 2022

Official pytorch implementation of paper "Image-to-image Translation via Hierarchical Style Disentanglement".

HiSD: Image-to-image Translation via Hierarchical Style Disentanglement Official pytorch implementation of paper "Image-to-image Translation

364 Dec 14, 2022

Code for Dual Contrastive Learning for Unsupervised Image-to-Image Translation, NTIRE, CVPRW 2021.

arXiv Dual Contrastive Learning Adversarial Generative Networks (DCLGAN) We provide our PyTorch implementation of DCLGAN, which is a simple yet powerf

119 Dec 4, 2022

This is the PyTorch implementation of GANs N’ Roses: Stable, Controllable, Diverse Image to Image Translation

Official PyTorch repo for GAN's N' Roses. Diverse im2im and vid2vid selfie to anime translation.

1.1k Jan 1, 2023

Unsupervised Image-to-Image Translation

UNIT: UNsupervised Image-to-image Translation Networks Imaginaire Repository We have a reimplementation of the UNIT method that is more performant. It

1.9k Dec 26, 2022

PyTorch implementation of "Image-to-Image Translation Using Conditional Adversarial Networks".

pix2pix-pytorch PyTorch implementation of Image-to-Image Translation Using Conditional Adversarial Networks. Based on pix2pix by Phillip Isola et al.

383 Dec 17, 2022

Image-to-Image Translation in PyTorch

CycleGAN and pix2pix in PyTorch New: Please check out contrastive-unpaired-translation (CUT), our new unpaired image-to-image translation model that e

19k Jan 7, 2023

Breaking the Dilemma of Medical Image-to-image Translation

Breaking the Dilemma of Medical Image-to-image Translation Supervised Pix2Pix and unsupervised Cycle-consistency are two modes that dominate the field

86 Dec 21, 2022

Maximum Spatial Perturbation for Image-to-Image Translation (Official Implementation)

MSPC for I2I This repository is by Yanwu Xu and contains the PyTorch source code to reproduce the experiments in our CVPR2022 paper Maximum Spatial Pe

51 Dec 14, 2022

Official PyTorch implementation of "ArtFlow: Unbiased Image Style Transfer via Reversible Neural Flows"

ArtFlow Official PyTorch implementation of the paper: ArtFlow: Unbiased Image Style Transfer via Reversible Neural Flows Jie An*, Siyu Huang*, Yibing

123 Dec 27, 2022

Two-Stage Peer-Regularized Feature Recombination for Arbitrary Image Style Transfer

Two-Stage Peer-Regularized Feature Recombination for Arbitrary Image Style Transfer Paper on arXiv Public PyTorch implementation of two-stage peer-reg

38 Oct 14, 2022

Pytorch implementation of CVPR2021 paper "MUST-GAN: Multi-level Statistics Transfer for Self-driven Person Image Generation"

MUST-GAN Code | paper The Pytorch implementation of our CVPR2021 paper "MUST-GAN: Multi-level Statistics Transfer for Self-driven Person Image Generat

46 Dec 26, 2022

(ICCV 2021) Official code of "Dressing in Order: Recurrent Person Image Generation for Pose Transfer, Virtual Try-on and Outfit Editing."

Dressing in Order (DiOr) ?? [Paper] ?? [Webpage] ?? [Running this code] The official implementation of "Dressing in Order: Recurrent Person Image Gene

277 Dec 28, 2022

This script runs neural style transfer against the provided content image.

Neural Style Transfer Content Style Output Description: This script runs neural style transfer against the provided content image. The content image m

0 Nov 25, 2021

U-2-Net: U Square Net - Modified for paired image training of style transfer

U2-Net: U Square Net Modified for paired image training of style transfer This is an unofficial repo making use of the code which was made available b

43 Oct 3, 2022

A series of convenience functions to make basic image processing operations such as translation, rotation, resizing, skeletonization, and displaying Matplotlib images easier with OpenCV and Python.

imutils A series of convenience functions to make basic image processing functions such as translation, rotation, resizing, skeletonization, and displ

4.3k Jan 8, 2023