Code for Dual Contrastive Learning for Unsupervised Image-to-Image Translation, NTIRE, CVPRW 2021.

Overview

arXiv

Dual Contrastive Learning Adversarial Generative Networks (DCLGAN)

We provide our PyTorch implementation of DCLGAN, which is a simple yet powerful model for unsupervised Image-to-image translation. Compared to CycleGAN, DCLGAN performs geometry changes with more realistic results. Compared to CUT, DCLGAN is usually more robust and achieves better performance. A viriant, SimDCL (Similarity DCLGAN) also avoids mode collapse using a new similarity loss.

DCLGAN is a general model performing all kinds of Image-to-Image translation tasks. It achieves SOTA performances in most tasks that we have tested.

Dual Contrastive Learning for Unsupervised Image-to-Image Translation
Junlin Han, Mehrdad Shoeiby, Lars Petersson, Mohammad Ali Armin
DATA61-CSIRO and Australian National University
In NTIRE, CVPRW 2021.

Our pipeline is quite straightforward. The main idea is a dual setting with two encoders to capture the variability in two distinctive domains.

Example Results

Unpaired Image-to-Image Translation

Qualitative results:

Quantitative results:

More visual results:

Prerequisites

Python 3.6 or above.

For packages, see requirements.txt.

Getting started

  • Clone this repo:
git clone https://github.com/JunlinHan/DCLGAN.git
  • Install PyTorch 1.4 or above and other dependencies (e.g., torchvision, visdom, dominate, gputil).

    For pip users, please type the command pip install -r requirements.txt.

    For Conda users, you can create a new Conda environment using conda env create -f environment.yml.

DCLGAN and SimDCL Training and Test

  • Download the grumpifycat dataset
bash ./datasets/download_cut_dataset.sh grumpifycat

The dataset is downloaded and unzipped at ./datasets/grumpifycat/.

  • To view training results and loss plots, run python -m visdom.server and click the URL http://localhost:8097.

Train the DCL model:

python train.py --dataroot ./datasets/grumpifycat --name grumpycat_DCL 

Or train the SimDCL model:

python train.py --dataroot ./datasets/grumpifycat --name grumpycat_SimDCL --model simdcl

We also support CUT:

python train.py --dataroot ./datasets/grumpifycat --name grumpycat_cut --model cut

and fastCUT:

python train.py --dataroot ./datasets/grumpifycat --name grumpycat_fastcut --model fastcut

and CycleGAN:

python train.py --dataroot ./datasets/grumpifycat --name grumpycat_cyclegan --model cycle_gan

The checkpoints will be stored at ./checkpoints/grumpycat_DCL/.

  • Test the DCL model:
python test.py --dataroot ./datasets/grumpifycat --name grumpycat_DCL

The test results will be saved to an html file here: ./results/grumpycat_DCL/latest_test/.

DCLGAN, SimDCL, CUT and CycleGAN

DCLGAN is a more robust unsupervised image-to-image translation model compared to previous models. Our performance is usually better than CUT & CycleGAN.

SIMDCL is a different version, it was designed to solve mode collpase. We recommend using it for small-scale, unbalanced dataset.

Datasets

Download CUT/CycleGAN/pix2pix datasets and learn how to create your own datasets.

Or download it here: https://people.eecs.berkeley.edu/~taesung_park/CycleGAN/datasets/.

Apply a pre-trained DCL model and evaluate

We provide our pre-trained DCLGAN models for:

Cat <-> Dog : https://drive.google.com/file/d/1-0SICLeoySDG0q2k1yeJEI2QJvEL-DRG/view?usp=sharing

Horse <-> Zebra: https://drive.google.com/file/d/16oPsXaP3RgGargJS0JO1K-vWBz42n5lf/view?usp=sharing

CityScapes: https://drive.google.com/file/d/1ZiLAhYG647ipaVXyZdBCsGeiHgBmME6X/view?usp=sharing

Download the pre-tained model, unzip it and put it inside ./checkpoints (You may need to create checkpoints folder by yourself if you didn't run the training code).

Example usage: Download the dataset of Horse2Zebra and test the model using:

python test.py --dataroot ./datasets/horse2zebra --name horse2zebra_dcl

For FID score, use pytorch-fid.

Test the FID for Horse-> Zebra:

python -m pytorch_fid ./results/horse2zebra_dcl/test_latest/images/fake_B ./results/horse2zebra_dcl/test_latest/images/real_B

and Zorse-> Hebra:

python -m pytorch_fid ./results/horse2zebra_dcl/test_latest/images/fake_A ./results/horse2zebra_dcl/test_latest/images/real_A

Citation

If you use our code or our results, please consider citing our paper. Thanks in advance!

@inproceedings{han2021dcl,
  title={Dual Contrastive Learning for Unsupervised Image-to-Image Translation},
  author={Junlin Han and Mehrdad Shoeiby and Lars Petersson and Mohammad Ali Armin},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops},
  year={2021}
}

If you use something included in CUT, you may also CUT.

@inproceedings{park2020cut,
  title={Contrastive Learning for Unpaired Image-to-Image Translation},
  author={Taesung Park and Alexei A. Efros and Richard Zhang and Jun-Yan Zhu},
  booktitle={European Conference on Computer Vision},
  year={2020}
}

Contact

[email protected] or [email protected]

Acknowledgments

Our code is developed based on pytorch-CycleGAN-and-pix2pix and CUT. We thank the awesome work provided by CycleGAN and CUT. We thank pytorch-fid for FID computation. Great thanks to the anonymous reviewers, from both the main CVPR conference and NTIRE. They provided invaluable feedbacks and suggestions.

Comments
  • About the Cityscapes FID

    About the Cityscapes FID

    Hi, thanks for your pretrained models!

    I downloaded it and used the model to generate city images.

    But I cannot get the result 49 as reported in your paper.

    I'm wondering on which testing set are you performing evaluation?

    Here are my testing results:

    <testA, fakeA>=55.49 (using testA) <realA, fakeA> = 55.47 (png format) <valA, valA->fakeA> = 140. (using the compressed validation set by CUT)

    Thanks.

    opened by veroveroxie 9
  • Script freezes

    Script freezes

    Thanks for making this project open access and sharing the code in a nice format. I sincerely appreciate it!

    I am running the code on an HPC server. Here I face the problem that the script freezes after 4-20 epochs. I do not receive any error messages and the HPC server does not terminate my script (which it usually does if for example if there is not enough RAM allocated). The job is still active on the server but I can see from the log file that no more epochs are done. It has frozen. I have also tried to follow the RAM usage through training and I have not found any memory leak during training.

    I am not sure if this problem is in the code or it is related to the HPC server. Have you encountered similar problems?

    opened by AndySAnker 3
  • Super Resolution Implementation

    Super Resolution Implementation

    Hi, thanks for your great work. I just check out your paper and it seem quite well on domain translation tasks and wanna try this work on super resolution task. I tried to set one generator model up sampling size to 0 and another one's up sampling size 2. But it gives me incorrect channel size error. Can you give me any advice about how can i implement this?

    opened by tprdk 2
  • GAN Loss

    GAN Loss

    Hello! I want to know why use hinge as gan loss ? minvalue = torch.min(prediction - 1, torch.zeros(prediction.shape).to(prediction.device)) The network output of discriminator D is not limited to 0-1, so I want to know the reason for using this loss function. If a pixel value in the prediction value is 1.03 after D, then the output of minvalue at this time is still 0, and the prediction result at this time is not correct, right? Maybe there are some questions in my understanding. I sincerely look forward to your reply!

    opened by dingyan1478 2
  • inference

    inference

    I don't understand why inference (test of the model) need testA and testB?

    Why can't we just do inference with the model using a single folder of image that we want to translate to B?

    Thanks for your help.

    Bests

    opened by remybonnav 3
  • colab not save the checkpoint directory pth files

    colab not save the checkpoint directory pth files

    When I use the "--model test" option to run a trained model on new input I get an error:

    No such file or directory: './checkpoints/apple2orange_DCL/latest_net_G_A.pth'

    The checkpoint directory has only apple2orange_DCL folder

    thanks

    opened by NONI75 1
  • How to resume a training

    How to resume a training

    Hello Junlin,

    Thank you very much for your code!

    I was just wondering, what is the command to resume training from a checkpoint?

    I tried this one, but it didn't work out: python3 train.py --dataroot ./datasets/NIR2RGB --name NIR2RGB_DCL --pretrained_name NIR2RGB_DCL

    Thank you.

    opened by VictorLEDEZ 2
  •  mode collapse

    mode collapse

    I have some problems with my training facdes dataset and maps dataset 1 1

    The translation result on facdes is always the same, and there will be no change 1_A 1_A

    The translation results on the maps dataset are shown in the figure below.

    opened by hd201708010401 4
Owner
Computer vision.
null
This is the official implementation code repository of Underwater Light Field Retention : Neural Rendering for Underwater Imaging (Accepted by CVPR Workshop2022 NTIRE)

Underwater Light Field Retention : Neural Rendering for Underwater Imaging (UWNR) (Accepted by CVPR Workshop2022 NTIRE) Authors: Tian Ye†, Sixiang Che

jmucsx 17 Dec 14, 2022
The 7th edition of NTIRE: New Trends in Image Restoration and Enhancement workshop will be held on June 2022 in conjunction with CVPR 2022.

NTIRE 2022 - Image Inpainting Challenge Important dates 2022.02.01: Release of train data (input and output images) and validation data (only input) 2

Andrés Romero 37 Nov 27, 2022
pytorch implementation of "Contrastive Multiview Coding", "Momentum Contrast for Unsupervised Visual Representation Learning", and "Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination"

Unofficial implementation: MoCo: Momentum Contrast for Unsupervised Visual Representation Learning (Paper) InsDis: Unsupervised Feature Learning via N

Zhiqiang Shen 16 Nov 4, 2020
[CVPRW 2021] Code for Region-Adaptive Deformable Network for Image Quality Assessment

RADN [CVPRW 2021] Code for Region-Adaptive Deformable Network for Image Quality Assessment [Paper on arXiv] Overview Update [2021/5/7] add codes for W

IIGROUP 53 Dec 28, 2022
Lowest memory consumption and second shortest runtime in NTIRE 2022 challenge on Efficient Super-Resolution

FMEN Lowest memory consumption and second shortest runtime in NTIRE 2022 on Efficient Super-Resolution. Our paper: Fast and Memory-Efficient Network T

null 33 Dec 1, 2022
Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set (CVPRW 2019). A PyTorch implementation.

Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set —— PyTorch implementation This is an unofficial offici

Sicheng Xu 833 Dec 28, 2022
Official PyTorch implementation of "IntegralAction: Pose-driven Feature Integration for Robust Human Action Recognition in Videos", CVPRW 2021

IntegralAction: Pose-driven Feature Integration for Robust Human Action Recognition in Videos Introduction This repo is official PyTorch implementatio

Gyeongsik Moon 29 Sep 24, 2022
CVPRW 2021: How to calibrate your event camera

E2Calib: How to Calibrate Your Event Camera This repository contains code that implements video reconstruction from event data for calibration as desc

Robotics and Perception Group 104 Nov 16, 2022
This repo is official PyTorch implementation of MobileHumanPose: Toward real-time 3D human pose estimation in mobile devices(CVPRW 2021).

Github Code of "MobileHumanPose: Toward real-time 3D human pose estimation in mobile devices" Introduction This repo is official PyTorch implementatio

Choi Sang Bum 203 Jan 5, 2023
Code for the paper "Unsupervised Contrastive Learning of Sound Event Representations", ICASSP 2021.

Unsupervised Contrastive Learning of Sound Event Representations This repository contains the code for the following paper. If you use this code or pa

Eduardo Fonseca 81 Dec 22, 2022
PyTorch version of the paper 'Enhanced Deep Residual Networks for Single Image Super-Resolution' (CVPRW 2017)

About PyTorch 1.2.0 Now the master branch supports PyTorch 1.2.0 by default. Due to the serious version problem (especially torch.utils.data.dataloade

Sanghyun Son 2.1k Jan 1, 2023
[CVPRW 2022] Attentions Help CNNs See Better: Attention-based Hybrid Image Quality Assessment Network

Attention Helps CNN See Better: Hybrid Image Quality Assessment Network [CVPRW 2022] Code for Hybrid Image Quality Assessment Network [paper] [code] T

IIGROUP 49 Dec 11, 2022
PyTorch implementation of the Deep SLDA method from our CVPRW-2020 paper "Lifelong Machine Learning with Deep Streaming Linear Discriminant Analysis"

Lifelong Machine Learning with Deep Streaming Linear Discriminant Analysis This is a PyTorch implementation of the Deep Streaming Linear Discriminant

Tyler Hayes 41 Dec 25, 2022
Revisiting Contrastive Methods for Unsupervised Learning of Visual Representations. [2021]

Revisiting Contrastive Methods for Unsupervised Learning of Visual Representations This repo contains the Pytorch implementation of our paper: Revisit

Wouter Van Gansbeke 80 Nov 20, 2022
Official implementation of "DSP: Dual Soft-Paste for Unsupervised Domain Adaptive Semantic Segmentation"

DSP Official implementation of "DSP: Dual Soft-Paste for Unsupervised Domain Adaptive Semantic Segmentation". Accepted by ACM Multimedia 2021. Authors

null 20 Oct 24, 2022
Unsupervised Image-to-Image Translation

UNIT: UNsupervised Image-to-image Translation Networks Imaginaire Repository We have a reimplementation of the UNIT method that is more performant. It

Ming-Yu Liu 劉洺堉 1.9k Dec 26, 2022
[CVPRW 21] "BNN - BN = ? Training Binary Neural Networks without Batch Normalization", Tianlong Chen, Zhenyu Zhang, Xu Ouyang, Zechun Liu, Zhiqiang Shen, Zhangyang Wang

BNN - BN = ? Training Binary Neural Networks without Batch Normalization Codes for this paper BNN - BN = ? Training Binary Neural Networks without Bat

VITA 40 Dec 30, 2022
CRLT: A Unified Contrastive Learning Toolkit for Unsupervised Text Representation Learning

CRLT: A Unified Contrastive Learning Toolkit for Unsupervised Text Representation Learning This repository contains the code and relevant instructions

XiaoMing 5 Aug 19, 2022
Contrastive Learning for Many-to-many Multilingual Neural Machine Translation(mCOLT/mRASP2), ACL2021

Contrastive Learning for Many-to-many Multilingual Neural Machine Translation(mCOLT/mRASP2), ACL2021 The code for training mCOLT/mRASP2, a multilingua

null 104 Jan 1, 2023