The repository offers the official implementation of our paper in PyTorch.

Related tags

Deep Learning CIT
Overview

Cloth Interactive Transformer (CIT)

Cloth Interactive Transformer for Virtual Try-On
Bin Ren1, Hao Tang1, Fanyang Meng2, Runwei Ding3, Ling Shao4, Philip H.S. Torr5, Nicu Sebe16.
1University of Trento, Italy, 2Peng Cheng Laboratory, China, 3Peking University Shenzhen Graduate School, China,
4Inception Institute of AI, UAE, 5University of Oxford, UK, 6Huawei Research Ireland, Ireland.

The repository offers the official implementation of our paper in PyTorch. The code and pre-trained models are tested with pytorch 0.4.1, torchvision 0.2.1, opencv-python 4.1, and pillow 5.4 (Python 3.6).

In the meantime, check out our recent paper XingGAN and XingVTON.

Usage

This pipeline is a combination of consecutive training and testing of Cloth Interactive Transformer (CIT) Matching block based GMM and CIT Reasoning block based TOM. GMM generates the warped clothes according to the target human. Then, TOM blends the warped clothes outputs from GMM into the target human properties, to generate the final try-on output.

  1. Install the requirements
  2. Download/Prepare the dataset
  3. Train the CIT Matching block based GMM network
  4. Get warped clothes for training set with trained GMM network, and copy warped clothes & masks inside data/train directory
  5. Train the CIT Reasoning block based TOM network
  6. Test CIT Matching block based GMM for testing set
  7. Get warped clothes for testing set, copy warped clothes & masks inside data/test directory
  8. Test CIT Reasoning block based TOM testing set

Installation

This implementation is built and tested in PyTorch 0.4.1. Pytorch and torchvision are recommended to install with conda: conda install pytorch=0.4.1 torchvision=0.2.1 -c pytorch

For all packages, run pip install -r requirements.txt

Data Preparation

For training/testing VITON dataset, our full and processed dataset is available here: https://1drv.ms/u/s!Ai8t8GAHdzVUiQQYX0azYhqIDPP6?e=4cpFTI. After downloading, unzip to your own data directory ./data/.

Training

Run python train.py with your specific usage options for GMM and TOM stage.

For example, GMM: python train.py --name GMM --stage GMM --workers 4 --save_count 5000 --shuffle. Then run test.py for GMM network with the training dataset, which will generate the warped clothes and masks in "warp-cloth" and "warp-mask" folders inside the "result/GMM/train/" directory. Copy the "warp-cloth" and "warp-mask" folders into your data directory, for example inside "data/train" folder.

Run TOM stage, python train.py --name TOM --stage TOM --workers 4 --save_count 5000 --shuffle

Evaluation

We adopt four evaluation metrics in our work for evaluating the performance of the proposed XingVTON. There are Jaccard score (JS), structral similarity index measure (SSIM), learned perceptual image patch similarity (LPIPS), and Inception score (IS).

Note that JS is used for the same clothing retry-on cases (with ground truth cases) in the first geometric matching stage, while SSIM and LPIPS are used for the same clothing retry-on cases (with ground truth cases) in the second try-on stage. In addition, IS is used for different clothing try-on (where no ground truth is available).

For JS

  • Step1: Runpython test.py --name GMM --stage GMM --workers 4 --datamode test --data_list test_pairs_same.txt --checkpoint checkpoints/GMM_pretrained/gmm_final.pth then the parsed segmentation area for current upper clothing is used as the reference image, accompanied with generated warped clothing mask then:
  • Step2: Runpython metrics/getJS.py

For SSIM

After we run test.py for GMM network with the testibng dataset, the warped clothes and masks will be generated in "warp-cloth" and "warp-mask" folders inside the "result/GMM/test/" directory. Copy the "warp-cloth" and "warp-mask" folders into your data directory, for example inside "data/test" folder. Then:

  • Step1: Run TOM stage test python test.py --name TOM --stage TOM --workers 4 --datamode test --data_list test_pairs_same.txt --checkpoint checkpoints/TOM_pretrained/tom_final.pth Then the original target human image is used as the reference image, accompanied with the generated retry-on image then:
  • Step2: Run python metrics/getSSIM.py

For LPIPS

  • Step1: You need to creat a new virtual enviriment, then install PyTorch 1.0+ and torchvision;
  • Step2: Run sh metrics/PerceptualSimilarity/testLPIPS.sh;

For IS

  • Step1: Run TOM stage test python test.py --name TOM --stage TOM --workers 4 --datamode test --data_list test_pairs.txt --checkpoint checkpoints/TOM_pretrained/tom_final.pth
  • Step2: Run python metrics/getIS.py

Inference

The pre-trained models are provided here. Download the pre-trained models and put them in this project (./checkpoints) Then just run the same step as Evaluation to test/inference our model.

Acknowledgements

This source code is inspired by CP-VTON, CP-VTON+. We are extremely grateful for their public implementation.

Citation

If you use this code for your research, please consider giving a star and citing our paper 🦖 :

CIT

@article{ren2021cloth,
  title={Cloth Interactive Transformer for Virtual Try-On},
  author={Ren, Bin and Tang, Hao and Meng, Fanyang and Ding, Runwei and Shao, Ling and Torr, Philip HS and Sebe, Nicu},
  journal={arXiv preprint arXiv:2104.05519},
  year={2021}
}

Contributions

If you have any questions/comments/bug reports, feel free to open a github issue or pull a request or e-mail to the author Bin Ren ([email protected]).

Comments
  • How much GPU memory is needed for TOM stage training?

    How much GPU memory is needed for TOM stage training?

    python train.py --name TOM --stage TOM --workers 4 --save_count 5000 --shuffle Namespace(batch_size=4, checkpoint='', checkpoint_dir='/home/admin/workspace/project/CIT/checkpoints', data_list='train_pairs.txt', datamode='train', dataroot='/home/admin/workspace/project/CIT/data', decay_step=100000, display_count=100, fine_height=256, fine_width=192, gpu_ids='0', grid_size=5, keep_step=100000, lr=0.0001, name='TOM', radius=5, save_count=5000, shuffle=True, stage='TOM', tensorboard_dir='/home/admin/workspace/project/CIT/tensorboard', workers=4) Start to train stage: TOM, named: TOM!

    ......

    RuntimeError: CUDA error: out of memory terminate called without an active exception Aborted

    opened by sunkaianna 1
  • Got some strange output.

    Got some strange output.

    Hii @Amazingren Thank you for your amazing work! I followed the readme and ran the test. without changing anything, just adding this line of code torch.backends.cudnn.benchmark = True Because I got this error: runtimeerror: cudnn error: cudnn_status_success. The output of stage 2 is so strange . image

    Could you please tell me what is the problem? and how can I solve it?

    opened by TAUIL-Abd-Elilah 0
  • "RuntimeError: CUDA error: out of memory" when train TOM with RTX2080Ti

    Hello, thank you for your amazing work! I tried to train the model, and I finished the GMM step. But when I tried to train the TOM with one GPU(2080Ti), I got a error: "RuntimeError: CUDA error: out of memory " , even if I changed the batch size to 1. And what I want to ask is : what kind of GPU you used to train ? 2022-03-29 14-50-10 的屏幕截图

    opened by kris-yangjs 1
  • Error while testing

    Error while testing

    @Ha0Tang @Amazingren Hello, Thank you for the amazing repository. I was evaluating the network for JS and other metrics. I ran into an error while running the test.py for GMM

    Screenshot 2021-09-30 at 4 01 12 PM

    Can you please help me?

    Thank you

    opened by SumanthJain2998 2
  • Colab Demo?

    Colab Demo?

    Hello, thank you for your great work! can you share a colab notebook demo of this amazing repo, it will help everyone to test it and it will be much appreciated thank you

    opened by Adeel-Intizar 8
Owner
Bingoren
Bingoren
This python-based package offers a way of creating a parametric OpenMC plasma source from plasma parameters.

openmc-plasma-source This python-based package offers a way of creating a parametric OpenMC plasma source from plasma parameters. The OpenMC sources a

Fusion Energy 10 Oct 18, 2022
Virtual Dance Reality Stage: a feature that offers you to share a stage with another user virtually

Portrait Segmentation using Tensorflow This script removes the background from an input image. You can read more about segmentation here Setup The scr

null 291 Dec 24, 2022
The pyrelational package offers a flexible workflow to enable active learning with as little change to the models and datasets as possible

pyrelational is a python active learning library developed by Relation Therapeutics for rapidly implementing active learning pipelines from data management, model development (and Bayesian approximation), to creating novel active learning strategies.

Relation Therapeutics 95 Dec 27, 2022
Official implementation of our paper "LLA: Loss-aware Label Assignment for Dense Pedestrian Detection" in Pytorch.

LLA: Loss-aware Label Assignment for Dense Pedestrian Detection This project provides an implementation for "LLA: Loss-aware Label Assignment for Dens

null 35 Dec 6, 2022
Official implementation of our CVPR2021 paper "OTA: Optimal Transport Assignment for Object Detection" in Pytorch.

OTA: Optimal Transport Assignment for Object Detection This project provides an implementation for our CVPR2021 paper "OTA: Optimal Transport Assignme

null 217 Jan 3, 2023
The official pytorch implementation of our paper "Is Space-Time Attention All You Need for Video Understanding?"

TimeSformer This is an official pytorch implementation of Is Space-Time Attention All You Need for Video Understanding?. In this repository, we provid

Facebook Research 1k Dec 31, 2022
This is the official pytorch implementation for our ICCV 2021 paper "TRAR: Routing the Attention Spans in Transformers for Visual Question Answering" on VQA Task

?? ERASOR (RA-L'21 with ICRA Option) Official page of "ERASOR: Egocentric Ratio of Pseudo Occupancy-based Dynamic Object Removal for Static 3D Point C

Hyungtae Lim 225 Dec 29, 2022
Official PyTorch implemention of our paper "Learning to Rectify for Robust Learning with Noisy Labels".

WarPI The official PyTorch implemention of our paper "Learning to Rectify for Robust Learning with Noisy Labels". Run python main.py --corruption_type

Haoliang Sun 3 Sep 3, 2022
The project is an official implementation of our CVPR2019 paper "Deep High-Resolution Representation Learning for Human Pose Estimation"

Deep High-Resolution Representation Learning for Human Pose Estimation (CVPR 2019) News [2020/07/05] A very nice blog from Towards Data Science introd

Leo Xiao 3.9k Jan 5, 2023
This project is the official implementation of our accepted ICLR 2021 paper BiPointNet: Binary Neural Network for Point Clouds.

BiPointNet: Binary Neural Network for Point Clouds Created by Haotong Qin, Zhongang Cai, Mingyuan Zhang, Yifu Ding, Haiyu Zhao, Shuai Yi, Xianglong Li

Haotong Qin 59 Dec 17, 2022
This is an official implementation of our CVPR 2021 paper "Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression" (https://arxiv.org/abs/2104.02300)

Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression Introduction In this paper, we are interested in the bottom-up paradigm of estima

HRNet 367 Dec 27, 2022
Official implementation of GraphMask as presented in our paper Interpreting Graph Neural Networks for NLP With Differentiable Edge Masking.

GraphMask This repository contains an implementation of GraphMask, the interpretability technique for graph neural networks presented in our ICLR 2021

Michael Schlichtkrull 29 Sep 2, 2022
The official implementation of our CVPR 2021 paper - Hybrid Rotation Averaging: A Fast and Robust Rotation Averaging Approach

Graph Optimizer This repo contains the official implementation of our CVPR 2021 paper - Hybrid Rotation Averaging: A Fast and Robust Rotation Averagin

Chenyu 109 Dec 23, 2022
The project is an official implementation of our paper "3D Human Pose Estimation with Spatial and Temporal Transformers".

3D Human Pose Estimation with Spatial and Temporal Transformers This repo is the official implementation for 3D Human Pose Estimation with Spatial and

Ce Zheng 363 Dec 28, 2022
Official implementation of our paper "Learning to Bootstrap for Combating Label Noise"

Learning to Bootstrap for Combating Label Noise This repo is the official implementation of our paper "Learning to Bootstrap for Combating Label Noise

null 21 Apr 9, 2022
PyTorch implementation of the Deep SLDA method from our CVPRW-2020 paper "Lifelong Machine Learning with Deep Streaming Linear Discriminant Analysis"

Lifelong Machine Learning with Deep Streaming Linear Discriminant Analysis This is a PyTorch implementation of the Deep Streaming Linear Discriminant

Tyler Hayes 41 Dec 25, 2022
Pytorch implementation of our paper under review — Lottery Jackpots Exist in Pre-trained Models

Lottery Jackpots Exist in Pre-trained Models (Paper Link) Requirements Python >= 3.7.4 Pytorch >= 1.6.1 Torchvision >= 0.4.1 Reproduce the Experiment

Yuxin Zhang 27 Jun 28, 2022
PyTorch implementation of our ICCV2021 paper: StructDepth: Leveraging the structural regularities for self-supervised indoor depth estimation

StructDepth PyTorch implementation of our ICCV2021 paper: StructDepth: Leveraging the structural regularities for self-supervised indoor depth estimat

SJTU-ViSYS 112 Nov 28, 2022