Official PyTorch code for the paper: "Point-Based Modeling of Human Clothing" (ICCV 2021)

Related tags

Deep Learning animation point-cloud pytorch clothing virtual-try-on re-targeting neural-rendering virtual-clothing point-based-graphics
Overview

Point-Based Modeling of Human Clothing

Paper | Project page | Video

This is an official PyTorch code repository of the paper "Point-Based Modeling of Human Clothing" (accepted to ICCV, 2021).

Setup

Build docker

  • Prerequisites: your nvidia driver should support cuda 10.2, Windows or Mac are not supported.
  • Clone repo:
    • git clone https://github.com/izakharkin/point_based_clothing.git
    • cd point_based_clothing
    • git submodule init && git submodule update
  • Docker setup:
  • Download 10_nvidia.json and place it in the docker/ folder
  • Create docker image:
    • Build on your own: run 2 commands
  • Inside the docker container: source activate pbc

Download data

  • Download the SMPL neutral model from SMPLify project page:
    • Register, go to the Downloads section, download SMPLIFY_CODE_V2.ZIP, and unpack it;
    • Move smplify_public/code/models/basicModel_neutral_lbs_10_207_0_v1.0.0.pkl to data/smpl_models/SMPL_NEUTRAL.pkl.
  • Download models checkpoints (~570 Mb): Google Drive and place them to the checkpoints/ folder;
  • Download a sample data we provide to check the appearance fitting (~480 Mb): Google Drive, unpack it, and place psp/ folder to the samples/ folder.

Run

We provide scripts for geometry fitting and inference and appearance fitting and inference.

Geometry (outfit code)

Fitting

To fit a style outfit code to a single image one can run:

python fit_outfit_code.py --config_name=outfit_code/psp

The learned outfit codes are saved to out/outfit_code/outfit_codes_<dset_name>.pkl by default. The visualization of the process is in out/outfit_code/vis_<dset_name>/:

  • Coarse fitting stage: four outfit codes initialized randomly and being optimized simultaneosly.

outfit_code_fitting_coarse

  • Fine fitting stage: mean of found outfit codes is being optimized further to possibly imrove the reconstruction.

outfit_code_fitting_fine

Note: visibility_thr hyperparameter in fit_outfit_code.py may affect the quality of result point cloud (e.f. make it more sparse). Feel free to tune it if the result seems not perfect.

vis_thr_360

Inference

outfit_code_inference

To further infer the fitted outfit style on the train or on new subjects please see infer_outfit_code.ipynb. To run jupyter notebook server from the docker, run this inside the container:

jupyter notebook --ip=0.0.0.0 --port=8087 --no-browser

Appearance (neural descriptors)

Fitting

To fit a clothing appearance to a sequence of frames one can run:

python fit_appearance.py --config_name=appearance/psp_male-3-casual

The learned neural descriptors ntex0_<epoch>.pth and neural rendering network weights model0_<epoch>.pth are saved to out/appearance/<dset_name>/<subject_id>/<experiment_dir>/checkpoints/ by default. The visualization of the process is in out/appearance/<dset_name>/<subject_id>/<experiment_dir>/visuals/.

Inference

appearance_inference

To further infer the fitted clothing point cloud and its appearance on the train or on new subjects please see infer_appearance.ipynb. To run jupyter notebook server from the docker, run this inside the container:

jupyter notebook --ip=0.0.0.0 --port=8087 --no-browser

Citation

If you find our work helpful, please do not hesitate to cite us:

@InProceedings{Zakharkin_2021_ICCV,
    author    = {Zakharkin, Ilya and Mazur, Kirill and Grigorev, Artur and Lempitsky, Victor},
    title     = {Point-Based Modeling of Human Clothing},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2021},
    pages     = {14718-14727}
}

Non-commercial use only.

Related projects

We also thank the authors of Cloth3D and PeopleSnapshot datasets.

Comments
  • Custom Data

    Custom Data

    Trying to run inference on custom data. However input seems to require additional segmentation data, that doesn't seem to be specified. What preprocessing steps are required?

    opened by sidk99 3
  • Cloth3D dataset

    Cloth3D dataset

    Hello sir, I am a student who is studying Point Cloud. Recently discovered papers 《Cloth3d: Clothed 3d humans》Dataset download address resource can not be downloaded successfully, can you provide other download channels for this dataset?

    opened by runnerfavourite 3
  • UV Coordinate

    UV Coordinate

    https://github.com/saic-vul/point_based_clothing/blob/3650fdae2ba9a5ced40f3075a4f0fd995442a64e/src/outfit_code/utils.py#L61 Hi, I found that the above code seems to generate transposed UV coordinates, i.e. the UV coordinates of the first row of the image ([height, width, channel] format) grid are [0, 0], [0, 1], [0, 2], ..., [ 0, H-1], which are actually "VU coordinates". And when the width and height of the image are different, it gives an error.

    opened by caiyongqi 0
  • Using rasterize from pytorch3d instead of nvdiffrast in get_smpl_rast ?

    Using rasterize from pytorch3d instead of nvdiffrast in get_smpl_rast ?

    Hi,

    Thank you for sharing the great work! Is it possible to replace the rasterize function of nvdiffrast with pytorch3d's in get_smpl_rast because my server is CentOS which is not supported by nvdiffrast?

    Thanks!

    opened by zhaofang0627 0
  • Point cloud location

    Point cloud location

    Trying to obtain the final point cloud generated during the outfit_coding module. Is there a way to save the 3D point clouds used to generate the output images/videos when running fit outfit code?

    opened by sidk99 0
Owner
Visual Understanding Lab @ Samsung AI Center Moscow
Visual Understanding Lab @ Samsung AI Center Moscow
GitHub https://saic-violet.github.io/point-based-clothing/
This is the official pytorch implementation for our ICCV 2021 paper "TRAR: Routing the Attention Spans in Transformers for Visual Question Answering" on VQA Task

?? ERASOR (RA-L'21 with ICRA Option) Official page of "ERASOR: Egocentric Ratio of Pseudo Occupancy-based Dynamic Object Removal for Static 3D Point C

Hyungtae Lim 225 Dec 29, 2022
Official Pytorch implementation of the paper "Action-Conditioned 3D Human Motion Synthesis with Transformer VAE", ICCV 2021

ACTOR Official Pytorch implementation of the paper "Action-Conditioned 3D Human Motion Synthesis with Transformer VAE", ICCV 2021. Please visit our we

Mathis Petrovich 248 Dec 23, 2022
Official code release for ICCV 2021 paper SNARF: Differentiable Forward Skinning for Animating Non-rigid Neural Implicit Shapes.

Official code release for ICCV 2021 paper SNARF: Differentiable Forward Skinning for Animating Non-rigid Neural Implicit Shapes.

null 235 Dec 26, 2022
Official PyTorch code of DeepPanoContext: Panoramic 3D Scene Understanding with Holistic Scene Context Graph and Relation-based Optimization (ICCV 2021 Oral).

DeepPanoContext (DPC) [Project Page (with interactive results)][Paper] DeepPanoContext: Panoramic 3D Scene Understanding with Holistic Scene Context G

Cheng Zhang 66 Nov 16, 2022
Official pytorch code for SSC-GAN: Semi-Supervised Single-Stage Controllable GANs for Conditional Fine-Grained Image Generation(ICCV 2021)

SSC-GAN_repo Pytorch implementation for 'Semi-Supervised Single-Stage Controllable GANs for Conditional Fine-Grained Image Generation'.PDF SSC-GAN:Sem

tyty 4 Aug 28, 2022
Official implementation of the paper Vision Transformer with Progressive Sampling, ICCV 2021.

Vision Transformer with Progressive Sampling This is the official implementation of the paper Vision Transformer with Progressive Sampling, ICCV 2021.

yuexy 123 Jan 1, 2023
Official Repository for the ICCV 2021 paper "PixelSynth: Generating a 3D-Consistent Experience from a Single Image"

PixelSynth: Generating a 3D-Consistent Experience from a Single Image (ICCV 2021) Chris Rockwell, David F. Fouhey, and Justin Johnson [Project Website

Chris Rockwell 95 Nov 22, 2022
Official implementation of the ICCV 2021 paper "Conditional DETR for Fast Training Convergence".

The DETR approach applies the transformer encoder and decoder architecture to object detection and achieves promising performance. In this paper, we handle the critical issue, slow training convergence, and present a conditional cross-attention mechanism for fast DETR training. Our approach is motivated by that the cross-attention in DETR relies highly on the content embeddings and that the spatial embeddings make minor contributions, increasing the need for high-quality content embeddings and thus increasing the training difficulty.

null 281 Dec 30, 2022
The Official Implementation of the ICCV-2021 Paper: Semantically Coherent Out-of-Distribution Detection.

SCOOD-UDG (ICCV 2021) This repository is the official implementation of the paper: Semantically Coherent Out-of-Distribution Detection Jingkang Yang,

Jake YANG 62 Nov 21, 2022
Official implementation of the ICCV 2021 paper: "The Power of Points for Modeling Humans in Clothing".

The Power of Points for Modeling Humans in Clothing (ICCV 2021) This repository contains the official PyTorch implementation of the ICCV 2021 paper: T

Qianli Ma 158 Nov 24, 2022
Official implementation of the ICCV 2021 paper "Joint Inductive and Transductive Learning for Video Object Segmentation"

JOINT This is the official implementation of Joint Inductive and Transductive learning for Video Object Segmentation, to appear in ICCV 2021. @inproce

Yunyao 35 Oct 16, 2022
An official implementation of "Exploiting a Joint Embedding Space for Generalized Zero-Shot Semantic Segmentation" (ICCV 2021) in PyTorch.

Exploiting a Joint Embedding Space for Generalized Zero-Shot Semantic Segmentation This is an official implementation of the paper "Exploiting a Joint

CV Lab @ Yonsei University 35 Oct 26, 2022
[ICCV 2021] Official Pytorch implementation for Discriminative Region-based Multi-Label Zero-Shot Learning SOTA results on NUS-WIDE and OpenImages

Discriminative Region-based Multi-Label Zero-Shot Learning (ICCV 2021) [arXiv][Project page >> coming soon] Sanath Narayan*, Akshita Gupta*, Salman Kh

Akshita Gupta 54 Nov 21, 2022
[ICCV 2021] Official Pytorch implementation for Discriminative Region-based Multi-Label Zero-Shot Learning SOTA results on NUS-WIDE and OpenImages

Discriminative Region-based Multi-Label Zero-Shot Learning (ICCV 2021) [arXiv][Project page >> coming soon] Sanath Narayan*, Akshita Gupta*, Salman Kh

Akshita Gupta 54 Nov 21, 2022
[ICCV 2021] Official PyTorch implementation for Deep Relational Metric Learning.

Deep Relational Metric Learning This repository is the official PyTorch implementation of Deep Relational Metric Learning. Framework Datasets CUB-200-

Borui Zhang 39 Dec 10, 2022
Official pytorch implementation of "Scaling-up Disentanglement for Image Translation", ICCV 2021.

Official pytorch implementation of "Scaling-up Disentanglement for Image Translation", ICCV 2021.

Aviv Gabbay 41 Nov 29, 2022
Official PyTorch implementation of N-ImageNet: Towards Robust, Fine-Grained Object Recognition with Event Cameras (ICCV 2021)

N-ImageNet: Towards Robust, Fine-Grained Object Recognition with Event Cameras Official PyTorch implementation of N-ImageNet: Towards Robust, Fine-Gra

null 32 Dec 26, 2022
(ICCV 2021) Official code of "Dressing in Order: Recurrent Person Image Generation for Pose Transfer, Virtual Try-on and Outfit Editing."

Dressing in Order (DiOr) ?? [Paper] ?? [Webpage] ?? [Running this code] The official implementation of "Dressing in Order: Recurrent Person Image Gene

Aiyu Cui 277 Dec 28, 2022
Official code release for "Learned Spatial Representations for Few-shot Talking-Head Synthesis" ICCV 2021

Official code release for "Learned Spatial Representations for Few-shot Talking-Head Synthesis" ICCV 2021

Moustafa Meshry 16 Oct 5, 2022