This repository contains the code for the paper "SCANimate: Weakly Supervised Learning of Skinned Clothed Avatar Networks"

Overview

SCANimate: Weakly Supervised Learning of Skinned Clothed Avatar Networks (CVPR 2021 Oral)

Paper

This repository contains the official PyTorch implementation of:

SCANimate: Weakly Supervised Learning of Skinned Clothed Avatar Networks

Full paper | 5min Presentation | Video | Project website | Poster

Installation

Please follow the instructions in ./installation.txt to install the environment and the SMPL model.

Run SCANimate

0. Activate the environment if it is not already activated:

$ source ./venv/scanimate/bin/activate

1. First download the pretrained model, some motion sequences and other files for the demo

  • Download an AIST++ dance motion sequence for test (CC BY 4.0 license):
$ . ./download_aist_demo_motion.sh

​ This script will create a data folder under current directory, please make sure to put it under the SCANimate directory.

  • Download pre-trained scanimats for animation test: Please visit https://scanimate.is.tue.mpg.de/download.php, register, login, read and agree to the license and then download some demo scanimats. Unzip the zip file into ./data directory

  • Download subset of CAPE data for training demo: Please visit https://scanimate.is.tue.mpg.de/download.php, register, login, read and agree to the license and then download the data for training demo. Unzip the zip file into ./data directory.

  • Now you should have a ./data directory under SCANimate. Within ./data you will have 5 directories: minimal_body, pretrained, pretrained_configs, test, and train.

Run animation demos:

2. Now you can run the test demo with the following command:

$ python -m apps.test_scanimate -c ./data/pretrained_configs/release_03223_shortlong.yaml -t ./data/test/gLO_sBM_cAll_d14_mLO1_ch05
  • You can replace the configuration file with other files under ./data/pretrained_configs/ to try other subjects.
  • You can also replace the test motions with others under ./data/test.
  • The result will be generated under ./demo_result/results_test.

3. The generated mesh sequences can be rendered with the code under ./demo_result:

First, install Open3D (for rendering the results) by:

$ pip install open3d==0.12.0

Then run:

$ python render/render_aist.py -i demo_result/results_test/release_03223_shortlong_test_gLO_sBM_cAll_d14_mLO1_ch05/ -o demo_result

Run training demo

2. Now you can run the demo training with

$ python -m apps.train_scanimate -c ./configs/example.yaml

The results can be found under ./demo_result/results/example.

3. Train on your own data Make your data the same structure as in the ./data/train/example_03375_shortlong, where a .ply file contains a T-pose SMPL body mesh and a folder containing training frames. Each frame corresponds to two files: one .npz files containing SMPL parameters that describes the body and one .ply file containing the clothed scan. The body should align with the scan. Then, change the ./configs/example.yaml to point to your data directory and you are good to go!

Citations

If you find our code or paper useful to your research, please consider citing:

@inproceedings{Saito:CVPR:2021,
  title = {{SCANimate}: Weakly Supervised Learning of Skinned Clothed Avatar Networks},
  author = {Saito, Shunsuke and Yang, Jinlong and Ma, Qianli and Black, Michael J.},
  booktitle = {Proceedings IEEE/CVF Conf.~on Computer Vision and Pattern Recognition (CVPR)},
  month = jun,
  year = {2021},
  month_numeric = {6}}
Comments
  • Train/Test with Color

    Train/Test with Color

    Hi, could you give a demo of using test_color.py and train_color.py? Is there any special requirement on the data?

    I made several attempts on training and testing the color predictor, I can see that the data in the .obj file is in a nx6 format. Is each row composed of vertex coordinates and the corresponding RGB value(in (0,1))?

    Could you recommend a proper way to render the mesh with color using the .obj file?

    Many thanks.

    opened by XiaodongGuan 6
  • question: how to get high frequency/resolution details in the generated animation mesh

    question: how to get high frequency/resolution details in the generated animation mesh

    I wanted to know if there is a reason why the generated animated mesh is flattened/smooth with only the low frequency details even though the input mesh has high frequency features. Is there a parameter or option to get the results in high frequency fidelity?

    Input Mesh - input_scene1 Generated Mesh - generated_mesh1

    opened by codesavory 4
  • How to get minimal body of my own data for test_scanimate.py?

    How to get minimal body of my own data for test_scanimate.py?

    Dear authors, thanks for your amazing work. I have a questions about minimal bofy: I found running test_ scanimate.py requires a canonical pose of 6890 vertices, but the number of vertices of cano.obj obtained by running the training code is greater than 6890, so it cannot be run on the test code. How to get minimal body of my own data for test_scanimate.py?

    opened by MrChangJiang 3
  • question: can I train using smplx instead of smpl when training for my own data

    question: can I train using smplx instead of smpl when training for my own data

    for training you mention that we need 3 things - input scan, SMPL parameters of the scan and a T-pose SMPL mesh. This works for SMPL. You have mentioned extensions to testing with smplx, but do you know what are the things I need to change to train with the SMPLx model? Do I need to change default.yaml and example.yaml?

    opened by codesavory 3
  • Test on unseen subjects

    Test on unseen subjects

    Dear authors,

    Many thanks for sharing such an amazing work. Without training, would it be possible to test this tool on 3D scans of unseen subjects in the training? The training data consisting of pairs of (raw scans, motion parameters) is not straightforward to obtain for a given subject.

    Thanks!

    opened by aniskacem 2
  • Questions about data usage.

    Questions about data usage.

    Dear authors, thanks for your amazing work. I am trying to follow your method, and have some questions about \data usage:

    1. Which subjects in CAPE did you use for the reported metrics? Are they beyond the publically available 20 raw scan sequences of 4 subjects in CAPE?

    2. You mentioned in the paper that "the motion sequences are randomly split into training (80%) and test (20%) sets". Do you mean "80% frames" or "80% sequences"? Is it true that at least a part of each sequence is used for training?

    Thank you!

    opened by ShenhanQian 2
  • smpl error

    smpl error

    code: model = smpl.create(opt['data']['smpl_dir'], model_type='smpl_vitruvian', gender=opt['data']['smpl_gender'], use_face_contour=False, ext='npz').to(cuda)
    The above line of code reports an error.

    File "/root/anaconda3/envs/pytorch3d/lib/python3.7/site-packages/smpl-0.1.13-py3.7.egg/smpl/body_models.py", line 94, in create ValueError: Unknown model type smpl_vitruvian, exiting!

    Is this a smpl installation error?

    opened by Passion-ysh 1
  • The demo training produces unsatisfactory results. How to obtain better results?

    The demo training produces unsatisfactory results. How to obtain better results?

    After training with the data of example_03375_shortlong, I obtain unsatisfactory results. There are many noise points. Further, the left upper leg is weird. Is it because the number of frames (16 only) is not enough? Do you have suggestions to obtain better results?

    Screenshot from 2021-12-22 19-24-09

    opened by HalleyJiang 1
  • GPU

    GPU

    想请问一下,如果设置多GPU 并行处理? 在代码中设置: os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID" os.environ['CUDA_VISIBLE_DEVICES'] = "4,6"; 开头: cuda = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") 中间: if torch.cuda.device_count() > 1: print("Let's use", torch.cuda.device_count(), "GPUs!") model = nn.DataParallel(model); model = model.module model.to(cuda); else: model.to(cuda);

    但是不起作用!

    opened by HDYYZDN 1
  • ground level value in load_trimesh()

    ground level value in load_trimesh()

    CapeDataset.py/load_trimesh()

    This function uses a fixed value of ground level(line 60). This creates problem for new datasets if the ground level for the new dataset is much lower.

    opened by garvita-tiwari 1
  • question: is there a recommended way or tool to get the pose and shape parameters of the input 3D scans

    question: is there a recommended way or tool to get the pose and shape parameters of the input 3D scans

    1. How to fit a SMPL model to the input 3D scan/mesh - how to extract the pose and shape parameters of the 3D scans is this done along-side scan generation or can it be done posterior to any arbitrary 3D mesh/3d scan
    2. Do you intend on providing any tools/suggest methods to extract the SMPL minimal and pose parameters for a given 3D mesh
    opened by codesavory 1
  • Evaluation and Metric Code

    Evaluation and Metric Code

    Hi, @shunsukesaito. I am trying to follow your amazing work. However, I encountered some problems with the evaluation.

    Would you mind releasing the Evaluation and Metric Code of Table 1 (Ds2m (cm) and Dn of Int. and Ex.)?

    opened by tpzou 0
  • Pretrainig skinning net does not converge on customized dataset

    Pretrainig skinning net does not converge on customized dataset

    Hi, I'm training SCANimate on ClothSeq dataset provided by Neural-GIF(https://github.com/garvita-tiwari/neuralgif). However, I met some problems and got bad results.

    Here are the logs of the first stage when training on a Cape sequence: cape

    And here are the logs of the first stage when training on a ClothSeq sequence: clothseq

    I notice that the pretraining of the skinning net converges very quickly on the Cape sequence, while the loss drops slowly on the ClothSeq sequence. I have checked the minimal_body file, the scans and SMPL parameters of each frame, all of them have the same format as the Cape dataset and are aligned well. So I'm wondering what the problem is? Do I need just more epochs in the first stage, or something else? I hope you can give me some advice, thank you!

    opened by bluestyle97 1
This repository contains codes on how to handle mouse event using OpenCV

Handling-Mouse-Click-Events-Using-OpenCV This repository contains codes on how t

Happy  N. Monday 3 Feb 15, 2022
Source code of our TPAMI'21 paper Dual Encoding for Video Retrieval by Text and CVPR'19 paper Dual Encoding for Zero-Example Video Retrieval.

Dual Encoding for Video Retrieval by Text Source code of our TPAMI'21 paper Dual Encoding for Video Retrieval by Text and CVPR'19 paper Dual Encoding

null 81 Dec 1, 2022
This repo contains a script that allows us to find range of colors in images using openCV, and then convert them into geo vectors.

Vectorizing color range This repo contains a script that allows us to find range of colors in images using openCV, and then convert them into geo vect

Development Seed 9 Jul 27, 2022
This repo contains several opencv projects done while learning opencv in python.

opencv-projects-python This repo contains both several opencv projects done while learning opencv by python and opencv learning resources [Basic conce

Fatin Shadab 2 Nov 3, 2022
Repository relating to the CVPR21 paper TimeLens: Event-based Video Frame Interpolation

TimeLens: Event-based Video Frame Interpolation This repository is about the High Speed Event and RGB (HS-ERGB) dataset, used in the 2021 CVPR paper T

Robotics and Perception Group 544 Dec 19, 2022
This repository provides train&test code, dataset, det.&rec. annotation, evaluation script, annotation tool, and ranking.

SCUT-CTW1500 Datasets We have updated annotations for both train and test set. Train: 1000 images [images][annos] Additional point annotation for each

Yuliang Liu 600 Dec 18, 2022
Repository of conference publications and source code for first-/ second-authored papers published at NeurIPS, ICML, and ICLR.

Repository of conference publications and source code for first-/ second-authored papers published at NeurIPS, ICML, and ICLR.

Daniel Jarrett 26 Jun 17, 2021
Code for the paper "DewarpNet: Single-Image Document Unwarping With Stacked 3D and 2D Regression Networks" (ICCV '19)

DewarpNet This repository contains the codes for DewarpNet training. Recent Updates [May, 2020] Added evaluation images and an important note about Ma

CVLab@StonyBrook 354 Jan 1, 2023
Code for the paper STN-OCR: A single Neural Network for Text Detection and Text Recognition

STN-OCR: A single Neural Network for Text Detection and Text Recognition This repository contains the code for the paper: STN-OCR: A single Neural Net

Christian Bartz 496 Jan 5, 2023
Code for AAAI 2021 paper: Sequential End-to-end Network for Efficient Person Search

This repository hosts the source code of our paper: [AAAI 2021]Sequential End-to-end Network for Efficient Person Search. SeqNet achieves the state-of

Zj Li 218 Dec 31, 2022
Code related to "Have Your Text and Use It Too! End-to-End Neural Data-to-Text Generation with Semantic Fidelity" paper

DataTuner You have just found the DataTuner. This repository provides tools for fine-tuning language models for a task. See LICENSE.txt for license de

null 81 Jan 1, 2023
Code for CVPR2021 paper "Learning Salient Boundary Feature for Anchor-free Temporal Action Localization"

AFSD: Learning Salient Boundary Feature for Anchor-free Temporal Action Localization This is an official implementation in PyTorch of AFSD. Our paper

Tencent YouTu Research 146 Dec 24, 2022
This is the code for our paper DAAIN: Detection of Anomalous and AdversarialInput using Normalizing Flows

Merantix-Labs: DAAIN This is the code for our paper DAAIN: Detection of Anomalous and Adversarial Input using Normalizing Flows which can be found at

Merantix 14 Oct 12, 2022
Code for the ACL2021 paper "Combining Static Word Embedding and Contextual Representations for Bilingual Lexicon Induction"

CSCBLI Code for our ACL Findings 2021 paper, "Combining Static Word Embedding and Contextual Representations for Bilingual Lexicon Induction". Require

Jinpeng Zhang 12 Oct 8, 2022
Code for the head detector (HeadHunter) proposed in our CVPR 2021 paper Tracking Pedestrian Heads in Dense Crowd.

Head Detector Code for the head detector (HeadHunter) proposed in our CVPR 2021 paper Tracking Pedestrian Heads in Dense Crowd. The head_detection mod

Ramana Subramanyam 76 Dec 6, 2022
Code release for our paper, "SimNet: Enabling Robust Unknown Object Manipulation from Pure Synthetic Data via Stereo"

SimNet: Enabling Robust Unknown Object Manipulation from Pure Synthetic Data via Stereo Thomas Kollar, Michael Laskey, Kevin Stone, Brijen Thananjeyan

null 68 Dec 14, 2022
code for our ICCV 2021 paper "DeepCAD: A Deep Generative Network for Computer-Aided Design Models"

DeepCAD This repository provides source code for our paper: DeepCAD: A Deep Generative Network for Computer-Aided Design Models Rundi Wu, Chang Xiao,

Rundi Wu 85 Dec 31, 2022
Dataset and Code for ICCV 2021 paper "Real-world Video Super-resolution: A Benchmark Dataset and A Decomposition based Learning Scheme"

Dataset and Code for RealVSR Real-world Video Super-resolution: A Benchmark Dataset and A Decomposition based Learning Scheme Xi Yang, Wangmeng Xiang,

Xi Yang 91 Nov 22, 2022