CVPR 2021 - Official code repository for the paper: On Self-Contact and Human Pose.

Overview

SMPLify-XMC

This repo is part of our project: On Self-Contact and Human Pose.
[Project Page] [Paper] [MPI Project Page]

Teaser SMPLify-XMC

License

Software Copyright License for non-commercial scientific research purposes. Please read carefully the following terms and conditions and any accompanying documentation before you download and/or use the TUCH data and software, (the "Data & Software"), including 3D meshes, images, videos, textures, software, scripts, and animations. By downloading and/or using the Data & Software (including downloading, cloning, installing, and any other use of the corresponding github repository), you acknowledge that you have read these terms and conditions, understand them, and agree to be bound by them. If you do not agree with these terms and conditions, you must not download and/or use the Data & Software. Any infringement of the terms of this agreement will automatically terminate your rights under this License.

Description and Demo

SMPLify-XMC adapts SMPLify-X to fit SMPL-X model to Mimic The Pose (MTP) data. To run SMPLify-XMC you need

  • an image of a person mimicking a presented pose
  • the presented pose parameters
  • the person's gender, height and weight
  • the OpenPose keypoints.

The code has been tested with Python 3.6.9, CUDA 10.1, CuDNN 7.5 and PyTorch 1.8.1 on Ubuntu 18.04.

Installation

1) Clone this repo

git clone [email protected]:muelea/smplify-xmc.git
cd smplify-xmc

2) Download body model

Download them SMPL-X body model from https://smpl-x.is.tue.mpg.de and save it in MODEL_FOLDER. You can replace model_folder: MODEL_FOLDER in the config file configs/fit_smplx_singleview.yaml or use an environment variable.

3) Download essentials

Download essentials from here and unpack to ESSENTIALS_DIR. Then create symlinks between the essentials and this repo:

ln -s $ESSENTIALS_DIR/smplify-xmc-essentials data/essentials

4) Create python virtual environment

python3 -m venv $YOUR_VENV_DIR/smplify-xmc
source $YOUR_VENV_DIR/smplify-xmc/bin/activate

5) Install requirements

pip install -r requirements.txt

6) Get dependencies

Clone self-contact repo, e.g. to YOUR_PYTHON_PACKAGE_DIR. Then use pip to install the package. Then you can import the self-contact functions from any place in your system. (make sure your venv is activated).

cd $YOUR_PYTHON_PACKAGE_DIR
git clone [email protected]:muelea/selfcontact.git
cd selfcontact
rm -r .git
pip install .
cd ..

Demo using our example data

You can find our example dataset in this repo under data/example_input. The following command will automatically save parameters, mesh, and image under output_dir:

python main_singleview.py --config configs/fit_smplx_singleview.yaml \
--dataset mtp_demo \
--input_base_dir data/example_input/singleview/subject1 \
--input_dir_poses data/example_input/presented_poses \
--output_dir data/example_output/singleview/subject1 \
--model_folder $MODELS_FOLDER

Process the MTP dataset:

Download MTP data from the TUCH website: https://tuch.is.tue.mpg.de and save the data in DS_DIR. You should now see a folder named $DS_DIR/mtp.

Read MTP data: python lib/dataextra/preprocess_mtp_mturk_dataset.py --ds_dir=$DS_DIR/mtp

Process the first item: python main_singleview.py --config configs/fit_smplx_singleview_mtp_dataset.yaml --db_file data/dbs/mtp_mturk.npz --output_dir data/example_output/mtp/ --model_folder=$MODEL_FOLDER --cluster_bs=1 --ds_start_idx=0

Process your own data:

Follow the structure of the example data in data/example_input. Create a folder PP_FOLDER for the presented poses:

PP_FOLDER
  ----pose_name1.pkl
  ----pose_name2.pkl

The pickle file should contain a dictionary with the pose parameters and the vertices. If you include the vertices ('v'), the vertices in contact will be computed automatically.

data = {
  'body_pose': ..
  'right_hand_pose': ..
  'left_hand_pose': ..
  'global_orient': ..
  'v': .. #vertices

}

Then create a folder MI_FOLDER for the mimicked images, following the structure below. Compute the keypoints for each image from OpenPose. The meta file should contain the gender, height and weight of the subject mimicking the pose.

MI_FOLDER
  ----subject_name1
    ----images
      ----pose_name1.png
      ----pose_name2.png
    ----keypoints
      ----pose_name1.json
      ----pose_name2.json
    ----meta.yaml

Finally, run the fitting code:

python main_singleview.py --config configs/fit_smplx_singleview.yaml \
--input_base_dir $MI_FOLDER/subject_name1 \
--input_dir_poses $PP_FOLDER \
--output_dir data/example_output/subject_name1

Citation

@inproceedings{Mueller:CVPR:2021,
  title = {On Self-Contact and Human Pose},
  author = {M{\"u}ller, Lea and Osman, Ahmed A. A. and Tang, Siyu and Huang, Chun-Hao P. and Black, Michael J.},
  booktitle = {Proceedings IEEE/CVF Conf.~on Computer Vision and Pattern Recogßnition (CVPR)},
  month = jun,
  year = {2021},
  doi = {},
  month_numeric = {6}
}

Acknowledgement

We thank Vassilis Choutas and Georgios Pavlakos for publishing the SMPLify-X code: https://github.com/vchoutas/smplify-x. This has allowed us to build our code on top of it and continue to use important features, such as the prior or optimization. Again, special thanks to Vassilis Choutas for his implementation of the generalized winding numbers and the measurements code. We also thank our data capture and admin team for their help with the extensive data collection on Mechanical Turk and in the Capture Hall. Many thanks to all subjects who contributed to this dataset in the scanner and on the Internet. Thanks to all PS members who proofread the script and did not understand it and the reviewers, who helped improving during the rebuttal. Lea Mueller and Ahmed A. A. Osman thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting them. We thank the wonderful PS department for their questions and support.

Contact

For questions, please contact [email protected]

For commercial licensing (and all related questions for business applications), please contact [email protected].

Comments
  • About the error in the demo file

    About the error in the demo file

    About the error in the demo file

    Hello. I am a student who has just started human 3d reconstruction research in Korea. I found smplify-xmc and it worked really well for what I was looking for. So I tried running the demo file, but an error occurred, but I don't know how to solve it, so I'm writing this.

    Thank you very much for your reply


    Traceback (most recent call last): File "main_singleview.py", line 244, in main(**args) File "main_singleview.py", line 234, in main **args) File "/home/ylab/smplify-xmc/lib/core/fit_single_frame.py", line 228, in fit_single_frame body_model) File "/home/ylab/smplify-xmc/lib/core/fitting.py", line 76, in run_fitting loss = optimizer.step(closure) File "/home/ylab/anaconda3/envs/smplify-xmc/lib/python3.7/site-packages/torch/optim/optimizer.py", line 89, in wrapper return func(*args, **kwargs) File "/home/ylab/smplify-xmc/lib/optimizers/lbfgs_ls.py", line 305, in step orig_loss = closure() File "/home/ylab/smplify-xmc/lib/core/fitting.py", line 132, in fitting_func gt_weight=weight, File "/home/ylab/anaconda3/envs/smplify-xmc/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(*input, **kwargs) File "/home/ylab/smplify-xmc/lib/core/losses.py", line 299, in forward self.sc_crit(vertices=body_model_output.vertices) File "/home/ylab/anaconda3/envs/smplify-xmc/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(*input, **kwargs) File "/home/ylab/anaconda3/envs/smplify-xmc/lib/python3.7/site-packages/selfcontact/losses/general_selfcontact_loss.py", line 80, in forward test_segments=self.test_segments File "/home/ylab/anaconda3/envs/smplify-xmc/lib/python3.7/site-packages/selfcontact/selfcontact/selfcontact.py", line 219, in segment_vertices vertices, v2v_min, incontact, exterior, test_segments) File "/home/ylab/anaconda3/envs/smplify-xmc/lib/python3.7/site-packages/selfcontact/selfcontact/selfcontact.py", line 339, in segment_hd_points hd_v2v[:, ~self.geomask] = inf_tensor IndexError: The shape of the mask [10475, 10475] at index 0 does not match the shape of the indexed tensor [1, 1593, 1593] at index 1

    bug 
    opened by mostar39 4
  •  Run demo, but report an error

    Run demo, but report an error

    Thanks for your great works! When I tried to run your demo, it reported the following error: RuntimeError: einsum() operands do not broadcast with remapped shapes [original->remapped]: [1, 20]->[1, 1, 1, 20] [10475, 3, 400]->[1, 10475, 3, 400] Can you help me see what's wrong? Thank you very much!

    opened by code-conquer 3
  • fix error from issue#1

    fix error from issue#1

    Fixed error from issue#1.

    • updated smplx library version.
    • updated some sources in losses.py which calls "lbs.blend_shapes". (reference : smplx)
    opened by woo1 0
  • Can we reconstruct a 3D human body from a single RGB image like simpliy-x?

    Can we reconstruct a 3D human body from a single RGB image like simpliy-x?

    Hello, author! It is very valuable to see that this work is aimed at improving self-contact images. But can this project work just by inputting a single RGB image like smplify-x? I see that I have to input a presented pose for this job. What if I don't have this?

    opened by zhuangzhuang000 0
  • Multi-view processing

    Multi-view processing

    Hello! I noticed in the code some mentions of multi-view processing. Is there any possibility to start it? Is this method capable of multi-view processing?

    opened by R1chrdson 0
  • data type issue (or GPU memory issue?) on

    data type issue (or GPU memory issue?) on "vert_ids_wt"

    Hello, I recently would like to reproduce your code & data in my environment, but I encountered a small problem. This is the spec of my PC & setting.

    • OS: Windows 10
    • CPU: Core i7-8700 (3.2GHz) / RAM 32G / GPU: GTX 1080 Ti (11GB)
    • Python: 3.8.10 (anaconda 2021.05) / Pytorch: 1.8.0 / CUDA 10.1.243

    I followed your instruction and run main_singleview.py.
    It seemed to work well until 67%... However, it suddenly stopped with the following error message:

    Traceback (most recent call last):
      File "main_singleview.py", line 244, in <module>
        main(**args)
      File "main_singleview.py", line 207, in main
        fit_single_frame(img,
      File "{path to git clone}\lib\core\fit_single_frame.py", line 224, in fit_single_frame
        final_loss_val = monitor.run_fitting(
      File "{path to git clone}\lib\core\fitting.py", line 76, in run_fitting
        loss = optimizer.step(closure)
      File "{my user account}\.conda\envs\venv_pt\lib\site-packages\torch\optim\optimizer.py", line 89, in wrapper
        return func(*args, **kwargs)
      File "{path to git clone}\lib\optimizers\lbfgs_ls.py", line 305, in step
        orig_loss = closure()
      File "{path to git clone}\lib\core\fitting.py", line 125, in fitting_func
        total_loss = loss(
      File "{my user account}\.conda\envs\venv_pt\lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
        result = self.forward(*input, **kwargs)
      File "{path to git clone}\lib\core\losses.py", line 299, in forward
        self.sc_crit(vertices=body_model_output.vertices)
      File "{my user account}\.conda\envs\venv_pt\lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
        result = self.forward(*input, **kwargs)
      File "{my user account}\.conda\envs\venv_pt\lib\site-packages\selfcontact-0.2.0-py3.8.egg\selfcontact\losses\general_selfcontact_loss.py", line 77, in forward
      File "{my user account}\.conda\envs\venv_pt\lib\site-packages\selfcontact-0.2.0-py3.8.egg\selfcontact\selfcontact.py", line 196, in segment_vertices
      File "{my user account}\.conda\envs\venv_pt\lib\site-packages\selfcontact-0.2.0-py3.8.egg\selfcontact\selfcontact.py", line 122, in triangles
    IndexError: tensors used as indices must be long, byte or bool tensors
    

    I am not familiar with this error, but it seemed that vert_ids_wt in selfcontact is problematic. https://github.com/muelea/selfcontact/blob/master/selfcontact/selfcontact.py#L78

    I examined the data from smplx_inner_mouth_bounds.pkl. It is just an array of integer values [8958 1666 1665 ... 8958] became np.int32 to torch.int32 (Int), so I forcefully changed it as Long like this:

    #self.register_buffer('vert_ids_wt', torch.from_numpy(vert_ids_wt))
    self.register_buffer('vert_ids_wt', torch.from_numpy(vert_ids_wt.astype(np.int64))) ## [QUICK FIX for Windows]
    

    However, I still encountered a problem at the same location with a different error message:

      File "{my user account}\.conda\envs\venv_pt\lib\site-packages\selfcontact-0.2.0-py3.8.egg\selfcontact\losses\general_selfcontact_loss.py", line 77, in forward
      File "{my user account}\.conda\envs\venv_pt\lib\site-packages\selfcontact-0.2.0-py3.8.egg\selfcontact\selfcontact.py", line 200, in segment_vertices
      File "{my user account}\.conda\envs\venv_pt\lib\site-packages\selfcontact-0.2.0-py3.8.egg\selfcontact\selfcontact.py", line 141, in get_intersection_mask
      File "{my user account}\.conda\envs\venv_pt\lib\site-packages\selfcontact-0.2.0-py3.8.egg\selfcontact\utils\mesh.py", line 223, in winding_numbers
      File "{my user account}\.conda\envs\venv_pt\lib\site-packages\selfcontact-0.2.0-py3.8.egg\selfcontact\utils\mesh.py", line 164, in solid_angles
    RuntimeError: CUDA out of memory. Tried to allocate 438.00 MiB (GPU 0; 11.00 GiB total capacity; 9.64 GiB already allocated; 0 bytes free; 9.65 GiB reserved in total by PyTorch)
    

    So, here are my two questions.

    1. Was it ok to change vert_ids_wt as np.int64/torch.int64? If not, do you have any suggestions to fix this issue?
    2. If it is, do I need more (GPU) memory to run this code?

    Thank you in advance.

    opened by stnoh 1
  • Fixed torch.exp inf error(measurements_loss)

    Fixed torch.exp inf error(measurements_loss)

    • When I tried with weight over 150kg, measurements_loss became inf. So I couldn't fit with the person.
    • I am not sure this is the best way to fix. But when I tried another method like log1pexp it doesn't have an effect
    • test data : https://drive.google.com/file/d/1Tea9d5VPuD0LTq19mPJlCm-MSgIlJyi1/view?usp=sharing
    opened by woo1 2
Owner
Lea Müller
PhD student in the Perceiving Systems Department at the Max Planck Institute for Intelligent Systems in Tübingen, Germany.
Lea Müller
CVPR 2021 - Official code repository for the paper: On Self-Contact and Human Pose.

TUCH This repo is part of our project: On Self-Contact and Human Pose. [Project Page] [Paper] [MPI Project Page] License Software Copyright License fo

Lea Müller 45 Jan 7, 2023
Repository for the paper "PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation", CVPR 2021.

PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation Code repository for the paper: PoseAug: A Differentiable Pose Augme

Pyjcsx 328 Dec 17, 2022
Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors, CVPR 2021

Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors Human POSEitioning System (H

Aymen Mir 66 Dec 21, 2022
Python scripts for performing 3D human pose estimation using the Mobile Human Pose model in ONNX.

Python scripts for performing 3D human pose estimation using the Mobile Human Pose model in ONNX.

Ibai Gorordo 99 Dec 31, 2022
This is an official implementation of our CVPR 2021 paper "Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression" (https://arxiv.org/abs/2104.02300)

Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression Introduction In this paper, we are interested in the bottom-up paradigm of estima

HRNet 367 Dec 27, 2022
Research code for CVPR 2021 paper "End-to-End Human Pose and Mesh Reconstruction with Transformers"

MeshTransformer ✨ This is our research code of End-to-End Human Pose and Mesh Reconstruction with Transformers. MEsh TRansfOrmer is a simple yet effec

Microsoft 473 Dec 31, 2022
This repository contains codes of ICCV2021 paper: SO-Pose: Exploiting Self-Occlusion for Direct 6D Pose Estimation

SO-Pose This repository contains codes of ICCV2021 paper: SO-Pose: Exploiting Self-Occlusion for Direct 6D Pose Estimation This paper is basically an

shangbuhuan 52 Nov 25, 2022
Official Pytorch implementation of "Beyond Static Features for Temporally Consistent 3D Human Pose and Shape from a Video", CVPR 2021

TCMR: Beyond Static Features for Temporally Consistent 3D Human Pose and Shape from a Video Qualtitative result Paper teaser video Introduction This r

Hongsuk Choi 215 Jan 6, 2023
This repository is the offical Pytorch implementation of ContextPose: Context Modeling in 3D Human Pose Estimation: A Unified Perspective (CVPR 2021).

Context Modeling in 3D Human Pose Estimation: A Unified Perspective (CVPR 2021) Introduction This repository is the offical Pytorch implementation of

null 37 Nov 21, 2022
The official TensorFlow implementation of the paper Action Transformer: A Self-Attention Model for Short-Time Pose-Based Human Action Recognition

Action Transformer A Self-Attention Model for Short-Time Human Action Recognition This repository contains the official TensorFlow implementation of t

PIC4SeRCentre 20 Jan 3, 2023
[CVPR 2022] PoseTriplet: Co-evolving 3D Human Pose Estimation, Imitation, and Hallucination under Self-supervision (Oral)

PoseTriplet: Co-evolving 3D Human Pose Estimation, Imitation, and Hallucination under Self-supervision Kehong Gong*, Bingbing Li*, Jianfeng Zhang*, Ta

null 256 Dec 28, 2022
Code for "Reconstructing 3D Human Pose by Watching Humans in the Mirror", CVPR 2021 oral

Reconstructing 3D Human Pose by Watching Humans in the Mirror Qi Fang*, Qing Shuai*, Junting Dong, Hujun Bao, Xiaowei Zhou CVPR 2021 Oral The videos a

ZJU3DV 178 Dec 13, 2022
SE3 Pose Interp - Interpolate camera pose or trajectory in SE3, pose interpolation, trajectory interpolation

SE3 Pose Interpolation Pose estimated from SLAM system are always discrete, and

Ran Cheng 4 Dec 15, 2022
《Unsupervised 3D Human Pose Representation with Viewpoint and Pose Disentanglement》(ECCV 2020) GitHub: [fig9]

Unsupervised 3D Human Pose Representation [Paper] The implementation of our paper Unsupervised 3D Human Pose Representation with Viewpoint and Pose Di

null 42 Nov 24, 2022
[WACV 2020] Reducing Footskate in Human Motion Reconstruction with Ground Contact Constraints

Reducing Footskate in Human Motion Reconstruction with Ground Contact Constraints Official implementation for Reducing Footskate in Human Motion Recon

Virginia Tech Vision and Learning Lab 38 Nov 1, 2022
Official project website for the CVPR 2021 paper "Exploring intermediate representation for monocular vehicle pose estimation"

EgoNet Official project website for the CVPR 2021 paper "Exploring intermediate representation for monocular vehicle pose estimation". This repo inclu

Shichao Li 138 Dec 9, 2022
Code for ICCV 2021 paper "HuMoR: 3D Human Motion Model for Robust Pose Estimation"

Code for ICCV 2021 paper "HuMoR: 3D Human Motion Model for Robust Pose Estimation"

Davis Rempe 367 Dec 24, 2022
Official code for the CVPR 2021 paper "How Well Do Self-Supervised Models Transfer?"

How Well Do Self-Supervised Models Transfer? This repository hosts the code for the experiments in the CVPR 2021 paper How Well Do Self-Supervised Mod

Linus Ericsson 157 Dec 16, 2022
The project is an official implementation of our paper "3D Human Pose Estimation with Spatial and Temporal Transformers".

3D Human Pose Estimation with Spatial and Temporal Transformers This repo is the official implementation for 3D Human Pose Estimation with Spatial and

Ce Zheng 363 Dec 28, 2022