CVPR 2021 - Official code repository for the paper: On Self-Contact and Human Pose.

Overview

selfcontact

This repo is part of our project: On Self-Contact and Human Pose.
[Project Page] [Paper] [MPI Project Page]

It includes the main function to segment the SMPL and SMPL-X meshes into inside / outside / in-contact vertices. It includes contact losses that are used with SMPLify-XMC and TUCH.

Result vertices in contact

Vertices in contact (blue) based on geodesic and euclidean distances. In this example, the geodesic threshold is 30 cm and the euclidean threshold is 2 cm. In red, detected self-intersecting vertices with segment tests.

License

Software Copyright License for non-commercial scientific research purposes. Please read carefully the following terms and conditions and any accompanying documentation before you download and/or use the TUCH data and software, (the "Data & Software"), including 3D meshes, images, videos, textures, software, scripts, and animations. By downloading and/or using the Data & Software (including downloading, cloning, installing, and any other use of the corresponding github repository), you acknowledge that you have read these terms and conditions, understand them, and agree to be bound by them. If you do not agree with these terms and conditions, you must not download and/or use the Data & Software. Any infringement of the terms of this agreement will automatically terminate your rights under this License.

Installation

1) Clone this repository

git clone [email protected]:muelea/selfcontact.git
cd selfcontact

2) Create python virtual environment and install requirements

python3 -m venv $YOUR_VENV_DIR/selfcontact
source $YOUR_VENV_DIR/selfcontact/bin/activate
pip install -r requirements.txt

To use the repo as python module, use pip install . instead and move on with step 3).

3) Download essentials

Download the essentials from here. These files are required and include for example the precomputed geodesic distances for the neutral SMPL-X and SMPL body modles. Unpack the essentions to ESSENTIALS_FOLDER

5) Run example script to test for self-intersections

# vertices in contact
python selfcontact/tutorial/find_vertices_in_contact.py --essentials_folder ESSENTIALS_FOLDER --output_folder OUPUT_FOLDER
# intersecting vertices
python selfcontact/tutorial/find_self_intersecting_vertices.py --essentials_folder ESSENTIALS_FOLDER --output_folder OUPUT_FOLDER

Result inside / outside segmentation without segment tests
Mesh with result inside / outside segmentation WITHOUT segment tests. Note, how natural intersections, e.g. in the belly or crook regions are detected.

Result inside / outside segmentation without segment tests
Mesh with result inside / outside segmentation WITH segment tests. With segment testing, these self-intersections are ignored.

Citation

@inproceedings{Mueller:CVPR:2021,
  title = {On Self-Contact and Human Pose},
  author = {M{\"u}ller, Lea and Osman, Ahmed A. A. and Tang, Siyu and Huang, Chun-Hao P. and Black, Michael J.},
  booktitle = {Proceedings IEEE/CVF Conf.~on Computer Vision and Pattern Recogßnition (CVPR)},
  month = jun,
  year = {2021},
  doi = {},
  month_numeric = {6}
}

Acknowledgement

We thank Vassilis Choutas for his implementation of the generalized winding numbers.

Contact

For questions, please contact [email protected]

For commercial licensing (and all related questions for business applications), please contact [email protected].

Comments
  • SelfContact - tensor issue (running on GPU)

    SelfContact - tensor issue (running on GPU)

    Hi Lea!

    Running:

    self_contact_module = SelfContact(essentials_folder=essentials)
    self_contact_loss = SelfContactLoss(self_contact_module, device='cuda')
    self_contact_loss(vertices.to(self_contact_loss.device))
    

    Fails with:

    Traceback (most recent call last):
      File "/disk2/guytevet/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_exec2.py", line 3, in Exec
        exec(exp, global_vars, local_vars)
      File "<input>", line 1, in <module>
      File "/disk2/guytevet/venvs/py3.7_vibe/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
        return forward_call(*input, **kwargs)
      File "/disk2/guytevet/selfcontact/selfcontact/losses/general_selfcontact_loss.py", line 80, in forward
        test_segments=self.test_segments
      File "/disk2/guytevet/selfcontact/selfcontact/selfcontact.py", line 214, in segment_vertices
        vertices, v2v_min, incontact, exterior, test_segments)
      File "/disk2/guytevet/selfcontact/selfcontact/selfcontact.py", line 242, in segment_hd_points
        nzv = (exp1 == exp2).any(1).reshape(-1, 3).any(1)
    RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
    

    From my understanding, this happens since the essentials are not loaded to self_contact_loss.device.

    Many Thanks,

    opened by GuyTevet 4
  • 3DCP Dataset

    3DCP Dataset

    Thanks for your great work! Looking at the released 3DCP dataset - Does it contain the reported 3DCP Mocap dataset and how can I distinct 3DCP Mocap from 3DCP Scan?

    Thanks:)

    opened by GuyTevet 4
  • SelfContactLoss - memory issues (running on GPU)

    SelfContactLoss - memory issues (running on GPU)

    Hi Lea:) Applying SelfContactLoss with default parameters on batch_size=32 (i.e. [32, 10475, 3] tensor for a smplx mesh), fails for trying to allocate 112GB of GPU memory. Is there a way to handle this?

    RuntimeError: CUDA out of memory. Tried to allocate 112.33 GiB (GPU 0; 10.76 GiB total capacity; 309.89 MiB already allocated; 9.30 GiB free; 346.00 MiB reserved in total by PyTorch)
    
    opened by GuyTevet 3
  • running run_selfcontact_optimization meet error

    running run_selfcontact_optimization meet error

    Thank you for sharing so excellent research. The error happens when I run run_selfcontact_optimization.py. At first, My system is ubuntu18.04, cuda11.0, pytorch1.8.1 the detailed error is following: File "/opt/conda/envs/selfcontact/lib/python3.6/site-packages/selfcontact/utils/mesh.py", line 66, in compute_vertex_normals vertices_faces[:, 0] - vertices_faces[:, 1])) RuntimeError: index_add_cuda_ does not have a deterministic implementation, but you set 'torch.use_deterministic_algorithms(True)'. You can turn off determinism just for this operation if that's acceptable for your application. You can also file an issue at https://github.com/pytorch/pytorch/issues to help us prioritize adding deterministic support for this operation.

    I set the command torch.use_deterministic_algorithms(True), but it's useless. Hoping for your answer, thanks! @paulchhuang @ahmedosman

    opened by qinb 1
  • running run_selfcontact_optimization.py in SMPL model

    running run_selfcontact_optimization.py in SMPL model

    Hi, I successfully run the run_selfcontact_optimization.py in model_type='smplx', when I change model_type from smplx to smpl, and modify some codes to be compatible with it, the mismatch still occurs. Finally, It may be a solution to utilize the smpl_to_smplx.pkl to convert to smplx by smpl model, but I don't how to use it. Could you offer some help? Thanks @paulchhuang @ahmedosman

    opened by qinb 2
  • running speed

    running speed

    Hi,

    Thanks for proposing this meaningful work! I wonder if the running speed can be improved so that the proposed loss functions, i.e. inside, outside, normal losses, can be incorporated into a training framework?

    Thanks,

    opened by zhangy76 2
Owner
Lea Müller
PhD student in the Perceiving Systems Department at the Max Planck Institute for Intelligent Systems in Tübingen, Germany.
Lea Müller
CVPR 2021 - Official code repository for the paper: On Self-Contact and Human Pose.

TUCH This repo is part of our project: On Self-Contact and Human Pose. [Project Page] [Paper] [MPI Project Page] License Software Copyright License fo

Lea Müller 45 Jan 7, 2023
Repository for the paper "PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation", CVPR 2021.

PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation Code repository for the paper: PoseAug: A Differentiable Pose Augme

Pyjcsx 328 Dec 17, 2022
Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors, CVPR 2021

Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors Human POSEitioning System (H

Aymen Mir 66 Dec 21, 2022
Python scripts for performing 3D human pose estimation using the Mobile Human Pose model in ONNX.

Python scripts for performing 3D human pose estimation using the Mobile Human Pose model in ONNX.

Ibai Gorordo 99 Dec 31, 2022
This is an official implementation of our CVPR 2021 paper "Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression" (https://arxiv.org/abs/2104.02300)

Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression Introduction In this paper, we are interested in the bottom-up paradigm of estima

HRNet 367 Dec 27, 2022
Research code for CVPR 2021 paper "End-to-End Human Pose and Mesh Reconstruction with Transformers"

MeshTransformer ✨ This is our research code of End-to-End Human Pose and Mesh Reconstruction with Transformers. MEsh TRansfOrmer is a simple yet effec

Microsoft 473 Dec 31, 2022
This repository contains codes of ICCV2021 paper: SO-Pose: Exploiting Self-Occlusion for Direct 6D Pose Estimation

SO-Pose This repository contains codes of ICCV2021 paper: SO-Pose: Exploiting Self-Occlusion for Direct 6D Pose Estimation This paper is basically an

shangbuhuan 52 Nov 25, 2022
Official Pytorch implementation of "Beyond Static Features for Temporally Consistent 3D Human Pose and Shape from a Video", CVPR 2021

TCMR: Beyond Static Features for Temporally Consistent 3D Human Pose and Shape from a Video Qualtitative result Paper teaser video Introduction This r

Hongsuk Choi 215 Jan 6, 2023
This repository is the offical Pytorch implementation of ContextPose: Context Modeling in 3D Human Pose Estimation: A Unified Perspective (CVPR 2021).

Context Modeling in 3D Human Pose Estimation: A Unified Perspective (CVPR 2021) Introduction This repository is the offical Pytorch implementation of

null 37 Nov 21, 2022
The official TensorFlow implementation of the paper Action Transformer: A Self-Attention Model for Short-Time Pose-Based Human Action Recognition

Action Transformer A Self-Attention Model for Short-Time Human Action Recognition This repository contains the official TensorFlow implementation of t

PIC4SeRCentre 20 Jan 3, 2023
[CVPR 2022] PoseTriplet: Co-evolving 3D Human Pose Estimation, Imitation, and Hallucination under Self-supervision (Oral)

PoseTriplet: Co-evolving 3D Human Pose Estimation, Imitation, and Hallucination under Self-supervision Kehong Gong*, Bingbing Li*, Jianfeng Zhang*, Ta

null 256 Dec 28, 2022
Code for "Reconstructing 3D Human Pose by Watching Humans in the Mirror", CVPR 2021 oral

Reconstructing 3D Human Pose by Watching Humans in the Mirror Qi Fang*, Qing Shuai*, Junting Dong, Hujun Bao, Xiaowei Zhou CVPR 2021 Oral The videos a

ZJU3DV 178 Dec 13, 2022
SE3 Pose Interp - Interpolate camera pose or trajectory in SE3, pose interpolation, trajectory interpolation

SE3 Pose Interpolation Pose estimated from SLAM system are always discrete, and

Ran Cheng 4 Dec 15, 2022
《Unsupervised 3D Human Pose Representation with Viewpoint and Pose Disentanglement》(ECCV 2020) GitHub: [fig9]

Unsupervised 3D Human Pose Representation [Paper] The implementation of our paper Unsupervised 3D Human Pose Representation with Viewpoint and Pose Di

null 42 Nov 24, 2022
[WACV 2020] Reducing Footskate in Human Motion Reconstruction with Ground Contact Constraints

Reducing Footskate in Human Motion Reconstruction with Ground Contact Constraints Official implementation for Reducing Footskate in Human Motion Recon

Virginia Tech Vision and Learning Lab 38 Nov 1, 2022
Official project website for the CVPR 2021 paper "Exploring intermediate representation for monocular vehicle pose estimation"

EgoNet Official project website for the CVPR 2021 paper "Exploring intermediate representation for monocular vehicle pose estimation". This repo inclu

Shichao Li 138 Dec 9, 2022
Code for ICCV 2021 paper "HuMoR: 3D Human Motion Model for Robust Pose Estimation"

Code for ICCV 2021 paper "HuMoR: 3D Human Motion Model for Robust Pose Estimation"

Davis Rempe 367 Dec 24, 2022
Official code for the CVPR 2021 paper "How Well Do Self-Supervised Models Transfer?"

How Well Do Self-Supervised Models Transfer? This repository hosts the code for the experiments in the CVPR 2021 paper How Well Do Self-Supervised Mod

Linus Ericsson 157 Dec 16, 2022
The project is an official implementation of our paper "3D Human Pose Estimation with Spatial and Temporal Transformers".

3D Human Pose Estimation with Spatial and Temporal Transformers This repo is the official implementation for 3D Human Pose Estimation with Spatial and

Ce Zheng 363 Dec 28, 2022