FrankMocap: A Strong and Easy-to-use Single View 3D Hand+Body Pose Estimator
FrankMocap pursues an easy-to-use single view 3D motion capture system developed by Facebook AI Research (FAIR). FrankMocap provides state-of-the-art 3D pose estimation outputs for body, hand, and body+hands in a single system. The core objective of FrankMocap is to democratize the 3D human pose estimation technology, enabling anyone (researchers, engineers, developers, artists, and others) can easily obtain 3D motion capture outputs from videos and images.
Btw, why the name FrankMocap? Our pipeline to integrate body and hand modules reminds us of Frankenstein's monster!
News:
- [2020/10/09] We have improved openGL rendering speed. It's about 40% faster. (e.g., body module: 6fps -> 11fps)
Key Features
- Body Motion Capture:
- Hand Motion Capture
- Egocentric Hand Motion Capture
- Whole body Motion Capture (body + hands)
Installation
- See INSTALL.md
A Quick Start
-
Run body motion capture
# using a machine with a monitor to show output on screen python -m demo.demo_bodymocap --input_path ./sample_data/han_short.mp4 --out_dir ./mocap_output # screenless mode (e.g., a remote server) xvfb-run -a python -m demo.demo_bodymocap --input_path ./sample_data/han_short.mp4 --out_dir ./mocap_output -
Run hand motion capture
# using a machine with a monitor to show outputs on screen python -m demo.demo_handmocap --input_path ./sample_data/han_hand_short.mp4 --out_dir ./mocap_output # screenless mode (e.g., a remote server) xvfb-run -a python -m demo.demo_handmocap --input_path ./sample_data/han_hand_short.mp4 --out_dir ./mocap_output -
Run whole body motion capture
# using a machine with a monitor to show outputs on screen python -m demo.demo_frankmocap --input_path ./sample_data/han_short.mp4 --out_dir ./mocap_output # screenless mode (e.g., a remote server) xvfb-run -a python -m demo.demo_frankmocap --input_path ./sample_data/han_short.mp4 --out_dir ./mocap_output -
Note:
- Above commands use openGL by default. If it does not work, you may try alternative renderers (pytorch3d or openDR).
- See the readme of each module for details
Joint Order
- See joint_order
Body Motion Capture Module
- See run_bodymocap
Hand Motion Capture Module
- See run_handmocap
Whole Body Motion Capture Module (Body + Hand)
- See run_totalmocap
License
- CC-BY-NC 4.0. See the LICENSE file.
References
- FrankMocap is based on the following research outputs:
@article{rong2020frankmocap,
title={FrankMocap: Fast Monocular 3D Hand and Body Motion Capture by Regression and Integration},
author={Rong, Yu and Shiratori, Takaaki and Joo, Hanbyul},
journal={arXiv preprint arXiv:2008.08324},
year={2020}
}
@article{joo2020eft,
title={Exemplar Fine-Tuning for 3D Human Pose Fitting Towards In-the-Wild 3D Human Pose Estimation},
author={Joo, Hanbyul and Neverova, Natalia and Vedaldi, Andrea},
journal={arXiv preprint arXiv:2004.03686},
year={2020}
}
- FrankMocap leverages many amazing open-sources shared in research community.
- SMPL, SMPLX
- Detectron2
- Pytorch3D (for rendering)
- OpenDR (for rendering)
- SPIN (for body module)
- 100DOH (for hand detection)
- lightweight-human-pose-estimation (for body detection)
Thanks a lot!
We would know if this orientation is the same for each bone and if each angle written in pred_body_pose is the resultant of the angle between the previous bone and the current on each axis or has a different logic.
We are approaching to body tracking right now.
In addition, if possible, can you provide me an image of the skeleton shape and points? We tried to use opengl commands, but with no result.
Thank you in advance and thank you for the amazing job.
Is there any documentation about it?
Thanks for your attention!
1.4k Jan 7, 2023
138 Dec 28, 2022
824 Jan 7, 2023
30 Nov 18, 2022
1 Aug 9, 2022
4 Dec 15, 2022
11 Feb 8, 2022
51 Oct 14, 2022
19 Feb 12, 2022
27 Dec 13, 2022
2 Jan 12, 2022
38 Oct 21, 2022
66 Dec 21, 2022
10 Nov 11, 2022
53 Dec 4, 2022
118 Dec 26, 2022
2 Sep 15, 2022
28 Dec 10, 2022
2 Mar 26, 2022