Real-time RGBD-based Extended Body Pose Estimation
This repository is a real-time demo for our paper that was published at WACV 2021 conference
The output of our module is in SMPL-X parametric body mesh model:
- RNN estimates body pose from joints detected by Azure Kinect Body Tracking API
- For face (expression and jaw) and hand pose we crop from rgb image:
- for hand model we use minimal-hand
- our face NN takes media-pipe keypoints as input
Combined system runs at 30 fps on a 2080ti GPU and 8 core @ 4GHz CPU.
How to use
Build
- Prereqs: your nvidia driver should support cuda 10.2, Windows or Mac are not supported.
- Clone repo:
git clone https://github.com/rmbashirov/rgbd-kinect-pose.git
cd rgbd-kinect-pose
git submodule update --force --init --remote
- Docker setup:
- Install docker engine
- Install nvidia-docker
- Set nvidia your default runtime for docker
- Make docker run without sudo: create docker group and add current user to it:
sudo groupadd docker sudo usermod -aG docker $USER
- reboot
- Build docker image: run 2 cmds
- Attach your Azure Kinect camera
- Check your Azure Kinect camera is working inside Docker container:
- Enter Docker container:
./run_local.sh
fromdocker
dir - Then run
python -m pyk4a.viewer --vis_color --no_bt --no_depth
inside docker container
- Enter Docker container:
Download data
- Download our data archive
smplx_kinect_demo_data.tar.gz
- Unzip:
mkdir /your/unpacked/dir
,tar -zxf smplx_kinect_demo_data.tar.gz -C /your/unpacked/dir
- Download models for hand, see link in "Download models from here" line in our fork, put to
/your/unpacked/dir/minimal_hand/model
- To download SMPL-X parametric body model go to this project website, register, go to the downloads section, download SMPL-X v1.1 model, put to
/your/unpacked/dir/pykinect/body_models/smplx
/your/unpacked/dir
should look like this- Set
data_dirpath
andoutput_dirpath
variables in config file:data_dirpath
is a path to/your/unpacked/dir
output_dirpath
is used to check timings or to store result images- ensure these paths are visible inside docker container, set
VOLUMES
variable here
Run
- Run demo: in
src
dir run./run_server.sh
, the latter will enter docker container and will use config file where shape of the person is loaded from an external file: in our work we did not focus on person's shape estimation
What else
Apart from our main body pose estimation contribution you can find this repository useful for:
- minimal_pytorch_rasterizer python package: CUDA non-differentiable mesh rasterization library for pytorch tensors with python bindings
- pyk4a python package: real-time streaming from Azure Kinect camera, this package also works in our provided docker environment
- multiprocessing_pipeline python package: set-up pipeline graph of python blocks running in parallel, see usage in server.py
Citation
If you find the project helpful, please consider citing us:
@inproceedings{bashirov2021real,
title={Real-Time RGBD-Based Extended Body Pose Estimation},
author={Bashirov, Renat and Ianina, Anastasia and Iskakov, Karim and Kononenko, Yevgeniy and Strizhkova, Valeriya and Lempitsky, Victor and Vakhitov, Alexander},
booktitle={Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision},
pages={2807--2816},
year={2021}
}
Non-commercial use only