Towards Multi-Camera 3D Human Pose Estimation in Wild Environment

Overview

PanopticStudio Toolbox

This repository has a toolbox to download, process, and visualize the Panoptic Studio (Panoptic) data.

Note:

  • Sep-21-2020: Currently our server is offline due to the power outage in the CMU Campus, and COVID-19 makes it difficult to access the server room. We will fix the issue as soon as possible. 
  • Sep-30-2020: Unfortunately, we found that our server has been broken and we are replacing it now. Please wait a couple of more weeks.
  • Oct-5-2020: Our server is back and online now!
  • May-18-2021: Currently our server is offline due to our server maintenance. Hopefully it will be back online in this week.  

Quick start guide

Follow these steps to set up a simple example:

1. Check out the codebase

git clone https://github.com/CMU-Perceptual-Computing-Lab/panoptic-toolbox
cd panoptic-toolbox

2. Download a sample data and other data

To download a dataset, named "171204_pose1_sample" in this example, run the following script.

./scripts/getData.sh 171204_pose1_sample

This bash script requires curl or wget.

This script will create a folder "./171204_pose1_sample" and download the following files.

  • 171204_pose1_sample/hdVideos/hd_00_XX.mp4 #synchronized HD video files (31 views)
  • 171204_pose1_sample/vgaVideos/KINECTNODE%d/vga_XX_XX.mp4 #synchrponized VGA video files (480 views)
  • 171204_pose1_sample/calibration_171204_pose1_sample.json #calibration files
  • 171204_pose1_sample/hdPose3d_stage1_coco19.tar #3D Body Keypoint Data (coco19 keypoint definition)
  • 171204_pose1_sample/hdFace3d.tar #3D Face Keypoint Data
  • 171204_pose1_sample/hdHand3d.tar #3D Hand Keypoint Data

Note that this sample example currently does not have VGA videos.

You can also download any other seqeunce through this script. Just use the the name of the target sequence: instead of the "171204_pose1panopticHD". r example,

./scripts/getData.sh 171204_pose1

for the full version of 171204_pose1 sequence:. You can also specify the number of videospanopticHDnt to donwload.

./scripts/getData.sh (sequenceName) (VGA_Video_Number) (HD_Video_Number)

For example, the following command will download 240 vga videos and 10 videos.

./scripts/getData.sh 171204_pose1_sample 240 10

Note that we have sorted the VGA camera order so that you download uniformly distributed view.

3. Downloading All Available Sequences

You can find the list of currently available sequences in the following link:

List of released sequences (ver1.2)

Downloading all of them (including videos) may take a long time, but downloading 3D keypoint files (body+face+hand upon their availability) should be "relatively" quick.

You can use the following script to download currently available sequences (ver 1.2):

./scripts/getDB_panopticHD_ver1_2.sh

The default setting is not downloading any videos. Feel free to change the "vgaVideoNum" and "hdVideoNum" in the script to other numbers if you also want to download videos.

You can see the example videos and other information of each sequence: in our website: Browsing dataset.

Check the 3D viewer in each sequence: page where you can visualize 3D skeletons in your web browser. For example: http://domedb.perception.cs.cmu.edu/panopticHDpose1.html

4. Extract the images & 3D keypoint data

This step requires ffmpeg.

./scripts/extractAll.sh 171204_pose1_sample

This will extract images, for example 171204_pose1_sample/hdImgs/00_00/00_00_00000000.jpg, and the corresponding 3D skeleton data, for example 171204_pose1_sample/hdPose3d_stage1_coco19/body3DScene_00000000.json.

extractAll.sh is a simple script that combines the following set of commands (you shouldn't need to run these again):

cd 171204_pose1_sample
../scripts/vgaImgsExtractor.sh # PNG files from VGA video (25 fps)
../scripts/hdImgsExtractor.sh # PNG files from HD video (29.97 fps)
tar -xf vgaPose3d_stage1.tar # Extract skeletons at VGA framerate
tar -xf hdPose3d_stage1.tar # Extract skeletons for HD
cd ..

5. Run demo programs

Python

This codes require numpy, matplotlib.

Visualizing 3D keypoints (body, face, hand):

cd python
jupyter notebook demo_3Dkeypoints_3dview.ipynb

The result should look like this.

Reprojecting 3D keypoints (body, face, hand) on a selected HD view:

cd python
jupyter notebook demo_3Dkeypoints_reprojection_hd.ipynb

The result should look like this.

This codes require numpy, matplotlib.

Visualizing 3D keypoints (body, face, hand):

cd python
jupyter notebook demo_3Dkeypoints_3dview.ipynb

The result should look like this.

Reprojecting 3D keypoints (body, face, hand) on a selected HD view:

cd python
jupyter notebook demo_3Dkeypoints_reprojection_hd.ipynb

The result should look like this.

Python + OpengGL

  • This codes require pyopengl.

  • Visualizing 3D keypoints (body, face, hand):

python glViewer.py

Matlab

Note: Matlab code is outdated, and does not handle 3D keypoint outputs (coco19 body, face, hand). Please see this code only for reference. We will update this later.

Matlab example (outdated):

>>> cd matlab
>>> demo

Skeleton Output Format

We reconstruct 3D skeleton of people using the method of Joo et al. 2018.

The output of each frame is written in a json file. For example,

{ "version": 0.7, 
"univTime" :53541.542,
"fpsType" :"hd_29_97",
"bodies" :
[
{ "id": 0,
"joints19": [-19.4528, -146.612, 1.46159, 0.724274, -40.4564, -163.091, -0.521563, 0.575897, -14.9749, -91.0176, 4.24329, 0.361725, -19.2473, -146.679, -16.1136, 0.643555, -14.7958, -118.804, -20.6738, 0.619599, -22.611, -93.8793, -17.7834, 0.557953, -12.3267, -91.5465, -6.55368, 0.353241, -12.6556, -47.0963, -4.83599, 0.455566, -10.8069, -8.31645, -4.20936, 0.501312, -20.2358, -147.348, 19.1843, 0.628022, -13.1145, -120.269, 28.0371, 0.63559, -20.1037, -94.3607, 30.0809, 0.625916, -17.623, -90.4888, 15.0403, 0.327759, -17.3973, -46.9311, 15.9659, 0.419586, -13.1719, -7.60601, 13.4749, 0.519653, -38.7164, -166.851, -3.25917, 0.46228, -28.7043, -167.333, -7.15903, 0.523224, -39.0433, -166.677, 2.55916, 0.395965, -30.0718, -167.264, 8.18371, 0.510041]
}
] }

Here, each subject has the following values.

id: a unique subject index within a sequence:. Skeletons with the same id across time represent temporally associated moving skeletons (an individual). However, the same person may have multiple ids joints19: 19 3D joint locations, formatted as [x1,y1,z1,c1,x2,y2,z2,c2,...] where each c ispanopticHDjoint confidence score.

The 3D skeletons have the following keypoint order:

0: Neck
1: Nose
2: BodyCenter (center of hips)
3: lShoulder
4: lElbow
5: lWrist,
6: lHip
7: lKnee
8: lAnkle
9: rShoulder
10: rElbow
11: rWrist
12: rHip
13: rKnee
14: rAnkle
15: lEye
16: lEar
17: rEye
18: rEar

Note that this is different from OpenPose output order, although our method is based on it.

Note that we used to use an old format (named mpi15 as described in our outdated document), but we do not this format anymore.

KinopticStudio Toolbox

Kinoptic Studio is a subsystem of Panoptic Studio, which is composed of 10 Kinect2 sensors. Please see: README_kinoptic

Panoptic 3D PointCloud DB ver.1

You can download all sequences included in our 3D PointCloud DB ver.1 using the following script:

./scripts/getDB_ptCloud_ver1.sh

Haggling DB

We have released the processed data for the haggling sequence. Please see Social Signal Processing repository.

Teaser Image

License

Panoptic Studio Dataset is freely available for non-commercial and research purpose only.

References

By using the dataset, you agree to cite at least one of the following papers.

@inproceedings{Joo_2015_ICCV,
author = {Joo, Hanbyul and Liu, Hao and Tan, Lei and Gui, Lin and Nabbe, Bart and Matthews, Iain and Kanade, Takeo and Nobuhara, Shohei and Sheikh, Yaser},
title = {Panoptic Studio: A Massively Multiview System for Social Motion Capture},
booktitle = {ICCV},
year = {2015} }

@inproceedings{Joo_2017_TPAMI,
title={Panoptic Studio: A Massively Multiview System for Social Interaction Capture},
author={Joo, Hanbyul and Simon, Tomas and Li, Xulong and Liu, Hao and Tan, Lei and Gui, Lin and Banerjee, Sean and Godisart, Timothy Scott and Nabbe, Bart and Matthews, Iain and Kanade, Takeo and Nobuhara, Shohei and Sheikh, Yaser},
journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
year={2017} }

@inproceedings{Simon_2017_CVPR,
title={Hand Keypoint Detection in Single Images using Multiview Bootstrapping},
author={Simon, Tomas and Joo, Hanbyul and Sheikh, Yaser},
journal={CVPR},
year={2017} }

@inproceedings{joo2019ssp,
  title={Towards Social Artificial Intelligence: Nonverbal Social Signal Prediction in A Triadic Interaction},
  author={Joo, Hanbyul and Simon, Tomas and Cikara, Mina and Sheikh, Yaser},
  booktitle={CVPR},
  year={2019}
}



Comments
  • Script returns

    Script returns "404 Not Found" when trying to download keypoint data

    When attempting to use the getData.sh script to download data, a "404 Not Found" error is returned for the keypoint data files (including hdPose3d, hdHand3d, and hdFoot3d). This happens with the sampleData dataset and every other data set I tested. The videos download fine.

    If I revert the getData.sh script to a version from a week ago (undoing the last change), I do get a hdPose3d_stage1.tar (which is no longer downloaded by the new script), but the hdPose3d_stage1_coco19.tar returns a 404 Not Found error.

    The errors are as follows:

    --2018-07-19 16:27:34-- http://domedb.perception.cs.cmu.edu/webdata/dataset/sampleData/hdPose3d_stage1_coco19.tar Resolving domedb.perception.cs.cmu.edu (domedb.perception.cs.cmu.edu)... 128.2.177.92 Connecting to domedb.perception.cs.cmu.edu (domedb.perception.cs.cmu.edu)|128.2.177.92|:80... connected. HTTP request sent, awaiting response... 404 Not Found 2018-07-19 16:27:34 ERROR 404: Not Found.

    rm: hdPose3d_stage1_coco19.tar: No such file or directory --2018-07-19 16:27:34-- http://domedb.perception.cs.cmu.edu/webdata/dataset/sampleData/hdFace3d.tar Resolving domedb.perception.cs.cmu.edu (domedb.perception.cs.cmu.edu)... 128.2.177.92 Connecting to domedb.perception.cs.cmu.edu (domedb.perception.cs.cmu.edu)|128.2.177.92|:80... connected. HTTP request sent, awaiting response... 404 Not Found 2018-07-19 16:27:34 ERROR 404: Not Found.

    rm: hdFace3d.tar: No such file or directory --2018-07-19 16:27:34-- http://domedb.perception.cs.cmu.edu/webdata/dataset/sampleData/hdHand3d.tar Resolving domedb.perception.cs.cmu.edu (domedb.perception.cs.cmu.edu)... 128.2.177.92 Connecting to domedb.perception.cs.cmu.edu (domedb.perception.cs.cmu.edu)|128.2.177.92|:80... connected. HTTP request sent, awaiting response... 404 Not Found 2018-07-19 16:27:34 ERROR 404: Not Found.

    rm: hdHand3d.tar: No such file or directory

    opened by thully 7
  • Potentially incorrect joint labels in `170228_haggling_a2`

    Potentially incorrect joint labels in `170228_haggling_a2`

    https://github.com/CMU-Perceptual-Computing-Lab/panoptic-toolbox

    Hello,

    The skeleton definitions for the left and right eyes and ears do not quite match up with what you've started in your repo. Is there another format I should be considering for hdPose3d_stage1_coco19?

    From my understanding, the ordering is as follows:

    ''' 0: Neck 1: Nose 2: BodyCenter (center of hips) 3: lShoulder 4: lElbow 5: lWrist, 6: lHip 7: lKnee 8: lAnkle 9: rShoulder 10: rElbow 11: rWrist 12: rHip 13: rKnee 14: rAnkle 15: lEye 16: lEar 17: rEye 18: rEar '''

    opened by RSKothari 5
  • Depth images

    Depth images

    Hi, Im trying to download the kinect depth maps. I followed http://domedb.perception.cs.cmu.edu/171204_pose1.html using /getData.sh 171204_pose1 but the command doesn't download the depth maps.

    How can I Download all data from kinect sensors. Thanks

    opened by erezposner 4
  • Aligning depth and color images

    Aligning depth and color images

    Hey folks, Thanks for the great work! I am trying to align the Kinect depth and color images using the intrinsic of color and depth camera. So far, I have not been successful. It generates the misaligned color and depth images. Please see the Matlab script for the alignment and the generated images. It would be very helpful if you can help me to resolve the issue.

    color depth fused

    align_kineoptic.zip

    opened by vinkle 4
  • tar file download is empty once opened in one of the downloads from getData.sh

    tar file download is empty once opened in one of the downloads from getData.sh

    hdPose3d_stage1_coco19.tar is empty in 60422_haggling1.

    The tar file takes up 3.8MB on hard disk space, but open opening, it is empty. I am making this an issue as it may be indicative of a larger problem. image

    opened by Emnolope 3
  • Total Capture

    Total Capture

    Hi! I hope you are doing great and once again thanks for your work! I was wondering if you were planning to release some meshes based on your "Total Capture" paper for some of the scenes or trained Frankestein/Adam models ?

    Thanks!

    opened by Kernelux 3
  • The problem of demo_kinoptic_gen_ptcloud.m

    The problem of demo_kinoptic_gen_ptcloud.m

    demo_kinoptic_gen_ptcloud.m When the above program is executed, hints appear on the screen as follows: PLY files will be saved in: F:/panoptic-toolbox/matlab/160422_haggling1/kinoptic_ptclouds hd_index: 102, UnivTime: 44295.918 idk: 1, -1661.1570 Skipping 1 idk: 2, -1668.9130 Skipping 2.776500e+01, depth-color diff idk: 3, -1641.4520 Skipping 3 idk: 4, -1645.5540 Skipping 4 idk: 5, -1635.2900 Skipping 5 idk: 6, -1664.9270 Skipping 2.665800e+01, depth-color diff idk: 7, -1635.9900 Skipping 7 idk: 8, -1635.0910 Skipping 8 idk: 9, -1664.5840 Skipping 2.658900e+01, depth-color diff idk: 10, -1657.6420 Skipping 10 But when I open the folder F:/panoptic-toolbox/matlab/160422_haggling1/kinoptic_ptclouds I found that This folder is empty, and there is no PLY file.

    So what's the problem? How to modify the problem? Thank you!

    opened by postgraduater 3
  • Keypoint order in annotation

    Keypoint order in annotation

    It seems to me that there is an error in README about the order of kpts annotation. I think you used the following: 0: "Neck", 1: "Nose", 2: "MidHip", 3: "LShoulder", 4: "LElbow", 5: "LWrist", 6: "LHip", 7: "LKnee", 8: "LAnkle", 9: "RShoulder", 10: "RElbow", 11: "RWrist", 12: "RHip", 13: "RKnee", 14: "RAnkle", 15: "LEye", 16: "LEar", 17: "REye", 18: "REar",

    opened by elmuz 2
  • Does anyone have the transcripts and speaker ID annotations?

    Does anyone have the transcripts and speaker ID annotations?

    The website said that "they will be available soon". Are these hidden somewhere that I didn't manage to find, or were they not yet released?

    opened by steph-tan 2
  • Sensor information

    Sensor information

    Hello, thank you for the nice toolbox. I would like to have some more information about the dataset, in particular either the HD camera/sensor model(s) or the sensor size in millimeters. Since I didn't find them in the paper, is it possible to know them? Thank you kindly

    opened by A7ocin 2
  • 160906_band4 ksynctable file's url error

    160906_band4 ksynctable file's url error

    Hi, could you please update the URL of the ksynctables.json file of the scene "160906_band4"? I cannot download the file from the url: http://domedb.perception.cs.cmu.edu/webdata/dataset/160906_band4/kinect_shared_depth/ksynctables.json. Thank you!

    opened by Yu-Yy 0
  • Focal Length Info

    Focal Length Info

    Hi, I was wondering what the focal length is for the hd cameras. I saw that someone mentioned it is a Canon XH-G1s but the effective pixels from that seem to be 1440x1080 but the video frames I extract from the hd videos are all 1920x1080. So I'm not certain they were the camera used. Maybe I just don't understand effective pixels though. Either which way, it would be very helpful to know the focal lengths, thanks!

    Also, the website seems to be inaccessible, I saw another post about this though so I didn't want to raise an issue.

    opened by KushalHaran 0
  • Failed to get the dataset

    Failed to get the dataset

    Hello! I encountered some problems and failed to get the dataset. I try to visit domedb.perception.cs.cmu.edu/ , however, it redirects me to https://domedb.perception.cs.cmu.edu:5001/, which needs to login.

    I also tried to get the dataset with the script ./scripts/getData.sh 171204_pose1_sample, and got the result as below:

    Connecting to domedb.perception.cs.cmu.edu (domedb.perception.cs.cmu.edu)|128.2.220.8|:5001... connected.
    ERROR: cannot verify domedb.perception.cs.cmu.edu's certificate, issued by '[email protected],CN=Synology Inc. CA,OU=Certificate Authority,O=Synology Inc.,L=Taipei,ST=Taiwan,C=TW':
      Unable to locally verify the issuer's authority.
        ERROR: certificate common name 'synology.com' doesn't match requested host name 'domedb.perception.cs.cmu.edu'.
    

    So how to get CMU Panoptic dataset? I am looking forward to your reply and thanks in advance!

    opened by Billccx 11
  • One video of one view is broken in 160906_band3

    One video of one view is broken in 160906_band3

    http://domedb.perception.cs.cmu.edu/webdata/dataset/160906_band3/videos/hd_shared_crf20/hd_00_03.mp4

    Please update it. I can't extract images from it.

    Thx!

    opened by Kairobo 0
  • 2D Data

    2D Data

    I downloaded hdPose3d_stage1_coco19.tar, but this file only has 3D body keypoints. Do you have the data of 2D body keypoints?Or how can I get 2D body keypoints?

    opened by TDIT-haha 1
Python scripts for performing 3D human pose estimation using the Mobile Human Pose model in ONNX.

Python scripts for performing 3D human pose estimation using the Mobile Human Pose model in ONNX.

Ibai Gorordo 99 Dec 31, 2022
PoseViz – Multi-person, multi-camera 3D human pose visualization tool built using Mayavi.

PoseViz – 3D Human Pose Visualizer Multi-person, multi-camera 3D human pose visualization tool built using Mayavi. As used in MeTRAbs visualizations.

István Sárándi 79 Dec 30, 2022
SE3 Pose Interp - Interpolate camera pose or trajectory in SE3, pose interpolation, trajectory interpolation

SE3 Pose Interpolation Pose estimated from SLAM system are always discrete, and

Ran Cheng 4 Dec 15, 2022
Repository for the paper "PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation", CVPR 2021.

PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation Code repository for the paper: PoseAug: A Differentiable Pose Augme

Pyjcsx 328 Dec 17, 2022
Official PyTorch implementation of "Camera Distance-aware Top-down Approach for 3D Multi-person Pose Estimation from a Single RGB Image", ICCV 2019

PoseNet of "Camera Distance-aware Top-down Approach for 3D Multi-person Pose Estimation from a Single RGB Image" Introduction This repo is official Py

Gyeongsik Moon 677 Dec 25, 2022
Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors, CVPR 2021

Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors Human POSEitioning System (H

Aymen Mir 66 Dec 21, 2022
Simple Pose: Rethinking and Improving a Bottom-up Approach for Multi-Person Pose Estimation

SimplePose Code and pre-trained models for our paper, “Simple Pose: Rethinking and Improving a Bottom-up Approach for Multi-Person Pose Estimation”, a

Jia Li 256 Dec 24, 2022
[ICCV 2021] Encoder-decoder with Multi-level Attention for 3D Human Shape and Pose Estimation

MAED: Encoder-decoder with Multi-level Attention for 3D Human Shape and Pose Estimation Getting Started Our codes are implemented and tested with pyth

ZiNiU WaN 176 Dec 15, 2022
MHFormer: Multi-Hypothesis Transformer for 3D Human Pose Estimation

MHFormer: Multi-Hypothesis Transformer for 3D Human Pose Estimation This repo is the official implementation of "MHFormer: Multi-Hypothesis Transforme

Vegetabird 281 Jan 7, 2023
Neural Reprojection Error: Merging Feature Learning and Camera Pose Estimation

Neural Reprojection Error: Merging Feature Learning and Camera Pose Estimation This is the official repository for our paper Neural Reprojection Error

Hugo Germain 78 Dec 1, 2022
Camera calibration & 3D pose estimation tools for AcinoSet

AcinoSet: A 3D Pose Estimation Dataset and Baseline Models for Cheetahs in the Wild Daniel Joska, Liam Clark, Naoya Muramatsu, Ricardo Jericevich, Fre

African Robotics Unit 42 Nov 16, 2022
Code for "Multi-View Multi-Person 3D Pose Estimation with Plane Sweep Stereo"

Multi-View Multi-Person 3D Pose Estimation with Plane Sweep Stereo This repository includes the source code for our CVPR 2021 paper on multi-view mult

Jiahao Lin 66 Jan 4, 2023
Code for ICCV2021 paper SPEC: Seeing People in the Wild with an Estimated Camera

SPEC: Seeing People in the Wild with an Estimated Camera [ICCV 2021] SPEC: Seeing People in the Wild with an Estimated Camera, Muhammed Kocabas, Chun-

Muhammed Kocabas 187 Dec 26, 2022
《Unsupervised 3D Human Pose Representation with Viewpoint and Pose Disentanglement》(ECCV 2020) GitHub: [fig9]

Unsupervised 3D Human Pose Representation [Paper] The implementation of our paper Unsupervised 3D Human Pose Representation with Viewpoint and Pose Di

null 42 Nov 24, 2022
This repository contains codes of ICCV2021 paper: SO-Pose: Exploiting Self-Occlusion for Direct 6D Pose Estimation

SO-Pose This repository contains codes of ICCV2021 paper: SO-Pose: Exploiting Self-Occlusion for Direct 6D Pose Estimation This paper is basically an

shangbuhuan 52 Nov 25, 2022
The project is an official implementation of our CVPR2019 paper "Deep High-Resolution Representation Learning for Human Pose Estimation"

Deep High-Resolution Representation Learning for Human Pose Estimation (CVPR 2019) News [2020/07/05] A very nice blog from Towards Data Science introd

Leo Xiao 3.9k Jan 5, 2023
Human head pose estimation using Keras over TensorFlow.

RealHePoNet: a robust single-stage ConvNet for head pose estimation in the wild.

Rafael Berral Soler 71 Jan 5, 2023
Deep Dual Consecutive Network for Human Pose Estimation (CVPR2021)

Deep Dual Consecutive Network for Human Pose Estimation (CVPR2021) Introduction This is the official code of Deep Dual Consecutive Network for Human P

null 295 Dec 29, 2022
Bottom-up Human Pose Estimation

Introduction This is the official code of Rethinking the Heatmap Regression for Bottom-up Human Pose Estimation. This paper has been accepted to CVPR2

null 108 Dec 1, 2022