Code for "Multi-View Multi-Person 3D Pose Estimation with Plane Sweep Stereo"

Jiahao Lin

Last update: Jan 4, 2023

Related tags

Deep Learning PlaneSweepPose

Overview

Multi-View Multi-Person 3D Pose Estimation with Plane Sweep Stereo

This repository includes the source code for our CVPR 2021 paper on multi-view multi-person 3D pose estimation. Please read our paper for more details at https://arxiv.org/abs/2104.02273. The project webpage is available here.

Bibtex:

@InProceedings{Lin_2021_CVPR,
    author    = {Lin, Jiahao and Lee, Gim Hee},
    title     = {Multi-View Multi-Person 3D Pose Estimation With Plane Sweep Stereo},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2021},
    pages     = {11886-11895}
}

Environment

Our code is tested on

Python 3.8.5
PyTorch 1.6.0 & torchvision 0.7.0
CUDA 11.2

Preparing Data

Download following data before using the code in this repository:

Annotations and 2D pose predictions for the Campus and the Shelf datasets can be downloaded here. Credit to VoxelPose.
Follow the instructions on the CMU Panoptic Github repo to download the annotations. 2D pose predictions can be downloaded here.
Pre-trained models can be downloaded here.

The data should be organized as follows:

    ROOTDIR/
        └── data/
            └── Campus/
                └── actorsGT.mat
                └── calibration_campus.json
                └── pred_campus_maskrcnn_hrnet_coco.pkl
            └── Shelf/
                └── actorsGT.mat
                └── calibration_shelf.json
                └── pred_shelf_maskrcnn_hrnet_coco.pkl
            └── Panoptic/
                └── 160224_haggling1/
                └── 160226_haggling1/
                └── ...
                └── keypoints_train_results.json
                └── keypoints_validation_results.json
            └── panoptic_training_pose.pkl
        └── output/
            └── campus_synthetic/mvmppe/config/model_best_pretrained.pth.tar
            └── shelf_synthetic/mvmppe/config/model_best_pretrained.pth.tar
            └── panoptic/mvmppe/config/model_best_pretrained.pth.tar
        └── ...

Training and Inference

Below are the commands for training our model on different datasets.

The Campus dataset:

    python run/train.py --cfg configs/campus/config.yaml

The Shelf dataset:

    python run/train.py --cfg configs/shelf/config.yaml

The CMU Panoptic dataset:

    python run/train.py --cfg configs/panoptic/config.yaml

Below are the commands for performing inference with our pre-trained models.

The Campus dataset:

    python run/validate.py --cfg configs/campus/config.yaml -t pretrained

The Shelf dataset:

    python run/validate.py --cfg configs/shelf/config.yaml -t pretrained

The CMU Panoptic dataset:

    python run/validate.py --cfg configs/panoptic/config.yaml -t pretrained

Comments

When to release the code

Hi, thanks for your inspirational work on multi-view pose! Is there any plan to release the code as well as the pretrained models and 2D pose results?

opened by wusize 2
Can the trained model be transfered into another scene?

Hi thanks for your amazing work, supposing I have 2 calibrated cameras, could I use a trained model from the synthetic campus dataset into a different scene, or should I rather retrain the model with different voxel space size and calibration parameters?

opened by gpastal24 4
Will you release a demo code that helps in visualizing the 3D human pose estimation?

Hi, How do I get the projection of the estimated 3D pose in each camera view (like in Figure 5). Can you please post the code for this?Hope for your help.

opened by IncludeHappy 7
Further explanation on the extrinsic parameters used in the campus dataset

Can you explain how did you obtain the translation parameter (T) in the calibration_campus.json? What special consideration should be made to obtain such values? How should they be interpreted?

From the original calib info in the campus dataset, I got T = [-1.787557e+00, 1.361094e+00, 5.226973e+00] for the cam 0 . But the T you provide in the json is way too different T=[1774.89, -5051.69, 1923.35]. It is the same for all three cameras, I am just mentioning cam0 as an example.

When I used the calib data you provide I was able to obtain 3d poses successfully, but I cannot plot the camera system coherently. When I use the original T parameters, I can plot the camera system but the model inference doesn't work.

I would appreciate if you can elaborate more on it

Thank you for your time!! And nice work, by the way

opened by AntonioEscamilla 1
The performance in Shelf

I used Shelf trained model to evaluate the Shelf, and get the pose results in https://github.com/jiahaoLjh/PlaneSweepPose/blob/35d8bc32155dc0488c0a8b37de1afa8a5069117e/lib/dataset/shelf.py#L332 , and then visualized it showed above.

The result shows a lot of noise, I don't know if this results is the same as paper shows, or I implemented incorrectly.

opened by baojunshan 1
Error when any two cameras have different numbers of people

Hi, I have run your code on the dataset Shelf. However, when the target camera and reference camera have different numbers of people, (e.g. there are 3 people in the target camera and there are 2 people in the reference camera), an error will happen in line 65 of mvmppe.py file.

poses_dist = torch.sum((pt - pr) ** 2, dim=-1) # [B, Npt, Npf, Nj, Nd]

Is there any solution to solve this problem?

opened by letian-zhang 3

Owner

Jiahao Lin

GitHub

TensorFlow code for the neural network presented in the paper: "Structural Language Models of Code" (ICML'2020)

SLM: Structural Language Models of Code This is an official implementation of the model described in: "Structural Language Models of Code" [PDF] To ap

73 Nov 6, 2022

Inference code for "StylePeople: A Generative Model of Fullbody Human Avatars" paper. This code is for the part of the paper describing video-based avatars.

NeuralTextures This is repository with inference code for paper "StylePeople: A Generative Model of Fullbody Human Avatars" (CVPR21). This code is for

Visual Understanding Lab @ Samsung AI Center Moscow

18 Oct 6, 2022

A code generator from ONNX to PyTorch code

onnx-pytorch Generating pytorch code from ONNX. Currently support onnx==1.9.0 and torch==1.8.1. Installation From PyPI pip install onnx-pytorch From

94 Jan 6, 2023

This is the code for our KILT leaderboard submission to the T-REx and zsRE tasks. It includes code for training a DPR model then continuing training with RAG.

KGI (Knowledge Graph Induction) for slot filling This is the code for our KILT leaderboard submission to the T-REx and zsRE tasks. It includes code fo

72 Jan 6, 2023

Convert Python 3 code to CUDA code.

Py2CUDA Convert python code to CUDA. Usage To convert a python file say named py_file.py to CUDA, run python generate_cuda.py --file py_file.py --arch

3 Jul 14, 2021

Empirical Study of Transformers for Source Code & A Simple Approach for Handling Out-of-Vocabulary Identifiers in Deep Learning for Source Code

Transformers for variable misuse, function naming and code completion tasks The official PyTorch implementation of: Empirical Study of Transformers fo

56 Nov 15, 2022

Reference implementation of code generation projects from Facebook AI Research. General toolkit to apply machine learning to code, from dataset creation to model training and evaluation. Comes with pretrained models.

This repository is a toolkit to do machine learning for programming languages. It implements tokenization, dataset preprocessing, model training and m

408 Jan 1, 2023

Code for the prototype tool in our paper "CoProtector: Protect Open-Source Code against Unauthorized Training Usage with Data Poisoning".

CoProtector Code for the prototype tool in our paper "CoProtector: Protect Open-Source Code against Unauthorized Training Usage with Data Poisoning".

1 Oct 26, 2021

Low-code/No-code approach for deep learning inference on devices

EzEdgeAI A concept project that uses a low-code/no-code approach to implement deep learning inference on devices. It provides a componentized framewor

7 Apr 5, 2022

Code for all the Advent of Code'21 challenges mostly written in python

Advent of Code 21 Code for all the Advent of Code'21 challenges mostly written in python. They are not necessarily the best or fastest solutions but j

4 May 26, 2022

Code to use Augmented Shapiro Wilks Stopping, as well as code for the paper "Statistically Signifigant Stopping of Neural Network Training"

This codebase is being actively maintained, please create and issue if you have issues using it Basics All data files are included under losses and ea

32 Nov 9, 2021

Opinionated code formatter, just like Python's black code formatter but for Beancount

beancount-black Opinionated code formatter, just like Python's black code formatter but for Beancount Try it out online here Features MIT licensed - b

16 Oct 11, 2022

a delightful machine learning tool that allows you to train, test and use models without writing code

igel A delightful machine learning tool that allows you to train/fit, test and use models without writing code Note I'm also working on a GUI desktop

3k Jan 5, 2023

Pytorch Lightning code guideline for conferences

Deep learning project seed Use this seed to start new deep learning / ML projects. Built in setup.py Built in requirements Examples with MNIST Badges

1k Jan 2, 2023

Automatically Build Multiple ML Models with a Single Line of Code. Created by Ram Seshadri. Collaborators Welcome. Permission Granted upon Request.

Auto-ViML Automatically Build Variant Interpretable ML models fast! Auto_ViML is pronounced "auto vimal" (autovimal logo created by Sanket Ghanmare) N

397 Dec 30, 2022

Applications using the GTN library and code to reproduce experiments in "Differentiable Weighted Finite-State Transducers"

gtn_applications An applications library using GTN. Current examples include: Offline handwriting recognition Automatic speech recognition Installing

68 Dec 29, 2022

Code for "Multi-View Multi-Person 3D Pose Estimation with Plane Sweep Stereo"

Related tags

Overview

Multi-View Multi-Person 3D Pose Estimation with Plane Sweep Stereo

Environment

Preparing Data

Training and Inference

Comments

When to release the code

Can the trained model be transfered into another scene?

Will you release a demo code that helps in visualizing the 3D human pose estimation?

Further explanation on the extrinsic parameters used in the campus dataset

The performance in Shelf

Error when any two cameras have different numbers of people

Owner

Jiahao Lin

TensorFlow code for the neural network presented in the paper: "Structural Language Models of Code" (ICML'2020)

Inference code for "StylePeople: A Generative Model of Fullbody Human Avatars" paper. This code is for the part of the paper describing video-based avatars.

A code generator from ONNX to PyTorch code

This is the code for our KILT leaderboard submission to the T-REx and zsRE tasks. It includes code for training a DPR model then continuing training with RAG.

Convert Python 3 code to CUDA code.

Empirical Study of Transformers for Source Code & A Simple Approach for Handling Out-of-Vocabulary Identifiers in Deep Learning for Source Code

Reference implementation of code generation projects from Facebook AI Research. General toolkit to apply machine learning to code, from dataset creation to model training and evaluation. Comes with pretrained models.

Code for the prototype tool in our paper "CoProtector: Protect Open-Source Code against Unauthorized Training Usage with Data Poisoning".

Low-code/No-code approach for deep learning inference on devices

Code for all the Advent of Code'21 challenges mostly written in python

Code to use Augmented Shapiro Wilks Stopping, as well as code for the paper "Statistically Signifigant Stopping of Neural Network Training"

Opinionated code formatter, just like Python's black code formatter but for Beancount

a delightful machine learning tool that allows you to train, test and use models without writing code

Pytorch Lightning code guideline for conferences

Automatically Build Multiple ML Models with a Single Line of Code. Created by Ram Seshadri. Collaborators Welcome. Permission Granted upon Request.

Code samples for my book "Neural Networks and Deep Learning"

Code for: https://berkeleyautomation.github.io/bags/

Code for our method RePRI for Few-Shot Segmentation. Paper at http://arxiv.org/abs/2012.06166

Applications using the GTN library and code to reproduce experiments in "Differentiable Weighted Finite-State Transducers"