PyMAF
This repository contains the code for the following paper:
3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop
Hongwen Zhang*, Yating Tian*, Xinchi Zhou, Wanli Ouyang, Yebin Liu, Limin Wang, Zhenan Sun
* Equal contribution
[Project Page] [ArXiv] [Paper]
Requirements
- Python 3.6.10
packages
-
PyTorch tested on version 1.1.0
-
torchvision tested on version 0.3.0
-
Neural Renderer (render densepose labels for training)
-
opendr (visualization in training)
-
pyrender (optional for demo)
-
other packages listed in
requirements.txt
necessary files
mesh_downsampling.npz & DensePose UV data
- Run the following script to fetch mesh_downsampling.npz & DensePose UV data from other repositories.
bash fetch_data.sh
SMPL model files
- Collect SMPL model files from https://smpl.is.tue.mpg.de and UP. Rename model files and put them into the
./data/smpldirectory.
Fetch preprocessed data from SPIN.
Download the pre-trained model and put it into the
./data/pretrained_modeldirectory.
After collecting the above necessary files, the directory structure of ./data is expected as follows.
./data
├── dataset_extras
│ └── .npz files
├── J_regressor_extra.npy
├── J_regressor_h36m.npy
├── mesh_downsampling.npz
├── pretrained_model
│ └── PyMAF_model_checkpoint.pt
├── smpl
│ ├── SMPL_FEMALE.pkl
│ ├── SMPL_MALE.pkl
│ └── SMPL_NEUTRAL.pkl
├── smpl_mean_params.npz
├── static_fits
│ └── .npy files
└── UV_data
├── UV_Processed.mat
└── UV_symmetry_transforms.mat
Demo
[UPDATE] You can first give it a try on Google Colab using the notebook we have prepared, which is no need to prepare the environment yourself:
Run the demo code.
python3 demo.py --checkpoint=data/pretrained_model/PyMAF_model_checkpoint.pt --vid_file ./flashmob.mp4
Frame by frame reconstruction. Video clipped from here.
Evaluation
Human3.6M / 3DPW
Run the evaluation code. Using --dataset to specify the evaluation dataset.
# Example usage:
# Human3.6M Protocol 2
python3 eval.py --checkpoint=data/pretrained_model/PyMAF_model_checkpoint.pt --dataset=h36m-p2 --log_freq=20
# 3DPW
python3 eval.py --checkpoint=data/pretrained_model/PyMAF_model_checkpoint.pt --dataset=3dpw --log_freq=20
COCO Keypoint Localization
-
Download the preprocessed data coco_2014_val.npz. Put it into the
./data/dataset_extrasdirectory. -
Run the COCO evaluation code.
python3 eval_coco.py --checkpoint=data/pretrained_model/PyMAF_model_checkpoint.pt
Training
To perform training, we need to collect preprocessed files of training datasets at first.
The preprocessed labels have the same format as SPIN and can be retrieved from here. Please refer to SPIN for more details about data preprocessing.
PyMAF is trained on Human3.6M at the first stage and then trained on the mixture of both 2D and 3D datasets at the second stage. Example usage:
# training on Human3.6M
python3 train.py --regressor pymaf_net --single_dataset --misc TRAIN.BATCH_SIZE 64
# training on mixed datasets
python3 train.py --regressor pymaf_net --pretrained_checkpoint path/to/checkpoint_file.pt --misc TRAIN.BATCH_SIZE 64
Running the above commands will use Human3.6M or mixed datasets for training, respectively. We can monitor the training process by setting up a TensorBoard at the directory ./logs.
Citation
If this work is helpful in your research, please cite the following paper.
@article{pymaf2021,
title={3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop},
author={Zhang, Hongwen and Tian, Yating and Zhou, Xinchi and Ouyang, Wanli and Liu, Yebin and Wang, Limin and Sun, Zhenan},
journal={arXiv preprint arXiv:2103.16507},
year={2021}
}
Acknowledgments
The code is developed upon the following projects. Many thanks to their contributions.
Is there any documentation about it?
Thanks for your attention!
73 Nov 6, 2022
18 Oct 6, 2022
94 Jan 6, 2023
72 Jan 6, 2023
3 Jul 14, 2021
56 Nov 15, 2022
408 Jan 1, 2023
1 Oct 26, 2021
7 Apr 5, 2022
4 May 26, 2022
32 Nov 9, 2021
16 Oct 11, 2022
3k Jan 5, 2023
1k Jan 2, 2023
397 Dec 30, 2022
13.9k Dec 26, 2022
121 Dec 30, 2022
138 Dec 12, 2022