Bottom-up Human Pose Estimation

Overview

Introduction

This is the official code of Rethinking the Heatmap Regression for Bottom-up Human Pose Estimation. This paper has been accepted to CVPR2021.

This repo is built on Bottom-up-Higher-HRNet.

Main Results

Results on COCO val2017 without multi-scale test

Method Backbone Input size #Params GFLOPs AP Ap .5 AP .75 AP (M) AP (L)
HigherHRNet HRNet-w32 512 28.6M 47.9 67.1 86.2 73.0 61.5 76.1
HigherHRNet + SWAHR HRNet-w32 512 28.6M 48.0 68.9 87.8 74.9 63.0 77.4
HigherHRNet HRNet-w48 640 63.8M 154.3 69.9 87.2 76.1 65.4 76.4
HigherHRNet + SWAHR HRNet-w48 640 63.8M 154.6 70.8 88.5 76.8 66.3 77.4

Results on COCO val2017 with multi-scale test

Method Backbone Input size #Params GFLOPs AP Ap .5 AP .75 AP (M) AP (L)
HigherHRNet HRNet-w32 512 28.6M 47.9 69.9 87.1 76.0 65.3 77.0
HigherHRNet + SWAHR HRNet-w32 512 28.6M 48.0 71.4 88.9 77.8 66.3 78.9
HigherHRNet HRNet-w48 640 63.8M 154.3 72.1 88.4 78.2 67.8 78.3
HigherHRNet + SWAHR HRNet-w48 640 63.8M 154.6 73.2 89.8 79.1 69.1 79.3

Results on COCO test-dev2017 without multi-scale test

Method Backbone Input size #Params GFLOPs AP Ap .5 AP .75 AP (M) AP (L)
OpenPose* - - - - 61.8 84.9 67.5 57.1 68.2
Hourglass Hourglass 512 277.8M 206.9 56.6 81.8 61.8 49.8 67.0
PersonLab ResNet-152 1401 68.7M 405.5 66.5 88.0 72.6 62.4 72.3
PifPaf - - - - 66.7 - - 62.4 72.9
Bottom-up HRNet HRNet-w32 512 28.5M 38.9 64.1 86.3 70.4 57.4 73.9
HigherHRNet HRNet-w32 512 28.6M 47.9 66.4 87.5 72.8 61.2 74.2
HigherHRNet + SWAHR HRNet-w32 512 28.6M 48.0 67.9 88.9 74.5 62.4 75.5
HigherHRNet HRNet-w48 640 63.8M 154.3 68.4 88.2 75.1 64.4 74.2
HigherHRNet + SWAHR HRNet-w48 640 63.8M 154.6 70.2 89.9 76.9 65.2 77.0

Results on COCO test-dev2017 with multi-scale test

Method Backbone Input size #Params GFLOPs AP Ap .5 AP .75 AP (M) AP (L)
Hourglass Hourglass 512 277.8M 206.9 63.0 85.7 68.9 58.0 70.4
Hourglass* Hourglass 512 277.8M 206.9 65.5 86.8 72.3 60.6 72.6
PersonLab ResNet-152 1401 68.7M 405.5 68.7 89.0 75.4 64.1 75.5
HigherHRNet HRNet-w48 640 63.8M 154.3 70.5 89.3 77.2 66.6 75.8
HigherHRNet + SWAHR HRNet-w48 640 63.8M 154.6 72.0 90.7 78.8 67.8 77.7

Results on CrowdPose test

Method AP Ap .5 AP .75 AP (E) AP (M) AP (H)
Mask-RCNN 57.2 83.5 60.3 69.4 57.9 45.8
AlphaPose 61.0 81.3 66.0 71.2 61.4 51.1
SPPE 66.0. 84.2 71.5 75.5 66.3 57.4
OpenPose - - - 62.7 48.7 32.3
HigherHRNet 65.9 86.4 70.6 73.3 66.5 57.9
HigherHRNet + SWAHR 71.6 88.5 77.6 78.9 72.4 63.0
HigherHRNet* 67.6 87.4 72.6 75.8 68.1 58.9
HigherHRNet + SWAHR* 73.8 90.5 79.9 81.2 74.7 64.7

'*' indicates multi-scale test

Installation

The details about preparing the environment and datasets can be referred to README.md.

Downlaod our pretrained weights from BaidunYun(Password: 8weh) or GoogleDrive to ./models.

Training and Testing

Testing on COCO val2017 dataset using pretrained weights

For single-scale testing:

python tools/dist_valid.py \
    --cfg experiments/coco/higher_hrnet/w32_512_adam_lr1e-3.yaml \
    TEST.MODEL_FILE models/pose_coco/pose_higher_hrnet_w32_512.pth

By default, we use horizontal flip. To test without flip:

python tools/dist_valid.py \
    --cfg experiments/coco/higher_hrnet/w32_512_adam_lr1e-3.yaml \
    TEST.MODEL_FILE models/pose_coco/pose_higher_hrnet_w32_512.pth \
    TEST.FLIP_TEST False

Multi-scale testing is also supported, although we do not report results in our paper:

python tools/dist_valid.py \
    --cfg experiments/coco/higher_hrnet/w32_512_adam_lr1e-3.yaml \
    TEST.MODEL_FILE models/pose_coco/pose_higher_hrnet_w32_512.pth \
    TEST.SCALE_FACTOR '[0.5, 1.0, 2.0]'

Training on COCO train2017 dataset

python tools/dist_train.py \
    --cfg experiments/coco/higher_hrnet/w32_512_adam_lr1e-3.yaml 

By default, it will use all available GPUs on the machine for training. To specify GPUs, use

CUDA_VISIBLE_DEVICES=0,1 python tools/dist_train.py \
    --cfg experiments/coco/higher_hrnet/w32_512_adam_lr1e-3.yaml 

Testing on your own images

python tools/dist_inference.py \
    --img_dir path/to/your/directory/of/images \
    --save_dir path/where/results/are/saved \
    --cfg experiments/coco/higher_hrnet/w32_512_adam_lr1e-3.yaml \
    TEST.MODEL_FILE models/pose_coco/pose_higher_hrnet_w32_512.pth \
    TEST.SCALE_FACTOR '[0.5, 1.0, 2.0]'

Citation

If you find this work or code is helpful in your research, please cite:

@inproceedings{LuoSWAHR,
  title={Rethinking the Heatmap Regression for Bottom-up Human Pose Estimation},
  author={Zhengxiong Luo and Zhicheng Wang and Yan Huang and Liang Wang and Tieniu Tan and Erjin Zhou},
  booktitle={CVPR},
  year={2021}
}
Comments
  • training batchsize

    training batchsize

    hello, how many gpus did you used? I found that your batchsize maybe is different from the original HigherHRNet, can you share the batchsize you used for training? Thanks a lot!

    opened by syusukee 3
  • How can I use my images to infer skeleton on your pre-trained models

    How can I use my images to infer skeleton on your pre-trained models

    When I run python tools/dist_inference.py \ --img_dir path/to/your/directory/of/images \ --save_dir path/where/results/are/saved \ --cfg experiments/coco/higher_hrnet/w32_512_adam_lr1e-3.yaml \ TEST.MODEL_FILE models/pose_coco/pose_higher_hrnet_w32_512.pth \ TEST.SCALE_FACTOR '[0.5, 1.0, 2.0]' It occurs that RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU. Besides, there are another problem Process Process-1: Traceback (most recent call last): File "/home/ubuntu/anaconda3/envs/mykaolin/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/home/ubuntu/anaconda3/envs/mykaolin/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "tools/dist_inference.py", line 123, in worker model.load_state_dict(torch.load(cfg.TEST.MODEL_FILE), strict=True) File "/home/ubuntu/anaconda3/envs/mykaolin/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1052, in load_state_dict self.__class__.__name__, "\n\t".join(error_msgs))) RuntimeError: Error(s) in loading state_dict for PoseHigherResolutionNet: Missing key(s) in state_dict: "conv_scale_layers.0.0.weight", "conv_scale_layers.0.0.bias", "conv_scale_layers.0.1.weight", "conv_scale_layers.0.1.bias", "conv_scale_layers.0.1.running_mean", "conv_scale_layers.0.1.running_var", "conv_scale_layers.0.3.weight", "conv_scale_layers.0.3.bias", "conv_scale_layers.1.0.weight", "conv_scale_layers.1.0.bias", "conv_scale_layers.1.1.weight", "conv_scale_layers.1.1.bias", "conv_scale_layers.1.1.running_mean", "conv_scale_layers.1.1.running_var", "conv_scale_layers.1.3.weight", "conv_scale_layers.1.3.bias". I am confused why GPU doesn't work.

    opened by Yamato-01 2
  • error : missing draw_skeleton.py from utils.vis

    error : missing draw_skeleton.py from utils.vis

    I get this error when i try to use dist_inference , seems like there is a file missing or did I failed the installation?

    Traceback (most recent call last):
      File "/home/kubler/data/HPE/SWAHR-HumanPose/tools/dist_inference.py", line 42, in <module>
        from utils.vis import draw_skeleton
    ImportError: cannot import name 'draw_skeleton' from 'utils.vis' (/home/kubler/data/HPE/SWAHR-HumanPose/tools/../lib/utils/vis.py)```
    
    
    Thanks for your help!
    
    Greg
    opened by greg-is-kub 2
  • One concern in the algorithm

    One concern in the algorithm

    As we can see, the scale network branch directly regresses alpha = 1/s-1. When using Taylor expansion as shown in Eq. 8, in fact, due to sigma_0 has changed to ssigma_0, the coverage of gaussian should change from 3sigma to 3s* sigma, where is not hold in Eq. 8.

    opened by AlanLuSun 2
  • Is it necessary to add nn.ReLU(True) in the fuse_layers?

    Is it necessary to add nn.ReLU(True) in the fuse_layers?

    Thanks for releasing the code. By comparing SWAHR-HumanPose and HigherHRNet, I notice that some nn.ReLU layers are added in the fuse_layers. I wonder what if we do not modify the original backbone network. Will it affect the final performance?

    opened by jin-s13 2
  • When training, 'torch.multiprocessing.spawn.ProcessRaisedException' occured.

    When training, 'torch.multiprocessing.spawn.ProcessRaisedException' occured.

    My Environment is Ubuntu 18.04LTS, pytorch-1.10, RTX2080Ti, cuda 11.0.

    Traceback (most recent call last): File "tools/dist_train.py", line 320, in main() File "tools/dist_train.py", line 115, in main args=(ngpus_per_node, args, final_output_dir, tb_log_dir) File "/home/lab/.conda/envs/pytorch-1.10/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 230, in spawn return start_processes(fn, args, nprocs, join, daemon, start_method='spawn') File "/home/lab/.conda/envs/pytorch-1.10/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 188, in start_processes while not context.join(): File "/home/lab/.conda/envs/pytorch-1.10/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 150, in join raise ProcessRaisedException(msg, error_index, failed_process.pid) torch.multiprocessing.spawn.ProcessRaisedException:

    -- Process 0 terminated with the following error: Traceback (most recent call last): File "/home/lab/.conda/envs/pytorch-1.10/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap fn(i, *args) File "/media/hdd/xxy/swahr/tools/dist_train.py", line 161, in main_worker rank=args.rank File "/home/lab/.conda/envs/pytorch-1.10/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 576, in init_process_group store, rank, world_size = next(rendezvous_iterator) File "/home/lab/.conda/envs/pytorch-1.10/lib/python3.7/site-packages/torch/distributed/rendezvous.py", line 183, in _tcp_rendezvous_handler store = _create_c10d_store(result.hostname, result.port, rank, world_size, timeout) File "/home/lab/.conda/envs/pytorch-1.10/lib/python3.7/site-packages/torch/distributed/rendezvous.py", line 158, in _create_c10d_store hostname, port, world_size, start_daemon, timeout, multi_tenant=True RuntimeError: Address already in use

    opened by calmisential 1
  • questions about the paper

    questions about the paper

    In section 3.2, you introduced the scale-adaptive heatmaps, but in formula 4, you write s.t. ||i-x||<3σ ||j-y||<3σ, dose this σ refer to σ0 or σ0s? At the first, i thought this σ refer to σ0s, but in formula 6, you write when Hσ0 == 0, Hσ0s=Hσ0, which means the region whose value is not zero in Hσ0 and Hσ0s is the same, it's inconsistent with what you said "which means that the region convered by this Gaussian kernel will also become larger", so what are the facts?

    opened by SatMa34 1
  • about computing resource

    about computing resource

    Dear author: How many GPUs is used during training? How many days does it take to train for 300 epochs? I have 8 GPUs, but the training seems to take 3 weeks... Thank you!

    opened by TianyuLee 0
  • cuda

    cuda

    RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.

    my torch.cuda.is_available() return true,why it can not work .thanks

    opened by yuan243212790 1
  • The problem of training using the crowd pose dataset.

    The problem of training using the crowd pose dataset.

    When using the code to train the model using the crowd pose dataset without any changes in the code, why is the AP 62.7, which is not consistent with the result (AP=71.6) presented in this paper?

    opened by cici203 0
Owner
null
This is an official implementation of our CVPR 2021 paper "Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression" (https://arxiv.org/abs/2104.02300)

Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression Introduction In this paper, we are interested in the bottom-up paradigm of estima

HRNet 367 Dec 27, 2022
Python scripts for performing 3D human pose estimation using the Mobile Human Pose model in ONNX.

Python scripts for performing 3D human pose estimation using the Mobile Human Pose model in ONNX.

Ibai Gorordo 99 Dec 31, 2022
3D Multi-Person Pose Estimation by Integrating Top-Down and Bottom-Up Networks

3D Multi-Person Pose Estimation by Integrating Top-Down and Bottom-Up Networks Introduction This repository contains the code and models for the follo

null 124 Jan 6, 2023
Repository for the paper "PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation", CVPR 2021.

PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation Code repository for the paper: PoseAug: A Differentiable Pose Augme

Pyjcsx 328 Dec 17, 2022
Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors, CVPR 2021

Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors Human POSEitioning System (H

Aymen Mir 66 Dec 21, 2022
SE3 Pose Interp - Interpolate camera pose or trajectory in SE3, pose interpolation, trajectory interpolation

SE3 Pose Interpolation Pose estimated from SLAM system are always discrete, and

Ran Cheng 4 Dec 15, 2022
《Unsupervised 3D Human Pose Representation with Viewpoint and Pose Disentanglement》(ECCV 2020) GitHub: [fig9]

Unsupervised 3D Human Pose Representation [Paper] The implementation of our paper Unsupervised 3D Human Pose Representation with Viewpoint and Pose Di

null 42 Nov 24, 2022
This repository contains codes of ICCV2021 paper: SO-Pose: Exploiting Self-Occlusion for Direct 6D Pose Estimation

SO-Pose This repository contains codes of ICCV2021 paper: SO-Pose: Exploiting Self-Occlusion for Direct 6D Pose Estimation This paper is basically an

shangbuhuan 52 Nov 25, 2022
The project is an official implementation of our CVPR2019 paper "Deep High-Resolution Representation Learning for Human Pose Estimation"

Deep High-Resolution Representation Learning for Human Pose Estimation (CVPR 2019) News [2020/07/05] A very nice blog from Towards Data Science introd

Leo Xiao 3.9k Jan 5, 2023
Human head pose estimation using Keras over TensorFlow.

RealHePoNet: a robust single-stage ConvNet for head pose estimation in the wild.

Rafael Berral Soler 71 Jan 5, 2023
Deep Dual Consecutive Network for Human Pose Estimation (CVPR2021)

Deep Dual Consecutive Network for Human Pose Estimation (CVPR2021) Introduction This is the official code of Deep Dual Consecutive Network for Human P

null 295 Dec 29, 2022
HPRNet: Hierarchical Point Regression for Whole-Body Human Pose Estimation

HPRNet: Hierarchical Point Regression for Whole-Body Human Pose Estimation Official PyTroch implementation of HPRNet. HPRNet: Hierarchical Point Regre

Nermin Samet 53 Dec 4, 2022
A large-scale video dataset for the training and evaluation of 3D human pose estimation models

ASPset-510 ASPset-510 (Australian Sports Pose Dataset) is a large-scale video dataset for the training and evaluation of 3D human pose estimation mode

Aiden Nibali 36 Oct 30, 2022
A large-scale video dataset for the training and evaluation of 3D human pose estimation models

ASPset-510 (Australian Sports Pose Dataset) is a large-scale video dataset for the training and evaluation of 3D human pose estimation models. It contains 17 different amateur subjects performing 30 sports-related actions each, for a total of 510 action clips.

Aiden Nibali 25 Jun 20, 2021
The project is an official implementation of our paper "3D Human Pose Estimation with Spatial and Temporal Transformers".

3D Human Pose Estimation with Spatial and Temporal Transformers This repo is the official implementation for 3D Human Pose Estimation with Spatial and

Ce Zheng 363 Dec 28, 2022
Code for "Human Pose Regression with Residual Log-likelihood Estimation", ICCV 2021 Oral

Human Pose Regression with Residual Log-likelihood Estimation [Paper] [arXiv] [Project Page] Human Pose Regression with Residual Log-likelihood Estima

JeffLi 347 Dec 24, 2022
PyTorch implementation for 3D human pose estimation

Towards 3D Human Pose Estimation in the Wild: a Weakly-supervised Approach This repository is the PyTorch implementation for the network presented in:

Xingyi Zhou 579 Dec 22, 2022
A PyTorch toolkit for 2D Human Pose Estimation.

PyTorch-Pose PyTorch-Pose is a PyTorch implementation of the general pipeline for 2D single human pose estimation. The aim is to provide the interface

Wei Yang 1.1k Dec 30, 2022
This repository is the offical Pytorch implementation of ContextPose: Context Modeling in 3D Human Pose Estimation: A Unified Perspective (CVPR 2021).

Context Modeling in 3D Human Pose Estimation: A Unified Perspective (CVPR 2021) Introduction This repository is the offical Pytorch implementation of

null 37 Nov 21, 2022