[CVPR 2022] PoseTriplet: Co-evolving 3D Human Pose Estimation, Imitation, and Hallucination under Self-supervision (Oral)

Overview

PoseTriplet: Co-evolving 3D Human Pose Estimation, Imitation, and Hallucination under Self-supervision

Kehong Gong*, Bingbing Li*, Jianfeng Zhang*, Tao Wang*, Jing Huang, Bi Mi, Jiashi Feng, Xinchao Wang

CVPR 2022 (Oral Presentation, arxiv)

Logo

Framework

Pose-triplet contains three components: estimator, imitator and hallucinator

The three components form dual-loop during the training process, complementing and strengthening one another. alt text

Improvement through co-evolving

Here is imitated motion of different rounds, the estimator and imitator get improved over the rounds of training, and thus the imitated motion becomes more accurate and realistic from round 1 to 3. alt text

Video demo

04806-supp.mp4

Comparasion

Here we compared our results with two recent works Yu et al. and Hu et al.

Installation

  • Please refer to README_env.md for the python environment setup.

Data Preparation

Training

Please refer to script-summary for the training process. We also provide a checkpoint folder here with better performance, which support that this framework has the potential to reach the same performance as fully-supervised approaches.
Note: checkpoint for the RL policy is not include due to the size limitation, please following the training code to train the policy.

Inference

We provide an inference code here. Please follow the instruction and download the pretrained model for inference on videos.

Talk

Here is a slidestalk (PPT in english, speak in chinese).

Citation

If you find this code useful for your research, please consider citing the following paper:

@inproceedings{gong2022posetriplet,
  title      = {PoseTriplet: Co-evolving 3D Human Pose Estimation, Imitation, and Hallucination under Self-supervision},
  author     = {Gong, Kehong and Li, Bingbing and Zhang, Jianfeng and Wang, Tao and Huang, Jing and Mi, Michael Bi and Feng, Jiashi and Wang, Xinchao},
  booktitle  = {CVPR},
  year       = {2022}
}
Comments
  • Can this model inference single image?

    Can this model inference single image?

    I want to find out whether this model can inference online. So does it need the frame from future when it inference current frames? or like the question as title, can this model inference single image?

    opened by tiger990111 10
  • ValueError in Inference

    ValueError in Inference

    When I ran the inference script: python videopose-j16-wild-eval_run.py I got folloing errors. Could you help me?

    -------------- prepare video clip spends 0.03 seconds
    -------------- load keypoint spends 0.05 seconds
    Loading checkpoint ./checkpoint/ckpt_ep_045.bin
    -------------- load 3D model spends 3.81 seconds
    -------------- generate reconstruction 3D data spends 0.53 seconds
    Loading checkpoint ./checkpoint/ckpt_ep_045.bin
    -------------- load 3D Traj model spends 0.16 seconds
    -------------- generate reconstruction 3D data spends 0.02 seconds
    Rendering... save to ./wild_eval/333_scale2D_010/bilibili-clip/kunkun_clip_alpha_pose.mp4
    ===========================> This video get 49 frames in total.
      2%|##2                                                                                                          | 1/49 [00:00<00:10,  4.57it/s]Traceback (most recent call last):
      File "videopose-j16-wild-eval_run.py", line 288, in <module>
        Vis.redering()
      File "videopose-j16-wild-eval_run.py", line 44, in redering
        self.visalizatoin(anim_output)
      File "videopose-j16-wild-eval_run.py", line 232, in visalizatoin
        input_video_skip=args.viz_skip)
      File "/mnt/zhoudeyu/project/save_video/dengyuanzhang/posetriplet/PoseTriplet-main/estimator_inference/common/visualization.py", line 195, in render_animation
        anim.save(output, writer=writer)
      File "/root/miniconda3/envs/alphapose/lib/python3.6/site-packages/matplotlib/animation.py", line 1174, in save
        writer.grab_frame(**savefig_kwargs)
      File "/root/miniconda3/envs/alphapose/lib/python3.6/contextlib.py", line 99, in __exit__
        self.gen.throw(type, value, traceback)
      File "/root/miniconda3/envs/alphapose/lib/python3.6/site-packages/matplotlib/animation.py", line 232, in saving
        self.finish()
      File "/root/miniconda3/envs/alphapose/lib/python3.6/site-packages/matplotlib/animation.py", line 358, in finish
        self.cleanup()
      File "/root/miniconda3/envs/alphapose/lib/python3.6/site-packages/matplotlib/animation.py", line 395, in cleanup
        out, err = self._proc.communicate()
      File "/root/miniconda3/envs/alphapose/lib/python3.6/subprocess.py", line 863, in communicate
        stdout, stderr = self._communicate(input, endtime, timeout)
      File "/root/miniconda3/envs/alphapose/lib/python3.6/subprocess.py", line 1525, in _communicate
        selector.register(self.stdout, selectors.EVENT_READ)
      File "/root/miniconda3/envs/alphapose/lib/python3.6/selectors.py", line 351, in register
        key = super().register(fileobj, events, data)
      File "/root/miniconda3/envs/alphapose/lib/python3.6/selectors.py", line 237, in register
        key = SelectorKey(fileobj, self._fileobj_lookup(fileobj), events, data)
      File "/root/miniconda3/envs/alphapose/lib/python3.6/selectors.py", line 224, in _fileobj_lookup
        return _fileobj_to_fd(fileobj)
      File "/root/miniconda3/envs/alphapose/lib/python3.6/selectors.py", line 39, in _fileobj_to_fd
        "{!r}".format(fileobj)) from None
    ValueError: Invalid file object: <_io.BufferedReader name=30>
      6%|######6                                                                                                      | 3/49 [00:00<00:07,  5.99it/s]
    
    opened by ChawDoe 8
  • About custom training and inference

    About custom training and inference

    感谢作者的优秀工作,并且在这分享实现给大家! 我刚刚看了你的视频演讲回放,讲得很棒! 我正在做3D pose的应用,有一些问题想请教一下:

    1. 因为是做应用,所以比较关注estimator。我们在推理的时候,能不能只用estimator?是否还得配合着imitator来用使得这个动作序列更逼真一些?用和不用imitator做推理(不是训练)在性能上差多少呢?
    2. 关于整个framework,我自己理解是这三个模块结构上是可拆解替换的是吗?也就是我可以自定义修改estimator的实现,然后用pretrained的imitator和hallucinator来单独训练estimator的是吗?
    opened by PGogo 5
  • Lower body skeleton issue

    Lower body skeleton issue

    Hi I am trying to get the 3 pose on upper body and is there a way to filter or remove lower body keypoints completely. I am getting the following output where the legs are distorted but I don't want lower body joints to be visible in my output.

    Screen Shot 2022-04-25 at 2 23 37 PM
    opened by Abi5678 4
  • no such file or directory: './data_cross/3dhp/3dhp_testset_bySub.pkl'

    no such file or directory: './data_cross/3dhp/3dhp_testset_bySub.pkl'

    Hello, I am very interesting in your excellenct work! When I run the code in script-summary-gt2d-v5.sh, I get the following error: no such file or directory: './data_cross/3dhp/3dhp_testset_bySub.pkl' How can I get this file? thank you! https://github.com/Garfield-kh/PoseTriplet/blob/eb93132f99161bd776dafbcb713e9fb43e501c36/imitator/script-summary-gt2d-v5.sh#L30

    opened by JiahongWu1995 3
  • multi-person 3D pose estimation

    multi-person 3D pose estimation

    Hi Gong,

    we are using your impressive work for 3D pose reconstruction from video. In general, it works quite well for single-person scenarios. However, when I tried to apply it to the multi-person scenarios, it seems that it only tracks the pose for one character and tracking is not consistent on one person but jumping around on different person. So is that possible to apply for your work on multi-person 3D pose estimation? Thank you!

    opened by dhhjx880713 3
  • Question about applying pose imitator for pose estimator step

    Question about applying pose imitator for pose estimator step

    Dear author,

    Thank you for your amazing work. Currently I'm learning about 3D pose estimation and I'm really impressed by your work. As I understand so far, your pose estimator_inference step (when test with videos) doesn't apply pose imitator. Could you tell me the reason why you dont apply it? And if I want to apply the imitator, can you tell me how can I do it? Sorry I am new to this pose estimation field. Thank you so much for your time.

    opened by SonNguyen2510 2
  • Original PPTX Presentation

    Original PPTX Presentation

    Hi guys,

    Love you work, super interesting!

    Do you have the original PPTX file for the slide presentation posted in this repo? The uploaded one: PPT is a PDF file.

    Thanks!

    opened by BrianG13 1
  • The depth info of legs is always bad

    The depth info of legs is always bad

    I found that the depth of leg is always different from other parts, which leads to lean-forward bad case. Like, this is my kpts2d on img, kpt2d

    And this is the result of PoseTriplet, printed with ax.view_init(0, 70) and ax.view_init(0, 0) kpts3d

    So, Have you found this issue when you research in this task? Would you give me some advise to avoid those bad cases?

    opened by tiger990111 1
  • pretrained models of imitator?

    pretrained models of imitator?

    Hi, just wanna ask, how to train on imitator? does there any instructions on how to run? Will there any pretrained weights that can be used to visualize imitator result?

    opened by jinfagang 0
  • Questions about ablation study on rounds of training

    Questions about ablation study on rounds of training

    First, I just want to say, thank you so much for the great work, and for open-sourcing your code as well!

    I was looking at the training script and noticed that it seems to contain more than 3 rounds of training, while the paper only included the results from the first 3 rounds. Have you tried more rounds of training and if so did it lead to further performance improvement?

    opened by JinchengWang 1
  • 3d prediction bad

    3d prediction bad

    HI, TAHNK for your work! when I test the my video, some bad 3d case like this: 捕获1 捕获2

    and the correspond 2D prediction: 捕获3

    what's wrong with me? can u give me any advice ?

    opened by chunniunai220ml 1
Owner
null
Python scripts for performing 3D human pose estimation using the Mobile Human Pose model in ONNX.

Python scripts for performing 3D human pose estimation using the Mobile Human Pose model in ONNX.

Ibai Gorordo 90 Sep 25, 2022
Repository for the paper "PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation", CVPR 2021.

PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation Code repository for the paper: PoseAug: A Differentiable Pose Augme

Pyjcsx 312 Sep 26, 2022
[CVPR 2022 Oral] EPro-PnP: Generalized End-to-End Probabilistic Perspective-n-Points for Monocular Object Pose Estimation

EPro-PnP EPro-PnP: Generalized End-to-End Probabilistic Perspective-n-Points for Monocular Object Pose Estimation In CVPR 2022 (Oral). [paper] Hanshen

 同济大学智能汽车研究所综合感知研究组 ( Comprehensive Perception Research Group under Institute of Intelligent Vehicles, School of Automotive Studies, Tongji University) 761 Sep 22, 2022
Mixup for Supervision, Semi- and Self-Supervision Learning Toolbox and Benchmark

OpenSelfSup News Downstream tasks now support more methods(Mask RCNN-FPN, RetinaNet, Keypoints RCNN) and more datasets(Cityscapes). 'GaussianBlur' is

AI Lab, Westlake University 241 Sep 27, 2022
Code for "Human Pose Regression with Residual Log-likelihood Estimation", ICCV 2021 Oral

Human Pose Regression with Residual Log-likelihood Estimation [Paper] [arXiv] [Project Page] Human Pose Regression with Residual Log-likelihood Estima

JeffLi 326 Sep 29, 2022
Learning trajectory representations using self-supervision and programmatic supervision.

Trajectory Embedding for Behavior Analysis (TREBA) Implementation from the paper: Jennifer J. Sun, Ann Kennedy, Eric Zhan, David J. Anderson, Yisong Y

null 55 Sep 26, 2022
Code for "Reconstructing 3D Human Pose by Watching Humans in the Mirror", CVPR 2021 oral

Reconstructing 3D Human Pose by Watching Humans in the Mirror Qi Fang*, Qing Shuai*, Junting Dong, Hujun Bao, Xiaowei Zhou CVPR 2021 Oral The videos a

ZJU3DV 175 Sep 23, 2022
Code for "Single-view robot pose and joint angle estimation via render & compare", CVPR 2021 (Oral).

Single-view robot pose and joint angle estimation via render & compare Yann Labbé, Justin Carpentier, Mathieu Aubry, Josef Sivic CVPR: Conference on C

Yann Labbé 50 Jul 13, 2022
Code repo for realtime multi-person pose estimation in CVPR'17 (Oral)

Realtime Multi-Person Pose Estimation By Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh. Introduction Code repo for winning 2016 MSCOCO Keypoints Cha

Zhe Cao 4.9k Sep 27, 2022
Code for "PVNet: Pixel-wise Voting Network for 6DoF Pose Estimation" CVPR 2019 oral

Good news! We release a clean version of PVNet: clean-pvnet, including how to train the PVNet on the custom dataset. Use PVNet with a detector. The tr

ZJU3DV 700 Sep 30, 2022
This repository contains codes of ICCV2021 paper: SO-Pose: Exploiting Self-Occlusion for Direct 6D Pose Estimation

SO-Pose This repository contains codes of ICCV2021 paper: SO-Pose: Exploiting Self-Occlusion for Direct 6D Pose Estimation This paper is basically an

shangbuhuan 50 Sep 21, 2022
This is an official implementation of our CVPR 2021 paper "Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression" (https://arxiv.org/abs/2104.02300)

Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression Introduction In this paper, we are interested in the bottom-up paradigm of estima

HRNet 337 Sep 23, 2022
This repository is the offical Pytorch implementation of ContextPose: Context Modeling in 3D Human Pose Estimation: A Unified Perspective (CVPR 2021).

Context Modeling in 3D Human Pose Estimation: A Unified Perspective (CVPR 2021) Introduction This repository is the offical Pytorch implementation of

null 36 Sep 13, 2022
[CVPR 2022 Oral] Versatile Multi-Modal Pre-Training for Human-Centric Perception

Versatile Multi-Modal Pre-Training for Human-Centric Perception Fangzhou Hong1  Liang Pan1  Zhongang Cai1,2,3  Ziwei Liu1* 1S-Lab, Nanyang Technologic

Fangzhou Hong 90 Sep 29, 2022
(CVPR 2022 - oral) Multi-View Depth Estimation by Fusing Single-View Depth Probability with Multi-View Geometry

Multi-View Depth Estimation by Fusing Single-View Depth Probability with Multi-View Geometry Official implementation of the paper Multi-View Depth Est

Bae, Gwangbin 104 Sep 29, 2022
[CVPR 2022] Pytorch implementation of "Templates for 3D Object Pose Estimation Revisited: Generalization to New objects and Robustness to Occlusions" paper

template-pose Pytorch implementation of "Templates for 3D Object Pose Estimation Revisited: Generalization to New objects and Robustness to Occlusions

Van Nguyen Nguyen 71 Sep 26, 2022
Implementation of the master's thesis "Temporal copying and local hallucination for video inpainting".

Temporal copying and local hallucination for video inpainting This repository contains the implementation of my master's thesis "Temporal copying and

David Álvarez de la Torre 1 Mar 16, 2022
CVPR 2021 - Official code repository for the paper: On Self-Contact and Human Pose.

selfcontact This repo is part of our project: On Self-Contact and Human Pose. [Project Page] [Paper] [MPI Project Page] It includes the main function

Lea Müller 67 Aug 25, 2022
CVPR 2021 - Official code repository for the paper: On Self-Contact and Human Pose.

SMPLify-XMC This repo is part of our project: On Self-Contact and Human Pose. [Project Page] [Paper] [MPI Project Page] License Software Copyright Lic

Lea Müller 81 Sep 14, 2022