Repository for the paper "PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation", CVPR 2021.

Overview

PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation

Code repository for the paper:
PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation
Kehong Gong*, Jianfeng Zhang*, Jiashi Feng
CVPR 2021 (oral presentation)
[paper (coming soon)] [project page (coming soon)]

alt text

Citation

If you find this code useful for your research, please consider citing the following paper:

@inproceedings{gong2021poseaug,
  title       = {PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation},
  author      = {Gong, Kehong and Zhang, Jianfeng and Feng, Jiashi},
  booktitle   = {CVPR},
  year        = {2021}
}

Acknowledgements

This code uses (SemGCN, SimpleBL, ST-GCN and VPose3D) as backbone. We gratefully appreciate the impact these libraries had on our work. If you use our code, please consider citing the original papers as well.

Comments
  • 3DPW cross-dataset comparison

    3DPW cross-dataset comparison

    From #9, I was wondering if you can provide the 3dpw file or perhaps the preprocessing code?

    I followed the SPIN and your 3dhp preprocessing script but the resulted in some PA-MPJPE around 120 using your models on the test npz I put together myself (16 joints). Thanks!

    opened by yangchris11 17
  • How could I run a demo wild video to see the visualization result

    How could I run a demo wild video to see the visualization result

    It‘s a great work! And can you upload the The corresponding pre-trained model to evaluate any wild video and to see the visualization result if possible! Best wishes!

    opened by hujb48 9
  • Error when training model

    Error when training model

    Hello. Thanks for your work. I am trying to train a GCN model using the command python3 run_baseline.py --note pretrain --dropout 0 --lr 2e-2 --epochs 100 --posenet_name 'gcn' --checkpoint './checkpoint/pretrain_baseline' --keypoints gt, but an error occurs. Traceback (most recent call last): File "run_baseline.py", line 102, in main(args) File "run_baseline.py", line 65, in main glob_step, args.lr_decay, args.lr_gamma, max_norm=args.max_norm) File "/home/hkuit155/Documents/PoseAug/function_baseline/model_pos_train.py", line 41, in train outputs_3d = model_pos(inputs_2d) File "/home/hkuit155/anaconda3/envs/poseaug/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call result = self.forward(*input, **kwargs) File "/home/hkuit155/Documents/PoseAug/models_baseline/gcn/sem_gcn.py", line 104, in forward out = self.gconv_input(x) File "/home/hkuit155/anaconda3/envs/poseaug/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call result = self.forward(*input, **kwargs) File "/home/hkuit155/anaconda3/envs/poseaug/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward input = module(input) File "/home/hkuit155/anaconda3/envs/poseaug/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call result = self.forward(*input, **kwargs) File "/home/hkuit155/Documents/PoseAug/models_baseline/gcn/sem_gcn.py", line 28, in forward x = self.gconv(x).transpose(1, 2) File "/home/hkuit155/anaconda3/envs/poseaug/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call result = self.forward(*input, **kwargs) File "/home/hkuit155/Documents/PoseAug/models_baseline/gcn/sem_graph_conv.py", line 43, in forward output = torch.matmul(adj * M, h0) + torch.matmul(adj * (1 - M), h1) RuntimeError: invalid argument 6: wrong matrix size at /pytorch/aten/src/THC/generic/THCTensorMathBlas.cu:492

    A same error also occurs when training ST-GCN

    Traceback (most recent call last): File "run_baseline.py", line 102, in main(args) File "run_baseline.py", line 65, in main glob_step, args.lr_decay, args.lr_gamma, max_norm=args.max_norm) File "/home/hkuit155/Documents/PoseAug/function_baseline/model_pos_train.py", line 41, in train outputs_3d = model_pos(inputs_2d) File "/home/hkuit155/anaconda3/envs/poseaug/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call result = self.forward(*input, **kwargs) File "/home/hkuit155/Documents/PoseAug/models_baseline/models_st_gcn/st_gcn_single_frame_test.py", line 461, in forward x = torch.matmul(x, C) # nx2x17 RuntimeError: invalid argument 6: wrong matrix size at /pytorch/aten/src/THC/generic/THCTensorMathBlas.cu:492

    opened by CheungBH 7
  • 3DHP evaluation

    3DHP evaluation

    Hi nice work and thanks for releasing your code.

    could you please point me to what data for MPI you use to train and validate on? Also, can you tell me what file contains your skeleton and MPI specific dataset class?

    Also, do you only evaluate with MpJpE or do you use PcK amd any other metrics on MPI?

    opened by alecda573 5
  • About raising pose from 2d to 3d

    About raising pose from 2d to 3d

    Hi. I am running your code aiming at lifting the 2d pose to 3d. However, in the pre-processing step, I found that camera parameters are in need. Can I just use the models to lift the 2d pose to the 3d pose without camera calibration? For example, I obtained an image with humans whose camera parameters are unknown, and using pose lifting model after obtaining the 2d pose.

    opened by CheungBH 5
  • I encountered some tricky problems

    I encountered some tricky problems

    Can this model run a single picture? When I ran a picture, there was some misalignment. The model used was MLP trained by PoseAug, and I found that the result was not very satisfactory. 20210908-104340 20210908-170632 20210908-170643

    opened by psw13583112 4
  • pretrain the baseline model

    pretrain the baseline model

    Your work is great! Thanks. when I run the command python3 run_baseline.py --note pretrain --lr 1e-3 --posenet_name 'videopose' --checkpoint './checkpoint/pretrain_baseline' --keypoints gt, the following error was generated image Can i trouble you to check what is wrong? Thanks!

    opened by PJ-Hunter 4
  • Regarding the 2D input

    Regarding the 2D input

    Hello,

    Sorry to bother you again, may I know which joint is not needed for your 2D input? Is it head, or nose? I understand that your input and output are both 16 joints, and nose is not predicted by the model.

    Many thanks, Shiyang

    opened by jacksoncsy 3
  • Evaluation on MPI-INF-3DHP

    Evaluation on MPI-INF-3DHP

    I am assuming your training data is H36M poses not the meshes and you are evaluating on pose on MPI-INF-3DHP as well, without using SMPL parameters from both. Further, you apply poseAug on 2D and 3D poses of H36M to get training pairs. At that time you would use 16 joint skeleton model of

    So, my questions are how do you evaluate given that the skeleton model of MPI-INF-3DHP is 23 joints models and some basic joints have no correspondence from H36M to MPI-INF-3DHP, e.g. Thorex, head_top, Spine are not there in H36M, however, H36M have Neck and Head joint.

    How did you evaluate MPI-INF-3DHP in a cross-dataset-testing-scenrio given skeleton model discrepancy?

    I guess you only use a subset of joints to evaluate?

    opened by gurkirt 3
  • About extra 2D pose datasets (MPII&LSP) data pre-processing and training problem

    About extra 2D pose datasets (MPII&LSP) data pre-processing and training problem

    Hi author. Thanks for the great work!

    I'm still not clear on data pre-processing. Could you please release the code or point me how to use additional in-the-wild 2D poses (MPII & LSP) to train the 2D discriminator? Which you mention in paper page 6 "Results on 3DHP (cross-scenario) part".

    opened by wenwen12321 2
  • How to make npy file?

    How to make npy file?

    Thank you for your excellent research.

    https://github.com/jfzhang95/PoseAug/blob/main/data_extra/bone_length_npy/hm36s15678_bl_templates.npy

    I wonder how this file was created.

    opened by redorangeyellowy 2
  • PosDiscriminator

    PosDiscriminator

    Hello! I want to ask some questions about PosDiscriminator. First, get_ BoneVecbypose3d and get_ Pose3dbyBoneVec can be extended to 20 key points, right? Second, local KCS uses indexes, such as hb_ Idx=[0, 3, 4, 5, 6]. This is the index of the joint points included in the right leg. I went to see the bone map and found that it does not match. Third, PosDiscriminator inputs 3D posture and outputs the precise posture. Fourth, what are the rules for the human body to split the five parts. Look forward to your reply, thank you! @Garfield-kh

    opened by jianlai123-123 2
  • trajectory model training

    trajectory model training

    I am trying to use this training frame to train a baseline trajectory model (predict root node camera space position). It works but loss seems weird: the test MPJPE is always around 110.

    Is there anything I miss? Have you tried train any trajectory model for PoseAug model?

    opened by darcula1993 2
Owner
Pyjcsx
Code 5 mins then debug 2 hours
Pyjcsx
Repository of our paper 'Refer-it-in-RGBD' in CVPR 2021

Refer-it-in-RGBD This is the repository of our paper 'Refer-it-in-RGBD: A Bottom-up Approach for 3D Visual Grounding in RGBD Images' in CVPR 2021 Pape

Haolin Liu 34 Nov 7, 2022
CVPR 2021 - Official code repository for the paper: On Self-Contact and Human Pose.

selfcontact This repo is part of our project: On Self-Contact and Human Pose. [Project Page] [Paper] [MPI Project Page] It includes the main function

Lea Müller 67 Oct 18, 2022
CVPR 2021 - Official code repository for the paper: On Self-Contact and Human Pose.

SMPLify-XMC This repo is part of our project: On Self-Contact and Human Pose. [Project Page] [Paper] [MPI Project Page] License Software Copyright Lic

Lea Müller 83 Oct 31, 2022
Official repository for the CVPR 2021 paper "Learning Feature Aggregation for Deep 3D Morphable Models"

Deep3DMM Official repository for the CVPR 2021 paper Learning Feature Aggregation for Deep 3D Morphable Models. Requirements This code is tested on Py

null 36 Nov 8, 2022
This repository contains a re-implementation of the code for the CVPR 2021 paper "Omnimatte: Associating Objects and Their Effects in Video."

Omnimatte in PyTorch This repository contains a re-implementation of the code for the CVPR 2021 paper "Omnimatte: Associating Objects and Their Effect

Erika Lu 721 Nov 14, 2022
CVPR 2021 - Official code repository for the paper: On Self-Contact and Human Pose.

TUCH This repo is part of our project: On Self-Contact and Human Pose. [Project Page] [Paper] [MPI Project Page] License Software Copyright License fo

Lea Müller 45 Nov 9, 2022
This repository contains the code for the CVPR 2021 paper "GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields"

GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields Project Page | Paper | Supplementary | Video | Slides | Blog | Talk If

null 1.1k Nov 26, 2022
[CVPR 21] Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting, IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021.

Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting, CVPR 2021. Ayan Kumar Bhunia, Pinaki nath Chowdhury, Yongxin Yan

Ayan Kumar Bhunia 43 Sep 26, 2022
[CVPR 2022] CoTTA Code for our CVPR 2022 paper Continual Test-Time Domain Adaptation

CoTTA Code for our CVPR 2022 paper Continual Test-Time Domain Adaptation Prerequisite Please create and activate the following conda envrionment. To r

Qin Wang 82 Nov 10, 2022
Repository for "Exploring Sparsity in Image Super-Resolution for Efficient Inference", CVPR 2021

SMSR Reposity for "Exploring Sparsity in Image Super-Resolution for Efficient Inference" [arXiv] Highlights Locate and skip redundant computation in S

Longguang Wang 221 Nov 8, 2022
This repository is the offical Pytorch implementation of ContextPose: Context Modeling in 3D Human Pose Estimation: A Unified Perspective (CVPR 2021).

Context Modeling in 3D Human Pose Estimation: A Unified Perspective (CVPR 2021) Introduction This repository is the offical Pytorch implementation of

null 36 Sep 13, 2022
This repository contains the code for the CVPR 2020 paper "Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision"

Differentiable Volumetric Rendering Paper | Supplementary | Spotlight Video | Blog Entry | Presentation | Interactive Slides | Project Page This repos

null 688 Nov 23, 2022
Official repository for the paper "Self-Supervised Models are Continual Learners" (CVPR 2022)

Self-Supervised Models are Continual Learners This is the official repository for the paper: Self-Supervised Models are Continual Learners Enrico Fini

Enrico Fini 69 Nov 18, 2022
PyTorch implementation of paper "Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes", CVPR 2021

Neural Scene Flow Fields PyTorch implementation of paper "Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes", CVPR 20

Zhengqi Li 573 Nov 24, 2022
Official pytorch implementation of paper "Inception Convolution with Efficient Dilation Search" (CVPR 2021 Oral).

IC-Conv This repository is an official implementation of the paper Inception Convolution with Efficient Dilation Search. Getting Started Download Imag

Jie Liu 68 Oct 20, 2022
Code for our CVPR 2021 paper "MetaCam+DSCE"

Joint Noise-Tolerant Learning and Meta Camera Shift Adaptation for Unsupervised Person Re-Identification (CVPR'21) Introduction Code for our CVPR 2021

FlyingRoastDuck 59 Oct 31, 2022
Official code of the paper "ReDet: A Rotation-equivariant Detector for Aerial Object Detection" (CVPR 2021)

ReDet: A Rotation-equivariant Detector for Aerial Object Detection ReDet: A Rotation-equivariant Detector for Aerial Object Detection (CVPR2021), Jiam

csuhan 333 Nov 24, 2022
Official code for the paper: Deep Graph Matching under Quadratic Constraint (CVPR 2021)

QC-DGM This is the official PyTorch implementation and models for our CVPR 2021 paper: Deep Graph Matching under Quadratic Constraint. It also contain

Quankai Gao 55 Nov 14, 2022
Code for CVPR 2021 paper: Anchor-Free Person Search

Introduction This is the implementationn for Anchor-Free Person Search in CVPR2021 License This project is released under the Apache 2.0 license. Inst

null 155 Nov 9, 2022