Code for "Reconstructing 3D Human Pose by Watching Humans in the Mirror", CVPR 2021 oral

Related tags

Deep Learning Mirrored-Human
Overview

Reconstructing 3D Human Pose by Watching Humans in the Mirror

report
Qi Fang*, Qing Shuai*, Junting Dong, Hujun Bao, Xiaowei Zhou
CVPR 2021 Oral


The videos are from Youtube and Douyin. Please contact us for any copyright issue.

News

  • We build a website for a fast preview of our dataset. The whole dataset will be released later.

Features

In this paper, we introduce the new task of reconstructing 3D human pose from a single image in which we can see the person and the person’s image through a mirror.

This implementation:

  • has the demo of our optimization-based approach implemented purely in PyTorch.
  • provides a method to estimate the surface normal of the mirror from vanishing points.
  • provides an annotator to label the mirror edges for the vanishing points.
  • can estimate the focal length of the Internet mirror images.

Installation

This repo has a close relation with EasyMocap. Please refer to our EasyMocap project for installation.

Demo

Download our zju-m-test.zip and run the following code:

# set the data path
data=<path_to_sample>/zju-m-demo
out=<path_to_sample>/zju-m-demo-output
# extract the video frames
python3 scripts/preprocess/extract_video.py ${data}
# Run demo on videos
python3 apps/demo/1v1p_mirror.py ${data} --out ${out} --vis_smpl --video

Mirrored-Human Dataset (Coming Soon)

Due to the license limitation, we cannot share the raw data directly. We are working hard to organize the Mirrored-Human dataset in terms of url links and timestamps.

See Build Your Internet Dataset if you can't wait for our release.

Annotator

We also provide the annotator metioned in our paper.

The first row shows that we label the edges of the mirror and calculate the vanishing point by the human body automaticly. The intrisic camera parameter can be calculated by this two vanishing points.

The second row shows that to obtain a more accurate vanishing points and camera parameters, we can label the parallel lines in the scene, for example the door, the grid in the ground, and the door.


See EasyMocap/apps/annotator for more instructions.

Build Custom Internet Dataset

See doc/internet.md for more instructions.

Build Custom Evaluation Dataset (Multi-View)

This part is provided for the researchers who want to:

  1. capture more accurate human motion with multiple cameras and a mirror
  2. build a different evaluation dataset

See doc/custom.md for more instructions.

Evaluation

To evaluate the reconstruction part in our paper, see doc/evaluation.md.

Contact

Please open an issue if you have any questions. We appreciate all contributions to improve our project.

If you find some videos that we didn't notice, please tell us.

Citation

@inproceedings{fang2021mirrored,
  title={Reconstructing 3D Human Pose by Watching Humans in the Mirror},
  author={Fang, Qi and Shuai, Qing and Dong, Junting and Bao, Hujun and Zhou, Xiaowei},
  booktitle={CVPR},
  year={2021}
}

Acknowledgement

This project is build on our EasyMocap. We also would like to thank Jianan Zhen and Yuhao Chen for their advice for the paper. Sincere thanks to the performers (Yuji Chen and Hao Xu) in the evaluation dataset and people who uploaded the mirror-human videos to the Internet.

Recommendations to other works from our group

Welcome to checkout our work on learning-based feature matching (LoFTR) and reconstruction (NeuralBody and NeuralRecon) in CVPR 2021.

Comments
  • The scripts for pose reconstructions are missing

    The scripts for pose reconstructions are missing

    Hi, I tried to download the zju-m-test.zip but only zju-m-demo.zip could be found via the link you provided. Did you delete all the scripts folders? Will you release them again in the future? I really appreciate your work. And I'm doing research in a similar area. Thank you.

    opened by acse-ll2819 4
  • Supplementary material for the paper

    Supplementary material for the paper

    Hi,

    Kudos for the great work and and thanks for sharing. I was looking for the supplementary material for the paper, however, cannot seem to find it. Could you point at the download link.

    Thanks

    opened by nicolasugrinovic 1
  • ImportError: cannot import name 'init_with_spin'

    ImportError: cannot import name 'init_with_spin'

    I got same error below (easymocap) C:\ResourceCode\EasyMocap>python apps/demo/1v1p_mirror.py data/zju-m-demo --out data/zju-m-demo-output --vis_smpl --video Traceback (most recent call last): File "apps/demo/1v1p_mirror.py", line 8, in from easymocap.estimator import SPIN, init_with_spin ImportError: cannot import name 'init_with_spin' I try to traceback filelvlp_mirror.py but I didn't know where is the model init_with_spin

    opened by QiFuChina 1
  • Images are not generated by extract_video.py

    Images are not generated by extract_video.py

    Thanks for the good work.

    I tried to run extract_video.py with zju-m-demo, but there were no images in the images folder. Therefore, openpose will fail to make 2D pose.

    If you know any solution, please tell me.

    I use anaconda 3 on windows10.

    opened by 283pm 0
  • 关于使用mirrored-human-base数据集进行训练的问题

    关于使用mirrored-human-base数据集进行训练的问题

    你好，我想使用mirrored-human-base数据集和human3.6m数据集一起训练模型以提高性能，但是发现mirrored-human-base数据集与human3.6m数据集的关键点标注不一样，比如mirrored数据集3d关键点标注有25个，而human3.6m只有17个。请问需要如何处理这个问题呢？

    opened by XiaoYuhao 1
  • ValueError: Cannot load file containing pickled data when allow_pickle=False

    ValueError: Cannot load file containing pickled data when allow_pickle=False

    (easymocap) M:\GITHUB\EASYMOCAP>python apps/demo/1v1p_mirror.py zju-m-demo/ --out zju-m-demo-output/ --vis_smpl --video

    Demo code for single view and one person with mirror:

    - Input : zju-m-demo/: [backflip, dance, kaluli]
- Output: zju-m-demo-output/
- Body  : smpl => neutral, body25

    -> [Loading smpl, neutral]: 4.8s -> [Loading SPIN ]: 399.0ms Traceback (most recent call last): File "apps/demo/1v1p_mirror.py", line 153, in device=body_model.device) File "C:\ProgramData\Miniconda3\envs\easymocap\lib\site-packages\easymocap\estimator\SPIN\spin_api.py", line 178, in init model = hmr(SMPL_MEAN_PARAMS).to(device) File "C:\ProgramData\Miniconda3\envs\easymocap\lib\site-packages\easymocap\estimator\SPIN\models.py", line 175, in hmr model = HMR(Bottleneck, [3, 4, 6, 3], smpl_mean_params, **kwargs) File "C:\ProgramData\Miniconda3\envs\easymocap\lib\site-packages\easymocap\estimator\SPIN\models.py", line 102, in init mean_params = np.load(smpl_mean_params) File "C:\ProgramData\Miniconda3\envs\easymocap\lib\site-packages\numpy\lib\npyio.py", line 445, in load raise ValueError("Cannot load file containing pickled data " ValueError: Cannot load file containing pickled data when allow_pickle=False

    opened by CGMikeG 2
Owner
ZJU3DV
ZJU3DV is a research group of State Key Lab of CAD&CG, Zhejiang University. We focus on the research of 3D computer vision, SLAM and AR.
ZJU3DV
GitHub
Code for "Single-view robot pose and joint angle estimation via render & compare", CVPR 2021 (Oral).

Single-view robot pose and joint angle estimation via render & compare Yann Labbé, Justin Carpentier, Mathieu Aubry, Josef Sivic CVPR: Conference on C

Yann Labbé 51 Oct 14, 2022
Code for CVPR 2021 oral paper "Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contexts"

Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contexts The rapid progress in 3D scene understanding has come with growing dem

Facebook Research 182 Dec 30, 2022
This is the code for CVPR 2021 oral paper: Jigsaw Clustering for Unsupervised Visual Representation Learning

JigsawClustering Jigsaw Clustering for Unsupervised Visual Representation Learning Pengguang Chen, Shu Liu, Jiaya Jia Introduction This project provid

DV Lab 73 Sep 18, 2022
Official pytorch implementation of paper "Inception Convolution with Efficient Dilation Search" (CVPR 2021 Oral).

IC-Conv This repository is an official implementation of the paper Inception Convolution with Efficient Dilation Search. Getting Started Download Imag

Jie Liu 111 Dec 31, 2022
Official PyTorch implementation of RobustNet (CVPR 2021 Oral)

RobustNet (CVPR 2021 Oral): Official Project Webpage Codes and pretrained models will be released soon. This repository provides the official PyTorch

Sungha Choi 173 Dec 21, 2022
Dynamic Slimmable Network (CVPR 2021, Oral)

Dynamic Slimmable Network (DS-Net) This repository contains PyTorch code of our paper: Dynamic Slimmable Network (CVPR 2021 Oral). Architecture of DS-

Changlin Li 197 Dec 9, 2022
[CVPR 2021 Oral] ForgeryNet: A Versatile Benchmark for Comprehensive Forgery Analysis

ForgeryNet: A Versatile Benchmark for Comprehensive Forgery Analysis ForgeryNet: A Versatile Benchmark for Comprehensive Forgery Analysis [arxiv|pdf|v

Yinan He 78 Dec 22, 2022
[CVPR 2021 Oral] Variational Relational Point Completion Network

VRCNet: Variational Relational Point Completion Network This repository contains the PyTorch implementation of the paper: Variational Relational Point

PL 121 Dec 12, 2022
Quasi-Dense Similarity Learning for Multiple Object Tracking, CVPR 2021 (Oral)

Quasi-Dense Tracking This is the offical implementation of paper Quasi-Dense Similarity Learning for Multiple Object Tracking. We present a trailer th

ETH VIS Research Group 327 Dec 27, 2022
Pytorch implementation for "Adversarial Robustness under Long-Tailed Distribution" (CVPR 2021 Oral)

Adversarial Long-Tail This repository contains the PyTorch implementation of the paper: Adversarial Robustness under Long-Tailed Distribution, CVPR 20

Tong WU 89 Dec 15, 2022
TAP: Text-Aware Pre-training for Text-VQA and Text-Caption, CVPR 2021 (Oral)

TAP: Text-Aware Pre-training TAP: Text-Aware Pre-training for Text-VQA and Text-Caption by Zhengyuan Yang, Yijuan Lu, Jianfeng Wang, Xi Yin, Dinei Flo

Microsoft 61 Nov 14, 2022
Official PyTorch Implementation of Convolutional Hough Matching Networks, CVPR 2021 (oral)

Convolutional Hough Matching Networks This is the implementation of the paper "Convolutional Hough Matching Network" by J. Min and M. Cho. Implemented

Juhong Min 70 Nov 22, 2022
Implementation of "Bidirectional Projection Network for Cross Dimension Scene Understanding" CVPR 2021 (Oral)

Bidirectional Projection Network for Cross Dimension Scene Understanding CVPR 2021 (Oral) [ Project Webpage ] [ arXiv ] [ Video ] Existing segmentatio

Hu Wenbo 135 Dec 26, 2022
Deep RGB-D Saliency Detection with Depth-Sensitive Attention and Automatic Multi-Modal Fusion (CVPR'2021, Oral)

DSA^2 F: Deep RGB-D Saliency Detection with Depth-Sensitive Attention and Automatic Multi-Modal Fusion (CVPR'2021, Oral) This repo is the official imp

如今我已剑指天涯 46 Dec 21, 2022
🔥RandLA-Net in Tensorflow (CVPR 2020, Oral & IEEE TPAMI 2021)

RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds (CVPR 2020) This is the official implementation of RandLA-Net (CVPR2020, Oral

Qingyong 1k Dec 30, 2022
git git《Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking》(CVPR 2021) GitHub:git2] 《Masksembles for Uncertainty Estimation》(CVPR 2021) GitHub:git3]

Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking Ning Wang, Wengang Zhou, Jie Wang, and Houqiang Li Accepted by CVPR

NingWang 236 Dec 22, 2022
Official code for "End-to-End Optimization of Scene Layout" -- including VAE, Diff Render, SPADE for colorization (CVPR 2020 Oral)

End-to-End Optimization of Scene Layout Code release for: End-to-End Optimization of Scene Layout CVPR 2020 (Oral) Project site, Bibtex For help conta

Andrew Luo 41 Dec 9, 2022
Code repo for realtime multi-person pose estimation in CVPR'17 (Oral)

Realtime Multi-Person Pose Estimation By Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh. Introduction Code repo for winning 2016 MSCOCO Keypoints Cha

Zhe Cao 4.9k Dec 31, 2022
Code for "PVNet: Pixel-wise Voting Network for 6DoF Pose Estimation" CVPR 2019 oral

Good news! We release a clean version of PVNet: clean-pvnet, including how to train the PVNet on the custom dataset. Use PVNet with a detector. The tr

ZJU3DV 722 Dec 27, 2022