C3DPO - Canonical 3D Pose Networks for Non-rigid Structure From Motion.

Overview

C3DPO: Canonical 3D Pose Networks for Non-Rigid Structure From Motion

By: David Novotny, Nikhila Ravi, Benjamin Graham, Natalia Neverova, Andrea Vedaldi

This is the official implementation of C3DPO: Canonical 3D Pose Networks for Non-Rigid Structure From Motion in PyTorch.

Link to paper | Project page

alt text

Dependencies

This is a Python 3.6 package. Required packages can be installed with e.g. pip and conda:

> conda create -n c3dpo python=3.6
> pip install -r requirements.txt

The complete list of dependencies:

  • pytorch (version==1.1.0)
  • numpy
  • tqdm
  • matplotlib
  • visdom
  • pyyaml
  • tabulate

Demo

demo.py downloads and runs a pre-trained C3DPO model on a sample skeleton from the Human36m dataset and generates a 3D figure with a video of the predicted 3D skeleton:

> python ./demo.py

Note that all the outputs are dumped to a local Visdom server. You can start a Visdom server with:

> python -m visdom.server

Images are also stored to the ./data directory. The video will get exported only if there's a functioning ffmpeg callable from the command line.

Downloading data / models

Whenever needed, all datasets / pre-trained models are automatically downloaded to various folders under the ./data directory. Hence, there's no need to bother with a complicated data setup :). In case you would like to cache all the datasets for your own use, simply run the evaluate.py which downloads all the needed data during its run.

Quick start = pre-trained network evaluation

Pre-trained networks can be evaluated by calling evaluate.py:

> python evaluate.py

Note that we provide pre-trained models that will get auto-downloaded during the run of the script to the ./data/exps/ directory. Furthermore, the datasets will also be automatically downloaded in case they are not stored in ./data/datasets/.

Network training + evaluation

Launch experiment.py with the argument cfg_file set to the yaml file corresponding the relevant dataset., e.g.:

> python ./experiment.py --cfg_file ./cfgs/h36m.yaml

will train a C3DPO model for the Human3.6m dataset.

Note that the code supports visualisation in Visdom. In order to enable Visdom visualisations, first start a visdom server with:

> python -m visdom.server

The experiment will output learning curves as well as visualisations of the intermediate outputs to the visdom server.

Furthermore, the results of the evaluation will be periodically updated after every training epoch in ./data/exps/c3dpo/<dataset_name>/eval_results.json. The metrics reported in the paper correspond to 'EVAL_MPJPE_best' and 'EVAL_stress'.

For the list of all possible yaml config files, please see the ./cfgs/ directory. Each config .yaml file corresponds to a training on a different dataset (matching the name of the .yaml file). Expected quantitative results are the same as for the evaluate.py script.

Reference

If you find our work useful, please cite it using the following bibtex reference.

@inproceedings{novotny2019c3dpo,
  title={C3DPO: Canonical 3D Pose Networks for Non-Rigid Structure From Motion},
  author={Novotny, David and Ravi, Nikhila and Graham, Benjamin and Neverova, Natalia and Vedaldi, Andrea},
  booktitle={Proceedings of the IEEE International Conference on Computer Vision},
  year={2019}
}

License

C3DPO is distributed under the MIT license, as found in the LICENSE file.

Expected outputs of evaluate.py

Below are the results of the supplied pre-trained models for all datasets:

dataset               MPJPE      Stress
--------------  -----------  ----------
h36m             95.6338     41.5864
h36m_hourglass  145.021      84.693
pascal3d_hrnet   56.8909     40.1775
pascal3d         36.6413     31.0768
up3d_79kp         0.0672771   0.0406902

Note that the models have better performance than published mainly due to letting the models to train for longer.

Notes for reproducibility

Note that the performance reported above was obtained with PyTorch v1.1. If you notice differences in performance make sure to use PyTorch v1.1.

Comments
  • No output when implementing evaluate.py

    No output when implementing evaluate.py

    Hi, I am trying to implement c3dpo_nrsfm as in github to learn it. I am neither getting any extracted 3D points nor an error. It stays as below image

    I tried the following commands in the sequence

    1. python -m visdom.server
    2. python ./experiment.py --cfg_file ./cfgs/h36m.yaml

    and

    1. python -m visdom.server
    2. python ./demo.py

    thanks Mritula

    opened by MritulaC 2
  • Applying to other dataset

    Applying to other dataset

    Hi, I have several images of humans in .jpg format. I am trying to extract 3D poses from these images. How am I to apply c3dpo_nrsfm to my input images?

    Thanks Mritula

    opened by MritulaC 2
  • Request error when running demo.py

    Request error when running demo.py

    Error info:

    requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8097): Max retries exceeded with url: /events (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x000002413CA92E10>: Failed to establish a new connection: [WinError 10061] 由于目标计算机积极拒绝,无法连接。',))

    opened by WinstonDeng 2
  • Dependencies updated in the README.md

    Dependencies updated in the README.md

    c3dpo_nrsfm_readme_ss

    While installing Dependencies from requirements.txt throwing Error c3dpo_nrsfm_error_ss So after some searching, I get to know that in order to install torch==1.1.0 we have to specifically provide "-f https://download.pytorch.org/whl/torch_stable.html" with pip command otherwise it throws errors because by default pip search for a package in https://pypi.org/simple only and torch 1.1.0 isn't available there that's why we give source URL for torch 1.1.0 separately( https://download.pytorch.org/whl/torch_stable.html ). So I added it in requirements.txt c3dpo_nrsfm_requir_txt_error_ss although it doesn't work. c3dpo_nrsfm_second_error_ss and after searching and trial&error i got this way working perfectly in installation. "pip install -r requirements.txt -f https://download.pytorch.org/whl/torch_stable.html" So I decided to update it in README.md file to save other people's time to eliminating errors in the installation. Thank you

    CLA Signed 
    opened by parthpatadiya 1
  • human scale

    human scale

    Hey, it seems that you assumed the scale of the person is known, so if we run the model on a new dataset, and don't have human scale, we can not predict absolute 3d keypoints coordinates, right?

    opened by xjturobocon 1
  • Visdom of demo program

    Visdom of demo program

    Hello, we are very interested in your research, but we need to download the sample file from visdom to run the program. Due to the limitation of Chinese network, visdom cannot be installed and used normally. Can you provide the static folder of visdom for us to manually replace the static file in the visdom directory. Thanks.

    opened by Vandejun 1
  • demo run error

    demo run error

    urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fb4322c4e10>: Failed to establish a new connection: [Errno 111] Connection refused

    opened by mathpopo 1
  • How to make it work on face

    How to make it work on face

    I am trying to train the model to make it work for 3D reconstruction for face. I used a 2D face landmark detector to detect 2D landmarks on face images. Then I trained the model using the same config file h36m.yaml except I set shape_basis_size=1000 as in your ICCV presentation talk slides. The training converges very fast, but when I test it, the result is not as good as seen from the image I pasted. Could you point me to how to make it work for 3D reconstruction for face? Thank you. image image

    opened by zhaoruiqiff 0
  • where is projection matrix

    where is projection matrix

    when set projection_type: perspective in exconfig.yaml and run demo.py but assert K is not None in def forward(self, kp_loc=None, kp_vis=None, class_mask=None, K=None, **kwargs):

    opened by azuryl 1
  • Dependencies updated in the README.md

    Dependencies updated in the README.md

    c3dpo_nrsfm_readme_ss

    While installing Dependencies from requirements.txt throwing Error c3dpo_nrsfm_error_ss

    So after some searching, I get to know that in order to install torch==1.1.0 we have to specifically provide "-f https://download.pytorch.org/whl/torch_stable.html" with pip command otherwise it throws errors because by default pip search for a package in https://pypi.org/simple only and torch 1.1.0 isn't available there that's why we give source URL for torch 1.1.0 separately( https://download.pytorch.org/whl/torch_stable.html ). So I added it in requirements.txt c3dpo_nrsfm_requir_txt_error_ss

    although it doesn't work. c3dpo_nrsfm_second_error_ss

    and after searching and trial&error i got this way working perfectly in installation. "pip install -r requirements.txt -f https://download.pytorch.org/whl/torch_stable.html" So I decided to update it in README.md file to save other people's time to eliminating errors in the installation. Thank you

    CLA Signed 
    opened by parthpatadiya 0
  • About visualization

    About visualization

    Hi, I run the demo.py but find the visualization results look totally different from the video of your project pages. (Your demo video is cooler! ) How can I draw the 3D pose just like the one on your project pages? Many thanks!

    opened by sjtuxcx 5
  • Visualizing PASCAL3D+ Results

    Visualizing PASCAL3D+ Results

    Hi, How do I visualize the results on PASCAL3D+ dataset. The models and the keypoint annotations given are with respect to 124 keypoints. However, for example for a car I wish to visualize only the valid keypoints (12 in the case of PASCAL3D+). When I try to visualize by changing the model in demo.py, it plots the invalid keypoints as well. Would really appreciate some help!

    opened by aadilmehdis 0
Owner
Meta Research
Meta Research
Official code release for ICCV 2021 paper SNARF: Differentiable Forward Skinning for Animating Non-rigid Neural Implicit Shapes.

Official code release for ICCV 2021 paper SNARF: Differentiable Forward Skinning for Animating Non-rigid Neural Implicit Shapes.

null 235 Dec 26, 2022
GPU Accelerated Non-rigid ICP for surface registration

GPU Accelerated Non-rigid ICP for surface registration Introduction Preivous Non-rigid ICP algorithm is usually implemented on CPU, and needs to solve

Haozhe Wu 144 Jan 4, 2023
A variational Bayesian method for similarity learning in non-rigid image registration (CVPR 2022)

A variational Bayesian method for similarity learning in non-rigid image registration We provide the source code and the trained models used in the re

daniel grzech 14 Nov 21, 2022
Official PyTorch implementation of CAPTRA: CAtegory-level Pose Tracking for Rigid and Articulated Objects from Point Clouds

CAPTRA: CAtegory-level Pose Tracking for Rigid and Articulated Objects from Point Clouds Introduction This is the official PyTorch implementation of o

Yijia Weng 96 Dec 7, 2022
SE3 Pose Interp - Interpolate camera pose or trajectory in SE3, pose interpolation, trajectory interpolation

SE3 Pose Interpolation Pose estimated from SLAM system are always discrete, and

Ran Cheng 4 Dec 15, 2022
Code for "Learning Canonical Representations for Scene Graph to Image Generation", Herzig & Bar et al., ECCV2020

Learning Canonical Representations for Scene Graph to Image Generation (ECCV 2020) Roei Herzig*, Amir Bar*, Huijuan Xu, Gal Chechik, Trevor Darrell, A

roei_herzig 24 Jul 7, 2022
Canonical Appearance Transformations

CAT-Net: Learning Canonical Appearance Transformations Code to accompany our paper "How to Train a CAT: Learning Canonical Appearance Transformations

STARS Laboratory 54 Dec 24, 2022
Finite-temperature variational Monte Carlo calculation of uniform electron gas using neural canonical transformation.

CoulombGas This code implements the neural canonical transformation approach to the thermodynamic properties of uniform electron gas. Building on JAX,

FermiFlow 9 Mar 3, 2022
PyTorch implementation DRO: Deep Recurrent Optimizer for Structure-from-Motion

DRO: Deep Recurrent Optimizer for Structure-from-Motion This is the official PyTorch implementation code for DRO-sfm. For technical details, please re

Alibaba Cloud 56 Dec 12, 2022
Deep Two-View Structure-from-Motion Revisited

Deep Two-View Structure-from-Motion Revisited This repository provides the code for our CVPR 2021 paper Deep Two-View Structure-from-Motion Revisited.

Jianyuan Wang 145 Jan 6, 2023
Video Autoencoder: self-supervised disentanglement of 3D structure and motion

Video Autoencoder: self-supervised disentanglement of 3D structure and motion This repository contains the code (in PyTorch) for the model introduced

null 157 Dec 22, 2022
COLMAP - Structure-from-Motion and Multi-View Stereo

COLMAP About COLMAP is a general-purpose Structure-from-Motion (SfM) and Multi-View Stereo (MVS) pipeline with a graphical and command-line interface.

null 4.7k Jan 7, 2023
Making Structure-from-Motion (COLMAP) more robust to symmetries and duplicated structures

SfM disambiguation with COLMAP About Structure-from-Motion generally fails when the scene exhibits symmetries and duplicated structures. In this repos

Computer Vision and Geometry Lab 193 Dec 26, 2022
SatelliteSfM - A library for solving the satellite structure from motion problem

Satellite Structure from Motion Maintained by Kai Zhang. Overview This is a libr

Kai Zhang 190 Dec 8, 2022
This repository contains the code for the paper "Hierarchical Motion Understanding via Motion Programs"

Hierarchical Motion Understanding via Motion Programs (CVPR 2021) This repository contains the official implementation of: Hierarchical Motion Underst

Sumith Kulal 40 Dec 5, 2022
Exploring Versatile Prior for Human Motion via Motion Frequency Guidance (3DV2021)

Exploring Versatile Prior for Human Motion via Motion Frequency Guidance This is the codebase for video-based human motion reconstruction in human-mot

Jiachen Xu 5 Jul 14, 2022
Official pytorch implementation for Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion (CVPR 2022)

Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion This repository contains a pytorch implementation of "Learning to Listen: Modeling

null 50 Dec 17, 2022
The official repo for OC-SORT: Observation-Centric SORT on video Multi-Object Tracking. OC-SORT is simple, online and robust to occlusion/non-linear motion.

OC-SORT Observation-Centric SORT (OC-SORT) is a pure motion-model-based multi-object tracker. It aims to improve tracking robustness in crowded scenes

Jinkun Cao 325 Jan 5, 2023
A Robust Non-IoU Alternative to Non-Maxima Suppression in Object Detection

Confluence: A Robust Non-IoU Alternative to Non-Maxima Suppression in Object Detection 1. 介绍 用以替代 NMS,在所有 bbox 中挑选出最优的集合。 NMS 仅考虑了 bbox 的得分,然后根据 IOU 来

null 44 Sep 15, 2022