PyTorch reimplementation of hand-biomechanical-constraints (ECCV2020)

Overview

Hand Biomechanical Constraints Pytorch

Unofficial PyTorch reimplementation of Hand-Biomechanical-Constraints (ECCV2020).

This project reimplement following components :

  1. 3 kinds of biomechanical soft constraints
  2. integrate BMC into training procedure (PyTorch version)

Usage

  • Retrieve the code
git clone https://github.com/MengHao666/Hand-BMC-pytorch
cd Hand-BMC-pytorch
  • Create and activate the virtual environment with python dependencies
conda env create --file=environment.yml
conda activate bmc

Download data

Download 3D joint location data joints.zip Google Drive or Baidu Pan (2pip), and . These statistics are from following datasets:

Note the data from these datasets under their own licenses.

Calculate BMC

BMC

Run the code

python calculate_bmc.py

You will get

  • bone_len_max.npy bone_len_min.npy for bone length limits
  • curvatures_max.npy curvatures_min.npy for Root bones' curvatures
  • PHI_max.npy PHI_min.npy for Root bones' angular distance
  • joint_angles.npy for Joint angles

And if u want to check the coordinate system, run the code

cd utils
python calculate_joint_angles.py
  • red ,green, blue arrows refer to X,Y,Z of local coordinate system respectively;
  • dark arrows refer to bones;
  • pink arrows refer to bone projection into X-Z plane of local coordinate system;
One view Another view

Run the code

python calculate_convex_hull.py

You will get CONVEX_HULLS.npy, i.e. convex hulls to encircle the anatomically plausible joint angles.

And you will also see every convex hull like following figure:

BMC

  • "Bone PIP" means the bone from MCP joint to PIP joint in thumb
  • flexion and abduction is two kinds of angle describing joint rotation
  • "ori_convex_hull" means the original convex hull calculated from all joint angle points
  • "rdp_convex_hull" means convex hull simplified by the Ramer-Douglas-Peucker algorithm, a polygon simplification algorithm
  • "del_convex_hull" means convex hull further simplified by a greedy algorithm
  • "rectangle" means the minimal rectangle to surround all joint angle points

Run the code

python plot.py

You will see all the convex hulls

BMC

Integrate BMC into training (PyTorch version)

Run the code

python weakloss.py

Experiment results

To check influence of BMC, instead of reimplementing the network of origin paper, I integrate BMC into my own project,

Train and evaluation curve

(AUC means 3D PCK, and ACC_HM means 2D PCK) teaser

3D PCK AUC Diffenence

Dataset DetNet DetNet+BMC
RHD 0.9339 0.9364
STB 0.8744 0.8778
DO 0.9378 0.9475
EO 0.9270 0.9182

Note

  • Adjusting training parameters carefully, longer training time might further boost accuracy.
  • As BMC is a weakly supervised method, it may only make predictions more physically plausible,but cannot boost AUC performance strongly when strong supervision is used.

Limitation

  • Due to time limitation, I didn't reimplement the network and experiments of original paper.
  • There is a little difference between original paper and my reimplementation. But most of them match.

Citation

This is the unofficial pytorch reimplementation of the paper "Weakly supervised 3d hand pose estimation via biomechanical constraints (ECCV 2020).

If you find the project helpful, please star this project and cite them:

@article{spurr2020weakly,
  title={Weakly supervised 3d hand pose estimation via biomechanical constraints},
  author={Spurr, Adrian and Iqbal, Umar and Molchanov, Pavlo and Hilliges, Otmar and Kautz, Jan},
  journal={arXiv preprint arXiv:2003.09282},
  volume={8},
  year={2020},
  publisher={Springer}
}
You might also like...
 a reimplementation of Optical Flow Estimation using a Spatial Pyramid Network in PyTorch
a reimplementation of Optical Flow Estimation using a Spatial Pyramid Network in PyTorch

pytorch-spynet This is a personal reimplementation of SPyNet [1] using PyTorch. Should you be making use of this work, please cite the paper according

PyTorch reimplementation of REALM and ORQA

PyTorch reimplementation of REALM and ORQA

 a reimplementation of UnFlow in PyTorch that matches the official TensorFlow version
a reimplementation of UnFlow in PyTorch that matches the official TensorFlow version

pytorch-unflow This is a personal reimplementation of UnFlow [1] using PyTorch. Should you be making use of this work, please cite the paper according

 a reimplementation of LiteFlowNet in PyTorch that matches the official Caffe version
a reimplementation of LiteFlowNet in PyTorch that matches the official Caffe version

pytorch-liteflownet This is a personal reimplementation of LiteFlowNet [1] using PyTorch. Should you be making use of this work, please cite the paper

a reimplementation of Holistically-Nested Edge Detection in PyTorch
a reimplementation of Holistically-Nested Edge Detection in PyTorch

pytorch-hed This is a personal reimplementation of Holistically-Nested Edge Detection [1] using PyTorch. Should you be making use of this work, please

Pytorch reimplementation of PSM-Net:
Pytorch reimplementation of PSM-Net: "Pyramid Stereo Matching Network"

This is a Pytorch Lightning version PSMNet which is based on JiaRenChang/PSMNet. use python main.py to start training. PSM-Net Pytorch reimplementatio

Unofficial PyTorch reimplementation of the paper Swin Transformer V2: Scaling Up Capacity and Resolution

PyTorch reimplementation of the paper Swin Transformer V2: Scaling Up Capacity and Resolution [arXiv 2021].

PyTorch reimplementation of the Smooth ReLU activation function proposed in the paper
PyTorch reimplementation of the Smooth ReLU activation function proposed in the paper "Real World Large Scale Recommendation Systems Reproducibility and Smooth Activations" [arXiv 2022].

Smooth ReLU in PyTorch Unofficial PyTorch reimplementation of the Smooth ReLU (SmeLU) activation function proposed in the paper Real World Large Scale

DC3: A Learning Method for Optimization with Hard Constraints

DC3: A learning method for optimization with hard constraints This repository is by Priya L. Donti, David Rolnick, and J. Zico Kolter and contains the

Comments
  • Normalizing bone length by Reference Bone Length

    Normalizing bone length by Reference Bone Length

    Hi @MengHao666 , first of all thank you for your implementation.

    I have a doubt with relation to the reference bone normalization in the BMCLoss Class. When you compute the min and max values for the bone lengths in the calculate_bmc.py code, all bones are normalized by the reference bone, as shown below:

    https://github.com/MengHao666/Hand-BMC-pytorch/blob/d9d1a5394ebc84079e9fc25885122c18a9ec1505/calculate_bmc.py#L50-L57

    But in the BMCLoss Class this step is not made, which could generate wrong results if the input data isn't normalized. Is this right?

    https://github.com/MengHao666/Hand-BMC-pytorch/blob/d9d1a5394ebc84079e9fc25885122c18a9ec1505/weakloss.py#L253-L278

    Also, did you think what is the easiest way to use the algorithm for the left hand?

    Thanks in advance.

    opened by thiagomcoutinho 1
  • Can't find the rdp algorithm

    Can't find the rdp algorithm

    Hello,thank you for your sharing. When I run the 'python calculate_convex_hull.py', it could't find the 'rdp' module. Thus it will be very greatfull if you can help this.

    opened by LiuYi-Up 1
  • Error when batch size=1

    Error when batch size=1

    File "/data_sdc/gzh/tmp/neural_render_learn/losses/BMC_loss.py", line 355, in loss_total, loss_dict = bmc.compute_loss(joints) File "/data_sdc/gzh/tmp/neural_render_learn/losses/BMC_loss.py", line 326, in compute_loss TIP_X_axis = torch.matmul(temp_R, DIP_X_axis.unsqueeze(3)).squeeze() IndexError: Dimension out of range (expected to be in range of [-3, 2], but got 3)

    opened by Guocode 1
Owner
Hao Meng
Master student at Beihang University , mainly interested in hand pose estimation.
Hao Meng
A hand tracking demo made with mediapipe where you can control lights with pinching your fingers and moving your hand up/down.

HandTrackingBrightnessControl A hand tracking demo made with mediapipe where you can control lights with pinching your fingers and moving your hand up

Teemu Laurila 19 Feb 12, 2022
Hand-distance-measurement-game - Hand Distance Measurement Game

Hand Distance Measurement Game This is program is made to calculate the distance

Priyansh 2 Jan 12, 2022
MohammadReza Sharifi 27 Dec 13, 2022
Code for the ECCV2020 paper "A Differentiable Recurrent Surface for Asynchronous Event-Based Data"

A Differentiable Recurrent Surface for Asynchronous Event-Based Data Code for the ECCV2020 paper "A Differentiable Recurrent Surface for Asynchronous

Marco Cannici 21 Oct 5, 2022
Official implementation of "Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervision" ECCV2020

XDVioDet Official implementation of "Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervision" ECCV2020. The proj

peng 64 Dec 12, 2022
Evaluation Pipeline for our ECCV2020: Journey Towards Tiny Perceptual Super-Resolution.

Journey Towards Tiny Perceptual Super-Resolution Test code for our ECCV2020 paper: https://arxiv.org/abs/2007.04356 Our x4 upscaling pre-trained model

Royson 6 Mar 30, 2022
Code for "Learning Canonical Representations for Scene Graph to Image Generation", Herzig & Bar et al., ECCV2020

Learning Canonical Representations for Scene Graph to Image Generation (ECCV 2020) Roei Herzig*, Amir Bar*, Huijuan Xu, Gal Chechik, Trevor Darrell, A

roei_herzig 24 Jul 7, 2022
text_recognition_toolbox: The reimplementation of a series of classical scene text recognition papers with Pytorch in a uniform way.

text recognition toolbox 1. 项目介绍 该项目是基于pytorch深度学习框架,以统一的改写方式实现了以下6篇经典的文字识别论文,论文的详情如下。该项目会持续进行更新,欢迎大家提出问题以及对代码进行贡献。 模型 论文标题 发表年份 模型方法划分 CRNN 《An End-t

null 168 Dec 24, 2022
PyTorch reimplementation of the paper Involution: Inverting the Inherence of Convolution for Visual Recognition [CVPR 2021].

Involution: Inverting the Inherence of Convolution for Visual Recognition Unofficial PyTorch reimplementation of the paper Involution: Inverting the I

Christoph Reich 100 Dec 1, 2022
A PyTorch Reimplementation of TecoGAN: Temporally Coherent GAN for Video Super-Resolution

TecoGAN-PyTorch Introduction This is a PyTorch reimplementation of TecoGAN: Temporally Coherent GAN for Video Super-Resolution (VSR). Please refer to

null 165 Dec 17, 2022