PyTorch reimplementation of minimal-hand (CVPR2020)

Overview

Minimal Hand Pytorch

Unofficial PyTorch reimplementation of minimal-hand (CVPR2020).

demo demo

you can also find in youtube or bilibili

This project reimplement following components :

  1. Training (DetNet) and Evaluation Code
  2. Shape Estimation
  3. Pose Estimation: Instead of IKNet in original paper, an analytical inverse kinematics method is used.

Offical project link: [minimal-hand]

Update

  • 2021/03/09 update about utils/LM.py, time cost drop from 12s/item to 1.57s/item

  • 2021/03/12 update about utils/LM.py, time cost drop from 1.57s/item to 0.27s/item

  • 2021/03/17 realtime perfomance is achieved when using PSO to estimate shape, coming soon

  • 2021/03/20 Add PSO to estimate shape. AUC is decreased by about 0.01 on STB and RHD datasets, and increased a little on EO and do datasets. Modifiy utlis/vis.py to improve realtime perfomance

  • 2021/03/24 Fixed some errors in calculating AUC. Update the 3D PCK AUC Diffenence.

Usage

  • Retrieve the code
git clone https://github.com/MengHao666/Minimal-Hand-pytorch
cd Minimal-Hand-pytorch
  • Create and activate the virtual environment with python dependencies
conda env create --file=environment.yml
conda activate minimal-hand-torch

Prepare MANO hand model

  1. Download MANO model from here and unzip it.

  2. Create an account by clicking Sign Up and provide your information

  3. Download Models and Code (the downloaded file should have the format mano_v*_*.zip). Note that all code and data from this download falls under the MANO license.

  4. unzip and copy the content of the models folder into the mano folder

  5. Your structure should look like this:

Minimal-Hand-pytorch/
   mano/
      models/
      webuser/

Download and Prepare datasets

Training dataset

Evaluation dataset

Processing

  • Create a data directory, extract all above datasets or additional materials in it

Now your data folder structure should like this:

data/

    CMU/
        hand143_panopticdb/
            datasets/
            ...
        hand_labels/
            datasets/
            ...

    RHD/
        RHD_published_v2/
            evaluation/
            training/
            view_sample.py
            ...

    GANeratedHands_Release/
        data/
        ...

    STB/
        images/
            B1Counting/
                SK_color_0.png
                SK_depth_0.png
                SK_depth_seg_0.png  <-- merged from STB_supp
                ...
            ...
        labels/
            B1Counting_BB.mat
            ...

    dexter+object/
        calibration/
        bbox_dexter+object.csv
        DO_pred_2d.npy
        data/
            Grasp1/
                annotations/
                    Grasp13D.txt
                    my_Grasp13D.txt
                    ...
                ...
            Grasp2/
                annotations/
                    Grasp23D.txt
                    my_Grasp23D.txt
                    ...
                ...
            Occlusion/
                annotations/
                    Occlusion3D.txt
                    my_Occlusion3D.txt
                    ...
                ...
            Pinch/
                annotations/
                    Pinch3D.txt
                    my_Pinch3D.txt
                    ...
                ...
            Rigid/
                annotations/
                    Rigid3D.txt
                    my_Rigid3D.txt
                    ...
                ...
            Rotate/
                                annotations/
                    Rotate3D.txt
                    my_Rotate3D.txt
                    ...
                ...
        

    EgoDexter/
        preview/
        data/
            Desk/
                annotation.txt_3D.txt
                my_annotation.txt_3D.txt
                ...
            Fruits/
                annotation.txt_3D.txt
                my_annotation.txt_3D.txt
                ...
            Kitchen/
                annotation.txt_3D.txt
                my_annotation.txt_3D.txt
                ...
            Rotunda/
                annotation.txt_3D.txt
                my_annotation.txt_3D.txt
                ...
        

Note

  • All code and data from these download falls under their own licenses.
  • DO represents "dexter+object" dataset; EO represents "EgoDexter" dataset
  • DO_supp and EO_supp are modified from original ones.
  • DO_pred_2d.npy are 2D predictions from 2D part of DetNet.
  • some labels of DO and EO is obviously wrong (u could find some examples with original labels from dexter_object.py or egodexter.py), when projected into image plane, thus should be omitted. Here come my_{}3D.txt and my_annotation.txt_3D.txt.

Download my Results

realtime demo

python demo.py

DetNet Training and Evaluation

Run the training code

python train_detnet.py --data_root data/

Run the evaluation code

python train_detnet.py --data_root data/  --datasets_test testset_name_to_test   --evaluate  --evaluate_id checkpoints_id_to_load 

or use my results

python train_detnet.py --checkpoint my_results/checkpoints  --datasets_test "rhd" --evaluate  --evaluate_id 106

python train_detnet.py --checkpoint my_results/checkpoints  --datasets_test "stb" --evaluate  --evaluate_id 71

python train_detnet.py --checkpoint my_results/checkpoints  --datasets_test "do" --evaluate  --evaluate_id 68

python train_detnet.py --checkpoint my_results/checkpoints  --datasets_test "eo" --evaluate  --evaluate_id 101

Shape Estimation

Run the shape optimization code. This can be very time consuming when the weight parameter is quite small.

python optimize_shape.py --weight 1e-5

or use my results

python optimize_shape.py --path my_results/out_testset/

Pose Estimation

Run the following code which uses a analytical inverse kinematics method.

python aik_pose.py

or use my results

python aik_pose.py --path my_results/out_testset/

Detnet training and evaluation curve

Run the following code to see my results

python plot.py --path my_results/out_loss_auc

(AUC means 3D PCK, and ACC_HM means 2D PCK) teaser

3D PCK AUC Diffenence

* means this project

Dataset DetNet(paper) DetNet(*) DetNet+IKNet(paper) DetNet+LM+AIK(*) DetNet+PSO+AIK(*)
RHD - 0.9339 0.856 0.9301 0.9310
STB 0.891 0.8744 0.898 0.8647 0.8671
DO 0.923 0.9378 0.948 0.9392 0.9342
EO 0.804 0.9270 0.811 0.9288 0.9277

Note

  • Adjusting training parameters carefully, longer training time, more complicated network or Biomechanical Constraint Losses could further boost accuracy.
  • As there is no official open source of original paper, above comparison is a little rough.

Citation

This is the unofficial pytorch reimplementation of the paper "Monocular Real-time Hand Shape and Motion Capture using Multi-modal Data" (CVPR 2020).

If you find the project helpful, please star this project and cite them:

@inproceedings{zhou2020monocular,
  title={Monocular Real-time Hand Shape and Motion Capture using Multi-modal Data},
  author={Zhou, Yuxiao and Habermann, Marc and Xu, Weipeng and Habibie, Ikhsanul and Theobalt, Christian and Xu, Feng},
  booktitle={Proceedings of the IEEE International Conference on Computer Vision},
  pages={0--0},
  year={2020}
}

Acknowledgement

  • Code of Mano Pytorch Layer was adapted from manopth.

  • Code for evaluating the hand PCK and AUC in utils/eval/zimeval.py was adapted from hand3d.

  • Part code of data augmentation, dataset parsing and utils were adapted from bihand and 3D-Hand-Pose-Estimation.

  • Code of network model was adapted from Minimal-Hand.

  • @Mrsirovo for the starter code of the utils/LM.py, @maitetsu update it later.

  • @maitetsu for the starter code of the utils/AIK.py

Issues
  • error in running demo.py

    error in running demo.py

    image

    i am getting this issue on running demo.py. can you help me in resolving this error.

    opened by kishan1823 12
  • what is the code  result ?

    what is the code result ?

    Hello,the final result of your code is a model of OBJ file (which can be imported into 3D Max) or is it just a hand of 3d image displayed in the picture, such as looking at the picture representation: 图片 or obj file like this: 图片

    opened by LingSmart 9
  • Cannot plot 3d points

    Cannot plot 3d points

    image

    Hi, i am not get the third window i.e 3d plot of keypoints of hand. Can you suggest any solution.

    opened by kishan1823 8
  • problems about the Delta map and L map groundtruth

    problems about the Delta map and L map groundtruth

    delta_map = kin_chain[:, :, np.newaxis, np.newaxis].repeat(32, axis=-2).repeat(32, axis=-1) It seems to make the value in W/H the same. is it right? We only need to train the corresponding position of the groundtruth. But we find the position of maximum value of the predicted heatmap to match the groundthuth.

    opened by aqsc 7
  • Unable to install some dependencies

    Unable to install some dependencies

    Hi! When I tried using the command 'conda env create --file=environment.yml', it threw me the following error:

    Collecting package metadata (repodata.json): done Solving environment: failed

    ResolvePackageNotFound:

    • pcre==8.44=he6710b0_0
    • pixman==0.40.0=h7b6447c_0
    • tornado==6.1=py37h27cfd23_0
    • libopencv==3.4.2=hb342d67_1
    • ld_impl_linux-64==2.33.1=h53a641e_7
    • qt==5.9.7=h5867ecd_1
    • libgfortran-ng==7.3.0=hdf63c60_0
    • freetype==2.10.4=h5ab3b9f_0
    • pandas==1.2.2=py37ha9443f7_0
    • sip==4.19.8=py37hf484d3e_0
    • libopus==1.3.1=h7b6447c_0
    • numpy-base==1.19.2=py37hfa32c7d_0
    • cffi==1.14.5=py37h261ae71_0
    • libxcb==1.14=h7b6447c_0
    • zstd==1.4.5=h9ceee32_0
    • readline==8.1=h27cfd23_0
    • libuv==1.40.0=h7b6447c_0
    • pyqt==5.9.2=py37h05f1152_2
    • harfbuzz==1.8.8=hffaf4a1_0
    • libuuid==1.0.3=h1bed415_2
    • gstreamer==1.14.0=h28cd5cc_2
    • torchvision==0.4.0=py37_cu100
    • libffi==3.3=he6710b0_2
    • ca-certificates==2021.1.19=h06a4308_0
    • lz4-c==1.9.3=h2531618_0
    • mkl_fft==1.3.0=py37h54f3939_0
    • expat==2.2.10=he6710b0_2
    • libtiff==4.1.0=h2733197_1
    • libgcc-ng==9.1.0=hdf63c60_0
    • libstdcxx-ng==9.1.0=hdf63c60_0
    • xz==5.2.5=h7b6447c_0
    • icu==58.2=he6710b0_3
    • glib==2.67.4=h36276a3_1
    • setuptools==52.0.0=py37h06a4308_0
    • pillow==8.1.0=py37he98fc37_0
    • termcolor==1.1.0=py37h06a4308_1
    • zlib==1.2.11=h7b6447c_3
    • libpng==1.6.37=hbc83047_0
    • python==3.7.10=hdb3f193_0
    • mkl==2020.2=256
    • scipy==1.6.1=py37h91f5cce_0
    • kiwisolver==1.3.1=py37h2531618_0
    • jasper==2.0.14=h07fcdf6_1
    • ncurses==6.2=he6710b0_1
    • mkl-service==2.3.0=py37he8ac12f_0
    • numpy==1.19.2=py37h54aff64_0
    • libxml2==2.9.10=hb55368b_3
    • pytorch==1.2.0=py3.7_cuda10.0.130_cudnn7.6.2_0
    • hdf5==1.10.2=hba1933b_1
    • py-opencv==3.4.2=py37hb342d67_1
    • ffmpeg==4.0=hcdf2ecd_0
    • pip==21.0.1=py37h06a4308_0
    • lcms2==2.11=h396b838_0
    • intel-openmp==2020.2=254
    • libvpx==1.7.0=h439df22_0
    • libglu==9.0.0=hf484d3e_1
    • matplotlib-base==3.3.4=py37h62a2d02_0
    • bzip2==1.0.8=h7b6447c_0
    • dbus==1.13.18=hb2f20db_0
    • mkl_random==1.1.1=py37h0573a6f_0
    • openssl==1.1.1j=h27cfd23_0
    • certifi==2020.12.5=py37h06a4308_0
    • fontconfig==2.13.1=h6c09931_0
    • graphite2==1.3.14=h23475e2_0
    • cudatoolkit==10.0.130=0
    • ninja==1.10.2=py37hff7bd54_0
    • tk==8.6.10=hbc83047_0
    • matplotlib==3.3.4=py37h06a4308_0
    • libedit==3.1.20191231=h14c3975_1
    • cairo==1.14.12=h8948797_3
    • freeglut==3.0.0=hf484d3e_5
    • sqlite==3.33.0=h62c20be_0
    • gst-plugins-base==1.14.0=h8213a91_2
    • jpeg==9b=h024ee3a_2

    Do I need to install all these dependencies individually or should the above command have worked as it is? Please help me out as installing them individually is also not working with pip. Further, let me know if I need to use some package manager other than pip for individual installation.

    Thanks in advance!

    opened by sidoodler 6
  • The filedata

    The filedata"STB" is missing and another file data “dexter+object” is different of data structure

    hello , there are three questions 1.your data is STB datasets I cant open the website,can you send my email ([email protected])? thank you! 2.another file data “dexter+object” is different of data structure see the follow picture. 图片 in my unzip file "dexter+object",there is missing file "bbox_dexter+object.csv" and "Do_pred_2d.npy"。 3.in code demo.py -"check_point = torch.load('new_check_point/ckp_detnet_83.pth', map_location=device)" run demo.py ,then result:Traceback (most recent call last): File "G:/Minimal-Hand-pytorch-main/demo.py", line 16, in check_point = torch.load('new_check_point/ckp_detnet_83.pth', map_location=device) File "G:\software-setup\python 3.6.7-setup\lib\site-packages\torch\serialization.py", line 366, in load load model start f = open(f, 'rb') FileNotFoundError: [Errno 2] No such file or directory: 'new_check_point/ckp_detnet_83.pth' "new_check_point/ckp_detnet_83.pth" the file i don't find it. please answer this .thank you !

    opened by LingSmart 4
  • trained model result is not good

    trained model result is not good

    Hi, I trained model following the instruction completely. But I got the result is that:

    图片

    another your model result: 图片

    opened by Gzzgz 4
  • about inference speed

    about inference speed

    Can you tell me that you test on GPU or CPU? THX

    opened by EEWenbinWu 4
  • Environment file import failed

    Environment file import failed

    Snipaste_2021-07-12_07-31-31

    opened by 14725836900 4
  • does analytical IK improves the performance over IKNet

    does analytical IK improves the performance over IKNet

    based on your experiments log, does analytical IK improves the performance over IKNet?

    opened by wangzheallen 2
  • shape_net still  have error

    shape_net still have error

    hello. I changed the path to an absolute path, so it should be no problem.but another question happen. i dont change this code.i dont know why this is . 图片

    opened by LingSmart 0
  • FileNotFoundError: [Errno 2] No such file or directory: 'mano/models\\MANO_RIGHT.pkl'

    FileNotFoundError: [Errno 2] No such file or directory: 'mano/models\\MANO_RIGHT.pkl'

    i have the file in this directory ,but still have this error,how can i solve it ? 图片

    opened by LingSmart 5
  • FileNotFoundError: [Errno 2] No such file or directory: 'out_testset/rhd_pso.npy'

    FileNotFoundError: [Errno 2] No such file or directory: 'out_testset/rhd_pso.npy'

    the file (rhd_pso.npy) is not find.I didn't find any pattern files,for example do_pso.npy,eo_pso.npy.stb_pso.npy 。 图片

    opened by LingSmart 1
  • Environment installation fails

    Environment installation fails

    Hey, I tried installing the environment, unfortunately, it fails either due to manopth or the MANO models. Can you help me out? Anything I need to specify differently?

    (minimal-hand-torch) [email protected]:~/git/minimal-hand-pytorch$ python demo.py
    Traceback (most recent call last):
      File "demo.py", line 3, in <module>
        from manopth import manolayer
      File "/home/AD/muellelu/xconda3/envs/minimal-hand-torch/lib/python3.7/site-packages/manopth/manolayer.py", line 7, in <module>
        from mano.webuser.smpl_handpca_wrapper_HAND_only import ready_arguments
      File "/home/AD/muellelu/git/minimal-hand-pytorch/mano/webuser/smpl_handpca_wrapper_HAND_only.py", line 137
        print 'FINITO'
                     ^
    SyntaxError: Missing parentheses in call to 'print'. Did you mean print('FINITO')?
    (minimal-hand-torch) [email protected]:~/git/minimal-hand-pytorch$ 
    
    opened by Haydnspass 7
  • ZeroDivisionError: float division by zero

    ZeroDivisionError: float division by zero

    我不能仅仅用“cmu”数据集去train,在hand143_panopticdb.py中,我发现他们不能正确的使用3D数据 raceback (most recent call last): File "/ssd3/hfang/Mini-hand2/train_detnet.py", line 547, in main(parser.parse_args()) File "/ssd3/hfang/Mini-hand2/train_detnet.py", line 160, in main args=args, loss_all=loss_all File "/ssd3/hfang/Mini-hand2/train_detnet.py", line 348, in train am_loss_dm.update(losses['det_dm'].item(), targets['batch_3d_size'].item()) File "/ssd3/hfang/Mini-hand2/utils/eval/evalutils.py", line 26, in update self.avg = self.sum / self.count ZeroDivisionError: float division by zero

    opened by fanghao123-qw 1
  • "ckp_siknet_synth_41.pth.tar"have a error

    there is a error,how to solve ?

    RuntimeError: G:/Minimal-Hand-pytorch/my_results/checkpoints/ckp_siknet_synth_41.pth.tar is a zip archive (did you mean to use torch.jit.load()?)

    图片

    opened by LingSmart 10
  • about the shape estimation

    about the shape estimation

    An issuefrom the official repo.

    Hi, I am wondering, once the pose and shape parameters are estimated, the absolute scale of the hand model will also be recovered. Does it mean that the network estimates an absolute-scale hand model from a single image? I am a new hand in this area, is the MANO model scale-independent?

    opened by zjhthu 7
Owner
Hao Meng
Master student at Beihang University , mainly interested in hand pose estimation. (LOOKING FOR RESEARCH INTERNSHIP NOW.)
Hao Meng
A hand tracking demo made with mediapipe where you can control lights with pinching your fingers and moving your hand up/down.

HandTrackingBrightnessControl A hand tracking demo made with mediapipe where you can control lights with pinching your fingers and moving your hand up

Teemu Laurila 18 Oct 24, 2021
A minimal solution to hand motion capture from a single color camera at over 100fps. Easy to use, plug to run.

Minimal Hand A minimal solution to hand motion capture from a single color camera at over 100fps. Easy to use, plug to run. This project provides the

Yuxiao Zhou 694 Nov 23, 2021
Pytorch implementation of CVPR2020 paper “VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation”

VectorNet Re-implementation This is the unofficial pytorch implementation of CVPR2020 paper "VectorNet: Encoding HD Maps and Agent Dynamics from Vecto

null 59 Nov 14, 2021
Code for the Active Speakers in Context Paper (CVPR2020)

Active Speakers in Context This repo contains the official code and models for the "Active Speakers in Context" CVPR 2020 paper. Before Training The c

null 34 Aug 26, 2021
Code for Referring Image Segmentation via Cross-Modal Progressive Comprehension, CVPR2020.

CMPC-Refseg Code of our CVPR 2020 paper Referring Image Segmentation via Cross-Modal Progressive Comprehension. Shaofei Huang*, Tianrui Hui*, Si Liu,

spyflying 42 Oct 4, 2021
This is an implementation for the CVPR2020 paper "Learning Invariant Representation for Unsupervised Image Restoration"

Learning Invariant Representation for Unsupervised Image Restoration (CVPR 2020) Introduction This is an implementation for the paper "Learning Invari

GarField 77 Nov 24, 2021
Densely Connected Search Space for More Flexible Neural Architecture Search (CVPR2020)

DenseNAS The code of the CVPR2020 paper Densely Connected Search Space for More Flexible Neural Architecture Search. Neural architecture search (NAS)

Jamin Fong 281 Nov 24, 2021
An unofficial implementation of "Unpaired Image Super-Resolution using Pseudo-Supervision." CVPR2020

UnpairedSR An unofficial implementation of "Unpaired Image Super-Resolution using Pseudo-Supervision." CVPR2020 turn RCAN(modified) --> xmodel(xilinx

JiaKui Hu 1 Nov 26, 2021
text_recognition_toolbox: The reimplementation of a series of classical scene text recognition papers with Pytorch in a uniform way.

text recognition toolbox 1. 项目介绍 该项目是基于pytorch深度学习框架,以统一的改写方式实现了以下6篇经典的文字识别论文,论文的详情如下。该项目会持续进行更新,欢迎大家提出问题以及对代码进行贡献。 模型 论文标题 发表年份 模型方法划分 CRNN 《An End-t

null 150 Nov 24, 2021
PyTorch reimplementation of the paper Involution: Inverting the Inherence of Convolution for Visual Recognition [CVPR 2021].

Involution: Inverting the Inherence of Convolution for Visual Recognition Unofficial PyTorch reimplementation of the paper Involution: Inverting the I

Christoph Reich 84 Nov 22, 2021
A PyTorch Reimplementation of TecoGAN: Temporally Coherent GAN for Video Super-Resolution

TecoGAN-PyTorch Introduction This is a PyTorch reimplementation of TecoGAN: Temporally Coherent GAN for Video Super-Resolution (VSR). Please refer to

null 121 Nov 29, 2021
a reimplementation of Optical Flow Estimation using a Spatial Pyramid Network in PyTorch

pytorch-spynet This is a personal reimplementation of SPyNet [1] using PyTorch. Should you be making use of this work, please cite the paper according

Simon Niklaus 230 Nov 27, 2021
PyTorch reimplementation of REALM and ORQA

PyTorch reimplementation of REALM and ORQA

Li-Huai (Allan) Lin 6 Nov 24, 2021
a reimplementation of UnFlow in PyTorch that matches the official TensorFlow version

pytorch-unflow This is a personal reimplementation of UnFlow [1] using PyTorch. Should you be making use of this work, please cite the paper according

Simon Niklaus 124 Nov 22, 2021
a reimplementation of LiteFlowNet in PyTorch that matches the official Caffe version

pytorch-liteflownet This is a personal reimplementation of LiteFlowNet [1] using PyTorch. Should you be making use of this work, please cite the paper

Simon Niklaus 326 Nov 22, 2021
a reimplementation of Holistically-Nested Edge Detection in PyTorch

pytorch-hed This is a personal reimplementation of Holistically-Nested Edge Detection [1] using PyTorch. Should you be making use of this work, please

Simon Niklaus 293 Dec 1, 2021
Pytorch reimplementation of PSM-Net: "Pyramid Stereo Matching Network"

This is a Pytorch Lightning version PSMNet which is based on JiaRenChang/PSMNet. use python main.py to start training. PSM-Net Pytorch reimplementatio

XIAOTIAN LIU 1 Nov 25, 2021
[ECCV 2020] Reimplementation of 3DDFAv2, including face mesh, head pose, landmarks, and more.

Stable Head Pose Estimation and Landmark Regression via 3D Dense Face Reconstruction Reimplementation of (ECCV 2020) Towards Fast, Accurate and Stable

Remilia Scarlet 146 Nov 21, 2021
Reimplementation of the paper `Human Attention Maps for Text Classification: Do Humans and Neural Networks Focus on the Same Words? (ACL2020)`

Human Attention for Text Classification Re-implementation of the paper Human Attention Maps for Text Classification: Do Humans and Neural Networks Foc

Shunsuke KITADA 16 Apr 28, 2021