HandTailor: Towards High-Precision Monocular 3D Hand Recovery

Related tags

Deep Learning HandTailor
Overview

HandTailor

This repository is the implementation code and model of the paper "HandTailor: Towards High-Precision Monocular 3D Hand Recovery" (arXiv)

Get the Code

git clone https://github.com/LyuJ1998/HandTailor.git
cd HandTailor

Install Requirements

Please install the dependencies listed in requirements.txt.

pip install -r requirements.txt

Download Pretrain Model

Download the pretrain model from Google Drive, and put the model.pt in ./checkpoints

Demo

To process the image provided in ./demo, run

python demo.py

You can also put your data in the fold ./demo, but remember to use the proper camera intrinsic like

python demo.py --fx=612.0206 --fy=612.2821 --cx=321.2842 --cy=235.8609

If camera information is unavailable, run

python demo_in_the_wild.py

We recommand you to utilize the camera intrinsic, which will improve the performance a lot.

Realtime Demo

To reconstruct the hand from image captured with a webcam,run the following command. Also remember to use the proper camera intrinsic, the following command is for RealSense D435

python app.py --fx=612.0206 --fy=612.2821 --cx=321.2842 --cy=235.8609

When camera information is absence

python app_in_the_wild.py

Citation

If you find this work helpful, please consider citing us

@article{lv2021handtailor,
  title={HandTailor: Towards High-Precision Monocular 3D Hand Recovery},
  author={Lv, Jun and Xu, Wenqiang and Yang, Lixin and Qian, Sucheng and Mao, Chongzhao and Lu, Cewu},
  journal={arXiv preprint arXiv:2102.09244},
  year={2021}
}
Comments
  • Questions on optimization/tailor module

    Questions on optimization/tailor module

    Hiya, I had a few questions about the tailor module

    Just as a high-level overview (just to make sure I correctly understand the approach). It seems that you want to refine the predicted pose parameters from the inverse kinematics network with the insight that the 2d keypoints will generally have higher accuracy/ reliability allowing for not only a more accurate prediction but also helping with bad initializations and occlusion problems in typical optimization-based techniques face (along with fewer iterations due to a good initial condition)

    1. Did you experiment with different optimizers in jax other than Adam? Such as the Levenberg-Marquat solver (not sure if this is implemented in jax) used in papers like MegaTrack/RGB2Hands/Online Optical Marker-based Hand Tracking with Deep Labels
    2. For the in-the-wild examples, I've noticed certain poses lead to failure cases for estimating the scale camera parameter, is this something you've noticed as well? usually when the hand is horizontal rather than vertical

    I enjoyed reading your paper greatly! Thank you so much for the contribution and I look forward to your response

    opened by pablovela5620 6
  • Got weird output if using different camera intrinsics

    Got weird output if using different camera intrinsics

    @lixiny I got pretty weird results, when used logitech c270 camera with it's intrisics.

    I used this command, python demo.py --fx 1078.03779 --fy 1084.50988 --cx 580.850545 --cy 245.959325

    I have checked the camera matrix multiple times, and it seems okay. Output Video link: https://drive.google.com/file/d/10oH3cL92bcvFLr1HTEfp5QOTn2WvBpVM/view?usp=sharing

    opened by coreqode 5
  • align_idx

    align_idx

    Hi, I was wondering that why we use align_idx=9, instead of root_idx 0. Is there any reason? If we will align by root before computing loss anyway, why don't we align by root in the beginning?

    opened by ZhengdiYu 3
  • Joints' indeces clarification

    Joints' indeces clarification

    Great work you have done with this research @LyuJ1998 . I just have a question on the joint's location converted here from uvd to xyz;

    • What is the index for each joint in the converted locations?
    opened by OlafenwaMoses 2
  • root depth prediction question

    root depth prediction question

    I'm trying to understand where you got common.DEPTH_RANGE and common. DEPTH_MIN that are used here. In your paper you specify that you scale normalized the predicted root depth when using the perspective projection. Are these values what are used to unnormalized the predicted root depth? I may have incorrectly thought that normalization had to do with the length from the root to the MCP joint (or whatever the chosen joint was), is that not the case when predicting root depth

    opened by pablovela5620 2
  • In the wild model scale error/typo

    In the wild model scale error/typo

    I believe this is a typo https://github.com/LyuJ1998/HandTailor/blob/080818f66c3c8eaadd7867c4c9db093fe1c8b745/model.py#L414 where you're using the u coordinate twice instead of u-v according to this

    image

    it should be

    s = torch.sqrt((deltaj[0,0]**2 + deltaj[0,1]**2)/(1 - deltaj[0,2]**2))
    opened by pablovela5620 1
Owner
Lv Jun
Lv Jun
GitHub
Quantization library for PyTorch. Support low-precision and mixed-precision quantization, with hardware implementation through TVM.

HAWQ: Hessian AWare Quantization HAWQ is an advanced quantization library written for PyTorch. HAWQ enables low-precision and mixed-precision uniform

Zhen Dong 293 Dec 30, 2022
A hand tracking demo made with mediapipe where you can control lights with pinching your fingers and moving your hand up/down.

HandTrackingBrightnessControl A hand tracking demo made with mediapipe where you can control lights with pinching your fingers and moving your hand up

Teemu Laurila 19 Feb 12, 2022
Hand-distance-measurement-game - Hand Distance Measurement Game

Hand Distance Measurement Game This is program is made to calculate the distance

Priyansh 2 Jan 12, 2022
Control-Raspberry-Pi-Robot-using-Hand-Gestures - A 4WD Robot car based on Raspberry Pi that controlled by hand gestures(using openCV and mediapipe)

Control-Raspberry-Pi-Robot-using-Hand-Gestures you can see all details about thi

MohammadReza Sharifi 27 Dec 13, 2022
Official code release for: EditGAN: High-Precision Semantic Image Editing

Official code release for: EditGAN: High-Precision Semantic Image Editing

null 565 Jan 5, 2023
code for paper"A High-precision Semantic Segmentation Method Combining Adversarial Learning and Attention Mechanism"

PyTorch implementation of UAGAN(U-net Attention Generative Adversarial Networks) This repository contains the source code for the paper "A High-precis

Tong 8 Apr 25, 2022
Towards uncontrained hand-object reconstruction from RGB videos

Towards uncontrained hand-object reconstruction from RGB videos Yana Hasson, Gül Varol, Ivan Laptev and Cordelia Schmid Project page Paper Table of Co

Yana 69 Dec 27, 2022
Boosting Monocular Depth Estimation Models to High-Resolution via Content-Adaptive Multi-Resolution Merging

Boosting Monocular Depth Estimation Models to High-Resolution via Content-Adaptive Multi-Resolution Merging This repository contains an implementation

Computational Photography Lab @ SFU 1.1k Jan 2, 2023
DeFMO: Deblurring and Shape Recovery of Fast Moving Objects (CVPR 2021)

Evaluation, Training, Demo, and Inference of DeFMO DeFMO: Deblurring and Shape Recovery of Fast Moving Objects (CVPR 2021) Denys Rozumnyi, Martin R. O

Denys Rozumnyi 139 Dec 26, 2022
Learning Energy-Based Models by Diffusion Recovery Likelihood

Learning Energy-Based Models by Diffusion Recovery Likelihood Ruiqi Gao, Yang Song, Ben Poole, Ying Nian Wu, Diederik P. Kingma Paper: https://arxiv.o

Ruiqi Gao 41 Nov 22, 2022
[ICCV'21] PlaneTR: Structure-Guided Transformers for 3D Plane Recovery

PlaneTR: Structure-Guided Transformers for 3D Plane Recovery This is the official implementation of our ICCV 2021 paper News There maybe some bugs in

null 73 Nov 30, 2022
Unsupervised 3D Human Mesh Recovery from Noisy Point Clouds

Unsupervised 3D Human Mesh Recovery from Noisy Point Clouds Xinxin Zuo, Sen Wang, Minglun Gong, Li Cheng Prerequisites We have tested the code on Ubun

null 41 Dec 12, 2022
(ICCV 2021) ProHMR - Probabilistic Modeling for Human Mesh Recovery

ProHMR - Probabilistic Modeling for Human Mesh Recovery Code repository for the paper: Probabilistic Modeling for Human Mesh Recovery Nikos Kolotouros

Nikos Kolotouros 209 Dec 13, 2022
Source code for Acorn, the precision farming rover by Twisted Fields

Acorn precision farming rover This is the software repository for Acorn, the precision farming rover by Twisted Fields. For more information see twist

Twisted Fields 198 Jan 2, 2023
A PyTorch Extension: Tools for easy mixed precision and distributed training in Pytorch

This repository holds NVIDIA-maintained utilities to streamline mixed precision and distributed training in Pytorch. Some of the code here will be included in upstream Pytorch eventually. The intention of Apex is to make up-to-date utilities available to users as quickly as possible.

NVIDIA Corporation 6.9k Jan 3, 2023
[ICLR 2021] "CPT: Efficient Deep Neural Network Training via Cyclic Precision" by Yonggan Fu, Han Guo, Meng Li, Xin Yang, Yining Ding, Vikas Chandra, Yingyan Lin

CPT: Efficient Deep Neural Network Training via Cyclic Precision Yonggan Fu, Han Guo, Meng Li, Xin Yang, Yining Ding, Vikas Chandra, Yingyan Lin Accep

null 26 Oct 25, 2022
EdMIPS: Rethinking Differentiable Search for Mixed-Precision Neural Networks

EdMIPS is an efficient algorithm to search the optimal mixed-precision neural network directly without proxy task on ImageNet given computation budgets. It can be applied to many popular network architectures, including ResNet, GoogLeNet, and Inception-V3.

Zhaowei Cai 47 Dec 30, 2022
Provide partial dates and retain the date precision through processing

Prefix date parser This is a helper class to parse dates with varied degrees of precision. For example, a data source might state a date as 2001, 2001

Friedrich Lindenberg 13 Dec 14, 2022
This is the pytorch implementation for the paper: Generalizable Mixed-Precision Quantization via Attribution Rank Preservation, which is accepted to ICCV2021.

GMPQ: Generalizable Mixed-Precision Quantization via Attribution Rank Preservation This is the pytorch implementation for the paper: Generalizable Mix

null 18 Sep 2, 2022