Forward Propagation, Backward Regression and Pose Association for Hand Tracking in the Wild (CVPR 2022)

Overview

HandLer

This repository contains the code and data for the following paper:

Forward Propagation, Backward Regression, and Pose Association for Hand Tracking in the Wild (CVPR 2022).

We develop a method to detect and track multiple hands in videos.

Installation

Install Torch 1.7.1

pip install torch==1.7.1+cu110 torchvision==0.8.2+cu110 torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html

Install other requirements

pip install -r requirements.txt

Install Detectron2 (version 0.4)

For CUDA 11.0 and Torch 1.7.x

python -m pip install detectron2==0.4 -f \
  https://dl.fbaipublicfiles.com/detectron2/wheels/cu110/torch1.7/index.html

Youtube-Hand Dataset

Please find the Youtube-Hand dataset in the project page.

Go to main folder

cd ./projects/HandLer

Model Weights

Please find the pretraind model weights here:

https://drive.google.com/file/d/1eaZn9E7lHvXY3Fh90f2eIK1P2yZwNfnP/view?usp=sharing

Please find the weights trained on Youtube-Hand training set here:

https://drive.google.com/file/d/1sK5bRTHt5zLOVPruUt-zyZ_AOrBvGQ9-/view?usp=sharing

Hand Tracking Demo: Run HandLer on custom videos

python demo.py --cfg configs/yt_trk.yaml --output output_vid.mp4 --weights /path/to/weights --input input_vid.mp4

Training on Youtube-Hand Dataset

For training on single GPU:

python train_trk.py --config-file configs/yt_trk.yaml

For training on multiple GPUs:

python train_trk.py --config-file configs/yt_trk.yaml --num_gpu 4

Testing on Youtube-Hand Test set

python tracking.py --cfg /path/to/cfg --out_dir /path/to/results/folders --weights /path/to/weights --root_path /path/to/dataset

Evaluating on Youtube-Hand Test set

python evaluations.py /path/to/results/folders

Improving Hand Tracking using Hand-Body Association

By associating hands with human bodies and linking human bodies across frames, we can establish correspondence between detected instances of the same hand across different frames, reducing identity switches in tracking. To know more about this, please check our work Whose Hands Are These? Hand Detection and Hand-Body Association in the Wild

Citation

If you find our code or dataset useful, please cite:

@inproceedings{handler_2022,
      title={Forward Propagation, Backward Regression and Pose Association for Hand Tracking in the Wild},
      author={Mingzhen Huang and Supreeth Narasimhaswamy and Saif Vazir and Haibin Ling and Minh Hoai},
      booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
      year={2022},
  }

@inproceedings{bodyhands_2022,
      title={Whose Hands Are These? Hand Detection and Hand-Body Association in the Wild},
      author={Supreeth Narasimhaswamy and Thanh Nguyen and Mingzhen Huang and Minh Hoai},
      booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
      year={2022},
  }
You might also like...
I was sick of having to hand my friends my phone, so I gave my Spotify some SMS features!

SMSpotifY Just a little tool so that my friends can text a phone number and add to my spotify queue for parties and such:) Features Roles / Access Con

Wrapper around the UPS API for creating shipping labels and fetching a package's tracking status.

ClassicUPS: A Useful UPS Library ClassicUPS is an Apache2 Licensed wrapper around the UPS API for creating shipping labels and fetching a package's tr

A discord bot for tracking Iranian Minecraft servers and showing the statistics of them

A discord bot for tracking Iranian Minecraft servers and showing the statistics of them

This is RequestTrackerBot and it used for tracking request made by user in a group
This is RequestTrackerBot and it used for tracking request made by user in a group

This is a Request Tracker Bot repo, It is for those who upload content like movies, anime, etc. It can be used for tracking request of content that your members asked for.

A discord bot with information and template tracking for pxls.space.
A discord bot with information and template tracking for pxls.space.

pyCharity A discord bot with information and template tracking for pxls.space. Inspired by Mikarific's Charity bot. Try out the beta version on your s

A mood based crypto tracking application.

Crypto Bud - API A mood based crypto tracking application. The main repository is private. I am creating the API before I connect everything to the ma

Fetch tracking numbers of Amazon orders, for the ease of the logistics.
Fetch tracking numbers of Amazon orders, for the ease of the logistics.

Amazon-Tracking-Number Fetch tracking numbers of Amazon orders, for the ease of the logistics. Read Me First (How to use this code): Get Amazon "Items

A GitHub Actions repo for tracking the dummies sending free money to Alex Jones + co.

A GitHub Actions repo for tracking the dummies sending free money to Alex Jones + co.

Python Client for MLflow Tracking Server

Python Client for MLflow Python client for MLflow REST API. Features: Unlike MLflow Tracking client all REST API methods are exposed to user. All clas

Comments
  • missing the config file

    missing the config file

    Hi, thanks for your great work on hand tracking. I am so interested in your work and want to run inference on my custom videos, but find the config file yt_trk.yaml is missing. When would you will share the config file, looking forward to your reply.

    documentation 
    opened by xinxiaozhe12345 1
  • Youtube-Hands Dataset does not contain pose_hm

    Youtube-Hands Dataset does not contain pose_hm

    Hello, there!! Thank you very much for your great work. It is very interesting. I have one question about Youtube-Hands dataset. When I try training the model on Youtube-Hands dataset, I receive an error saying "No such file or directory: /Youtube-Hands/pose_hm/train/kids_playing_v11/img1/frame000556.npy" Could you please guide me where I can find such file? Thank you very much in advance for your consideration.

    📚 Documentation

    • Links to the relevant documentation/comment:
    documentation 
    opened by Matus-Tanonwong 4
Owner
CVLab@StonyBrook
Computer Vision Lab at Stony Brook University
CVLab@StonyBrook
🤖 The bot that runs the official Fairfield Programming Association Discord server.

?? The bot that runs the official Fairfield Programming Association Discord server.

Fairfield Programming Association 1 Jan 7, 2022
Upload on Doodstream by Url, File and also by direct forward post from other channel...

Upload on Doodstream by Url, File and also by direct forward post from other channel...

Pʀᴇᴅᴀᴛᴏʀ 8 Aug 10, 2022
A library for demo trading | backtest and forward test simulation

Trade Engine a library for demo trading | backtest and forward test simulation Features Limit/Market orders: you can place a Limit or Market order in

Ali Moradi 7 Jul 2, 2022
A telegram bot to forward messages automatically when they arrived.

Telegram Message Forwarder Bot A telegram bot, which can forward messages from channel, group or chat to another channel, group or chat automatically.

Adnan Ahmad 181 Jan 7, 2023
A simple telegram bot to forward files from one channel to other.

Forward_2.0 Bot to forward messages from one channel to other without admin permission in source channel. Can be used for both private and Public chan

SUBIN 56 Dec 29, 2022
A script to forward mass number of media to another group/channel. Heroku deploy

Telegram Forward Script ?? This is a Script to Forward Large Number of Files to Another Telegram Channel. Star එකක් දාල fork එකක් ගහපියව් ?? If You Tr

Anjana Madu 17 Oct 21, 2022
Auto file forward bot with python

Auto-File-Forward-Bot Auto file forward bot. Without Admin Permission in FROM_CHANNEL Only Give Permission In your Telegram Personal Channel Please fo

Milas 1 Oct 15, 2021
A simple telegram bot to help you to remove forward tag from post from any messages . Maded in python3 using @Pyrogram . Developed by @Kunal-Diwan

Frwd-Tag-Remover Telegram Bot to Remove forward tag from any Post . If you need any more modes in repo or If you find out any bugs, mention in @Develo

Kunal Diwan 2 Oct 14, 2022
An API serving data on all creatures, monsters, materials, equipment, and treasure in The Legend of Zelda: Breath of the Wild

Hyrule Compendium API An API serving data on all creatures, monsters, materials, equipment, and treasure in The Legend of Zelda: Breath of the Wild. B

Aarav Borthakur 116 Dec 1, 2022
OGE-2022-na-Python - Solving problems in python for the OGE 2022

OGE-2022-na-Python Решение задачек на питоне для ОГЭ 2022 Тут разобраны разные в

Slava 0 Oct 14, 2022