Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Time

Overview

Semi Hand-Object

Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Time (CVPR 2021). report

Project Page with Videos Teaser

Installation

  • Clone this repository:
    git clone https://github.com/stevenlsw/Semi-Hand-Object.git
  • Install the dependencies by the following command:
    pip install -r requirements.txt

Quick Demo (update soon)

Training and Evaluation on HO3D Dataset

Preparation

  • Download the MANO model files (mano_v1_2.zip) from MANO website. Unzip and put mano/models/MANO_RIGHT.pkl into assets/mano_models.

  • Download the YCB-Objects used in HO3D dataset. Put unzipped folder object_models under assets.

  • The structure should look like this:

Semi-Hand-Object/
  assets/
    mano_models/
      MANO_RIGHT.pkl
    object_models/
      006_mustard_bottle/
        points.xyz
        textured_simple.obj
      ......
  • Download and unzip HO3D dataset to path you like, the unzipped path is referred as $HO3D_root.

Evaluation

The hand & object pose estimation performance on HO3D dataset. We evaluate hand pose results on the official CodaLab challenge. The hand metric below is mean joint/mesh error after procrustes alignment, the object metric is average object vertices error within 10% of object diameter (ADD-0.1D).

In our model, we use transformer architecture to perform hand-object contextual reasoning.

Please download the trained model and save to path you like, the model path is refered as $resume.

trained-model joint↓ mesh↓ cleanser↑ bottle↑ can↑ ave↑
link 0.99 0.95 92.2 80.4 55.7 76.1
  • Testing with trained model

   python traineval.py --evaluate --HO3D_root={path to the dataset} --resume={path to the model} --test_batch=24 --host_folder=exp_results

The testing results will be saved in the $host_folder, which contains the following files:

  • option.txt (saved options)
  • object_result.txt (object pose evaluation performance)
  • pred.json (zip -j pred.zip pred.json and submit to the offical challenge for hand evaluation)

Training

Please download the preprocessed files to train HO3D dataset. The downloaded files contains training list and labels generated from the original dataset to accelerate training. Please put the unzipped folder ho3d-process to current directory.

    python traineval.py --HO3D_root={path to the dataset} --train_batch=24 --host_folder=exp_results

The models will be automatically saved in $host_folder

Citation

@inproceedings{liu2021semi,
  title={Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Time},
  author={Liu, Shaowei and Jiang, Hanwen and Xu, Jiarui and Liu, Sifei and Wang, Xiaolong},
  booktitle={Proceedings of the IEEE conference on computer vision and pattern recognition},
  year={2021}
}

TODO

  • Google colab demo

Acknowledgments

We thank:

Comments
  • Evaluating the model on SSV2

    Evaluating the model on SSV2

    Hi, I am trying to perform the evaluation on SSV2 (sth-sth) dataset, I've successfully evaluated the trained model on HO3D, however, I am working on a small project for a project at the university and would like to run it on SSV2. Would you be able to point me in the right direction - what part of the dataloader should be changed to be able work with SSV2 and how can I do that? Amazing work.

    opened by aligator022 9
  • How to draw joint figures in your paper?

    How to draw joint figures in your paper?

    Thanks for your great work! I want to know how to draw joint figures in your paper. image

    When I ran your code, I find the 2d joints result (your model output "pred_joints") is below/ image It seems like it's not in the pixel frame. I don't know how to visualize it (like joint figures in your paper). If it's not convenient to show me your code, please just tell me the procedure to do it! Thanks a lot!

    opened by buaacyw 3
  • Some questions about get the results provided by your paper in Table 4(supervised with CR))

    Some questions about get the results provided by your paper in Table 4(supervised with CR))

    Thank you for your excellent work! There some diffculties that I can't get the same or even closed results provided by your paper in Table 4(supervised with CR).And I find the hyper-parameters in the code is not configed as your paper.Do your mand providing your hyper-parameters to help me ? Thank you. Here my logs.

    ====== Options ====== HO3D_root: /datassd/huanyao/semi-hand/Semi-Hand-Object-master/assets/data/ho3d_v2 blocks: 1 channels: 256 epochs: 60 evaluate: False host_folder: /datassd/huanyao/semi-hand/Semi-Hand-Object-master/host_folder inp_res: 512 lambda_joints2d: 100.0 lambda_objects: 500.0 lr: 0.0001 lr_decay_gamma: 0.7 lr_decay_step: 10 mano_lambda_joints3d: 10000.0 mano_lambda_manopose: 10 mano_lambda_manoshape: 0.1 mano_lambda_regulpose: 1 mano_lambda_regulshape: 100.0 mano_lambda_verts3d: 10000.0 mano_neurons: [1024, 512] mano_root: assets/mano_models manual_seed: 0 momentum: 0.9 network: honet_transformer obj_model_root: assets/object_models resume: /datassd/huanyao/semi-hand/Semi-Hand-Object-master/model.pth save_results: True snapshot: 5 stacks: 1 test_batch: 16 test_freq: 10 train_batch: 24 transformer_depth: 1 transformer_head: 1 use_cuda: 1 weight_decay: 0.0005 workers: 16

    opened by julyiii 5
  • YCB objects

    YCB objects

    Hi, your work is inspiring me a lot.

    I found that you are using different a pointcloud from the original data for YCB objects. How did you get it?

    Did you random sample from the original point cloud?

    opened by hongsukchoi 4
  • ho3d-process preprocessing

    ho3d-process preprocessing

    Hello! It's a great project. I am wondering if you could provide the code or some instructions for preprocessing. So that I could use it in other tasks? Thanks in advance!

    opened by BatFaceWayne 2
  • No such file

    No such file "ho3d-process"

    Hi @stevenlsw ,

    I've been trying to evaluate the trained model, but got result shown as below:

     Traceback (most recent call last):
    
        File "traineval.py", line 104, in <module>
            main(args)
        File "traineval.py", line 35, in main
            train_dat = get_dataset(args, mode="train")
        File "/home/yilin/Hand_est/Semi-Hand-Object/utils/utils.py", line 133, in get_dataset
            train_label_root="ho3d-process", mode=mode, inp_res=args.inp_res)
        File "/home/yilin/Hand_est/Semi-Hand-Object/dataset/ho3d.py", line 55, in __init__
            self.set_list = ho3d_util.load_names(os.path.join(train_label_root, "train.txt"))
        File "/home/yilin/Hand_est/Semi-Hand-Object/dataset/ho3d_util.py", line 14, in load_names
            with open(image_path) as f:
        FileNotFoundError: [Errno 2] No such file or directory: 'ho3d-process/train.txt'
    

    And by tracing back to ho3d.py, it seems that train.txt in the HO3D dataset some preprocessed files train_K.json, train_joint.json, train_mano.json and train_obj.json are supposed to be loaded to the ho3d-process path. Am I missing some preprocessing process? Also, is it ok to use HO3D_v3 for training and testing?

    opened by YiLin32Wang 1
Owner
null
A real-time motion capture system that estimates poses and global translations using only 6 inertial measurement units

TransPose Code for our SIGGRAPH 2021 paper "TransPose: Real-time 3D Human Translation and Pose Estimation with Six Inertial Sensors". This repository

Xinyu Yi 261 Dec 31, 2022
A hand tracking demo made with mediapipe where you can control lights with pinching your fingers and moving your hand up/down.

HandTrackingBrightnessControl A hand tracking demo made with mediapipe where you can control lights with pinching your fingers and moving your hand up

Teemu Laurila 19 Feb 12, 2022
Hand-distance-measurement-game - Hand Distance Measurement Game

Hand Distance Measurement Game This is program is made to calculate the distance

Priyansh 2 Jan 12, 2022
MohammadReza Sharifi 27 Dec 13, 2022
An automated algorithm to extract the linear blend skinning (LBS) from a set of example poses

Dem Bones This repository contains an implementation of Smooth Skinning Decomposition with Rigid Bones, an automated algorithm to extract the Linear B

Electronic Arts 684 Dec 26, 2022
Some useful blender add-ons for SMPL skeleton's poses and global translation.

Blender add-ons for SMPL skeleton's poses and trans There are two blender add-ons for SMPL skeleton's poses and trans.The first is for making an offli

犹在镜中 154 Jan 4, 2023
An addon uses SMPL's poses and global translation to drive cartoon character in Blender.

Blender addon for driving character The addon drives the cartoon character by passing SMPL's poses and global translation into model's armature in Ble

犹在镜中 153 Dec 14, 2022
[ICCV-2021] An Empirical Study of the Collapsing Problem in Semi-Supervised 2D Human Pose Estimation

An Empirical Study of the Collapsing Problem in Semi-Supervised 2D Human Pose Estimation (ICCV 2021) Introduction This is an official pytorch implemen

rongchangxie 42 Jan 4, 2023
Re-implementation of the Noise Contrastive Estimation algorithm for pyTorch, following "Noise-contrastive estimation: A new estimation principle for unnormalized statistical models." (Gutmann and Hyvarinen, AISTATS 2010)

Noise Contrastive Estimation for pyTorch Overview This repository contains a re-implementation of the Noise Contrastive Estimation algorithm, implemen

Denis Emelin 42 Nov 24, 2022
UniMoCo: Unsupervised, Semi-Supervised and Full-Supervised Visual Representation Learning

UniMoCo: Unsupervised, Semi-Supervised and Full-Supervised Visual Representation Learning This is the official PyTorch implementation for UniMoCo pape

dddzg 49 Jan 2, 2023
Project looking into use of autoencoder for semi-supervised learning and comparing data requirements compared to supervised learning.

Project looking into use of autoencoder for semi-supervised learning and comparing data requirements compared to supervised learning.

Tom-R.T.Kvalvaag 2 Dec 17, 2021
PyTorch code for ICLR 2021 paper Unbiased Teacher for Semi-Supervised Object Detection

Unbiased Teacher for Semi-Supervised Object Detection This is the PyTorch implementation of our paper: Unbiased Teacher for Semi-Supervised Object Detection

Facebook Research 366 Dec 28, 2022
[CVPR 2021] MiVOS - Mask Propagation module. Reproduced STM (and better) with training code :star2:. Semi-supervised video object segmentation evaluation.

MiVOS (CVPR 2021) - Mask Propagation Ho Kei Cheng, Yu-Wing Tai, Chi-Keung Tang [arXiv] [Paper PDF] [Project Page] [Papers with Code] This repo impleme

Rex Cheng 106 Jan 3, 2023
Semi-Supervised Learning, Object Detection, ICCV2021

End-to-End Semi-Supervised Object Detection with Soft Teacher By Mengde Xu*, Zheng Zhang*, Han Hu, Jianfeng Wang, Lijuan Wang, Fangyun Wei, Xiang Bai,

Microsoft 789 Dec 27, 2022
Instant-Teaching: An End-to-End Semi-Supervised Object Detection Framework

This repo is the official implementation of "Instant-Teaching: An End-to-End Semi-Supervised Object Detection Framework". @inproceedings{zhou2021insta

null 34 Dec 31, 2022
Data-Uncertainty Guided Multi-Phase Learning for Semi-supervised Object Detection

An official implementation of paper Data-Uncertainty Guided Multi-Phase Learning for Semi-supervised Object Detection

null 11 Nov 23, 2022
CVPR2022 paper "Dense Learning based Semi-Supervised Object Detection"

[CVPR2022] DSL: Dense Learning based Semi-Supervised Object Detection DSL is the first work on Anchor-Free detector for Semi-Supervised Object Detecti

Bhchen 69 Dec 8, 2022
Group R-CNN for Point-based Weakly Semi-supervised Object Detection (CVPR2022)

Group R-CNN for Point-based Weakly Semi-supervised Object Detection (CVPR2022) By Shilong Zhang*, Zhuoran Yu*, Liyang Liu*, Xinjiang Wang, Aojun Zhou,

Shilong Zhang 129 Dec 24, 2022
Code in conjunction with the publication 'Contrastive Representation Learning for Hand Shape Estimation'

HanCo Dataset & Contrastive Representation Learning for Hand Shape Estimation Code in conjunction with the publication: Contrastive Representation Lea

Computer Vision Group, Albert-Ludwigs-Universität Freiburg 38 Dec 13, 2022