S2HAND: Model-based 3D Hand Reconstruction via Self-Supervised Learning
S2HAND presents a self-supervised 3D hand reconstruction network that can jointly estimate pose, shape, texture, and the camera viewpoint. Specifically, we obtain geometric cues from the input image through easily accessible 2D detected keypoints. To learn an accurate hand reconstruction model from these noisy geometric cues, we utilize the consistency between 2D and 3D representations and propose a set of novel losses to rationalize outputs of the neural network. For the first time, we demonstrate the feasibility of training an accurate 3D hand reconstruction network without relying on manual annotations. For more details, please see our paper, video, and project page.
Code
Environment
Training is implemented with PyTorch. This code was developed under Python 3.6 and Pytorch 1.1.
Please compile the extension modules by running:
pip install tqdm tensorboardX transforms3d chumpy scikit-image
git clone https://github.com/TerenceCYJ/neural_renderer.git
cd neural_renderer
python setup.py install
rm -r neural_renderer
Note that we modified the neural_renderer/lighting.py compared to daniilidis-group/neural_renderer.
Data
For example, for 3D hand reconstruction task on the FreiHAND dataset:
- Download the FreiHAND dataset from the website.
- Modify the input and output directory accordingly in
examples/config/FreiHAND/*.json.
For HO3D dataset:
- Download the HO3D dataset from the website.
- Modify the input and output directory accordingly in
examples/config/HO3D/*.json.
Offline 2D Detection
- Offline 2D keypoint detection use a off-the-shelf detector like pytorch-openpose.
-
We also provide detected 2D keypoints for FreiHAND training set. You may downlad and change the
self.open_2dj_listsin theexamples/data/dataset.pyaccordingly. -
Or Download the
hand_pose_model.pthprovided by pytorch-openpose, and put the file toexamples/openpose_detector/src. Then use the following script and modify the input and output directory accordingly.python example/openpose_detector/hand_dectect.py
-
Training and Evaluation
HO3D
Evaluation: download the pretrained model [texturehand_ho3d.t7], and modify the "pretrain_model" in examples/config/HO3D/evaluation.json.
cd S2HAND
python3 ./examples/train.py --config_json examples/config/HO3D/evaluation.json
Training:
Stage-wise training:
python3 ./examples/train.py --config_json examples/config/HO3D/SSL-shape.json
python3 ./examples/train.py --config_json examples/config/HO3D/SSL-kp.json
python3 ./examples/train.py --config_json examples/config/HO3D/SSL-finetune.json
Or end-to-end training:
python3 ./examples/train.py --config_json examples/config/HO3D/SSL-e2e.json
Note: remember to check and inplace the dirs and files in the *.json files.
FreiHAND
Evaluation: download the pretrained model [texturehand_freihand.t7], and modify the "pretrain_model" in examples/config/FreiHAND/evaluation.json.
cd S2HAND
python3 ./examples/train.py --config_json examples/config/FreiHAND/evaluation.json
Training: refer to HO3D traing scripts.
Citation
If you find our work useful in your research, please consider citing:
@inproceedings{chen2021s2hand,
title={Model-based 3D Hand Reconstruction via Self-Supervised Learning},
author={Chen, Yujin and Tu, Zhigang and Kang, Di and Bao, Linchao and Zhang, Ying and Zhe, Xuefei and Chen, Ruizhi and Yuan, Junsong},
booktitle={Conference on Computer Vision and Pattern Recognition},
year={2021}
}
In your paper, I think the Camera Intrinsic Parameters were predicted from networks.
And I also found clues in line 210 'https://github.com/TerenceCYJ/S2HAND/blob/main/examples/utils/freihandnet.py'
Then train only the neural renderer and encoder for using lr = 0.001 for 30 epochs:
But then after 60 epochs finetune, there exists joint output but there are no rendered images shown :
150 Dec 28, 2022
92 Nov 30, 2022
19 Feb 12, 2022
2 Jan 12, 2022
117 Dec 28, 2022
69 Dec 27, 2022
304 Jan 3, 2023
282 Jan 9, 2023
274 Jan 5, 2023
151 Dec 26, 2022
19 Dec 12, 2022
36 Sep 14, 2022
2 Oct 31, 2022
3 Feb 25, 2022
96 Dec 27, 2022
833 Dec 28, 2022
99 Dec 26, 2022
46 Dec 26, 2022
35 Nov 20, 2022