Im confused about code:

rotate_finger = lambda j1, j2 : rvec2rmat(np.array( [0,np.arctan((j1[2]-j2[2])/(j1[0]-j2[0])),0])) # Rotate about y axis Z[0:2,0:3] = rotate_finger(Jrest[1,:],Jrest[0,:])[1:3,:] # 0:MCP abduct/adduct, 1:MCP flex/ext

why rotate y axis can get A/A & F/E movement

Mesh result is not correct by running utils_mpi_model.py using freihand ground truth data.

    ### test freihand dataset ###
    freihand_root_dir = "FreiHAND_pub_v2/"
    freihand_K_path = os.path.join(freihand_root_dir, "training_K.json")
    freihand_mano_path = os.path.join(freihand_root_dir, "training_mano.json")
    freihand_xyz_path = os.path.join(freihand_root_dir, "training_xyz.json")

    K_list = json.load(open(freihand_K_path, 'r'))
    mano_list = json.load(open(freihand_mano_path, 'r'))
    xyz_list = json.load(open(freihand_xyz_path, 'r'))
    img_path_list = [os.path.join(freihand_root_dir, "training/rgb/%08d.jpg" % idx) \
                     for idx in range(len(mano_list))]

    select_index = 0
    img_path = img_path_list[select_index]
    K_param = K_list[select_index]
    mano_param = mano_list[select_index]
    mano_param = np.array(mano_param)[0]
    xyz_param = xyz_list[select_index]

    pose_param = mano_param[:48]
    shape_param = mano_param[48:58]
    uv_root_param = mano_param[58:60]
    scale_param = mano_param[60:]

    beta = torch.Tensor(shape_param).reshape([bs, 10])
    pose = torch.Tensor(pose_param[3:]).reshape([bs, 15, 3])
    rvec = torch.Tensor(pose_param[:3]).reshape([bs, 3])
    tvec = torch.Tensor(mano_param[58:61]).reshape([bs, 1, 3])
    xyz_root = torch.Tensor(xyz_param[0])

Hi, very great work !

I feel some puzzles about the generation of Z mat.

# Convert rotation vector (3 by 1 or 1 by 3) to rotation matrix (3 by 3)
rvec2rmat = lambda rvec : cv2.Rodrigues(rvec)[0]
# Note: j1 is the joint of interest and j2 is its parent (1 by 3)
rotate_finger = lambda j1, j2 : rvec2rmat(np.array([0, np.arctan((j1[2]-j2[2])/(j1[0]-j2[0])),0])) # Rotate about y axis

Z[0:2,0:3] = rotate_finger(Jrest[1,:],Jrest[0,:])[1:3,:] # 0:MCP abduct/adduct, 1:MCP flex/ext

If I understand correctly, the return of function rotate_finger is a rotation matrix, so why Z[0:2,0:3] = rotate_finger(...)[1:3,:] ? (bs, 23) dot (23, 45) = (bs, 45) and the pose of mano input should be 45 axis-angle values, I just can not understand why the Z mat can be generated by the rotation matrix such as Z[0:2,0:3] = rotate_finger(...)[1:3,:], are the results of (bs, 23) dot (23, 45) 45 axis-angle values？.

Hi, I feel a little confused about the joint angle representation, can you please give me more details? (1) How does the 23 degrees defined and trained, are there any ground truth data? (2) How is the matrix T that transform the local joint angles to match MANO pose computed ?

In Training stage, whether it should remove Chumpy from MANO model?

Does utils_mpi_model.py file need to be modified during the training phase?

What is the difference between the utils_mpi_model.py and manopth?

Hi @gmntu , Thanks for opening source this program. I found that the original paper of this program is hosted on researchgate but I cannot reach it because of "some unusual traffic from my network". Could you upload the original paper to arixiv or send a copy of it by email to me? Sorry for this abnormal request.

I found the cam params (scale tx, ty) converge worse with or without the cam loss, and i use weak perspective in my code, in which kpy_2d = scale(kyp3d[, :2]) + txy. I think the key reason is the focal length of the freihand dataset is different with each image, and it range from 400 mm to 800 mm. So does the scale (focal / global_z) differs with each image. So maybe the network cannot regress the scale well? I want to how to make cam params converget better? is there any tricks in training? thanks!

hi, i try to replace resnet netwrok in HMR by mobilenet v3, according to your work. however, i found the training becomes not convernge and the loss becomes larger. I used the pretrained v3 small model by imagenet. Can u tell me where i missed?

How to get 3D coordinates of camera coordinate system？

Considering that the absolute coordinates of hand are not obtained, how can 3d hand tracking be applied to augmented reality scenes such as AR glasses?