As your paper declares, for face pose evaluation, you fine-tune the model on 300W-LP dataset. However, I cannot find the corresponding code in your repository, did I miss it?

Thanks again for your work.

I wonder whether the scales of RPN would change the performance too much. The original code you provided use 5 scales at all, have you ever tried 3 scales?

I want to use img2pose to extract the camera pose (for the purpose of using it as the input for NeRF). On page 3 of the paper it is stated that this can be obtained from the 6DoF pose h by "standard means," but I'm struggling to figure out how this is done. I'm especially struggling with how to determine the t of the [R¦t] matrix; I did manage to extract the R.

In short: How can one obtain a camera pose from the output 6DoF pose of this model?

Edit: My specific use case is an input video of a single talking head; I would like to get a camera pose determined by the head pose for each frame; i.e. interpret head movement as camera movement instead.

Hi,

I am trying to render (x,y,z) axis based on network output instead of using provided renderer.

Currently using this code:

def draw_axis(img, euler_angle, center, size=80, thickness=3,
              angle_const=np.pi / 180, copy=False):
    if copy:
        img = img.copy()

    euler_angle *= angle_const
    sin_pitch, sin_yaw, sin_roll = np.sin(euler_angle)
    cos_pitch, cos_yaw, cos_roll = np.cos(euler_angle)

    axis = np.array([
        [cos_yaw * cos_roll,
         cos_pitch * sin_roll + cos_roll * sin_pitch * sin_yaw],
        [-cos_yaw * sin_roll,
         cos_pitch * cos_roll - sin_pitch * sin_yaw * sin_roll],
        [sin_yaw,
         -cos_yaw * sin_pitch]
    ])
    axis *= size
    axis += center

    axis = axis.astype(np.int)
    print('axis', axis)

    tp_center = tuple(center.astype(np.int))

    cv2.line(img, tp_center, tuple(axis[0]), (0, 0, 255), thickness)
    cv2.line(img, tp_center, tuple(axis[1]), (0, 255, 0), thickness)
    cv2.line(img, tp_center, tuple(axis[2]), (255, 0, 0), thickness)

    return img

According to readme poses variable contains 6 values per detected face

      [
        0.013399183198461687,
        0.0015700862562677677,
        -0.0008193041494016704,
        0.1667461395263672,
        -7.139801979064941,
        53.44799041748047
      ]

Is it true that first one is pitch then yaw then roll other 3 are horizontal, vertical translation and scale?

Based on provided rendering code, i get pretty static axis always pointing in same direction, but the face mask shows orientation

Thanks for your work and help. Now I have got a lite-weight model using MobileNetV3-small as backbone. This lite-weight model can achieve the same pose evaluation performance on AFLW2000 as your model (both without fine-tuning on 300W-LP).

Now I am focused on fine-tuning. In your paper, you said:

Training pose rotation labels are obtained by converting the 300W-LP ground-truth Euler angles to rotation vectors, and pose translation labels are created using the ground-truth landmarks, using standard means.

I open this issue to confirm several things, and I will be very grateful if you can help.

Here are my questions:

1. How did you define the face bounding box for each image in 300W-LP since in some images there can be two or more faces detected? Did you just use one bounding box, which has landmark annotations? How did you get the bounding box? Did you use a face detector, like InsightFace?
2. Since the only one face in each image of 300W-LP are annotated with 68 points, can I directly use the labeled landmarks to make the JSON files, and choose to use self.threed_68_points in the code here to generate the lmdb file?

That's all. Thank you so much.

Great work! I have a question on focal length.

In your paper, Pose conversion methods were explained in the end.

But I have no idea on this statement because the larger focal length means a smaller field of view,

I just wonder why this is a "zoom-out" operation instead of "zoom-in" operation

Dear Vítor Albiero,

Thanks for your helpful comments in the previous git issues. It was great help in understanding the paper, img2pose.

I have additional questions.

1. What is the definition of the proposed idea's result (i.e. img2pose prediction value) ? According to the equation (2) in your paper, 6D vector h_i consists of Euler angles and 3D face translation vectors. Also, you let me know the pose_pred includes rotation vectors, not the Euler angles. (Both information can be easily converted.) I clearly understand your comment and also cannot find the code where the rotation can convert to the Euler angles [2]. Thus, I hope to fully understand your paper, so I politely ask you again. Is it correct that the whole proposed method (i.e. network and post processing) makes the global pose, h_{i}^{img} that consists of a 3D face translation vector and 3D rotation vector, not Euler information in the given entire image, not an image corp B that is defined in the Appendix A..

2. Questions related to the rendered results. Please refer to the below images in the [3]. Please note that the name of the image is the 27_Spa_Spa_27_32.jpg in the WIDERFACE training dataset. In other words, I think the model may already used the image in the training phase. I get the values for the boxes, labels and dofs from the lmdb dataset that is obtained by your guide. I think the values might be obtained correctly because the both functions, random_crop and random_clip are turned off. However, the result in [3]-b is a little odd. It makes me confused. If I get the GT values correctly, it is impressive that the generalization of the model is well trained by numerous other GTs and the results are nice although GT used for training may be inaccurate as far as you can see. For reference, please note that both the rendering results using prediction and GT values corresponding to the other images obtained in the same way as above were nice unlike [3], although I did not attach a picture.

   • Is it correct that the GT - dof values in the lmdb dataset corresponds to h_{i}^{img*} for the given entire image (i.e. whole image) in the Fig. 4?
   • What do you think that which factor determines the size of face? It's difficult to understand why the rotation vector determines the size of the face. Could you tell me your explanation for this?

[1] https://github.com/vitoralbiero/img2pose/issues/27#issuecomment-804506944 [2] In inference mode (i.e. test_own_images.ipynb), the proposed network performs transform module, transform = GeneralizedRCNNTransform(min_size, max_size, image_mean, image_std), at the end. However the transform does not include any rotation - Euler conversion method.
[3] Rendered result for the image, 27_Spa_Spa_27_32.jpg. a) The rendered result of the img2pose. b) The rendered result using the GT values that are obtained in the train.py using below additional codes:

• dict = targets[0]
• boxes = data_dict['boxes'].numpy().tolist()
• labels = data_dict['labels'].numpy().tolist()
• dofs = data_dict['dofs'].numpy().tolist()

Thanks again for your contribution.

I have another question, how did you get the pose reference files, i.e. reference_3d_5_points_trans.npy, reference_3d_68_points_trans.npy, triangles.npy and vertices_trans.npy ?

First of all, thank you for your work. I am trying to run "test_own_images.ipynb" evaluation but I am getting the error "the size of tensor a must match the size of tensor b at non-singleton dimension 0" at line "res = img2pose_model.predict([transform(img)])[0]". I am trying to use images from CASIA webface and MultiPIE dataset. Please let me know I should I solve this.

Hello, I m not quite familiar with eval tools. How to get mAP and plot curves in python scripts, instead of eval tools, upon getting results/WIDER_FACE/Val.

Hellow! Thanks for your perfect project! I wonder to know how to train on my own dataset! I notice that you use the five landmarks to generate 6DoF pose labels by using standard means. Can you share the code about these methods? Thanks for your beautiful work! I am waiting for your reply！

I'm focusing on the fine-tuning work of img2pose。I refer to the steps you suggested on issues in github to fine tuned based on the model“img2pose_v1.pth”you provided. However，the head pose estimation were worse after the fine-tuning than before. my steps are as follows: 1、use the 300wlp annotations "300W_LP_annotations_train.txt" you provided in github and dowload 300wlp datasets on the official website. 2、use the json_loader_300wlp.py in your codes to create "300W_LP_annotations_train.lmdb", "300W_LP_annotations_train_pose_mean.npy" and "300W_LP_annotations_train_pose_stddev.npy" by execute "convert_json_list_to_lmdb.py" 3、in models.py change rpn_batch_size_per_image to 2 (proposals), and box_detections_per_img to 4 (head samples)。 4、 use "300W_LP_annotations_train.lmdb" to train without augmentations, use "300W_LP_annotations_train_pose_mean.npy" and "300W_LP_annotations_train_pose_stddev.npy" as pose_mean and pose_stddev. 5、 lr = 0.001 for 2 epochs 6、Other parameters are set as follows:       "--pose_mean", "./datasets/lmdb/WIDER_train_annotations_pose_mean.npy",                 "--pose_stddev", "./datasets/lmdb/WIDER_train_annotations_pose_stddev.npy",                 "--pretrained_path", "./models/img2pose_v1.pth",                 "--workspace", "./workspace/",                 "--train_source", "./datasets/lmdb/300W_LP_annotations_train.lmdb",                 "--prefix", "trial_1",                 "--batch_size", "2",                 "--max_size", "1400",

however, after 2 epochs，the head pose estimation were worse , and the MAE in aflw2000 is "Yaw: 20.656 Pitch: 17.178 Roll: 13.957 MAE: 17.264; H. Trans.: 0.179 V. Trans.: 0.363 Scale: 1.465 MAE: 0.669". I don't know which step went wrong. I would appreciate it if you could help me.