I am running HMR on a video sequence. After completion, I get Theta (85X1) for each frame. For visualization, I used Theta[3:75] for SMPL pose and Theta[75:85] for SMPL shape. The visualized mesh sequence seems to be incorrect and I think it is because I am not taking care of the inferred camera parameters i.e. Theta[0:3]. I am not sure what exactly get_original function in renderer.py does, but I suspected it compensates for the camera parameters. So , I tried visualizing the vertices given by get_original function and it looks a bit better. Am I correct?

My question is: In the neutrMosh/neutrSMPL_H3.6 dataset and your processed tf_records_human36m, the SMPL pose's first 3 parameters are different. How to process it?

Below is what the author answered me: Quick answer is that those are two different mosh is in global coordinate space while the individual TF records for human 3.6m are in the camera coordinate frame. I think I rotated it using the provided camera information from human3.6m. You can also do it by solving the procrustes problem between the meshes to figure out the alignment.

I tried the following code and the visualization results show that the global rotation of the ground truth mosh data doesn't correspond to the image. Did I get something wrong or the data is incorrect?

flength = 1000.
renderer = SMPLRenderer(img_size=224, flength=flength)
smpl_model = SMPL('neutral_smpl_with_cocoplus_reg.pkl')

fqueue = tf.train.string_input_producer(
    ['/data/tf_datasets/tf_records_human36m_wjoints/train/h36m_train_mixed_0000.tfrecord'])
reader = tf.TFRecordReader()
_, example_serialized = reader.read(fqueue)
image_, image_size_, label_, center_, fname_, pose_, shape_, gt3d_, has_smpl3d_ = data_utils.parse_example_proto(
    example_serialized, has_3d=True)

pose_ph = tf.placeholder(tf.float32, [None, 72])
shape_ph = tf.placeholder(tf.float32, [None, 10])
verts_, joints_, Rs_ = smpl_model(shape_ph, pose_ph, True)
init = tf.global_variables_initializer()
sess = tf.train.MonitoredTrainingSession()
sess.run(init)

while 1:
    image, image_size, label, center, fname, pose, shape, gt3d, has_smpl3d = sess.run(
        [image_, image_size_, label_, center_, fname_, pose_, shape_, gt3d_, has_smpl3d_])
    verts = sess.run(verts_, feed_dict={pose_ph: np.expand_dims(pose, 0), shape_ph: np.expand_dims(shape, 0)})
    vert = verts[0]
    vert_shift = np.array([[0., 0., flength / 112.]])
    vert = vert + vert_shift
    rendered_img = renderer(vert, do_alpha=False)
    cv2.imshow('a', rendered_img)
    cv2.imshow('b', cv2.cvtColor((image*255).astype(np.uint8), cv2.COLOR_RGB2BGR))
    cv2.waitKey()

Hi, I wonder how did you get the parameters for the cocoplus_regressor? Some joints are shared by both SMPL and cocoplus, I check that row of 6890-d vectors from J_regressor and cocoplus_regressor, and found that they are different (cocoplus_regressor seems to have fewer valid vertices for each joint). So I am quite curious how you computed the parameters for cocoplus_regressor (for the last five I know that you just directly pick the corresponding vertex).

Hello @akanazawa and thank you for releasing the code for the paper. I was trying to figure out how to get the 3D distance between the camera and the predicted 3D joints. Is there a way to do that?

As for now, I've understood that HMR is object-centric, that's why the mesh is always positioned at (0,0,0) in the 3D world. Another thing I've seen is that the 3D skeleton is flipped, but a solution to that is mentioned in another issue.

The final step for me is to understand how to retrieve the 3D (x,y,z) of the mesh with respect to the camera. Is that possible? Maybe using the axis-angle 24 joints instead of the 19 ones?

Thank you so much

Hi @akanazawa,

I'm working on a tool trying to eliminate learning a new regressor but rather just clicking certain vertices and therefore defining a new keypoint, i.e.

Then by having the newly defined joint just selecting the N closest vertices and solving the linear matrix equation using least-squares solution.

Could you explain why the original cocoplus regressor is normalized 0 to 1 where the vertices are not?

System details:

Ubuntu 18.04

I have been trying to run the demo file since last night. but It still not showing anything in output. it just stop wihout any error starting python console.

Here is the terminal logs. mirrorsize@mirrorsize-Latitude-E6420:~/HMR$ python demo.py --img_path /home/mirrorsize/HMR/data/random.jpg Iteration 0 Iteration 1 Reuse is on! Iteration 2 Reuse is on! Restoring checkpoint /home/mirrorsize/HMR/src/../models/model.ckpt-667589.. Resizing so the max image size is 224.. /home/mirrorsize/HMR/src/util/renderer.py:313: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. if np.issubdtype(image.dtype, np.float): --Return-- None

/home/mirrorsize/HMR/demo.py(91)visualize() 90 import ipdb ---> 91 ipdb.set_trace() 92

ipdb>

Can anyone help with this?

Hi @akanazawa,

thanks for the great work. I'm currently working on a TF2.0 implementation of your work in a Keras Model Based version. While evaluating, I am not sure which training setting I really have to use and therefore compare the performance witch the results published in the paper.

In Section 3. Model of the paper the unpaired setting is defined as

Additionally we assume that there is a pool of 3D meshes of human bodies of varying shape and pose. Since these meshes do not necessarily have a corresponding image, we refer to this data as unpaired [55].

But then later in Section 4.3. Without Paired 3D Supervision it is said to be

So far we have used paired 2D-to-3D supervision, i.e. L3D whenever available. Here we evaluate a model trained without any paired 3D supervision. We refer to this setting as HMR unpaired and report numerical results in all the tables.

Could you please clarify this?

Thanks

Hello, I read your new paper "Learning 3D Human Dynamics from Video", but I don't know where to download the Penn Action Dataset. I forces on the sports datset. By the way, when your code will be released? Thanks for your work!

Hi @akanazawa It's really great job and helps me a lot. However, when I try to get the joints of human and use them for other computer vision tasks, I'm confused by the actual meaning of joints, joints3d and joints_ori.

So I'm curious to know, if I directly use joints and joints3d, will they match the joints on the original 2D image?

Thanks a lot!

Hello akanazawa,

First of all thanks a lot for making this awesome work into public.

I have download the Mosh Data (named as mosh_data.tar.gz if you have tried) dataset mentioned in https://github.com/akanazawa/hmr/blob/master/doc/train.md.

I just found only two sub-folders inside this (neutrSMPL_CMU and neutrSMPL_jointLim) and neutrSMPL_H3.6 is missing.

I am just wondering is it missing because I am looking it for wrong place or I missed something.

Thanks and Regards, Shafeeq E

how do you prepare(not download) the gender neutral mean SMPL parameters i.e. the neutral_smpl_mean_params.h5 file?

How did you prepare the file? What does this file exactly contain?

I have tried to run this project onthis Colab page and updated the readme file as you can see below:

But I get some bugs which i have asked here:

Failed to load the native TensorFlow runtime. (Colab error) and here

So I guess after solving the bugs, the project will works on Colab page and it could be tested online and fast which is good.

Hi, @akanazawa sorry to bother you. I am confused w.r.t the predicted parameters of the weak perspective projection.

1. As you mentioned that scale s that HMR recovers is essentially focal_length/z, but the following line https://github.com/akanazawa/hmr/blob/bce0ef9b90bd36871d2aff8688b2682170cd365a/src/util/renderer.py#L247 suggests that 0.5 * img_size comes into play, why?

2. This line code https://github.com/akanazawa/hmr/blob/bce0ef9b90bd36871d2aff8688b2682170cd365a/src/util/renderer.py#L249 suggests that verts and trans, which is trans = np.hstack([cam_pos, tz]), are in the some but what space?

Thus, could you elaborate a little bit on the parameters of this weak perspective projection?

Thanks in advance.

Hi Angjoo,

I am not sure if I am precisely following why there is a plus 2 here. I might be wrong but does it refer to theta and beta parameters from the factorized SMPL model?

"In all we train K + 2 discriminators"