Hi, thank you for releasing the code!

I have a few questions on the triplet loss function as described in Section 3.2.2. It seems the triplet loss is trying to evaluate how well a generated pose sequence in comparison to a 'real' sequence. The anchor and positive samples in a triplet tuple are selected by using two consecutive results produced by the body encoder with 'real sequence' as input, and another result is treated as the negative sample which is produced by the body encoder part with 'fake' or 'generated' results. According to Eq.4 and Eq.5, both sequences produced by feeding the 'original' input and 'limb-scaled' input can be treated as 'real' output. Therefore, the anchor and positive sample can be selected from both streams in your proposed model? Is that correct the triplet loss is trying to guide the body encoder to generated the 'same' body structure when changing the scale of the input pose?

BTW, is that a bug in the calculation of temporal pairwise cosine similarity of the seqs_b in Line 43 (loss.py)? because the temporal similarity should be also calculated within the seqs_b to obtain the similarity between an 'anchor' and a 'positive' sample?

https://github.com/yzhq97/transmomo.pytorch/blob/5d766fb16f511b77f446abda3697b626595a2b2d/lib/loss.py#L42-L44

Thanks!

Hi, first of all, thank you for releasing such a great code! My question regarding the latent space interpolation is that how the background of the interpolated pose changes. For example Figure 8 in the paper, I can see body structures can be successfully interpolated, but what about the synthesized frames conditioned on them? It would be helpful if you can provide or describe the synthesized video frame. (Is it just a mixture of two frames??)

Hello, I am interested in running your code. The test.py runs fine, however when I try to run the infer_pair.py file with the following command

python infer_pair.py --config configs/transmomo.yaml --checkpoint transmomo_mixamo_36_800_24/checkpoints/autoencoder_00200000.pt --source data/mixamo/36_800_24/test_random_rotate/SPORTY_GRANY/Pulling_A_Rope/Pulling_A_Rope.npy --target "data/mixamo/36_800_24/test_random_rotate/TY/Golf_Tee_Up_(1)/Golf_Tee_Up_(1).npy" --source_width 1280 --source_height 720 --target_height 1080 --target_width 1920

I get the following error

Traceback (most recent call last):
  File "infer_pair.py", line 87, in <module>
    main(config, args)
  File "infer_pair.py", line 77, in main
    x_cross = postprocess(x_cross, mean_pose, std_pose, unit=1.0, start=x_src_start)
  File "/home/rk/projects/transmomo.pytorch/lib/util/motion.py", line 27, in postprocess
    motion = globalize_motion(motion, start=start)
  File "/home/rk/projects/transmomo.pytorch/lib/util/motion.py", line 100, in globalize_motion
    centers += start.reshape([2, 1])
ValueError: cannot reshape array of size 3 into shape (2,1)

It's also the same error with other pairs of motion.

Is there something wrong? Thanks.

Thanks for your excellent work. And I what to know how many GPU memory it Need during training phase and testing phase respectively? And how long does it take to train this network.

Hello, thank you for sharing the code.

I am trying to get 3d key joints by calling reconstruct3d() in network.py. It returns an array in this format : (45, length)

What's the order of these 45 data points?

Thanks

Q1: The result is amazing no matter what people is small or large. But I don't find the code you choose which point as the center for scaling structure, so all limbs will shifting together, it is not reasonable. And if we scaling every target's skeleton frame by frame independently, whether the generated result will jittering?

Q2: For all target video, do we just need one common model for generating results?

I have fetched the keypoints of an image/video using detectron2 as suggested in the readme. But the format is Cocoo. This project expects the input to be in Body_25 format. How do I convert Cocoo to Body_25?

Also, how do I visualize the output of infer.py? I know that I'm supposed to use motion2video utility, but it doesn't seem to work directly on the output motion of infer.py. I am not sure what format it's expecting the data in.

I want to convert the video like this and save it as Npy:

The file should contain an array with shape 15 x 2 x length. The first dimension (15) corresponds the 15 body joint defined here. The second dimension (2) corresponds to x and y coordinates. The third dimension (length) is the temporal dimension.

Hi! I've noticed in the paper that the 2D pose can be generated by projecting 3D into 2D, and the retargeting inference that the repo provides is 2D pose generation. I wonder if there are functions which can directly generate 3D pose(which means every point has x,y,z coordinate instead of x,y). Thanks a lot!

Hi @yzhq97, I am facing issue regarding the limb normalisation. I have been trying to normalise each to the limb lengths in scale between 0-1. But it seems that it is not straight forward. I have till now understood that limb normalisation requires multiple dependents of joints to be normalised. But limb_norm(x_a,x_b) takes two different motions to scale x_a according to target x_b skeleton structure. How can I scale limbs between 0-1 since I am not trying to adjust structure according to target skeleton by using limb_norm()?

I want to know what is unit meaning in this function. Hope for your reply.

`def preprocess_test(motion, meanpose, stdpose, unit=128): motion = motion * unit

motion[1, :, :] = (motion[2, :, :] + motion[5, :, :]) / 2
motion[8, :, :] = (motion[9, :, :] + motion[12, :, :]) / 2

start = motion[8, :, 0]
motion = localize_motion(motion)
motion = normalize_motion(motion, meanpose, stdpose)

return motion, start

`