Hi, @liruilong940607 I have some questions about the input and output of transformer network.

1. As the paper said, during training, the input of the Audio Transformer is music feature Y with shape (240, 35) and the input of Motion Transformer is motion data X with shape (120, 219) , so for all of Q,K,V are same, include the query vector of decoder of these transformer. Is this right?
2. For the cross-modal transformer, assume I get H(X) (240, 512) and H(Y) (120, 512) after audio and motion transformer, how to fuse features from music and motion features? Just concatenate both in sequence dimension resulting in features with shape (240 + 120, 512)?
3. What's the query vector of cross-modal transformer decoder? How to get a motion output with 20 frames?
4. What is the representation of motion prediction? Is it same as the input motion data with 24*3*3 + 3 = 219. But I think it is not good to regress a 3x3 rotation matrix directly? And for the regression of global translation, do you directly regress the absolute translation (x,y,z) in 3D space? Or just the offset relative to the first frame of seed motion? Thanks!

Hi there! Hope you are well! :)

I am desperately trying to visualize the SMPL pose parameters from your dataset but it seems that your axis convention does not meet mine.

I have got inverted and inconsistent rotations as in the following picture :

I have tried to swap (x, y z) axis, oppose signs in the x, y or z direction without success.

Do you have any idea of the transformation I need to make to visualize the animation properly ? Could you please provide the axis convention used for your skeleton please ?

Thank you :D

I am trying to project the keypoints viewed from the Camera of (0,0,0) exactly as the image through the world_to_camera method and additional methods. For example:

In above image, assuming that the camera was shot at (0,0,0) in the current 3d space, I would like to match the rotation and translation of the keypoints as shown in the photo above. (The photo above was adjusted manually and was not correct. Also, don't mind scattered points of any color other than the human keypoint below)

However, when the camera translations of the annotations are displayed in 3d space, they appear as follows compared to other keypoints. You can see that the height does not match with the human keypoints, and it is not surrounded by 360 degrees and is clustered in a specific area.

Looking at it a little further, it looks like this:

I think the coordinate scale of camera translation and human keypoints translation is different or there are other reasons. If what I didn't aware about something can you explain?

Thank you very much for providing the dataset.

Hi,

Your work is excellent and I am interested in it! Now I'm trying my hands on visualizing the SMPL joints.

But I have no idea about where the file "SMPL_MALE.pkl" is and what function it serves as. Could you please tell me about it?

Sincerely, Shuhong

Hi, thanks for your amazing dataset! I found this dataset does not contain the shape parameters of SMPL model. Also, the learning rate is set to 0 for all stages. Also, shape parameters will affect the scale optimization. So, which shape parameters should I use for visualizing the SMPL mesh?

Hi，

Thanks for your. wonderful datasets.

I recently do some thing about 3D humane pose and I tried to download your dataset. However I always meet the error 'multiprocessing.pool.MaybeEncodingError: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x7f344db94160>'. Reason: 'TypeError("cannot serialize '_io.BufferedReader' object",)'.

Thus I am not sure whether I have finished the downloading. I find a video file list in downloader.py, so I download it individually. I find the video_list contains 12669 items, but it not equal to the annotations which only contains 1408 SMPL files.

I wonder how to check the dataset? One more thing, how can I fixed the error when I using?

Looking forward to hearing from you,

I Downloaded the Keypoints 3D only and it says "all annotations are 60 fps", however when i open them they seems to be 30fps for example a motion of duration : 1 minute and 35 seconds is around 2800 frames not 5600 is it right ? should i treat the keypoints3d as 30fps.

Plus: if it is so and i want to extract audio_features with the same fps from audio, is it right to use a sample rate = 48000and hope_size = 800 ?

Hi, Thanks a lot for releasing this dataset. You are providing visualization of keypoints, that's great. However you are not providing a way to visualize SMPL from each camera view point, would it be possible to incorporate this in the run_vis.py file? For the moment the SMPL poses is independant to camera viewpoints, how to you make it dependant from a specific camera viewpoint? Thanks a lot for your help. Fabien

I can't find the download button of SMPL model on the download page of https://smpl.is.tue.mpg.de/downloads.

Is the download button of SMPL model has been cancelled？

Thanks!

Hi, Is it possible to post a script that aligned the music and motion recordings? As I can see, in the AIST dataset the audio fragments do not match the length of the corresponding fragments, and when extracting audio directly from a video fragment it is not clear how to properly align the individual features of the audio and movement frames.

Timur Sokhin

Hi，thanks for sharing such a great dataset. I have downloaded the dataset but failed to find the ignore_list.txt which is mentioned in AIST++ websites ?

We noticed that your paper stated that the dataset contained 3D joint rot data, but we cannot find it on the web page. So I was wondering if there are any accesses to download the 3D joint rot data. We hope to do some research with this data.
I look forward to hearing from you.

I'm very interested in this dataset so I did some experiments with SMPL backbone. But the result turns out quite surprising, so I further analyzed the dataset. My approach is to use run_vis.py to re-run unreasonable videos on my model. And here are my results:

• gBR_sFM_c09_d04_mBR4_ch07

with the setting

flags.DEFINE_string(
    'video_name',
    'gBR_sFM_c09_d04_mBR4_ch07',
    'input video name to be visualized.')
flags.DEFINE_enum(
    'mode', 'SMPL', ['2D', '3D', 'SMPL', 'SMPLMesh'],
    'visualize 3D or 2D keypoints, or SMPL joints on image plane.')

• gBR_sBM_c05_d04_mBR0_ch08

with the setting

flags.DEFINE_string(
    'video_name',
    'gBR_sBM_c05_d04_mBR0_ch08',
    'input video name to be visualized.')
flags.DEFINE_enum(
    'mode', 'SMPL', ['2D', '3D', 'SMPL', 'SMPLMesh'],
    'visualize 3D or 2D keypoints, or SMPL joints on image plane.')

• gJB_sBM_c06_d07_mJB3_ch05

with the setting

flags.DEFINE_string(
    'video_name',
    'gJB_sBM_c06_d07_mJB3_ch05',
    'input video name to be visualized.')
flags.DEFINE_enum(
    'mode', 'SMPLMesh', ['2D', '3D', 'SMPL', 'SMPLMesh'],
    'visualize 3D or 2D keypoints, or SMPL joints on image plane.')

I'm wondering if anyone else has similar results? Or did I make a mistake on the code running? Cause I found hundreds of videos like above.

Description: Skip already downloaded files (helpful to resume interrupted downloads)

Additional Known Issues: With this method, it will not resume partially downloaded files. Workout suggestions for this issue in the future. -- If there is away to know expected file size then we could check for it. -- Each thread can save a last downloading file and remove them when done. Before resuming, delete all the incomplete files.

Hi, when i use ffmpeg_video_read to extract the image,i specified a frame rate of 60,but i found the number of frames and the number of corresponding motions are unequal. while if i don't specify fps,the number are equal. i use gHO_sBM_c01/02...09_d19_mHO2_ch02 .mp4 and get 481 images,but smpl poses only have 480.so do i need to specify the frame rate? Thanks！