Thanks for your fancy work. I run evaluator.py file and the output file is .npy. Could you please give me some suggestions about transferring the output file to video?

Hi, thanks for your fancy work. I'm new in 3D visualization. Just curious how you visualize the generate 3D motion with character from Mixamo. Do you use blender or something? Could you possibly refer some helpful websites or something like that?

Thanks in advance.

Visualizing the output .npy files of evaluator.py according to README with the provided checkpoints seems like some of the angles (e.g. shoulders) are wrong, maybe flipped. In this clip, green is the original sample from AIST++ and red is MINT inference.

To visualize, I implement the opposite operation described here, then used Blender's SMPLX addon to visualize. Here's my code:

    rotations = mint_data[:seq_len, 9:] # trim first 9 entries according to https://github.com/google-research/mint - mint/tools/preprocessing.py +161
    rotations = rotations.reshape([-1, 3, 3])
    rotations = R.from_matrix(rotations).as_rotvec().reshape([seq_len, (joint_dim-1)//9, 3])
    body_pose = rotations[:, :NUM_SMPLX_BODYJOINTS]  # FIXME - not sure about that (trimming last 3 joints from smpl 24, to smplx 21)

Hi Ruilong,

Is it possible that providing instructions on bvh_writer? I didn't find the skeleton_csv_filename and joints_to_ignore_csv_filename.

Best, Wenjie

I'm sorry I meet a problem. where can I get this file of crossmodal_train.txt In this code, it seems in /mnt/data/AIST/music/, but i can't find where to download .

When I completed the data preprocessing and got the tfrecord file, however, when I tried to train the model, the error occurred like this. It seems to be the problem of tf.data.TFRecordDataset. Thank you in advance!

https://github.com/google-research/mint/blob/b8f8bdfbbe3fbfa67831a2ef9bcf71a6a9e74552/mint/utils/inputs_util.py#L101-L103

It should be:

example["audio_input"] = example["audio_sequence"][start * audio_sample_rate:start * audio_sample_rate + 
                                                   audio_input_length, :] 

Hello, is there any guide or things to note when trying to run the AI choreographer on colab? I already get a few errors when trying to run the dependencies, I would be thankful for any help on running this on colab

Hi, authors, thank you for your nice work! I am learning your FID calculation code. I found that in Line 170 in preprocessing.py, the seq_name repeats 10 times, so that the evaluator run 10 times on testval set. I am not understand this design. https://github.com/google-research/mint/blob/b8f8bdfbbe3fbfa67831a2ef9bcf71a6a9e74552/tools/preprocessing.py#L170

And can I reproduce the FID scores in your paper with the provided checkpoints?

Thank you! Best

my eval result using the given checkpoint has only a few second valid dance. after a few second, the result became totally nonsence. is this the real case or did I do something wrong?

Hi. After evaluation, I just tried to run calculate_beat_scores.py. However, the default result_files is '/mnt/data/aist_paper_results/*.pkl' which doesn't work. Is there anyone could tell me how to generate the motion and replace the default result files? Thank you very much!

本人用Pytorch复刻了一版，也增加了valid过程，结果发现，FID_k最好的是第21个epoch，为101，后面越训练，FID_k的值越大，波动比较大，也不知道什么原因。 loss值降到0.0002左右，基本就不收敛了，FID_k的值能达到7000多, FID_g的值25左右。 使用原作者tensorflow原代码，从头开始训练，loss值降到0.0001后，基本就不收敛了，FID_k的值达到700多，FID_g也就30多，也是完全复现不了原作者放出的训练好的模型。 @liruilong940607

TF: 2.3 cuda: 10.1

pytorch: 1.9.1 cuda: 10.1

阶段进展更新了，pytorch复刻版训练已经收敛，复现了原作者的指标参数，解决方法如下：

1、每一层的初始化方法与TF版保持一致，注意检查每一层的默认初始化方法 2、由于训练集只有952个视频，如果多个GPU同时训练，batch_size设置为32，那么每个epoch只包含几个迭代，建议方法是加载列表后，把列表复制10倍或20倍，这样每个epoch的迭代次数就变为原来的10倍或20倍 3、训练足够多的epoch，如果训练的时候，迭代器中把训练数据列表复制了10倍或20倍，那么loss要收敛到0.00011级别，至少训练800个epoch，0-200 epoch是lr=1e-4，200-500 epoch时lr=1e-5，500-800 是1e-6

另外，我在最后一个CrossTransformer后面，最后一个fc前面，增加了一个2层双向的LSTM，训练下来，loss收敛到0.00011左右，FIDk最小可以达到22.3，比原作者给出的指标还小，但这个指标小，并不代表着生成的舞蹈动作很好，效果如原作者给出的模型类似，只有少数几个效果还行，其他的都不怎么样。