To ensure audio emotion and speech content are disentangled, you design a Cross-Reconstructed Emotion Disentanglement module in paper. In my opinion, emotion encoder and content encoder should be freeze once the disentanglement training if finished. But i found that the two pretrained models of two different subjects you provide has totally different weights in the emotion encoder and content encoder. Thus i guess that you finetune these two encoders together with other parts when you train your audio2lm module, but how can you guarantee the disentanglement once you finetune these two encoders?

Thank you for the great work. Could you please share the detail of calculating one specific person's landmark mean and pca? Take a specific person's video data in MEAD for example. We have multiple viewpoints, emotions and intensities. Could you please tell which videos are used to calculate the landmark and produce mean and pca? Like front view with intensity 3? or front view with intensity 2? or some other combination or even all the videos.

I am trying to pre-process the MEAD dataset and do some face reenactment work, I would appreciate it a lot if you may share the pre-processing details. Looking forward to the reply!

Enjoyed the paper - thanks! Looking forward to seeing the training code :) - A few things:

1. in test.py line 208 add_audio(config['video_dir'], opt.in_file) throws an error - AttributeError: 'Namespace' object has no attribute 'in_file' (the coide runs if this line is commented out).
2. Line 3 in M003_test.yaml video_dir: audio2lm/result/M003.mp4 throws an error - OpenCV: FFMPEG: tag 0x44495658/'XVID' is not supported with codec id 12 and format 'mp4 / MP4 (MPEG-4 Part 14)' that can be fixed by changing it to video_dir: audio2lm/result/M003.avi
3. In section 2 - running python lm2video/lm2map.py results in FileNotFoundError: [Errno 2] No such file or directory: './base_weight.pickle' - not sure how this file is generated or obtained. Any help much appreciated - 谢谢你

Hi @jixinya, Thanks for your excellent work! I am curious about the implementation of Cross-Reconstructed Emotion Disentanglement. In the paper, you say, "Given four audio samples Xi,m, Xj,n, Xj,m, Xi,n" for disentangling. However, the implementation in this project is a little different: you sample 2 emotions and 3 contents and set X11, X21, X12, X23 as inputs, and X11, X11, X12, X12 as four targets to calculate the cross reconstruction loss and self reconstruction loss. (as below) https://github.com/jixinya/EVP/blob/990ea8b085a450b6fcc2c28b817989191e173218/train/disentanglement/code/dataload.py#L103-L107 Could you please explain this? Hope for your response.

How are the data in mfcc and face key point landmark files related and whether they are time aligned |--train |---landmark |---dataSet-M030 --landmark ---mfcc

Hi,

I am trying to test with my own dataset but have some trouble with generating 3dmm parameters. I am wondering if you can explain the procedure of how you get the parameters.

Thanks,

I am running python landmark / code / preProcess.py. Then, this path does not exist. How should it be operated

Traceback (most recent call last): File "/home/user/PycharmProjects/EVP/EVP/train/landmark/code/preprocess.py", line 145, in a = np.load(path) File "/home/user/anaconda3/envs/evp/lib/python3.6/site-packages/numpy/lib/npyio.py", line 416, in load fid = stack.enter_context(open(os_fspath(file), "rb")) FileNotFoundError: [Errno 2] No such file or directory: '/home/user/PycharmProjects/EVP/EVP/train/landmark/dataset_M030/landmark/M030_fear_3_026/5.npy'

There are some problems when I running python train/disentanglement/dtw/MFCC_dtw.py.

1. Can you tell the function in 132-140? https://github.com/jixinya/EVP/blob/1f725b8e23f5e29f6211d74e3c08636de7053239/train/disentanglement/dtw/MFCC_dtw.py#L132-L140
2. the dimension of mfcc_k is (13, ) is right? The dimension of mfcc_k can not feed into network.(it will cause dismatch dimension)

Waiting your reply. Thanks.