Hey @Hangz-nju-cuhk @liuziwei7 @liuyuisanai! I am trying to understand and reproduce the results of this repository end-to-end, So I could create a docker file and contribute. I have already read the paper thoroughly and analyzing the code now. But I am having a problem with pre-processing the data. Could you please guide how could I do this? Step by step process. Looking forward to you. Thank you.

I want to build a dataset of Chinese characters to train this model. I applied speech recognition on some Chinese news videos (by CCTV). The recognition part was fine, but I found that Chinese characters are too short in terms of pronounce time because each of them has only one syllable. The average number of video frames it takes to show the lip movement of a single Chinese character is only 5 (fps=25), and It can be even as low as 2 frames. This is much less than the required 29 frames. Obviously, interpolation won't work well in this case. So I would like to know if you guys have considered Chinese? Will this model work? Is there any workaround?

Hi,Huang zhou,I just add some new images into the demo_images for test，and find that the result image fake frames 's variation is not like the four demo images , does this repo code support any other else image test? or Should I do some preprocssing work on my own images ?

你好，我是中国科学技术大学的一位本科生，最近我和一些小伙伴想学习一些talking face generation 方面的工作，我们选择复现您的代码，预处理方面还比较顺利地按照您的思想完成了，但是train方面的代码错误很多，我们最终在改动多处地方后使得train代码可以正常运行了。忽然发现您在10月5日提交了一个修改，而且这个修改和我们当初的修改相同，难道您最近也在修改train方面的代码吗？请问您是否有新版本的代码？我们很想和您交流一下 如果您愿意，这是我的邮箱[email protected] 如果您愿意，我们将不胜感激

https://github.com/Hangz-nju-cuhk/Talking-Face-Generation-DAVS/blob/c0233ace95be15fb1665dfcd056d82117822a797/preprocess/savemfcc.m#L7

In the Readme it is suggested that you use a similar audio pre-processing as Zimmerman et al. However, they use 40 filterbank channels across their code (e.g. in the yousaidthat repository https://github.com/joonson/yousaidthat/blob/98b51812894497cb6c2b65a7ae147067609fc6ca/run_demo.m#L22) I was wondering if there was a reason for choosing 13, or if it had just been mixed up with the number of cepstral coefficients.

Thanks,

Hello, I use the savemfcc.m to generate *.bin file but when I execute the code, an error occurred.

savemfcc('~/talkingface//20180619_1_M.wav','~/talkingface/tlkface/wav')
Undefined function or variable 'vec2frames'.

Error in mfcc (line 151) frames = vec2frames( speech, Nw, Ns, 'cols', window, false );

Error in runmfcc (line 5) [ CC, FBE, frames ] = mfcc( speech, opt.fs, opt.Tw, opt.Ts, opt.alpha, hamming, opt.R, opt.M, N, opt.L );

Error in savemfcc (line 17) [ MFCCs, ~, ~ ] = runmfcc( Speech, opt );

Could you please tell me where to find 'vec2frames'?

In the test command: python test_all.py --test_root ./0572_0019_0003/video --test_type video --test_audio_video_length 99 --test_resume_path CHECKPOINT_PATH What's the meaning of the parameter --test_audio_video_length?

Hi, firstly I want to thank you for sharing such a great project. However, I noticed that you wrote 'The pretraining training code is temporarily not provided.' in README.md, so I was wondering if my understanding is right about the classification part. Here is my own assumption:

1. Use the subset of the MS-Celeb-1M dataset to train the ID_encoder part.
2. Use the optimize_parameters_no_generation() function in Gen_final_v1.py and LRW dataset to train the lip_feature_encoder, mfcc_encoder and model_fusion part. Moreover, when I read and try to train the model using train.py, I find some small errors. For example , opt.isTrain and opt.eval_freq are not defined in Options.py and pair in lip_reading_loader() should be (2,25), since there are only 24 files in /mfcc20. So I want to know if you will update the project later which will be of a great help to me.

I have just try some other audios for test(I use the matlab taking the *.wav file into mfcc bin files),find that most of the wav files did not make the modle generate the images, only a few can. Can this mode support this test?

@Hangz-nju-cuhk LRW is used to train the model according to your paper, but there are audio video offset in LRW videos. And [11] used SyncNet when pre-processing dataset to correct the offset. Did you consider this problem when preparing dataset? Thank you!

[11] Joon Son Chung, Amir Jamaludin, and Andrew Zisserman. You said that? arXiv preprint arXiv:1705.02966, 2017.

Hi~ Thanks for your code, but I encountered some problems when I run the testing script to generate videos from audio.

1. In Test_load_audio.py，there seems config has no require_audio attribute, because when I run python test_all.py --test_root './0572_0019_0003/audio' --test_type 'audio' --test_audio_video_length 99 --test_resume_path, I got the error AttributeError: 'Namespace' object has no attribute 'require_audio
2. When will you release the complete code?

Hi, Why you use Variable(self.lip_embedding_norm.data) instead of direct use self.lip_embedding_norm here? And why denominator is 2*batch_size instead of batch_size^2?

I downloaded pretrain checkpoint from the link: https://drive.google.com/file/d/1UW22xm4r9AewNoySyPd2fyUab0nqymBR/view. But when I tried to extract it, it report the following error: tar: This does not look like a tar archive tar: Skipping to next header tar: Exiting with failure status due to previous errors

Would you please check the tarball?

大家好，想跟大家请教一下关于mouth.txt 的数据处理过程，mouth.txt 的数据分别表示什么，多少维度的，我没有下载作者使用的数据集，我想使用新闻主持人的数据，但是我不知道如果生成mouth.txt 文件。如果有人处理处mouth.txt的数据，希望能告诉我mouth.txt的数据是多少维度的 分别表示什么意思，谢谢@liuziwei7 @liuyuisanai @Hangz-nju-cuhk

Thank you for your questions and solutions for your great help. Now, I have successfully got test_all (video and audio) output result, the question now is how to combine the output result with the original audio to form a complete video, and to ensure that the mouth shape and audio in the video are aligned @Hangz-nju-cuhk

hi, thank you for your sharing. It's profitable for me, in face_align.py, I wound not understand how this points is generated points = [[70, 112], [110, 112], [90, 150]] I am looking forward to your reply, thank you.