Hi guys ,

I've already synthesized my own video, and the resolution seems good! However, the voice sometimes not sync well with the video (I used mandarin tts speech).

https://user-images.githubusercontent.com/53510802/149249264-89a4c5cb-5ef7-49f7-ba87-c999e7a8470c.mp4

Does anyone ever have similar problem? Any suggestion is appreciated.

Thanks a lot

首先感谢作者公开项目代码供大家研究。费尽千辛万苦总算把demo脚本和公开的train脚本都跑通了，而且是在CUDA10.1、tf2.3环境。但在使用新的语音生成视频的过程中，发现几个问题。希望作者有空给予解答。 1，face2vid在执行 test_video.py 时，我使用RTX 2080Ti 大约为15帧/秒。作者曾称使用了1080Ti，应该不会比这个更快。 我使用1分钟的音频，预处理声音耗时3分钟左右，合成画面环节又耗时3分钟左右。整体耗时相当于声音时长的6倍以上，在效率上距离实时合成画面应该还有很大差距。作者曾说可以做到近似实时，能否在效率改进等方面提供一些思路？

2，在声音预处理过程采用了deepspeech 0.1.0的模型，已经非常陈旧了。这个项目已经更新到了0.9.3版本，并且有了支持中文的pbmm模型，为什么不采用更新版本的模型而要用最初版本的pb模型呢？作者能否调整一下代码以提供对新版本的支持？

Thanks for your sharing @zhangchenxu528 。 I get the bad results by using my own data, the results seems to be affected by "latest_net_G.pth". How to train a custom latest_net_G.pth? Or do I need to do some other work? .

Hi. Hello to everyone at the repo! Much kudos to Professor @zhangchenxu528 for his great research. I just used a new video to generate a 3DFaceReconstruction with PyTorch (Original notebook used Tensorflow) as detailed here:

https://github.com/sicxu/Deep3DFaceRecon_pytorch

The generated .mat file has different keys from the video_preprocess provided. For example when running the file, handle_netface.py, there is this line of code:

faceshape[i-1,:] = loadmat(os.path.join(param_folder,str(i)+'.mat'))['coeff'][0,:]

This means that the .mat file is expected to have the key 'coeff'! However the .mat from the PyTorch version only has the following keys: 'angle', 'exp', 'gamma', 'id', 'tex', and 'trans'. When one concats these keys, its dimensions become equal to that of the original required 'coeff' key.

Here is what I have done so far! However, I am not sure that the keys are concatenated in the correct order.

coeffs = np.concatenate((mat_contents['angle'][0],
                        mat_contents['exp'][0],
                        mat_contents['gamma'][0],
                        mat_contents['id'][0],
                        mat_contents['tex'][0],
                        mat_contents['trans'][0]

Can someone please confirm if this is the correct order?

I have attached the two mat files here. 1.mat.orig.zip contains the .mat file in video_preprocess that has a 'coeff' key. The file 1.mat.zip contains the .mat file that doesn't have a 'coeff' key, but rather the following keys: 'angle', 'exp', 'gamma', 'id', 'tex', and 'trans'.

1.mat.orig.zip 1.mat.zip

Hi, thank your awesome work! I try to use your method to train the custom dataset, but the video fps of trainet is 25. I don’t want to force frame rate of those videos to to 30. Can I achieve this by modifying the original code?

https://github.com/zhangchenxu528/FACIAL/blob/95eaf8f40e44e8c1ad5ed047b874881fb6115b66/audio2face/audio_preprocessing.py#L41 Do I need to change only this line of code? Looking forward to your advice.

Hi, I tried to make my own dataset. But I have no idea to create the mat file. When I use the deep3dface to make the mat file, it asked a txt files which record the coordinates of eye, noise and mouth. But the deep3dface author doesn't show how to make the txt file for every image. Could you please show us the method to make the mat file? or the method to make .txt file for every iamge?

Hi @zhangchenxu528 , Thankyou for the wonderful project ! Looked at your demo video and output (for even higher resolution) in colab. I have trained for custom video model using your training colab notebook. Don't know why, but didnt get proper output video for a custom video of 4 mins. Except the preprocessed data, rest of everything I used, is same as provided in the colab notebook.

Hi, I have read your paper. It's a good idea to use 3d construction to do aud2face task. I plan to make my own Chinese speech dataset but I don't know how what's the standard of making a dataset. Your paper mentioned that you used 450 clips for training. I want to know will the background music allowed in video clips or just one person is talking?

Hi, I am a little confused about the Openface file. Is it related to the Deep3DFaceReconstruction files? Then how can I create the csv file? Looking forward to your reply.Thanks!

Thanks for the great work. However, when I run the python audio_preprocessing.py code, I face error: Traceback (most recent call last): File "/home/ubuntu/muhiddin/FACIAL/audio2face/audio_preprocessing.py", line 61, in processed_audio = process_audio(ds_fname, audio4deepspeech, fps) # generate audio feature File "/home/ubuntu/muhiddin/FACIAL/audio2face/audio_preprocessing.py", line 39, in process_audio processed_audio = audio_handler.process(audio, fps) File "/home/ubuntu/muhiddin/FACIAL/audio2face/_utils/audio_handler.py", line 58, in process return self.convert_to_deepspeech(audio, fps) File "/home/ubuntu/muhiddin/FACIAL/audio2face/_utils/audio_handler.py", line 107, in convert_to_deepspeech input_tensor = graph.get_tensor_by_name('deepspeech/input_node:0') File "/home/ubuntu/anaconda3/envs/audio_face/lib/python3.10/site-packages/tensorflow/python/framework/ops.py", line 4150, in get_tensor_by_name return self.as_graph_element(name, allow_tensor=True, allow_operation=False) File "/home/ubuntu/anaconda3/envs/audio_face/lib/python3.10/site-packages/tensorflow/python/framework/ops.py", line 3974, in as_graph_element return self._as_graph_element_locked(obj, allow_tensor, allow_operation) File "/home/ubuntu/anaconda3/envs/audio_face/lib/python3.10/site-packages/tensorflow/python/framework/ops.py", line 4014, in _as_graph_element_locked raise KeyError("The name %s refers to a Tensor which does not " KeyError: "The name 'deepspeech/input_node:0' refers to a Tensor which does not exist. The operation, 'deepspeech/input_node', does not exist in the graph." Can you give some solution for this issue?

When I run the python train.py --blink_path '../video_preprocess/train1_openface/train1_512_audio.csv' --name train3 --model pose2vid --dataroot ./datasets/train3/ --netG local --ngf 32 --num_D 3 --tf_log --niter_fix_global 0 --label_nc 0 --no_instance --save_epoch_freq 2 --lr=0.0001 --resize_or_crop resize --no_flip --verbose --n_local_enhancers 1 faced this issue: RuntimeError: CUDA error: out of memory CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. I am using a Linux server with #GPU is 2, is it related to this? Please, any solution?

Tensorflow 1.x is not supported in Google colab now.

ValueError: Tensorflow 1 is unsupported in Colab.

With tensorflow 2.x, I modified the code in audioface/_util/audio_handler.py import tensorflow as tf to import tensorflow.compat.v1 as tf

But after finished the first several steps, I encountered error as below during "python audio_preprocessing.py"

Traceback (most recent call last): File "audio_preprocessing.py", line 60, in processed_audio = process_audio(ds_fname, audio4deepspeech, fps) # generate audio feature File "audio_preprocessing.py", line 38, in process_audio processed_audio = audio_handler.process(audio, fps) File "/content/FACIAL/audio2face/_utils/audio_handler.py", line 59, in process return self.convert_to_deepspeech(audio, fps) File "/content/FACIAL/audio2face/_utils/audio_handler.py", line 108, in convert_to_deepspeech input_tensor = graph.get_tensor_by_name('deepspeech/input_node:0') File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 4128, in get_tensor_by_name return self.as_graph_element(name, allow_tensor=True, allow_operation=False) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 3952, in as_graph_element return self._as_graph_element_locked(obj, allow_tensor, allow_operation) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 3992, in _as_graph_element_locked raise KeyError("The name %s refers to a Tensor which does not " KeyError: "The name 'deepspeech/input_node:0' refers to a Tensor which does not exist. The operation, 'deepspeech/input_node', does not exist in the graph."

Any body solved such problem?

In this paper I find Dataset statistics. The proposed dataset contains rich samples of more than 450 video clips which are collected from the videos used by Agarwal et al. [1]. Each video clip lasts for around 1 minute. We re-normalize all videos to 30 FPS, forming 535,400 frames in total. Can you seed me a link for this viodes，thanks.Or can you give me a drvie to download the pre-train for 450 video！ thank you again！

您好，非常感谢您分享的优秀的工作和代码！ 我注意到在您分享的train a new person on Google Colab中，video-preprocess的deep3dreconstruction部分，其中的.mat文件是1*257维的，请问您如何从BFM或者其它模型中获取这些数据？谢谢！

Hi,

I've generated the face model .mat files using Deep3DFaceRecon_pytorch, and the facial features using OpenFace, however the resulting rendered images used for training are cropped heavily. I am not sure if it is an issue with render_netface_fitpose.py or fit_headpose.py, or if the Deep3DFaceRecon_pytorch .mat files are still incorrect (I modified handle_netface.py as mentioned in #26, as the pytorch version uses different keys). I have attached an example output and the corresponding true image. Thank you very much for your help.

Hi, I have been working on you project for video synthesis. I have successfully been able to generate the video with good quality lip syncing. But the output video seems to have some un-natural shoulder movements can you please suggest what should I do to keep them as it is in the input video.