Hierarchical Cross-modal Talking Face Generation with Dynamic Pixel-wise Loss （ATVGnet）

Hi @lelechen63. When I ran the demo, I encountered Segmentation fault. After debugging, it was found that torch and Dlib should be the cause. Can you share the version of Dlib?

I am trying to create data for this project. Is there some images or landmark*.npy can be downloaded? just as one processed landmark npy and image region~ thank you

Hi, I clone the code and run the demo.py, nothing error, but the video and the images generated are totally in wrong type like this.

img
motion
attention

By the way, I run in CPU mode. thanks in advance.

Dear Authors, Thanks for sharing the code. I just wanted to know whether the pre-trained model which you have released can be used on GRID dataset or not? I am only interested to run your demo.py. For example, I want to

1. give a frame from GRID dataset as a target frame
2. provide a .wav audio file from GRID

Do you think that should work. Or do I need to take any special care for GRID dataset demo.

We are trying to deploy this project on an android application. In order to do so, we need to convert the pretrained pytorch model (atnet_lstm_18.pth and generator_23.pth) into tensorflow but it shows an error of 'state_dict'. When i load the pre trained models, it only gives the weights but not the architecture. Can you guide me where i can find the Model architecture? And how to convert it to tensorflow?

Hello, I learned about ATVGNet at CVPR2019 site.And this is a very interesting work！ But when I read code after that,some place made me confuse. 1.What is the use of mean_shape_norm.npy & S.npy & S_3d.npy, I know it try to normazile landmark,but What is the individual function of the parts(mean_shape_norm.npy & S.npy & S_3d.npy)? Knowing this might give me a better understanding of the code.

2.How to get mean_shape_norm.npy & S.npy & S_3d.npy?

look forward to you reply

作者您好，感谢您出色的工作。 我有两个疑问比较疑惑 1.留意到您在AT-net中的dataset处理中，将mfcc特征向量堆叠为16个拼接在一起。 想问一下堆叠为16的依据是什么?(我们在deepspeech中也有留意到相似的做法) "t_mfcc =mfcc[(r + ind - 3)*4: (r + ind + 4)*4, 1:]" 这一步操作已经是选取了前后3帧共计280ms的特征向量，为什么还要将之拼接16次呢？ 2."landmark =lmark[r+1 : r + 17,:]" 正常来讲，我们的label不是以当前帧为中心，为什么这里选取了当前帧往后的16帧的landmark作为标签？

期待您的解答，感激不尽！

Dear Authors, Thanks for the awesome release of the paper and code.

I was trying to compare our result with yours on the GRID dataset for the LMD metric. Can you please tell me that in the paper

1. Which subjects IDs of GRID did you use for testing.

2. How many keypoints did you use for each subject ? I usually use a dlib detector which gives me 68 keypoints.

3. Do you perform any normalization of the keypoints (after getting raw pixel coordinates using a dlib detector) to get rid of scale effects before calculating the difference on real and synthetic faces?

4. Lastly, when you report the SSIM and PSNR: do you calculate those metrics on the entire frame or just cropped out face regions. I just wanted to make sure that we compare fairly with you. So, keenly looking forward to your kind reply.

Thanks, Avisek Lahiri

Im trying to run this script.

import torchvision import torch from somefile import modelarchitecture model = modelarchitecture() model.load_state_dict(torch.load(???)) model.eval()

Please guide me which values will be set for "model" and path inside torch.load?

In demo.py, you multiply the example_landmark by 5 before applying PCA,

example_landmark = example_landmark * 5.0
example_landmark = example_landmark - mean.expand_as(example_landmark)
example_landmark = torch.mm(example_landmark, pca)

And for fake_landmarks, you multiply 2 times 1.1~1.5 before applying PCA

fake_lmark[:, 1:6] *= 2 * torch.FloatTensor(np.array([1.1, 1.2, 1.3, 1.4, 1.5])).cuda()
fake_lmark = torch.mm(fake_lmark, pca.t())
fake_lmark = fake_lmark + mean.expand_as(fake_lmark)

so (1) do you apply different scaling parameters for example_landmark and fake_landmarks? (2) how are those scaling parameters (5 vs 2.2~3.0) being selected?

Hi: I'm testing to get the landmarks from the output facial images. Since the VG output images are cropped, the landmarks not so stable from dlib. Is that possible to generate the landmarks directly from VG net? Since we already have a landmark input for the vg net. thanks

Traceback (most recent call last): File "demo.py", line 486, in test() File "demo.py", line 465, in test fake_store = restore_image(orgImage,rect,fake_store,indx) File "demo.py", line 196, in restore_image cv2.normalize(img, img, 0, 255, cv2.NORM_MINMAX) cv2.error: OpenCV(4.5.5) :-1: error: (-5:Bad argument) in function 'normalize'

Overload resolution failed:

• Layout of the output array dst is incompatible with cv::Mat
• Expected Ptrcv::UMat for argument 'dst'

https://github.com/lelechen63/ATVGnet/blob/2d4d1b03df1c706c6575b942fd2a1585347c4aab/requirement.txt#L9

The above line in requirements.txt throws a pretty long error that is not worth reproducing in its entirety:

Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /home/arta/anaconda3/envs/py2/bin/python /home/arta/anaconda3/envs/py2/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmp88lNX5
       cwd: /tmp/pip-install-hhbrzT/opencv-python

I found this website, that, when translated to English, describes a workaround whereby you specify a version of opencv that is compatible with Python 2.7:

python -m pip install opencv-python==4.2.0.32

I hope this helps.

Hello @lelechen63,

I found your work very interesting. I want to use it for the finetuning on my own data, however I could not find a pretrained discriminator for the model. And when I trying to fine-tune with only generator the model breaks. Could you please advice me on where it can be found. In the link provided on github only a generetor part is available.
Thank you in advance

while training vgnet.py, the model didn't find the file " new_img_full_gt_train.pkl", what should this file contain? and how to create it? could anyone who worked on it help?

thank you in advance.