Hi, thank you for your amazing work.

I am just wondering if it's possible to render without a mouth frame and just based on the audio?(similar to what Wav2Lip does)

If so, can you tell me how to do it? Because I've been trying to figure it out if it's possible and keep on running into Exception: None Image error if I put the paths for mouth frames to None and amount of frames to 0 in demo.csv

Thanks for sharing your work! Where is the sricpt for constrast learning for image feature and audio feature. I found a class in "models\networks\loss.py": SoftmaxContrastiveLoss, is this the realization of contrast learning?

Thanks so much for the great work and codes. When I read the paper and codes, I get confused about the pose space learning part.

As the training strategy said in the paper, firstly pre-train the identity encoder and speech content space and then loaded to the overall framework to train the generator and pose space learning, I could understand the training procedure, however, for learning the pose space I am confused if you use the loss (compute_diff_loss)

https://github.com/Hangz-nju-cuhk/Talking-Face_PC-AVS/blob/23585e281360872fe2d1e1eec8ff49176ea0183d/models/av_model.py#L401

when training the whole generator. If so, it is consistent with the codes to compute the l1 loss on pose differences and pose_feature_audio.

Last but not least, congratulations on the research progress, I think it is a great breakthrough to disentangle the sync information and head pose in the feature representation. Looking forward to your reply!

Hi,

Congratulations on this great work and thanks for releasing the codes! The results are mighty impressive! Any plans on releasing the training code too?

Hi, thanks for your greate work. what about the parameters in file "demo.csv" (160, 363)

misc/Input/517600055 1 misc/Pose_Source/517600078 160 misc/Audio_Source/681600002.mp3 misc/Mouth_Source/681600002 363 dummy

In the demo video on the project page, there are generated faces of Obama and Biden，of time stamp 2:26. It seems that there exists a little identity mismatch problem, which may be a potential improvement direction. Could you please tell us the source videos you use in this situation or the location in VoxCeleb2(maybe) for better reference? Thanks in advance and appreciate the great work PC-AVS!

i notice that you mentioned the paper "Neural Head Reenactment with Latent Pose Descriptors" in an issue about non-id space. i tried the augmentation used in their code, but it seems different from yours, with fewer changes. this part seems vital considering the non-id space is vital, promising the model to distangle the id feature and pose feature. so i would like to know the augmentation details in your paper. can you share it to us?

Thank you for the great work. In the Equation 2, you use F_c^v and F_c^a to calculate the loss function L_c^v2a. However, in the code "av_model.py", when mode == 'sync', you use function **sync(self, augmented, spectrogram)**to train the speech content space. If i am not wrong, in the function, you use F_n of Non-Identity space and F_c^a to calculate the loss function L_c^v2a.

Does it play the same role as equation 2? Looking for your reply and best wishes!

Hello, I have configured the project as suggested here. I am able to run it on the demo images placed inside ./misc/input/some_id My question is how can I run the project on my own image.

For example when I am creating a folder inside ./misc/input/ of name 123456 and then placing my own image of size 224x224 inside it with name 000000.jpg (the complete path is ./misc/input/123456/000000.jpg), and then after changing the demo.csv file when I am running the code then I am not getting desired results.

Please help me.

Hi Hang, did you combine the two dataset for training the lip-sync and test separately on each, or did you separate the training dataset - i wonder if voxceleb2 alone can achieve the good performance in terms of the lip-sync. Thanks.

I'm just curious whether if its possible to just have mouth_source and input to generate a video, where the audio source can be ignored (be None). Similar to how lips sync works. I tested on mine and it didn't work.

Sorry to bother you again. I tried to figure out by looking through the codes but that didn't help.

Some trouble during “sudo apt install ffmpeg“

sudo add-apt-repository ppa:savoury1/ffmpeg4 sudo add-apt-repository ppa:savoury1/graphics

https://blog.csdn.net/jn10010537/article/details/124078608

self.target_frame_inds = np.arange(2, len(self.spectrogram) // self.audio.num_bins_per_frame - 2) In voxtest_dataset.py L107, target frame index starts from 2. But in the paper, it starts from 1 (it means start from 0 in python). I don't understand why it is. Thanks.

The author Hangz answered other's question about training code. He answered that losses or differenct modules for training are available on av_models.py. But is there any difference b.t.w training and inference when I have to make it on my own?

Hi, thanks for sharing this great work!

I understand the main pipeline, i.e., encoding the speech content features, id features, and pose features respectively then feeding them to the generator for the driven results. But I am a little bit confused after reading the inference code. https://github.com/Hangz-nju-cuhk/Talking-Face_PC-AVS/blob/23585e281360872fe2d1e1eec8ff49176ea0183d/models/av_model.py#L473-L484

As can be seen, the mel-spectrogram is encoded by the audio encoder first in Line 473 and is ready to be fused with the pose feature in Line 483. However, in the merge_mouthpose() function: https://github.com/Hangz-nju-cuhk/Talking-Face_PC-AVS/blob/23585e281360872fe2d1e1eec8ff49176ea0183d/models/av_model.py#L454-L461

I found the audio features are further embedded, what is the intuition behind that? In my view, the netE.mouth_embed would be used to embed the mouth features for the video but NOT for the audio. If anything is wrong, please correct me. Thanks in advance.