Hello, I am trying to reconstruct the training code and there are several questions I have:

1. From what I saw from audio2feature_model.py, in the forward module, the size for self.audio_feats is [b, 1, nfeats, nwins] while in audio2feature.py, the dimension for audio_features is [b, T, ndim]. From my understanding(correct me if I was wrong), for batch_size=32, T=240*2, ndim=512(the APC feature dimension), the input batch for Audio2Feature model should be [32, 480, 512] (480 because mel_frame is n_frames * 2) and output size is [32, 240, 75]. Is that right?

2. Furthermore, from your paper in section 3.2，a delay d=18 is added during training but not reflected in the code. How that works in training? For example, m0 is inferred by h0, h1,....h18?

3. In audiovisual_dataset.py, you seemed like clipping the audio into many pieces and extract APC feature for each audio. What is the number of clips for a certain dataset, eg. 4 mins 60fps video?

There might be some stupid questions as I am not very familiar with audio processing field, just correct me if I made mistakes, thanks!

大佬您好， 看到论文中提到的“ Audio to Mouth-related Motion” ，这里的音频特征是使用基于中文数据集训练的（Mandarin Chinese part of the Common Voice dataset）APC模型，然后 audio2feature部分是要针对每个目标人物重新训练一遍是吗？ 但是这里每个目标人物的数据可能只有几分钟（3-5分钟），这样子会不会对泛化效果影响很大，例如训练的时候使用的是一个女性角色，测试的时候使用男性去测试？ 因为我目前尝试的一个方案，是基于ATnet在大量数据集上训练从音频到人脸关键点的映射，然后利用ATnet提取的音频特征，后面的操作与大佬论文中相似，也是用3-5分钟的视频微调 音频特征到人脸表情相关参数的映射，但是目前测试的效果，感觉泛化能力并不理想。因此想请教大佬的看法

Hi I noticed that for the discriminator inputs you have 23 channels.

Why is this? I would have thought the input would have 3 channels for each of the RGB channels in an image

Thanks

1. Can you explain the difference between a and b? I understand that a is the average of the 3d landmark, and b is the 3d landmark。Please understand I'm so rubbish。THANKS！

Impressive job!

I wonder how to preprocess the images. Specifically, could you please share the scripts on choosing the four candidate images from the sequences and how to draw the shoulder edges since the landmark detectors I have found are all face landmark detectors.

Thanks !

Hi , i want to try to train a model of personal speaker. Can I train only the Audio2Feature model and Audio2Headpose model based on your APC model weights？（In other words, is the APC model generalizable?） Can you give me some advice, thanks a lot

Hi Thank you for amazing lib and open source code,

Helping me learn a lot. One question I had was with regards to the target speech representation database. Is it simply the embedding of several speech from target speaker and then the inputted speech is essentially mapped to the closest point within those embeddings?

Eg: Extract embedding from 50 obama utterances -> input arbitrary speech sample -> map embedding of arbitrary speech sample to the closest obama representation

Thank you

dear fellow I found manifold projection use APC_feature_base.npy each person, but not clear how to generate this file. Is that use target person voice to train audio2feature_model?

cv2.error: OpenCV(4.5.4) :-1: error: (-5:Bad argument) in function 'line'

Overload resolution failed:

• Can't parse 'pt1'. Sequence item with index 0 has a wrong type
• Can't parse 'pt1'. Sequence item with index 0 has a wrong type I got this error when run the demo. Any solutions, please?

6. Image2Image translation & Saving results... Image2Image translation inference: 0%| | 0/672 [00:00<?, ?it/s] Traceback (most recent call last): File "demo.py", line 264, in facedataset.dataset.image_pad) File "C:\Users\23046\LiveSpeechPortraits\datasets\face_dataset.py", line 280, in get_data_test_mode feature_map = torch.from_numpy(self.get_feature_image(landmarks, (self.opt.loadSize, self.opt.loadSize), shoulder, pad)[np.newaxis, :].astype(np.float32)/255.) File "C:\Users\23046\LiveSpeechPortraits\datasets\face_dataset.py", line 287, in get_feature_image im_edges = self.draw_face_feature_maps(landmarks, size) File "C:\Users\23046\LiveSpeechPortraits\datasets\face_dataset.py", line 317, in draw_face_feature_maps im_edges = cv2.line(im_edges, tuple(keypoints[edge[i]]), tuple(keypoints[edge[i+1]]), 255, 2) cv2.error: OpenCV(4.5.4) :-1: error: (-5:Bad argument) in function 'line'

Overload resolution failed:

• Can't parse 'pt1'. Sequence item with index 0 has a wrong type
• Can't parse 'pt1'. Sequence item with index 0 has a wrong type

Thanks for sharing such nice work Can I run this code for person arbitrary? How the information in the folder is obtained for each person?

Thank you for your help!

hey, I cloned the repo. Its working fine with the pretrained model and data generated. But, I wish to train model on new face, and how to generate data for the new face. Is there any source code available?? if yes, whether it is included in this repo? If not can you please provide the same.

If anyone has worked on personalized datasets. please do comment. Your help is appreciated. Thanks

I get the following error trying to execute the demo. Searching on the error I find suggestions for changing the numba/librosa versions but would appreciate guidance for this project.

(LSP) C:\Users\Kris\source\repos\LiveSpeechPortraits>python demo.py --id May --driving_audio .\data\Input\00083.wav --device cuda Traceback (most recent call last): File "demo.py", line 8, in import librosa File "C:\Users\Kris\anaconda3\envs\LSP\lib\site-packages\librosa_init_.py", line 13, in from . import core File "C:\Users\Kris\anaconda3\envs\LSP\lib\site-packages\librosa\core_init_.py", line 114, in from .time_frequency import * # pylint: disable=wildcard-import File "C:\Users\Kris\anaconda3\envs\LSP\lib\site-packages\librosa\core\time_frequency.py", line 10, in from ..util.exceptions import ParameterError File "C:\Users\Kris\anaconda3\envs\LSP\lib\site-packages\librosa\util_init_.py", line 73, in from . import decorators File "C:\Users\Kris\anaconda3\envs\LSP\lib\site-packages\librosa\util\decorators.py", line 9, in from numba.decorators import jit as optional_jit ModuleNotFoundError: No module named 'numba.decorators'

We followed the README instructions for the LSP environment, instaled PyTorch 1.7 and the requirements like so:

conda install pytorch==1.7.0 torchvision==0.8.0 torchaudio==0.7.0 cudatoolkit=9.2 -c pytorch pip install -r requirements.txt

Here are the versions of the packages:

Requirement already satisfied: tqdm in c:\users\kris\anaconda3\envs\lsp\lib\site-packages (from -r requirements.txt (line 1)) (4.64.0) Requirement already satisfied: librosa==0.7.0 in c:\users\kris\anaconda3\envs\lsp\lib\site-packages (from -r requirements.txt (line 2)) (0.7.0) Requirement already satisfied: scikit_image in c:\users\kris\anaconda3\envs\lsp\lib\site-packages (from -r requirements.txt (line 3)) (0.17.2) Requirement already satisfied: opencv_python==4.4.0.40 in c:\users\kris\anaconda3\envs\lsp\lib\site-packages (from -r requirements.txt (line 4)) (4.4.0.40) Requirement already satisfied: scipy in c:\users\kris\anaconda3\envs\lsp\lib\site-packages (from -r requirements.txt (line 5)) (1.5.4) Requirement already satisfied: dominate in c:\users\kris\anaconda3\envs\lsp\lib\site-packages (from -r requirements.txt (line 6)) (2.7.0) Requirement already satisfied: albumentations==0.5.2 in c:\users\kris\anaconda3\envs\lsp\lib\site-packages (from -r requirements.txt (line 7)) (0.5.2) Requirement already satisfied: numpy in c:\users\kris\anaconda3\envs\lsp\lib\site-packages (from -r requirements.txt (line 8)) (1.19.2) Requirement already satisfied: beautifulsoup4 in c:\users\kris\anaconda3\envs\lsp\lib\site-packages (from -r requirements.txt (line 9)) (4.11.1) Requirement already satisfied: decorator>=3.0.0 in c:\users\kris\anaconda3\envs\lsp\lib\site-packages (from librosa==0.7.0->-r requirements.txt (line 2)) (4.4.2) Requirement already satisfied: soundfile>=0.9.0 in c:\users\kris\anaconda3\envs\lsp\lib\site-packages (from librosa==0.7.0->-r requirements.txt (line 2)) (0.10.3.post1) Requirement already satisfied: six>=1.3 in c:\users\kris\anaconda3\envs\lsp\lib\site-packages (from librosa==0.7.0->-r requirements.txt (line 2)) (1.16.0) Requirement already satisfied: numba>=0.38.0 in c:\users\kris\anaconda3\envs\lsp\lib\site-packages (from librosa==0.7.0->-r requirements.txt (line 2)) (0.53.1) Requirement already satisfied: scikit-learn!=0.19.0,>=0.14.0 in c:\users\kris\anaconda3\envs\lsp\lib\site-packages (from librosa==0.7.0->-r requirements.txt (line 2)) (0.24.2) Requirement already satisfied: resampy>=0.2.0 in c:\users\kris\anaconda3\envs\lsp\lib\site-packages (from librosa==0.7.0->-r requirements.txt (line 2)) (0.4.0) Requirement already satisfied: joblib>=0.12 in c:\users\kris\anaconda3\envs\lsp\lib\site-packages (from librosa==0.7.0->-r requirements.txt (line 2)) (1.1.0) Requirement already satisfied: audioread>=2.0.0 in c:\users\kris\anaconda3\envs\lsp\lib\site-packages (from librosa==0.7.0->-r requirements.txt (line 2)) (3.0.0) Requirement already satisfied: opencv-python-headless>=4.1.1 in c:\users\kris\anaconda3\envs\lsp\lib\site-packages (from albumentations==0.5.2->-r requirements.txt (line 7)) (4.6.0.66) Requirement already satisfied: PyYAML in c:\users\kris\anaconda3\envs\lsp\lib\site-packages (from albumentations==0.5.2->-r requirements.txt (line 7)) (6.0) Requirement already satisfied: imgaug>=0.4.0 in c:\users\kris\anaconda3\envs\lsp\lib\site-packages (from albumentations==0.5.2->-r requirements.txt (line 7)) (0.4.0) Requirement already satisfied: importlib-resources in c:\users\kris\anaconda3\envs\lsp\lib\site-packages (from tqdm->-r requirements.txt (line 1)) (5.4.0) Requirement already satisfied: colorama in c:\users\kris\anaconda3\envs\lsp\lib\site-packages (from tqdm->-r requirements.txt (line 1)) (0.4.5) Requirement already satisfied: imageio>=2.3.0 in c:\users\kris\anaconda3\envs\lsp\lib\site-packages (from scikit_image->-r requirements.txt (line 3)) (2.15.0) Requirement already satisfied: pillow!=7.1.0,!=7.1.1,>=4.3.0 in c:\users\kris\anaconda3\envs\lsp\lib\site-packages (from scikit_image->-r requirements.txt (line 3)) (8.4.0) Requirement already satisfied: tifffile>=2019.7.26 in c:\users\kris\anaconda3\envs\lsp\lib\site-packages (from scikit_image->-r requirements.txt (line 3)) (2020.9.3) Requirement already satisfied: matplotlib!=3.0.0,>=2.0.0 in c:\users\kris\anaconda3\envs\lsp\lib\site-packages (from scikit_image->-r requirements.txt (line 3)) (3.3.4) Requirement already satisfied: PyWavelets>=1.1.1 in c:\users\kris\anaconda3\envs\lsp\lib\site-packages (from scikit_image->-r requirements.txt (line 3)) (1.1.1) Requirement already satisfied: networkx>=2.0 in c:\users\kris\anaconda3\envs\lsp\lib\site-packages (from scikit_image->-r requirements.txt (line 3)) (2.5.1) Requirement already satisfied: soupsieve>1.2 in c:\users\kris\anaconda3\envs\lsp\lib\site-packages (from beautifulsoup4->-r requirements.txt (line 9)) (2.3.2.post1) Requirement already satisfied: Shapely in c:\users\kris\anaconda3\envs\lsp\lib\site-packages (from imgaug>=0.4.0->albumentations==0.5.2->-r requirements.txt (line 7)) (1.8.2) Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3 in c:\users\kris\anaconda3\envs\lsp\lib\site-packages (from matplotlib!=3.0.0,>=2.0.0->scikit_image->-r requirements.txt (line 3)) (3.0.9) Requirement already satisfied: python-dateutil>=2.1 in c:\users\kris\anaconda3\envs\lsp\lib\site-packages (from matplotlib!=3.0.0,>=2.0.0->scikit_image->-r requirements.txt (line 3)) (2.8.2) Requirement already satisfied: kiwisolver>=1.0.1 in c:\users\kris\anaconda3\envs\lsp\lib\site-packages (from matplotlib!=3.0.0,>=2.0.0->scikit_image->-r requirements.txt (line 3)) (1.3.1) Requirement already satisfied: cycler>=0.10 in c:\users\kris\anaconda3\envs\lsp\lib\site-packages (from matplotlib!=3.0.0,>=2.0.0->scikit_image->-r requirements.txt (line 3)) (0.11.0) Requirement already satisfied: setuptools in c:\users\kris\anaconda3\envs\lsp\lib\site-packages (from numba>=0.38.0->librosa==0.7.0->-r requirements.txt (line 2)) (58.0.4) Requirement already satisfied: llvmlite<0.37,>=0.36.0rc1 in c:\users\kris\anaconda3\envs\lsp\lib\site-packages (from numba>=0.38.0->librosa==0.7.0->-r requirements.txt (line 2)) (0.36.0) Requirement already satisfied: threadpoolctl>=2.0.0 in c:\users\kris\anaconda3\envs\lsp\lib\site-packages (from scikit-learn!=0.19.0,>=0.14.0->librosa==0.7.0->-r requirements.txt (line 2)) (3.1.0) Requirement already satisfied: cffi>=1.0 in c:\users\kris\anaconda3\envs\lsp\lib\site-packages (from soundfile>=0.9.0->librosa==0.7.0->-r requirements.txt (line 2)) (1.15.1) Requirement already satisfied: pycparser in c:\users\kris\anaconda3\envs\lsp\lib\site-packages (from cffi>=1.0->soundfile>=0.9.0->librosa==0.7.0->-r requirements.txt (line 2)) (2.21) Requirement already satisfied: zipp>=3.1.0 in c:\users\kris\anaconda3\envs\lsp\lib\site-packages (from importlib-resources->tqdm->-r requirements.txt (line 1)) (3.6.0)

is there some resource? or some fake code? https://www.mathworks.com/help/vision/ref/estimategeometrictransform3d.html https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.transform.Rotation.align_vectors.html
http://learning.aols.org/aols/3D_Affine_Coordinate_Transformations.pdf https://en.wikipedia.org/wiki/Kabsch_algorithm

consider the security/licence of original project,i recreat the alg base on deca and LiveSpeechPotraits and the other block is comming. there are still some problem, welcome discuss and pr. here: lsp

in inference, your camer model is fixed, which means camera_intrinsic, sacle etc is constant. but when i use deca, every image has its own camera paremeters, so in traing data, camer parameter are different for ervey frame. how to fix this problem?