Clone a voice in 5 seconds to generate arbitrary speech in real-time

Overview

WechatIMG2968

MIT License

This repository is forked from Real-Time-Voice-Cloning which only support English.

English | 中文

Features

🌍 Chinese supported mandarin and tested with dataset: aidatatang_200zh

🤩 PyTorch worked for pytorch, tested in version of 1.9.0(latest in August 2021), with GPU Tesla T4 and GTX 2060

🌍 Windows + Linux tested in both Windows OS and linux OS after fixing nits

🤩 Easy & Awesome effect with only newly-trained synthesizer, by reusing the pretrained encoder/vocoder

DEMO VIDEO

Quick Start

1. Install Requirements

Follow the original repo to test if you got all environment ready. **Python 3.7 or higher ** is needed to run the toolbox.

  • Install PyTorch.
  • Install ffmpeg.
  • Run pip install -r requirements.txt to install the remaining necessary packages.

2. Train synthesizer with aidatatang_200zh

  • Download aidatatang_200zh dataset and unzip: make sure you can access all .wav in train folder

  • Preprocess with the audios and the mel spectrograms: python synthesizer_preprocess_audio.py

  • Preprocess the embeddings: python synthesizer_preprocess_embeds.py /SV2TTS/synthesizer

  • Train the synthesizer: python synthesizer_train.py mandarin /SV2TTS/synthesizer

  • Go to next step when you see attention line show and loss meet your need in training folder synthesizer/saved_models/.

FYI, my attention came after 18k steps and loss became lower than 0.4 after 50k steps. attention_step_20500_sample_1 step-135500-mel-spectrogram_sample_1

3. Launch the Toolbox

You can then try the toolbox:

python demo_toolbox.py -d
or
python demo_toolbox.py

TODO

  • Add demo video
  • Add support for more dataset
  • Upload pretrained model
  • 🙏 Welcome to add more
Comments
  • 求助执行requirements.txt时报No module named 'pyworld'是什么问题??

    求助执行requirements.txt时报No module named 'pyworld'是什么问题??

    已是最新代码 E:\MockingBird\MockingBird>python demo_toolbox.py Traceback (most recent call last): File "E:\MockingBird\MockingBird\demo_toolbox.py", line 2, in from toolbox import Toolbox File "E:\MockingBird\MockingBird\toolbox_init_.py", line 9, in from utils.f0_utils import compute_f0, f02lf0, compute_mean_std, get_converted_lf0uv File "E:\MockingBird\MockingBird\utils\f0_utils.py", line 3, in import pyworld ModuleNotFoundError: No module named 'pyworld'

    bug 
    opened by bricklayers 14
  • 训练合成器时无法收敛

    训练合成器时无法收敛

    问题简述 使用自己的数据集训练合成器模型的时候的时候,在预处理之后训练合成器并将合成器替换成既有model后产生的图并没有收敛。

    复现与环境

    参照www.bilibili.com/video/BV1dq4y137pH 进行的复现。代码版本为main branch,首先进行数据预处理之后参考视频里的首先进行合成器训练,然后用pretrained-11-7-21 替换掉当前mode 继续进行训练。发现图并没有收敛。 截图 qX67L9.png qX6bZR.png

    opened by akiaki1996 13
  • 關於 Train synthesizer 的問題,求指導 !

    關於 Train synthesizer 的問題,求指導 !

    你好 我已經下載了aidatatang_200zh這個數據集,並且把 aidatatang_200zh\corpus\train 底下的檔案都解壓縮完畢 但是當我要開始執行 python synthesizer_preprocess_audio.py D:\google download(我把檔案放在 D:\google download 這個路徑下 ) 卻發生以下狀況: D:\python_demo\Realtime-Voice-Clone-Chinese>python synthesizer_preprocess_audio.py D:\google download\ D:\python_demo\Realtime-Voice-Clone-Chinese\encoder\audio.py:13: UserWarning: Unable to import 'webrtcvad'. This package enables noise removal and is recommended. warn("Unable to import 'webrtcvad'. This package enables noise removal and is recommended.") usage: synthesizer_preprocess_audio.py [-h] [-o OUT_DIR] [-n N_PROCESSES] [-s] [--hparams HPARAMS] [--no_trim] [--no_alignments] [--dataset DATASET] datasets_root synthesizer_preprocess_audio.py: error: unrecognized arguments: download\

    請問我可以怎麼解決問題呢? 我有查看之前 issues 的討論並沒有發現有類似問題,以下是我想到可能有問題的地方,還請作者為我解答,謝謝!

    1.我只有解壓縮 aidatatang_200zh\corpus\train 底下的檔案,是否其他資料夾下的檔案也要解壓縮? 2.是不是只需要將所有 wav 檔單獨拉出來放在 aidatatang_200zh\corpus\train 底下然後再執行python synthesizer_preprocess_audio.py D:\google download ? 3. 輸入的指令不對 4. wav 檔 與 txt 檔是不是要預先處理,而我沒有進行處理?

    opened by XiuChen-Liu 13
  • 用社区分享的模型训练报错 不知道原因

    用社区分享的模型训练报错 不知道原因

    用社区分享的模型训练报错 不知道原因 而且不知道咋保存模型 是不是必须要每500步才会自动保存 求各位大佬解惑 感谢! RuntimeError: The size of tensor a (1024) must match the size of tensor b (3) at non-singleton dimension 3 屏幕截图 2021-11-28 021942

    opened by johnwestin 12
  • 训练模型时显存爆了

    训练模型时显存爆了

    Variable._execution_engine.run_backward(RuntimeError: CUDA out of memory. Tried to allocate 88.00 MiB (GPU 0; 4.00 GiB totalcapacity; 2.68 GiB already allocated; 0 bytes free; 2.85 GiB reserved in total by PyTorch)

    能不能提供一个调batch_size的参数? 我目前用的显卡显存只有4G(GTX1050Ti),默认参数正常训练时经常爆掉显存....

    opened by cronfox 11
  • 如何解决运行python synthesizer_preprocess_audio.py时报错 DLL load failed:页面文件太小,无法完成操作

    如何解决运行python synthesizer_preprocess_audio.py时报错 DLL load failed:页面文件太小,无法完成操作

    我在运行 python synthesizer_preprocess_audio.py时遇到如上错误 ,在CSDN上找到解决方法:1.如果python 运行环境不在C盘 查看高级系统设置->高级->性能 设置->高级->虚拟内存->更改 ->取消自动管理所有驱动器的分页文件大小-> 自定义大小 ->初始大小和最大值设为10240 2. 更改DateLoade 中的参数num_worker 改为0 但我现在不清楚具体怎样把参数设为0

    opened by 9527-567 11
  • capturable=False,报错

    capturable=False,报错

    Win11 GPU:3060laptop
    Python 3.9.13

    +----------------+------------+---------------+------------------+ | Steps with r=2 | Batch Size | Learning Rate | Outputs/Step (r) | +----------------+------------+---------------+------------------+ | 101k Steps | 16 | 3e-06 | 2 | +----------------+------------+---------------+------------------+

    Could not load symbol cublasGetSmCountTarget from cublas64_11.dll. Error code 127 Traceback (most recent call last): File "G:\AIvioce\MockingBird\synthesizer_train.py", line 37, in train(**vars(args)) File "G:\AIvioce\MockingBird\synthesizer\train.py", line 216, in train optimizer.step() File "C:\Users\Mark\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\optim\optimizer.py", line 109, in wrapper return func(*args, **kwargs) File "C:\Users\Mark\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context return func(*args, **kwargs) File "C:\Users\Mark\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\optim\adam.py", line 157, in step adam(params_with_grad, File "C:\Users\Mark\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\optim\adam.py", line 213, in adam func(params, File "C:\Users\Mark\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\optim\adam.py", line 255, in _single_tensor_adam assert not step_t.is_cuda, "If capturable=False, state_steps should not be CUDA tensors." AssertionError: If capturable=False, state_steps should not be CUDA tensors.

    opened by MarkIzhao 10
  • 求助!!!在下载剩余的包 pip install -r requirements.txt 报错了 大佬知道咋解决吗

    求助!!!在下载剩余的包 pip install -r requirements.txt 报错了 大佬知道咋解决吗

    以下是报错代码: Building wheels for collected packages: ctc-segmentation, pyworld Building wheel for ctc-segmentation (setup.py) ... error error: subprocess-exited-with-error

    × python setup.py bdist_wheel did not run successfully. │ exit code: 1 ╰─> [12 lines of output] running bdist_wheel running build running build_py creating build creating build\lib.win-amd64-3.7 creating build\lib.win-amd64-3.7\ctc_segmentation copying ctc_segmentation\ctc_segmentation.py -> build\lib.win-amd64-3.7\ctc_segmentation copying ctc_segmentation\partitioning.py -> build\lib.win-amd64-3.7\ctc_segmentation copying ctc_segmentation_init_.py -> build\lib.win-amd64-3.7\ctc_segmentation running build_ext building 'ctc_segmentation.ctc_segmentation_dyn' extension error: Microsoft Visual C++ 14.0 is required. Get it with "Build Tools for Visual Studio": https://visualstudio.microsoft.com/downloads/ [end of output]

    note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for ctc-segmentation Running setup.py clean for ctc-segmentation Building wheel for pyworld (pyproject.toml) ... error error: subprocess-exited-with-error

    × Building wheel for pyworld (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [13 lines of output] running bdist_wheel running build running build_py creating build creating build\lib.win-amd64-3.7 creating build\lib.win-amd64-3.7\pyworld copying pyworld_init_.py -> build\lib.win-amd64-3.7\pyworld running build_ext skipping 'pyworld\pyworld.cpp' Cython extension (up-to-date) building 'pyworld.pyworld' extension C:\Users\ADMINI~1\AppData\Local\Temp\pip-build-env-_y7fbfzj\overlay\Lib\site-packages\setuptools\dist.py:741: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead % (opt, underscore_opt) error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/ [end of output]

    note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for pyworld Failed to build ctc-segmentation pyworld ERROR: Could not build wheels for pyworld, which is required to install pyproject.toml-based projects

    opened by frankl07 10
  • 训练模型时这个问题怎么办??疑似N卡内存不够。 CUDA out of memory. Tried to allocate 122.00 MiB (GPU 0; 4.00 GiB total capacity; 3.15 GiB already allocated; 0 bytes free; 3.45 GiB reserved in total by PyTorch

    训练模型时这个问题怎么办??疑似N卡内存不够。 CUDA out of memory. Tried to allocate 122.00 MiB (GPU 0; 4.00 GiB total capacity; 3.15 GiB already allocated; 0 bytes free; 3.45 GiB reserved in total by PyTorch

    Summary[问题简述(一句话)] 训练模型时这个问题怎么办??疑似N卡内存不够。 CUDA out of memory. Tried to allocate 122.00 MiB (GPU 0; 4.00 GiB total capacity; 3.15 GiB already allocated; 0 bytes free; 3.45 GiB reserved in total by PyTorch

    Env & To Reproduce[复现与环境] python3.9、NVIDIA GeForce GTX 1050Ti(4GB)

    Screenshots[截图(如有)] image image

    opened by pzhyyd 9
  • AttributeError: module 'setuptools._distutils' has no attribute 'version'

    AttributeError: module 'setuptools._distutils' has no attribute 'version'

    F:\VideoCentTools\MockingBird-main>python synthesizer_train.py offhen F:\VideoCentTools/SV2TTS/synthesizer Traceback (most recent call last): File "F:\VideoCentTools\MockingBird-main\synthesizer_train.py", line 2, in from synthesizer.train import train File "F:\VideoCentTools\MockingBird-main\synthesizer\train.py", line 5, in from torch.utils.tensorboard import SummaryWriter File "C:\Users\Administrator\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\utils\tensorboard_init_.py", line 4, in LooseVersion = distutils.version.LooseVersion AttributeError: module 'setuptools._distutils' has no attribute 'version'

    opened by Dustwinddd 9
  • FileNotFoundError: [Errno 2] No such file or directory: 'encoder\\saved_models\\pretrained.pt'

    FileNotFoundError: [Errno 2] No such file or directory: 'encoder\\saved_models\\pretrained.pt'

    我把已经下载好的模型,放到了文件D:\声音克隆\MockingBird-main\synthesizer\saved_models下 并且还在D:\声音克隆\MockingBird-main\encoder\saved_models里也放了一个把模型my_run,py改名为pretrained.pt的文件 然后运行web.py文件

    (base) C:\Users\13549>python D:\声音克隆\MockingBird-main\web.py Loaded synthesizer models: 0 Traceback (most recent call last): File "D:\声音克隆\MockingBird-main\web.py", line 6, in app = webApp() File "D:\声音克隆\MockingBird-main\web_init_.py", line 33, in webApp encoder.load_model(Path("encoder/saved_models/pretrained.pt")) File "D:\声音克隆\MockingBird-main\encoder\inference.py", line 33, in load_model checkpoint = torch.load(weights_fpath, _device) File "D:\anaconda\lib\site-packages\torch\serialization.py", line 525, in load with _open_file_like(f, 'rb') as opened_file: File "D:\anaconda\lib\site-packages\torch\serialization.py", line 212, in _open_file_like return _open_file(name_or_buffer, mode) File "D:\anaconda\lib\site-packages\torch\serialization.py", line 193, in init super(_open_file, self).init(open(name, mode)) FileNotFoundError: [Errno 2] No such file or directory: 'encoder\saved_models\pretrained.pt'

    请问一下该怎么办呢,即使我把模型文件名修改成pretrained仍然会报同样的错误

    opened by qinan-nlx 9
  • pyworld版本过高导致报错ValueError: numpy.ndarray size changed...

    pyworld版本过高导致报错ValueError: numpy.ndarray size changed...

    今天刚搭建的环境,使用VC拟音时遇到 ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject 此时python=3.9.15, numpy=1.19.3, pyworld=0.3.2 查看报错信息发现和pyworld有关,将pyworld版本降低到0.3.0后不再报错。可能需要在requirements.txt限制pyworld版本?

    opened by carseny 0
  • 训练到18K,attention图不显示

    训练到18K,attention图不显示

    Summary[问题简述(一句话)] A clear and concise description of what the issue is.

    训练到18K,attention图不显示

    Env & To Reproduce[复现与环境] 描述你用的环境、代码版本、模型

    win10,anaconda虚拟环境python=3.9.7,代码版本是在2022年12月20日下载的,数据集是aidatatang_200zh。

    电脑是MacBook Pro 2019

    CPU: Intel(R) Core(TM) i9-9880H @2.3GHz

    Screenshots[截图(如有)] If applicable, add screenshots to help attention_step_18500_sample_1

    step-18500-mel-spectrogram_sample_1

    opened by zijubk 0
  • 求助:AttributeError: module 'umap' has no attribute 'UMAP'

    求助:AttributeError: module 'umap' has no attribute 'UMAP'

    Summary[问题简述(一句话)] 在Windows上运行出现问题:module 'umap' has no attribute 'UMAP' 在训练声码器和启动demo时都出现了这个问题

    Env & To Reproduce[复现与环境]

    Traceback (most recent call last):
      File "D:\MockingBird\encoder_train.py", line 46, in <module>
        train(**vars(args))
      File "D:\MockingBird\encoder\train.py", line 100, in train
        vis.draw_projections(embeds, utterances_per_speaker, step, projection_fpath)
      File "D:\MockingBird\encoder\visualizations.py", line 164, in draw_projections
        reducer = umap.UMAP()
    AttributeError: module 'umap' has no attribute 'UMAP'
    
    opened by heziyu2025 0
  • 小白求教:运行工具箱时报错“AttributeError: 'Toolbox' object has no attribute 'selected_source_utterance'”

    小白求教:运行工具箱时报错“AttributeError: 'Toolbox' object has no attribute 'selected_source_utterance'”

    微信截图_20221215215700 微信截图_20221215215735

    加载数据集运行工具箱时报错: AttributeError: 'Toolbox' object has no attribute 'selected_source_utterance'

    工具箱右上边,输入框下边的Vocode only按钮也是灰色的,只能合成没有声音输出

    工具箱左下角 Toolbox outpup 也是无法加载选项

    opened by love530love 0
  • 请更新依赖 requirements.txt ,web.py 所需的一些包并未涵盖在其中

    请更新依赖 requirements.txt ,web.py 所需的一些包并未涵盖在其中

    当前版本:@main-b402f9d 时间:2022-12-15

    当前 requirements.txt 中所缺失的包:

    fastapi==0.88.0
      pydantic==1.10.2
        typing_extensions==4.4.0
      starlette==0.22.0
        anyio==3.6.2
          idna==3.4
          sniffio==1.3.0
        typing_extensions==4.4.0
    loguru==0.6.0
      colorama==0.4.6
      win32-setctime==1.1.0
    typer==0.7.0
      click==8.0.0
        colorama==0.4.6
    

    注:其中倒数第二行, click 版本应注明保持在 8.0.0 ,新版本会导致 "get_os_args" 的报错。其他包最新版本仍可用

    opened by Golevka2001 0
Releases(v0.0.1)
Owner
Weijia Chen
ex-Facebook, Tencent Engineer. Focus on the cutting-edge SaaS/IaaS/ Cloud Service, expertise in Distributed System, AI.
Weijia Chen
Chinese real time voice cloning (VC) and Chinese text to speech (TTS).

Chinese real time voice cloning (VC) and Chinese text to speech (TTS). 好用的中文语音克隆兼中文语音合成系统,包含语音编码器、语音合成器、声码器和可视化模块。

Kuang Dada 6 Nov 8, 2022
Live Speech Portraits: Real-Time Photorealistic Talking-Head Animation (SIGGRAPH Asia 2021)

Live Speech Portraits: Real-Time Photorealistic Talking-Head Animation This repository contains the implementation of the following paper: Live Speech

OldSix 575 Dec 31, 2022
A Python wrapper for simple offline real-time dictation (speech-to-text) and speaker-recognition using Vosk.

Simple-Vosk A Python wrapper for simple offline real-time dictation (speech-to-text) and speaker-recognition using Vosk. Check out the official Vosk G

null 2 Jun 19, 2022
💬 Open source machine learning framework to automate text- and voice-based conversations: NLU, dialogue management, connect to Slack, Facebook, and more - Create chatbots and voice assistants

Rasa Open Source Rasa is an open source machine learning framework to automate text-and voice-based conversations. With Rasa, you can build contextual

Rasa 15.3k Dec 30, 2022
💬 Open source machine learning framework to automate text- and voice-based conversations: NLU, dialogue management, connect to Slack, Facebook, and more - Create chatbots and voice assistants

Rasa Open Source Rasa is an open source machine learning framework to automate text-and voice-based conversations. With Rasa, you can build contextual

Rasa 15.3k Jan 3, 2023
💬 Open source machine learning framework to automate text- and voice-based conversations: NLU, dialogue management, connect to Slack, Facebook, and more - Create chatbots and voice assistants

Rasa Open Source Rasa is an open source machine learning framework to automate text-and voice-based conversations. With Rasa, you can build contextual

Rasa 10.8k Feb 18, 2021
This project converts your human voice input to its text transcript and to an automated voice too.

Human Voice to Automated Voice & Text Introduction: In this project, whenever you'll speak, it will turn your voice into a robot voice and furthermore

Hassan Shahzad 3 Oct 15, 2021
Every Google, Azure & IBM text to speech voice for free

TTS-Grabber Quick thing i made about a year ago to download any text with any tts voice, over 630 voices to choose from currently. It will split the i

null 16 Dec 7, 2022
Vad-sli-asr - A Python scripts for a speech processing pipeline with Voice Activity Detection (VAD)

VAD-SLI-ASR Python scripts for a speech processing pipeline with Voice Activity

Dynamics of Language 14 Dec 9, 2022
Neural building blocks for speaker diarization: speech activity detection, speaker change detection, overlapped speech detection, speaker embedding

⚠️ Checkout develop branch to see what is coming in pyannote.audio 2.0: a much smaller and cleaner codebase Python-first API (the good old pyannote-au

pyannote 2.2k Jan 9, 2023
Speech Recognition for Uyghur using Speech transformer

Speech Recognition for Uyghur using Speech transformer Training: this model using CTC loss and Cross Entropy loss for training. Download pretrained mo

Uyghur 11 Nov 17, 2022
Silero Models: pre-trained speech-to-text, text-to-speech models and benchmarks made embarrassingly simple

Silero Models: pre-trained speech-to-text, text-to-speech models and benchmarks made embarrassingly simple

Alexander Veysov 3.2k Dec 31, 2022
PyTorch implementation of Microsoft's text-to-speech system FastSpeech 2: Fast and High-Quality End-to-End Text to Speech.

An implementation of Microsoft's "FastSpeech 2: Fast and High-Quality End-to-End Text to Speech"

Chung-Ming Chien 1k Dec 30, 2022
Simple Speech to Text, Text to Speech

Simple Speech to Text, Text to Speech 1. Download Repository Opsi 1 Download repository ini, extract di lokasi yang diinginkan Opsi 2 Jika sudah famil

Habib Abdurrasyid 5 Dec 28, 2021
A Python module made to simplify the usage of Text To Speech and Speech Recognition.

Nav Module The solution for voice related stuff in Python Nav is a Python module which simplifies voice related stuff in Python. Just import the Modul

Snm Logic 1 Dec 20, 2021
Code for ACL 2022 main conference paper "STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translation".

STEMM: Self-learning with Speech-Text Manifold Mixup for Speech Translation This is a PyTorch implementation for the ACL 2022 main conference paper ST

ICTNLP 29 Oct 16, 2022
A method to generate speech across multiple speakers

VoiceLoop PyTorch implementation of the method described in the paper VoiceLoop: Voice Fitting and Synthesis via a Phonological Loop. VoiceLoop is a n

Facebook Archive 873 Dec 15, 2022