gentle forced aligner

Related tags

Audio gentle
Overview

Gentle

Robust yet lenient forced-aligner built on Kaldi. A tool for aligning speech with text.

Getting Started

There are three ways to install Gentle.

  1. Download the pre-built Mac application. This package includes a GUI that will start the server and a browser. It only works on Mac OS.

  2. Use the Docker image. Just run docker run -P lowerquality/gentle. This works on all platforms supported by Docker.

  3. Download the source code and run ./install.sh. Then run python3 serve.py to start the server. This works on Mac and Linux.

Using Gentle

By default, the aligner listens at http://localhost:8765. That page has a graphical interface for transcribing audio, viewing results, and downloading data.

There is also a REST API so you can use Gentle in your programs. Here's an example of how to use the API with CURL:

curl -F "[email protected]" -F "[email protected]" "http://localhost:8765/transcriptions?async=false"

If you've downloaded the source code you can also run the aligner as a command line program:

git clone https://github.com/lowerquality/gentle.git
cd gentle
./install.sh
python3 align.py audio.mp3 words.txt

The default behaviour outputs the JSON to stdout. See python3 align.py --help for options.

Comments
  • Gentle Stops Working on Second Pass of Server Mode, or Align.py With Longer Files

    Gentle Stops Working on Second Pass of Server Mode, or Align.py With Longer Files

    Hello,

    I am running the latest version of Gentle on a Mac. In server mode, it runs through the first pass fine, but when it runs the second pass (the realignment phase), it gets to about half way and then it stops processing. There are no errors, but the k3 background processes all go away, except for one, and CPU usage drops to 0%. I think it might be a problem with the multipass.realign() function, standard_kaldi.py, or even the k3.cc file itself.

    This seems to only occur on longer audio files.

    Thank you @strob

    EDIT: I did some more testing, and align.py also will not finish if the audio file is too long. (Do note the test files work fine, it's just longer files I have problems with. In addition, the 2017 version of Gentle did not have this issue)

    opened by RomanScott 22
  • Ubuntu installation from source not working

    Ubuntu installation from source not working

    I am having an issue installing from source on Ubuntu 18.04.4 LTS. While installing Kaldi, the makefile doesn't get created. I have tried all suggested fixes in #192, as well as Kaldi issue 3395 (https://github.com/kaldi-asr/kaldi/issues/3395).

    opened by tfburns 10
  • Unable to compile Kaldi version after El Capitan

    Unable to compile Kaldi version after El Capitan

    Since upgrading to El Capitan I haven't been able to install the included version of Kaldi. There's an issue with gcc-fortran. Were you able to get it compiling?

    opened by maxhawkins 9
  • Install on Centos7

    Install on Centos7

    Hello, thanks for gentle!

    I'm trying to install on centos 7, installed dependencies with yum install, here is my install.sh:

    #!/bin/bash
    
    set -e
    git submodule init
    git submodule update
    
    #./install_deps.sh
    pip install .
    (cd ext && ./install_kaldi.sh)
    ./install_models.sh
    ./install_language_model.sh
    cd ext && make depend -j 2 && make -j 2
    

    But somewhre in the make process this happens (after ./install_language_model.sh is executed):

    make[1]: Leaving directory `/root/src/gentle/ext/kaldi/src/util'
    g++ -rdynamic   k3.o kaldi/src/online2/kaldi-online2.a kaldi/src/ivector/kaldi-ivector.a kaldi/src/nnet3/kaldi-nnet3.a kaldi/src/chain/kaldi-chain.a kaldi/src/nnet2/kaldi-nnet2.a kaldi/src/lat/kaldi-lat.a kaldi/src/decoder/kaldi-decoder.a kaldi/src/cudamatrix/kaldi-cudamatrix.a kaldi/src/feat/kaldi-feat.a kaldi/src/transform/kaldi-transform.a kaldi/src/gmm/kaldi-gmm.a kaldi/src/hmm/kaldi-hmm.a kaldi/src/tree/kaldi-tree.a kaldi/src/matrix/kaldi-matrix.a kaldi/src/fstext/kaldi-fstext.a kaldi/src/util/kaldi-util.a kaldi/src/base/kaldi-base.a   /root/src/gentle/ext/kaldi/tools/openfst/lib/libfst.a /root/src/gentle/ext/kaldi/tools/OpenBLAS/install/lib/libopenblas.a -lgfortran -lm -lpthread -ldl  -o k3
    m3.cc: In function 'int main(int, char**)':
    m3.cc:174:28: error: no matching function for call to 'AddSelfLoops(kaldi::TransitionModel&, std::vector<int>&, float&, bool, bool, fst::VectorFst<fst::ArcTpl<fst::TropicalWeightTpl<float> > >*)'
                       &hclg_fst);
                                ^
    m3.cc:174:28: note: candidates are:
    In file included from kaldi/src/fstext/pre-determinize.h:94:0,
                     from kaldi/src/fstext/fstext-utils-inl.h:29,
                     from kaldi/src/fstext/fstext-utils.h:448,
                     from kaldi/src/fstext/context-fst-inl.h:23,
                     from kaldi/src/fstext/context-fst.h:535,
                     from m3.cc:1:
    

    and later on:

    make: *** [m3.o] Error 1
    make: *** Waiting for unfinished jobs....
    kaldi/src/ivector/kaldi-ivector.a(ivector-extractor.o): In function `kaldi::IvectorExtractor::ComputeDerivedVars()':
    /root/src/gentle/ext/kaldi/src/ivector/ivector-extractor.cc:198: undefined reference to `kaldi::g_num_threads'
    kaldi/src/ivector/kaldi-ivector.a(ivector-extractor.o): In function `kaldi::IvectorExtractorStats::IvectorExtractorStats(kaldi::IvectorExtractor const&, kaldi::IvectorExtractorStatsOptions const&)':
    /root/src/gentle/ext/kaldi/src/ivector/ivector-extractor.cc:807: undefined reference to `kaldi::Mutex::Mutex()'
    

    Any advice? (by the way, the the latest cuda (9.2 as of writing this) is not supported by gentle's kaldi i guess)

    Thanks a lot.

    opened by spinicrus 8
  • (SOLVED!) FileNotFoundError: [Errno 2] No such file or directory: 'ext/m3'

    (SOLVED!) FileNotFoundError: [Errno 2] No such file or directory: 'ext/m3'

    Hey, I'm trying to run a gentle command. But when I do, I get this error.

    Can anyone pls explain what I should do?

    (simple explanations cus im a total noob)

    INFO:root:converting audio to 8K sampled wav
    INFO:root:starting alignment
    Traceback (most recent call last):
      File "align.py", line 54, in <module>
        aligner = gentle.ForcedAligner(resources, transcript, nthreads=args.nthreads, disfluency=args.disfluency, conservative=args.conservative, disfluencies=disfluencies)
      File "gentle/forced_aligner.py", line 18, in __init__
        gen_hclg_filename = language_model.make_bigram_language_model(ks, resources.proto_langdir, **kwargs)
      File "gentle/language_model.py", line 126, in make_bigram_language_model
        raise e
      File "gentle/language_model.py", line 116, in make_bigram_language_model
        subprocess.check_output([MKGRAPH_PATH,
      File "/usr/local/Cellar/[email protected]/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 424, in check_output
        return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
      File "/usr/local/Cellar/[email protected]/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 505, in run
        with Popen(*popenargs, **kwargs) as process:
      File "/usr/local/Cellar/[email protected]/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 951, in __init__
        self._execute_child(args, executable, preexec_fn, close_fds,
      File "/usr/local/Cellar/[email protected]/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 1821, in _execute_child
        raise child_exception_type(errno_num, err_msg, err_filename)
    FileNotFoundError: [Errno 2] No such file or directory: 'ext/m3'
    

    any help is appreciated. thx!

    opened by coderserif 7
  • i could not execute serve.py , it gives an error in multipass.py file at line 15

    i could not execute serve.py , it gives an error in multipass.py file at line 15

    i installed this aligner on linux which at last shows this:

    Fetched 5,296 kB in 1min 6s (80.2 kB/s)                                        
    Reading package lists... Done
    Reading package lists... Done
    Building dependency tree       
    Reading state information... Done
    Package ffmpeg is not available, but is referred to by another package.
    This may mean that the package is missing, has been obsoleted, or
    is only available from another source
    
    E: Package 'ffmpeg' has no installation candidate
    

    after that when i execute python serve.py it gives this error:

    Traceback (most recent call last):
      File "serve.py", line 23, in <module>
        from gentle import multipass
      File "/home/ucertify/Desktop/gentle/gentle/multipass.py", line 15, in <module>
        with open(vocab_path) as f:
    IOError: [Errno 2] No such file or directory: 'PROTO_LANGDIR/graphdir/words.txt'
    

    Please help me...

    opened by saquibajaz 7
  • How to run it in Colab?

    How to run it in Colab?

    I do the following, but the installation is going forever:

    !git clone -q https://github.com/lowerquality/gentle.git > /dev/null
    %cd /content/gentle
    !./install.sh
    
    opened by qo4on 5
  • Error finding kaldi files.

    Error finding kaldi files.

    When I run install.sh on Linux, it eventually stops at this error.

    Makefile:5: kaldi/src/kaldi.mk: No such file or directory Not building with cuda!!! make: *** No rule to make target 'kaldi/src/kaldi.mk'. Stop.

    the ext/kaldi/src direct has a bunch of other things but is indeed missing kaldi.mk file.

    opened by dhuv 5
  • Refactor: factor algorithm out of server; bundle resource lookups into a single object

    Refactor: factor algorithm out of server; bundle resource lookups into a single object

    Hi, thanks for making Gentle available!

    I'm looking into using Gentle, but without running a server. This PR includes:

    • align.py, a shell command that runs the forced alignment algorithm and outputs JSON
    • To do that cleanly, I factored out the algorithm from serve.py, creating two classes ForcedAligner and FullTranscriber
    • And to do that cleanly, I bundled all resource lookups into a single GentleResources instance which I pass around; and once I had that I propagated it into the other bits of code that used those resources.

    I took my best shot at organizing and naming and whatnot, but of course I'm open to changes that'd you'd find more suitable.

    [Oh I should also point out that I haven't tested FullTranscriber since the install didn't seem to include data/graph/HCLG.fst. It "should" work since the code is just moved over from where it had been earlier, but of course bugs could have easily crept in.]

    opened by ronen 5
  • DMG crashes on boot

    DMG crashes on boot

    gentle0.03.dmg does not start; may be related to spurious homebrew packages on my system.

    Crashed Thread:        0  Dispatch queue: com.apple.main-thread
    
    Exception Type:        EXC_BAD_INSTRUCTION (SIGILL)
    Exception Codes:       0x0000000000000001, 0x0000000000000000
    Exception Note:        EXC_CORPSE_NOTIFY
    
    Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
    0   org.python.python               0x000000010324d060 PyTuple_New + 112
    1   org.python.python               0x000000010325049e PyType_Ready + 199
    2   org.python.python               0x0000000103250460 PyType_Ready + 137
    3   org.python.python               0x000000010323d19a _Py_ReadyTypes + 16
    4   org.python.python               0x00000001032a6475 Py_InitializeEx + 395
    5   org.pythonmac.unspecified.gentle    0x00000001000024ac 0x100000000 + 9388
    6   org.pythonmac.unspecified.gentle    0x000000010000117a main + 650
    7   org.pythonmac.unspecified.gentle    0x0000000100000be4 start + 52
    
    opened by strob 5
  • List of phones Gentle uses and what they map to

    List of phones Gentle uses and what they map to

    I'm trying to use Gentle to create a lip-flap animator, and while the software works for me, I don't really know how to use its outputs.

    In my program, each phoneme of the English language will be assigned to a photo, and so I need to know what each phoneme uttered is. Gentle gives me this data, but the phoneme names don't seem to match any digitization of, say, the IPA. Instead, I get obscure phoneme names like dh_B and ah_E. dh is usually transcribed as th so this caught me off-guard at first, and I have no idea what E means.

    If I could get a list of all of Gentle's possible phonemes and their mappings to the IPA, that would make my life a lot easier.

    opened by SuperSonicHub1 4
  • "RuntimeError: No resource directory /Users/.../gentle/exp. Check GENTLE_RESOURCES_ROOT environment variable?"

    I am getting this error when trying to call align.py from the command line on Mac OS 10.12.6 Sierra, Python 3.11:

    $ python3 align.py /Volumes/.../5077_FINAL.WAV /Volumes/.../RAY15077_cleaned.txt Traceback (most recent call last): File "/Users/.../gentle/align.py", line 49, in resources = gentle.Resources() ^^^^^^^^^^^^^^^^^^ File "/Users/.../gentle/gentle/resources.py", line 19, in init require_dir(self.proto_langdir) File "/Users/.../gentle/gentle/resources.py", line 16, in require_dir raise RuntimeError("No resource directory %s. Check %s environment variable?" % (path, ENV_VAR)) RuntimeError: No resource directory /Users/.../gentle/exp. Check GENTLE_RESOURCES_ROOT environment variable?

    It looks like the issue is that the gentle.Resources() method is looking for a path that ends in "gentle/exp", but that folder isn't in the directory structure.

    Does anyone have any suggestions to run this from the command line on a Mac? I don't currently have access to Docker.

    opened by OmniControlRobot 1
  • An option to substitute OpenAI's Whisper models for Kaldi?

    An option to substitute OpenAI's Whisper models for Kaldi?

    I'm not a developer but I do find Gentle very useful.

    Since OpenAI released their Whisper models last week, I've been wondering if anyone with development skills would be interested in enabling an option to utilize Whisper instead of Kaldi when running Gentle.

    I know that language support for spoken languages beyond English has been a long-standing request for Gentle. Whisper appears to be pointedly multi-lingual, so perhaps this would make support for languages beyond English more easily achievable for Gentle?

    Anyway, please let me know what scale of an undertaking this would be. Thanks in advance.

    opened by natelawrence 2
  • Mac App not working

    Mac App not working

    The prebuilt Mac application will not open on MacOS 12.5. I have option clicked selected open and it simply flickers and immediately closes.

    Model Name: MacBook Pro Model Identifier: MacBookPro18,2 Chip: Apple M1 Max Total Number of Cores: 10 (8 performance and 2 efficiency) Memory: 64 GB

    opened by awindsor 1
  • Phonemes in output - is it actual or is it from the dictionary?

    Phonemes in output - is it actual or is it from the dictionary?

    Hi I am trying to find out if the phonemes that are listed in the output are the actual phone utterances or just listed from the dictionary. I am trying to find out if phoneme was correctly pronounced? Is there a way to identify the mispronounced phonemes in Gentle output

    opened by tap111 0
  • Trained tdnn model causes broken pipe error

    Trained tdnn model causes broken pipe error

    I trained a kaldi tdnn model and and modified it for gentle format I tried running align.py by it causes the following error ! Can anyone help me with this?

    /var/folders/bn/gms2j96s50x1ts27vgwp1jfm0000gp/T/tmpv3dkq1cp_HCLG.fst ERROR:gentle.standard_kaldi:hclg_path does not exist: /var/folders/bn/gms2j96s50x1ts27vgwp1jfm0000gp/T/tmpv3dkq1cp_HCLG.fst /Users/adminuser/Documents/gentle/gentle/exp/tdnn_7b_chain_online/ /var/folders/bn/gms2j96s50x1ts27vgwp1jfm0000gp/T/tmpv3dkq1cp_HCLG.fst ERROR:gentle.standard_kaldi:hclg_path does not exist: /var/folders/bn/gms2j96s50x1ts27vgwp1jfm0000gp/T/tmpv3dkq1cp_HCLG.fst Traceback (most recent call last): File "/Users/adminuser/Documents/gentle/gentle/align.py", line 58, in result = aligner.transcribe(wavfile, progress_cb=on_progress, logging=logging) File "/Users/adminuser/Documents/gentle/gentle/gentle/forced_aligner.py", line 24, in transcribe words, duration = self.mtt.transcribe(wavfile, progress_cb=progress_cb) File "/Users/adminuser/Documents/gentle/gentle/gentle/transcriber.py", line 51, in transcribe pool.map(transcribe_chunk, range(n_chunks)) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 364, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 771, in get raise self._value File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 125, in worker result = (True, func(*args, **kwds)) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar return list(map(*args)) File "/Users/adminuser/Documents/gentle/gentle/gentle/transcriber.py", line 38, in transcribe_chunk k.push_chunk(buf) File "/Users/adminuser/Documents/gentle/gentle/gentle/standard_kaldi.py", line 39, in push_chunk self._p.stdin.write(buf) #arr.tostring()) BrokenPipeError: [Errno 32] Broken pipe Exception ignored in: <function Kaldi.del at 0x7fcdccd76c10> Traceback (most recent call last): File "/Users/adminuser/Documents/gentle/gentle/gentle/standard_kaldi.py", line 79, in del File "/Users/adminuser/Documents/gentle/gentle/gentle/standard_kaldi.py", line 73, in stop File "/Users/adminuser/Documents/gentle/gentle/gentle/standard_kaldi.py", line 30, in _cmd BrokenPipeError: [Errno 32] Broken pipe Exception ignored in: <function Kaldi.del at 0x7fcdccd76c10> Traceback (most recent call last): File "/Users/adminuser/Documents/gentle/gentle/gentle/standard_kaldi.py", line 79, in del File "/Users/adminuser/Documents/gentle/gentle/gentle/standard_kaldi.py", line 73, in stop File "/Users/adminuser/Documents/gentle/gentle/gentle/standard_kaldi.py", line 30, in _cmd BrokenPipeError: [Errno 32] Broken pipe Exception ignored in: <function Kaldi.del at 0x7fcdccd76c10> Traceback (most recent call last): File "/Users/adminuser/Documents/gentle/gentle/gentle/standard_kaldi.py", line 79, in del File "/Users/adminuser/Documents/gentle/gentle/gentle/standard_kaldi.py", line 73, in stop File "/Users/adminuser/Documents/gentle/gentle/gentle/standard_kaldi.py", line 30, in _cmd BrokenPipeError: [Errno 32] Broken pipe Exception ignored in: <function Kaldi.del at 0x7fcdccd76c10> Traceback (most recent call last): File "/Users/adminuser/Documents/gentle/gentle/gentle/standard_kaldi.py", line 79, in del File "/Users/adminuser/Documents/gentle/gentle/gentle/standard_kaldi.py", line 73, in stop File "/Users/adminuser/Documents/gentle/gentle/gentle/standard_kaldi.py", line 30, in _cmd BrokenPipeError: [Errno 32] Broken pipe

    Initially I got a called subprocess error I tried to decode the output by changing line 116 in language_model.py to

    try: devnull = open(os.devnull, 'wb')

        print([MKGRAPH_PATH,
                        proto_langdir,
                        txt_fst_file.name,
                        hclg_filename])
    
        sub_result = subprocess.check_output([MKGRAPH_PATH,
                        proto_langdir,
                        txt_fst_file.name,
                        hclg_filename],
                        stderr=devnull)
    except subprocess.CalledProcessError as e:
        sub_result = e.output.decode() 
    
    opened by vikraman22 0
Releases(0.10.1)
A sentence aligner for comparable corpora

About Yalign is a tool for extracting parallel sentences from comparable corpora. Statistical Machine Translation relies on parallel corpora (eg.. eur

Machinalis 128 Aug 24, 2022
Lingtrain Aligner — ML powered library for the accurate texts alignment.

Lingtrain Aligner ML powered library for the accurate texts alignment in different languages. Purpose Main purpose of this alignment tool is to build

Sergei Averkiev 76 Dec 14, 2022
dragmap-meth: Fast and accurate aligner for bisulfite sequencing reads using dragmap

dragmap_meth (dragmap_meth.py) Alignment of BS-Seq reads using dragmap. Intro This works for single-end reads and for paired-end reads from the direct

Shaojun Xie 3 Jul 14, 2022