Fixes #99

First pass at adding this feature. Not extensively tested yet.

I took the approach @mmmaat suggested, which made the changes pretty minimal. But I'd be happy to rewrite it the way @hadware proposed if it's decided that's better.

echo "하늘은 파랗게 구름은 하얗게 실바람도 불어와 부풀은 내 마음 나뭇잎 푸르게 강물도 푸르게 아름다운 이곳에 내가 있고 네가 있네 우리는 이 땅 위에 우리는 태어나고 아름다운 이곳에 자랑스런 이곳에 살리라 찬란하게 빛나는 붉은 태양이 비추고 하얀 물결 넘치는 저 바다와 함께 있네 그 얼마나 좋은가 우리 사는 이곳에 사랑하는 그대와 노래하리 빰빠밤빠밤 빠바밤 빠바밤 빰빠 빠바바바바밤 오늘도 너를 만나러 가야지 말해야지 먼 훗날에 너와 나 살고 지고 영원한 이곳에 우리의 새 꿈을 만들어 보고파 찬란하게 빛나는
붉은 태양이 비추고 하얀 물결 넘치는 저 바다와 함께 있네 그 얼마나 좋은가 우리 사는 이곳에 사랑하는 그대와 사랑하며 노래하리 빰빠밤빠밤 빠바밤 빠바밤 빰빠 빠바바바바밤 빰빠밤빠밤 빠바밤 빠바밤 빰빠 빠바바바바밤 빰빠밤빠밤 빠바밤 빠바밤 빰빠 빠바바바바밤 빰빠밤빠밤 빠바밤 빠바밤 빰빠 빠바바바바밤 오오오오 봄여름이 지나면 가을 겨울이 온다네 아름다운 강산 너의 마음은 나의 마음 나의 마음은 너의 마음 너와 나는 한마음 너와 나 우리 영원히 영원히 사랑 영원히 영원히 우리 모두 다 모두 다 끝없이 다정해 end of the sentence" | phonemize
[WARNING] 1 utterances containing language switches on lines 1
[WARNING] extra phones may appear in the "en-us" phoneset
[WARNING] language switch flags have been kept (applying "keep-flags" policy)
(ko)hɐnɯɾɯn phɐɾɐtkhe ɡuɾɯmɯn hɐjɐtkhe siɫbɐɾɐmdo puɾʌwɐ puphuɾɯnnɛmɐɯmnɐmunnip phuɾɯqe ɡɐŋmuɫdo phuɾɯqe ɐɾɯmdɐun iqosenɛqɐ itkoneqɐ inne uɾinɯn i tɐŋ wie uɾinɯn thɛʌnɐqo ɐɾɯmdɐun iqose tɕɐɾɐŋsɯɾʌn iqose sɐliɾɐ tʃhɐnɾɐnhɐqe pinnɐnɯn pulɡɯn thɛjɐŋi pitʃhuqo hɐjɐnmuɫqjʌɫnʌmtʃhinɯn tɕʌ pɐdɐwɐ hɐmqe inne ɡɯ ʌɫmɐnɐ tɕot(enus) (ko)hɐjɐnmuɫqjʌɫnʌmtʃhinɯn tɕʌ pɐdɐwɐ hɐmqe inne ɡɯ ʌɫmɐnɐ tɕoɯnqɐ uɾi sɐnɯn iqose sɐɾɐŋhɐnɯn ɡɯdɛwɐ sɐɾɐŋhɐmjʌnoɾɛhɐɾi pɐmpɐbɐmpɐbɐm pɐbɐbɐm pɐbɐbɐm pɐmpɐ pɐbɐbɐbɐbɐbɐm pɐmpɐbɐmpɐbɐm pɐbɐbɐm pɐbɐbɐm pɐmpɐ pɐbɐbɐbɐbɐbɐm pɐmpɐbɐmpɐbɐm pɐbɐbɐm pɐbɐbɐm pɐmpɐ pɐbɐbɐbɐbɐbɐm pɐmpɐbɐmpɐbɐm pɐbɐbɐm pɐbɐ(enus)

I'm using WSL to preprocess korean to ipa . for some reason the phonemizer takes only part of the sentence as input and do not preprocess characters after that . I tried using cat,echo,and phonemizer I/O(using option -o) but the result are all same

Is your feature request related to a problem? Please describe. I'd like more flexibility in defining punctuation, ideally by having access directly to the regex.

Specifically, instead of defining the characters to be counted as punctuation, I think it'd be more useful to me to define which characters are words to be phonemized, and treat everything else as punctuation.

Describe the solution you'd like Something as broad as [^\p{L}\p{M}0-9'] could work as a default, which from what I understand would capture everything that's not a number, unicode letter or its diacritics.

That may be overly broad, though, because I've run into trouble with espeak and characters from Cyrillic and Korean sets already, and I'd imagine characters from other less-supported languages could also be problematic.

Describe alternatives you've considered In my local copy of phonemizer, I've played with hard-coding the punctuation regex like so:

marks = "[^a-zA-ZÀ-ÖØ-öø-ÿ0-9',.$@&+\\-=/\\\]"
self._marks_re = re.compile(fr'(\s*{marks}+\s*|\s*(?<!\d),\s*|\s*(?<!\d)\.(?!\d)\s*)+')

which captures everything that's not a latin character or a set of marks that the backends can pronounce, like "æt" for "@".

The back half of this is also an attempt to handle the problem raised in #87, though I haven't tested it much, and there may be some cases where it breaks.

This reworks a few things related to preserving punctuation.

• Fixes the inconsistency described in #106. Now word separators appear in the same locations whether preserve_punctuation is True or False.
• Addresses part of the problem described in #104 (but doesn't return the output to what it was in 0.3, which I believe was incorrect – they'd also need to use a non-None word separator, maybe, to get what they're after?)
• Fixes #108 by refactoring the restore method to be iterative instead of recursive

A number of the tests had to be updated due to that first bullet point.

Adding the feature I requested in #95.

Not stripping out the empty lines was causing problems with the festival backend and preserve_punctuation, so I took the approach of stripping out the empty lines pre-phonemization and then reinserting them afterward.

I want to flag that I changed the conversion of the input text to a list from a generator to list comprehension here since I needed to run through the list a second time to preserve the empty lines, and I thought this made the code easier to read than making another generator. I'm assuming that this won't make a big difference on performance given how I think phonemizer is generally used.

when i run phonemize on windows 10 , python 3.6 i still have issue. it looks the temp file doesn't created at all. (the backslash issue fixed i can see)

>>> ph=phonemize('Hello World',strip=False,njobs=1,backend='espeak')
Failed to read file 'C:\Users\cinetec\AppData\Local\Temp\tmp5sigu2vf'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\cinetec\AppData\Local\Programs\Python\Python36\lib\site-packages\phonemizer-1.0.1-py3.6.egg\phonemizer\phonemize.py", line 94, in phonemize
    text, separator=separator, strip=strip, njobs=njobs)
  File "C:\Users\cinetec\AppData\Local\Programs\Python\Python36\lib\site-packages\phonemizer-1.0.1-py3.6.egg\phonemizer\backend.py", line 130, in phonemize
    out = self._phonemize_aux(self._list2str(text), separator, strip)
  File "C:\Users\cinetec\AppData\Local\Programs\Python\Python36\lib\site-packages\phonemizer-1.0.1-py3.6.egg\phonemizer\backend.py", line 235, in _phonemize_aux
    shlex.split(command, posix=False)).decode('utf8')
  File "C:\Users\cinetec\AppData\Local\Programs\Python\Python36\lib\subprocess.py", line 356, in check_output
    **kwargs).stdout
  File "C:\Users\cinetec\AppData\Local\Programs\Python\Python36\lib\subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['espeak', '-ven-us', '--ipa=3', '-q', '-f', 'C:\\Users\\cinetec\\AppData\\Local\\Temp\\tmp5sigu2vf']' returned non-zero exit status 1.

Part of a review at openjournals/joss-reviews#3958

• [ ] State of the field: Do the authors describe how this software compares to other commonly-used packages?

The manuscript identifies a number of related software packages with which this program could interface, but there is a gap in its references to the force alignment literature. Within linguistics, especially phonetic analysis, force alignment is an important part of the research pipeline whereby an acoustic signal is segmented and aligned with a text transcript. This then allows corpus queries and phonetic analysis of segments. The paper briefly touched on this when discussing Kaldi (Povey, et al. 2011), but the state of the field is broader and this program has important implications for that field. I believe the paper would be improved by further review of that literature.

The most impactful piece of software in that field is the force Alignment and Vowel Extraction (FAVE) toolkit (Rosenfelder, et al. 2014) which converts orthographic transcriptions to phonetic transcriptions through dictionary lookups using the CMU pronunciation dictionary. This has the downside of not being able to handle out-of-dictionary words requiring experimenter transcription or data exclusion.

Other researchers have been trying to improve coverage of force alignment to underdocumented languages and a major problem is the lack of grapheme-to-phoneme mappings (Barth, et al. 2020) or comprehensive pronunciation dictionaries (Johnson, Di Paolo, and Bell 2018). These can be substantial work and the language, orthographic system, or researcher time can limit the utility of these approaches.

These programs require a task similar to the one performed by this package, but do it in a seemingly different way. Comparing this package to the methods used in those packages will improve the paper by connecting it to a wider body of literature and identifying new potential areas of impact.

This is a really interesting project, and I'm excited to look further into the code!

How does one access one's files from inside the interactive session? That is, if I sudo docker run -it phonemizer /bin/bash I get transported inside a different space, where my files are not available. Right? And this is the only way to call phonemizer in mac?

Minor suggestion: In the instructions for docker users, add the explicit instruction to do git clone https://github.com/bootphon/phonemizer.git.

Hi, I heard about espeak-ng and am considering installing it: https://github.com/espeak-ng/espeak-ng Do you know if phonemizer will work with espeak-ng? regards, Andrew

Describe the bug

When phonemizing a text whick has more than 100k utterances, it will always gives a "RuntimeError" include "espeak not installed on your system"，“failed to find espeak library” and "invalid voice code 'cmn' " at around 900 utterances.

Phonemizer version phonemizer-3.0 available backends: espeak-ng-1.49.2, espeak-mbrola, festival-2.5.0, segments-2.2.0

System cat /proc/version: Linux version 4.15.0-106-generic (buildd@lcy01-amd64-016) (gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04))

python: Python 3.9.1 (default, Dec 11 2020, 14:32:07) [GCC 7.3.0] :: Anaconda, Inc. on linux

To reproduce

txtdict = txt2dict(text_path)

with open(scp_path) as f:
    for line in f.readlines():
        txt = txtdict.get(line[0])
        phone = phonemize(txt, backend='espeak', language='cmn', 
                        separator=Separator(word='/', phone=' ', syllable="-"))
        rows.append([wav, new_wav, txt, phone, new_phone])

Expected behavior

When run phonemize with -l cmn or zh, the phoneme is International Phonetic Alphabet instead of Mandarin phoneme

Is this a code problem or a bug? How can i convert Chinese to Mandarin phoneme? Look forward to your reply, thanks!

Expected behavior I try to convert Chinese to Mandarin phoneme with this code：

cat chinese.txt | PHONEMIZER_ESPEAK_PATH=$(which espeak) phonemize -o train_out.phn -p ' ' -w '' -l zh -j 70 --language-switch remove-flags

result: fatal error: language "zh" is not supported by the espeak backend then i check: espeak --voices it include Mandarin phonemize --list-languages it said：cmn -> Chinese (Mandarin)

Phonemizer version phonemizer-3.0 espeak-ng-1.50, espeak-mbrola, festival-2.5.0, segments-2.2.0

System ubuntu 20.04

Describe the bug When using the default preserve_punctuation=False, the Festival backend ignores text that only contains punctuation, whereas the Espeak backend returns the empty string.

Phonemizer version

phonemizer-3.2.1
available backends: espeak-ng-1.50, espeak-mbrola, festival-2.5.0, segments-2.2.1

System Ubuntu 20.04.4 Linux kernel 5.15.0 Python 3.8.10

To reproduce

from phonemizer import phonemize

print(phonemize([".", "."], language="en-us", backend="festival"))
print(phonemize([".", "."], language="en-us", backend="espeak"))
print(phonemize([".", "."], language="mb-us1", backend="espeak-mbrola"))

Yields output

[]
['', '']
['', '']

Expected behavior Should output:

['', '']
['', '']
['', '']

Describe the bug When calling phonemize on an instance of EspeakBackend with the character "ꪁ", the backend enters a corrupted state where all succeeding phonemization (including in the sentence with "ꪁ") is incorrect.

Phonemizer version Phonemizer 3.2.1 Espeak NG 1.50

System Reproduced the bug both on Win10 and Ubuntu

To reproduce

from phonemizer.backend import EspeakBackend

texts = [
    "a, b, c, d, e, f, p, w, y, z",
    "ꪁ",
    "a, b, c, d, e, f, p, w, y, z"
]

backend = EspeakBackend(
    language="en-us", preserve_punctuation=True, with_stress=True,
    language_switch="remove-flags", words_mismatch="ignore"
)

for text in texts:
    print(backend.phonemize([text])[0])

Expected behavior Expected output:

ˈeɪ , bˈiː , sˈiː , dˈiː , ˈiː , ˈɛf , pˈiː , dˈʌbəljˌuː , wˈaɪ , zˈiː 

ˈeɪ , bˈiː , sˈiː , dˈiː , ˈiː , ˈɛf , pˈiː , dˈʌbəljˌuː , wˈaɪ , zˈiː

Actual output:

ˈeɪ , bˈiː , sˈiː , dˈiː , ˈiː , ˈɛf , pˈiː , dˈʌbəljˌuː , wˈaɪ , zˈiː 

ˈʌ , bˈʌ , sˈʌ , dˈʌ , ˈʌ , ˈʌf , pˈʌ , dˈʌbd-jʌ , wˈʌ , zˈʌ 

Describe the bug It seems that instantiation of multiple EspeakBackend objects is not correctly handled. All the objects start operating with the language used to instantiate the last object. Please refer to the example below.

Phonemizer version 3.0.1

System macOS 11.6.4 python 3.8.9 [Clang 13.0.0 (clang-1300.0.29.30)] on darwin

To reproduce

from phonemizer.backend import EspeakBackend

en_backend = EspeakBackend(
    "en-us",
    preserve_punctuation=True,
    with_stress=True,
    language_switch="remove-flags",
    words_mismatch="ignore",
)
en_sentence = ["I love to eat pizza everyday"]
print(en_backend.phonemize(en_sentence, njobs=1, strip=True)) # ['aɪ lˈʌv tʊ ˈiːt pˈiːtsə ˈɛvɹɪdˌeɪ']

de_backend = EspeakBackend(
    "de",
    preserve_punctuation=True,
    with_stress=True,
    language_switch="remove-flags",
    words_mismatch="ignore",
)
de_sentence = ["ich esse jeden tag gerne pizza."]
print(de_backend.phonemize(de_sentence, njobs=1, strip=True)) # ['ɪç ˈɛsə jˈeːdən tˈɑːk ɡˈɛɾnə pˈɪtsɑː.']

incorrect_en = en_backend.phonemize(en_sentence, njobs=1, strip=True)
en_with_de = de_backend.phonemize(en_sentence, njobs=1, strip=True)

assert en_with_de == incorrect_en
print(incorrect_en, en_with_de) 
# ['ˈiː lˈoːvə tˈoː eːˈɑːt pˈɪtsɑː ˈeːveːrˌyːdɛɪ'] 
# ['ˈiː lˈoːvə tˈoː eːˈɑːt pˈɪtsɑː ˈeːveːrˌyːdɛɪ']

Expected behavior Notice that incorrect_en is equal to en_with_de and not equal to en_sentence.

Additional context This problem happens only with njobs=1 and doesn't appear with njobs>1

Currently, to get the language-specific SAMPA form of each phoneme, a working (and espeak-friendly) installation of mbrola is required. This is problematic for several reasons:

• it requires an additional system-wide package install on linux platform (even though mbrola is available on most distributions)
• it requires the installation of a corresponding mbrola voice, which is either unpractical and/or quite heavy. Moreover, the entirety of the voice's speech data isn't actually used by espeak for the phonemization.
• the OSX/windows support for mbrola is very bad.

The foldings are all here: https://github.com/espeak-ng/espeak-ng/tree/master/phsource/mbrola

echo "bonjour le monde" | phonemize -b espeak-mbrola -l mb-fr1 -p ' ' -w '/w ' when running this command it is giving error

fatal error: language "mb-fr4" is not supported by the espeak-mbrola backend