the apostrophe is needed to write Ukrainian, as in ім'я ("name")

https://en.wiktionary.org/wiki/%D1%96%D0%BC%27%D1%8F#Ukrainian

h/t @robinhad

Hi, thanks for the practical toolkit for CV data preprocessing!

I recently utilized this toolkit to validate data of different languages, but found the Validator failed to initialize, i.e. it. After checking the code I found, the initialization of Validator demands data/$lang/validate.tsv to be given.

Thus my question is: 1) Will the missing data be updated recently? and 2) How to prepare the data/$lang/validate.tsv file from the scratch?

Thanks in advance!

the letter ß is part of the german alphabet and definetely part of the german corpus. It is woth noting that this letter is not used in the swiss version of high german.

This happened when I try to pip install it in the Windows Anaconda cmd (Windows 10 US English version, Python 3.9 and 3.8 tested).

c:\Users\xxxx> pip install git+https://github.com/ftyers/commonvoice-utils.git
Collecting git+https://github.com/ftyers/commonvoice-utils.git
  Cloning https://github.com/ftyers/commonvoice-utils.git to c:\temp1\pip-req-build-06f01lti
  Running command git clone -q https://github.com/ftyers/commonvoice-utils.git 'C:\TEMP1\pip-req-build-06f01lti'
  Resolved https://github.com/ftyers/commonvoice-utils.git to commit c738e7f8031cd2e1ca83fdbd6dd3e8a5db1ad583
    ERROR: Command errored out with exit status 1:
     command: 'D:\Anaconda\Anaconda3\envs\p39_c112\python.exe' -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\TEMP1\\pip-req-build-06f01lti\\setup.py'"'"'; __file__='"'"'C:\\TEMP1\\pip-req-build-06f01lti\\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base 'C:\TEMP1\pip-pip-egg-info-mvngstdl'
         cwd: C:\TEMP1\pip-req-build-06f01lti\
    Complete output (9 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\TEMP1\pip-req-build-06f01lti\setup.py", line 8, in <module>
        README = (HERE / "README.md").read_text()
      File "D:\Anaconda\Anaconda3\envs\p39_c112\lib\pathlib.py", line 1267, in read_text
        return f.read()
      File "D:\Anaconda\Anaconda3\envs\p39_c112\lib\encodings\cp1252.py", line 23, in decode
        return codecs.charmap_decode(input,self.errors,decoding_table)[0]
    UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 2697: character maps to <undefined>
    ----------------------------------------
WARNING: Discarding git+https://github.com/ftyers/commonvoice-utils.git. Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

It seems setup.py needs to configure encoding as UTF-8, but I'm a noob here...

At the moment the alphabet for Chatino includes sequences of numerals for the tone characters. This is an artefact of the original dataset used to generate the data. The official orthography uses superscript uppercase letters.

It should be possible to use Unicode superscript letters and implement the conversion within covo, but first we need a mapping from sequence of numerals → superscript uppercase letter.

Looks like that you forgot to commit it.

~$ python3 -m  pip install git+https://github.com/ftyers/commonvoice-utils.git
Defaulting to user installation because normal site-packages is not writeable
Collecting git+https://github.com/ftyers/commonvoice-utils.git
  Cloning https://github.com/ftyers/commonvoice-utils.git to /tmp/pip-req-build-4ptidukg
  Running command git clone -q https://github.com/ftyers/commonvoice-utils.git /tmp/pip-req-build-4ptidukg
Building wheels for collected packages: commonvoice-utils
  Building wheel for commonvoice-utils (setup.py) ... done
  Created wheel for commonvoice-utils: filename=commonvoice_utils-0.2.7-py3-none-any.whl size=142813 sha256=541d42fa2c786d602f4ca04e6f1ad8848a57ded5376f69a19629b1b602577fc7
  Stored in directory: /tmp/pip-ephem-wheel-cache-x_86ocb3/wheels/56/67/73/4bf2d8a681334251a44405673d52e767f646121bbd89c8b7fa
Successfully built commonvoice-utils
Installing collected packages: commonvoice-utils
Successfully installed commonvoice-utils-0.2.7
~$ python3 -c "from cvutils import Alphabet"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/selimcan/.local/lib/python3.8/site-packages/cvutils/__init__.py", line 10, in <module>
    from transliterator import Transliterator
ModuleNotFoundError: No module named 'transliterator'

I don't think it's this package https://pypi.org/project/transliterator/ that's required ?

Hi! I think I found an error here for Italian, the ' is a valid character for Italian and in cvutils/data/it/validate.tsv#L6 it is replaced with blank, even if there seem to be other rules below in the same file converting other symbols to ' and later allowing '. Maybe I'm not seeing a reason to have that replacement rule there on line 6? Thanks for the useful tools!

This adds Basque support to the segmenter.

• abbr.tsv:
  • Added common abbreviations from different sources:
    • Elhuyar: http://www.euskara.euskadi.net/appcont/elhuyar/laburdura_zerrenda.pdf
    • Euskaltzaindia: https://www.euskaltzaindia.eus/index.php?option=com_ebe&view=bilaketa&Itemid=1161&task=bilaketa&id=985
    • Euskaljakintza: https://euskaljakintza.com/kontsultategia/laburdurak/
  • Added two regex rules to avoid splitting sentences with . (-garren) in ordinal numbers.
• validate.tsv: normalize diacritics from Spanish and French that are frequent in Basque texts due to common names or foreign words.

With this segmenter, I re-trained your previous models adding the Wikipedia corpus here: https://github.com/coqui-ai/STT-models/pull/25

Processed TDK's list to extract abbr which uses periods (removed 2 instances already existing here). These are added with 1000 in the first column. https://tdk.gov.tr/wp-content/uploads/2019/01/Kısaltmalar_Dizini.pdf

Also added single capital letters with periods, which are used to shorten names. These are added with 999 in the first column. (we already had T. on the list)

I've made some updates to the DV specifications. I think with these additions the phonemiser output is sensible enough for now. Some of the rules are obviously simplified since their actual rules are more complex and probably require fst

The following changes were made:

• Add missing char ޱ in alphabet.txt
• Fix phons.tsv
  • Add rules for pre-nasalised consonants
  • Make އް and ށް default to stops. This is a sensible default for them. Should sound fine 90% of the time
  • default ން to voiced retroflex nasal.
  • fix vowel rules that were previously tied to consonant އ, preventing proper placement
  • Other minor adjustments to consonant maps

At https://github.com/ftyers/commonvoice-utils/blob/main/cvutils/data/uk/alphabet.txt should be абвгґдеєжзиіїйклмнопрстуфхцчшщьюя-ʼ. ы is a Russian letter.

Is it possible to implement optional "--exclude-xxx fn" flags to exclude recordings during cv export?

--exclude-voices voices.txt            // E.g. to measure the effect of a single person recording too much
--exclude-sentences sentences.txt             // E.g. to exclude reported sentences
--exclude-gender [male|female|other|empty]             // E.g. to train with male voices and test with female voices
etc

That would very much ease any experiments on biasing effects.

PS: The correct place to implement these would be CorporaCreator but it is not actively maintained as you know.

Similar can be implemented for opus corpora.

Bülent

Although Korean is not fully enabled on Common Voice yet, it only lacks 1500 sentences. If added, we can start using alphabet/normalization support provided by covo.

Hi. I tried using the g2p tool to phonemize hindi words, but there was some encoding issues.

from cvutils import Phonemiser p = Phonemiser('hi') p.phonemise('अवकाशग्रहण')

At first, the error message was like:

UnicodeDecodeError Traceback (most recent call last) C:\Users\MAGICD~1\AppData\Local\Temp/ipykernel_10860/951158175.py in 1 from cvutils import Phonemiser ----> 2 p = Phonemiser('hi') 3 p.phonemise('अवकाशग्रहण')

~\Anaconda3\lib\site-packages\cvutils\phonemiser.py in init(self, lang) 22 print('[Phonemiser] Function not implemented', file=sys.stderr) 23 try: ---> 24 self.validator = Validator(self.lang) 25 except FileNotFoundError: 26 pass

~\Anaconda3\lib\site-packages\cvutils\validator.py in init(self, lang) 13 self.nfkd = False 14 try: ---> 15 self.load_data() 16 except FileNotFoundError: 17 print('[Validator] Function not implemented', file=sys.stderr)

~\Anaconda3\lib\site-packages\cvutils\validator.py in load_data(self) 24 self.lower = False 25 data_dir = os.path.abspath(os.path.dirname(file)) + '/data/' ---> 26 for line in open(data_dir + self.lang + '/validate.tsv').readlines(): 27 if line[0] == '#': 28 continue

UnicodeDecodeError: 'gbk' codec can't decode byte 0x98 in position 50: illegal multibyte sequence

Then I set encoding='utf-8 in the line 26 of '~\Anaconda3\lib\site-packages\cvutils\validator.py' , but it didn't work. It still went like:

UnicodeDecodeError Traceback (most recent call last) ... ~\Anaconda3\lib\site-packages\cvutils\validator.py in load_data(self) 24 self.lower = False 25 data_dir = os.path.abspath(os.path.dirname(file)) + '/data/' ---> 26 for line in open(data_dir + self.lang + '/validate.tsv',encoding='utf-8').readlines(): 27 if line[0] == '#': 28 continue

UnicodeDecodeError: 'gbk' codec can't decode byte 0x98 in position 50: illegal multibyte sequence

Is there anything I did wrong? And I wonder is there any other method to solve the encoding issue? Thanks!

Perhaps something like PASS to basically return whatever was input and REPL for removing punctuation.

Another option would be something like CB for check Unicode Block.