A fantastic model! Thank you for sharing it on huggingface. Is it possible to upload also a model based on xml-roberta-base trained on all the languages in the Europarl data set? I have noticed it generalizes well to other languages for some of the more difficult cases, such commas at the end of subordinate clauses, but it misses some cases where the comma is always or almost always present in some languages. For example in Bulgarian: "Отивам на село, защото искам да си почина."; "Той каза, че ще дойде по-късно"; "Мислех си кой може да свърши работата с моя помощ и се сетих, че имам един приятел, който е съдружник във фирма, която прави подобни неща." Is it going to worsen signficantly the performance for English?

Is it possible to utilise a gpu when processing text with this script ? I have it working fine but it uses my CPU and I have very large chunks of text I want to process and it take to long with just the cpu.

Code model = PunctuationModel() labled_words = model.predict(data)

Expected output: (something like this) [['My', '0', 0.9999887], ['name', '0', 0.99998665], ['is', '0', 0.9998579], ['Clara', '0', 0.6752215], ['and', '0', 0.99990904], ['I', '0', 0.9999877], ['live', '0', 0.9999839], ['in', '0', 0.9999515], ['Berkeley', ',', 0.99800044], ['California', '.', 0.99534047], ['Ist', '0', 0.99998784], ['das', '0', 0.99999154], ['eine', '0', 0.9999918], ['Frage', ',', 0.99622655], ['Frau', '0', 0.9999889], ['Müller', '?', 0.99863917]]

Actual Outpu: (like this) [['S', '0', 0.85909706], ['p', '0', 0.79623544], ['i', '0', 0.97123384], ['d', '0', 0.9573735], ['e', '0', 0.7520275], ['r', '0', 0.83077455], ['-', '0', 0.912434], ['M', '0', 0.79773486], ['a', '0', 0.8560487], ['n', '0', 0.80509317], [' ', 0, 0.80509317], ['i', '0', 0.9247775], ['s', '0', 0.93199074], [' ', 0, 0.93199074], ['a', '0', 0.93133676]]

Hello : )

First of all, thank you for your work! We are currently using it for a group project at university and it’s super helpful.

There is just one small issue: When we are using the restore_punctuation function on a large enumeration that is missing commas, we get a “Chunk size too large, text got clipped” error as shown below:

We can't seem to find a solution. Do you have any idea how to fix this? Thanks in advance!

Phyllis

In case you want to replicate the error, this is the text snipped we are using: “This is a list: Diindolylmethane (DIM) Dimethylaminoethanol (DMAE) Dong Quai Echinacea Eclipta Alba Egg Shell Calcium Elderberry Eleutherosides Emblica Essential Fatty Acids Energy EDTA Eurycoma Longifolia Evodia Extract Eye Health Ferulic Acid Fish Oils GABA Gamma Tocopherol Garlic Ginger Extract Ginkgo Biloba Ginseng Glucosamine Glutathione Goji Berry Gotu Kola GPC Choline Grape Seed Extract Green Coffee Bean Green Tea Gymnema Sylvestre Health Commentaries Health Questionnaires Heart Attacks Heart Health Herbs Hormone Support Huperzine Immune System Support Inflammation Joint Pain Kohki Tea L-Arginine L-Citrulline L-Cysteine L-Theanine L-Tyrosine Licorice Extract Life Extension Lipoic Acid Liver Support Longevity and Anti-Aging Lutein Luteolin Lycium Berry Lycopene Lysine Magnesium Magnolia Malic Acid Mastic Gum Medium-Chain Triglycerides Melatonin Memory Menopause Menaquinone-7 Men's Health Mental Health Metabolic Syndrome Minerals Mixed Tocopherols Multi-Vitamins Myricetin N-Acetyl Cysteine (NAC) N-Acetyl-carnosine Naringenin Nattokinase Nettle Notoginseng Nutrition Olive Leaf Oolong Tea Omega-3 Fatty Acids OKG Ornithine Oral Chelation Oral Health - Teeth and Gums Osteoporosis Overall Health Pain Relief Parkinson's Disease Passionflower Peony Persimmon Phenylalanine Pheromones Phosphatidylserine Phytosterol Pine Bark Policosanol Pomegranate Pregnenolone Probiotics Prostate Health Pueraria Mirifica Pumpkin Pygeum Africanum R-Lipoic Acid Red Wine Resveratrol Rhodiola Rosea Rosemary Salacia Reticulata Salvia Miltiorrhiza Schisandra Berry Selenium Sexual Health Silymarin Skin Hair and Nails Sleep Soy Phytosterol St John's Wort Stinging Nettle Stress Strontium Citrate Suntheanine Taurine Terminalia Chebula The Common Cold Turmeric Root Tyrosine Uridine Urinary Tract Health Urtica Dioica Root Varicose Veins Vinpocetine Vision Health"

Great lib, found this small bug:

text = "das , ist fies "

leads to error

TypeError: unsupported operand type(s) for +: 'int' and 'str'

when converting the results to text.

I followed your tutorial and works great! however i get the following Warning coming from hugging face pipeline:

/usr/local/lib/python3.8/dist-packages/transformers/pipelines/token_classification.py:159: UserWarning: `grouped_entities` is deprecated and will be removed in version v5.0.0, defaulted to `aggregation_strategy="none"` instead.
  warnings.warn(

Here is the required doc: Hugging Face pipeline

And the line of code to change is: From the PunctuationModel (inside the __init__ method...) both pipeline() calls

Would you consider updating those lines in order to get rid of the warning? Thank you in advance!

Hi and thanks a lot for the great tool!

Seems that in the original punctuation removal step, punctuation in numbers is intentionally kept. Perhaps due to decimal point issues or ordinal number representation in some languages.

This, however, results in extra punctuation being predicted when a number is at the end of a sentence: 'The Answer to the Ultimate Question of Life, the Universe, and Everything is 42.' becomes 'The Answer to the Ultimate Question of Life, the Universe, and Everything is 42..'

Not sure what would be an elegant solution to this. The punctuation-stripping regex can't tell apart ordinal marks from sentence-final full-stops. Would be nice to trust the LM to predict all the punctuation, i.e., remove all of it in the pre-processing step.