Hashtag segmentation is the task of automatically inserting the missing spaces between the words in a hashtag.
Hashformers applies Transformer models to hashtag segmentation. It is built on top of the transformers library and the lm-scorer and mlm-scoring packages.
Try it right now on Google Colab.
Paper: Zero-shot hashtag segmentation for multilingual sentiment analysis
Basic usage
from hashformers import WordSegmenter
ws = WordSegmenter(
segmenter_model_name_or_path="gpt2",
reranker_model_name_or_path="bert-base-uncased",
use_reranker=True
)
segmentations = ws.segment([
"#myoldphonesucks",
"#latinosinthedeepsouth",
"#weneedanationalpark",
"#LandoftheLost",
"#icecold",
"#Heartbreaker",
"#TheRiseGuys"
])
print(segmentations)
# ['my old phone sucks',
# 'latinos in the deep south',
# 'we need a national park',
# 'Land of the Lost',
# 'ice cold',
# 'Heartbreaker',
# 'The Rise Guys']
Installation
Installation steps are described on this notebook. A Docker image is coming soon.
Examples
Applications of hashtag segmentation to tweet sentiment analysis and the automatic translation of tweets can be found on the examples folder.
Contributing
Pull requests are welcome! We need to improve on the documentation and code quality of this repository. It's also a good idea to implement more sophisticated ensembling techniques. Read our paper for more details on the inner workings of our framework.
Citation
@misc{rodrigues2021zeroshot,
title={Zero-shot hashtag segmentation for multilingual sentiment analysis},
author={Ruan Chaves Rodrigues and Marcelo Akira Inuzuka and Juliana Resplande Sant'Anna Gomes and Acquila Santos Rocha and Iacer Calixto and Hugo Alexandre Dantas do Nascimento},
year={2021},
eprint={2112.03213},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
13 Oct 9, 2022
161 Dec 8, 2022
1.4k Jan 4, 2023
112 Dec 16, 2022
687 Jan 7, 2023
897 Jan 5, 2023
182 Dec 19, 2022
172 Dec 20, 2022
1.4k Dec 31, 2022
102 Jan 5, 2023
186 Dec 30, 2022
105 Dec 6, 2022
32 Sep 21, 2022
134 Dec 19, 2022
147 Dec 3, 2022
610 Dec 28, 2022
14 Aug 30, 2022
97 Dec 17, 2022
179 Jan 2, 2023