A pre-trained language model for social media text in Spanish

Related tags

Deep Learning robertuito
Overview

RoBERTuito

A pre-trained language model for social media text in Spanish

READ THE FULL PAPER Github Repository

RoBERTuito is a pre-trained language model for user-generated content in Spanish, trained following RoBERTa guidelines on 500 million tweets. RoBERTuito comes in 3 flavors: cased, uncased, and uncased+deaccented.

We tested RoBERTuito on a benchmark of tasks involving user-generated text in Spanish. It outperforms other pre-trained language models for this language such as BETO, BERTin and RoBERTa-BNE. The 4 tasks selected for evaluation were: Hate Speech Detection (using SemEval 2019 Task 5, HatEval dataset), Sentiment and Emotion Analysis (using TASS 2020 datasets), and Irony detection (using IrosVa 2019 dataset).

model hate speech sentiment analysis emotion analysis irony detection score
robertuito-uncased 0.801 ± 0.010 0.707 ± 0.004 0.551 ± 0.011 0.736 ± 0.008 0.699
robertuito-deacc 0.798 ± 0.008 0.702 ± 0.004 0.543 ± 0.015 0.740 ± 0.006 0.696
robertuito-cased 0.790 ± 0.012 0.701 ± 0.012 0.519 ± 0.032 0.719 ± 0.023 0.682
roberta-bne 0.766 ± 0.015 0.669 ± 0.006 0.533 ± 0.011 0.723 ± 0.017 0.673
bertin 0.767 ± 0.005 0.665 ± 0.003 0.518 ± 0.012 0.716 ± 0.008 0.667
beto-cased 0.768 ± 0.012 0.665 ± 0.004 0.521 ± 0.012 0.706 ± 0.007 0.665
beto-uncased 0.757 ± 0.012 0.649 ± 0.005 0.521 ± 0.006 0.702 ± 0.008 0.657

We release the pre-trained models on huggingface model hub:

Usage

IMPORTANT -- READ THIS FIRST

RoBERTuito is not yet fully-integrated into huggingface/transformers. To use it, first install pysentimiento

pip install pysentimiento

and preprocess text using pysentimiento.preprocessing.preprocess_tweet before feeding it into the tokenizer

','▁Esto','▁es','▁un','▁tweet','▁estoy','▁usando','▁','▁hashtag','▁','▁ro','bert','uito','▁@usuario','▁','▁emoji','▁cara','▁revolviéndose','▁de','▁la','▁risa','▁emoji',''] "> 
from transformers import AutoTokenizer
from pysentimiento.preprocessing import preprocess_tweet

tokenizer = AutoTokenizer.from_pretrained('pysentimiento/robertuito-base-cased')

text = "Esto es un tweet estoy usando #Robertuito @pysentimiento 🤣"
preprocessed_text = preprocess_tweet(text, ha)

tokenizer.tokenize(preprocessed_text)
# ['','▁Esto','▁es','▁un','▁tweet','▁estoy','▁usando','▁','▁hashtag','▁','▁ro','bert','uito','▁@usuario','▁','▁emoji','▁cara','▁revolviéndose','▁de','▁la','▁risa','▁emoji','']

We are working on integrating this preprocessing step into a Tokenizer within transformers library

Development

Installing

We use python==3.7 and poetry to manage dependencies.

pip install poetry
poetry install

Benchmarking

To run benchmarks

python bin/run_benchmark.py <model_name> --times 5 --output_path <output_path>

Check RUN_BENCHMARKS for all experiments

Smoke test

Test the benchmark running

./smoke_test.sh

Citation

If you use RoBERTuito, please cite our paper:

@misc{perez2021robertuito,
      title={RoBERTuito: a pre-trained language model for social media text in Spanish},
      author={Juan Manuel Pérez and Damián A. Furman and Laura Alonso Alemany and Franco Luque},
      year={2021},
      eprint={2111.09453},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
You might also like...
The DL Streamer Pipeline Zoo is a catalog of optimized media and media analytics pipelines.

The DL Streamer Pipeline Zoo is a catalog of optimized media and media analytics pipelines. It includes tools for downloading pipelines and their dependencies and tools for measuring their performace.

null 8 Dec 4, 2022
A 1.3B text-to-image generation model trained on 14 million image-text pairs
A 1.3B text-to-image generation model trained on 14 million image-text pairs

minDALL-E on Conceptual Captions minDALL-E, named after minGPT, is a 1.3B text-to-image generation model trained on 14 million image-text pairs for no

Kakao Brain 604 Dec 14, 2022
Code, Data and Demo for Paper: Controllable Generation from Pre-trained Language Models via Inverse Prompting

InversePrompting Paper: Controllable Generation from Pre-trained Language Models via Inverse Prompting Code: The code is provided in the "chinese_ip"

THUDM 101 Dec 16, 2022
Source code and dataset for ACL2021 paper: "ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning".

ERICA Source code and dataset for ACL2021 paper: "ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive L

THUNLP 75 Nov 2, 2022
Source code for paper: Knowledge Inheritance for Pre-trained Language Models
Source code for paper: Knowledge Inheritance for Pre-trained Language Models

Knowledge-Inheritance Source code paper: Knowledge Inheritance for Pre-trained Language Models (preprint). The trained model parameters (in Fairseq fo

THUNLP 31 Nov 19, 2022
The code repository for EMNLP 2021 paper
The code repository for EMNLP 2021 paper "Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarization".

Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarization [Paper] accepted at the EMNLP 2021: Vision Guided Genera

CAiRE 42 Jan 7, 2023
CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation

CPT This repository contains code and checkpoints for CPT. CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Gener

fastNLP 341 Dec 29, 2022
Pre-trained BERT Models for Ancient and Medieval Greek, and associated code for LaTeCH 2021 paper titled -
Pre-trained BERT Models for Ancient and Medieval Greek, and associated code for LaTeCH 2021 paper titled - "A Pilot Study for BERT Language Modelling and Morphological Analysis for Ancient and Medieval Greek"

Ancient Greek BERT The first and only available Ancient Greek sub-word BERT model! State-of-the-art post fine-tuning on Part-of-Speech Tagging and Mor

Pranaydeep Singh 22 Dec 8, 2022
CLIP (Contrastive Language–Image Pre-training) trained on Indonesian data

CLIP-Indonesian CLIP (Radford et al., 2021) is a multimodal model that can connect images and text by training a vision encoder and a text encoder joi

Galuh 17 Mar 10, 2022
Comments
  • Will the corpus be openly published?

    Will the corpus be openly published?

    Given how good this model works, I'd be interested in having access to the original corpus with which it was trained. Will that be possible? I mean, is there any plan for uploading it to huggingface/datasets or publish it in any other form?

    Thank you very much in advance :)

    opened by avacaondata 4
  • CUDA error, possibly related to max_length

    CUDA error, possibly related to max_length

    I think the library should be installed isolated from transformers, because if one has another version of transformers with custom models or whatever, this breaks the environment, unnecessarily.

    But the important point here is that it's not possible to train robertuito:

    :\cb\pytorch_1000000000000\work\aten\src\ATen\native\cuda\Indexing.cu:699: block: [142,0,0], thread: [64,0,0] Assertion `srcIndex < srcSelectDimSize` failed. 
C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\cuda\Indexing.cu:699: block: [142,0,0], thread: [65,0,0] Assertion `srcIndex < srcSelectDimSize` failed.                | 73/666 [00:38<00:45, 13.07it/s] 
C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\cuda\Indexing.cu:699: block: [142,0,0], thread: [66,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\cuda\Indexing.cu:699: block: [142,0,0], thread: [67,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\cuda\Indexing.cu:699: block: [142,0,0], thread: [68,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\cuda\Indexing.cu:699: block: [142,0,0], thread: [69,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\cuda\Indexing.cu:699: block: [142,0,0], thread: [70,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\cuda\Indexing.cu:699: block: [142,0,0], thread: [71,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\cuda\Indexing.cu:699: block: [142,0,0], thread: [72,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\cuda\Indexing.cu:699: block: [142,0,0], thread: [73,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\cuda\Indexing.cu:699: block: [142,0,0], thread: [74,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\cuda\Indexing.cu:699: block: [142,0,0], thread: [75,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\cuda\Indexing.cu:699: block: [142,0,0], thread: [76,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\cuda\Indexing.cu:699: block: [142,0,0], thread: [77,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\cuda\Indexing.cu:699: block: [142,0,0], thread: [78,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\cuda\Indexing.cu:699: block: [142,0,0], thread: [79,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\cuda\Indexing.cu:699: block: [142,0,0], thread: [80,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\cuda\Indexing.cu:699: block: [142,0,0], thread: [81,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\cuda\Indexing.cu:699: block: [142,0,0], thread: [82,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\cuda\Indexing.cu:699: block: [142,0,0], thread: [83,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\cuda\Indexing.cu:699: block: [142,0,0], thread: [84,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\cuda\Indexing.cu:699: block: [142,0,0], thread: [85,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\cuda\Indexing.cu:699: block: [142,0,0], thread: [86,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\cuda\Indexing.cu:699: block: [142,0,0], thread: [87,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\cuda\Indexing.cu:699: block: [142,0,0], thread: [88,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\cuda\Indexing.cu:699: block: [142,0,0], thread: [89,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\cuda\Indexing.cu:699: block: [142,0,0], thread: [90,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\cuda\Indexing.cu:699: block: [142,0,0], thread: [91,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\cuda\Indexing.cu:699: block: [142,0,0], thread: [92,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\cuda\Indexing.cu:699: block: [142,0,0], thread: [93,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\cuda\Indexing.cu:699: block: [142,0,0], thread: [94,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\cuda\Indexing.cu:699: block: [142,0,0], thread: [95,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[W 2022-02-20 18:25:42,448] Trial 0 failed because of the following error: RuntimeError('CUDA error: device-side assert triggered\nCUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.')
Traceback (most recent call last):
  File "C:\Users\Usuario\anaconda3\envs\rigobenchmarks\lib\site-packages\optuna\study\_optimize.py", line 213, in _run_trial
    value_or_values = func(trial)
  File "C:\Users\Usuario\anaconda3\envs\rigobenchmarks\lib\site-packages\transformers\integrations.py", line 150, in _objective
    trainer.train(resume_from_checkpoint=checkpoint, trial=trial)
  File "C:\Users\Usuario\anaconda3\envs\rigobenchmarks\lib\site-packages\transformers\trainer.py", line 1365, in train
    tr_loss_step = self.training_step(model, inputs)
  File "C:\Users\Usuario\anaconda3\envs\rigobenchmarks\lib\site-packages\transformers\trainer.py", line 1950, in training_step
    self.scaler.scale(loss).backward()
  File "C:\Users\Usuario\anaconda3\envs\rigobenchmarks\lib\site-packages\torch\_tensor.py", line 307, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "C:\Users\Usuario\anaconda3\envs\rigobenchmarks\lib\site-packages\torch\autograd\__init__.py", line 156, in backward
    allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Trying models on dataset exist22_1_es:   0%|                                                                                                                                             | 0/1 [00:49<?, ?it/s] 
Iterating over datasets...: 0it [00:49, ?it/s]
Traceback (most recent call last):
  File "run_experiments.py", line 3279, in <module>
    benchmarker()
  File "run_experiments.py", line 1196, in __call__
    self.optuna_hp_search()
  File "run_experiments.py", line 1470, in optuna_hp_search
    sampler=TPESampler(seed=420)
  File "C:\Users\Usuario\anaconda3\envs\rigobenchmarks\lib\site-packages\transformers\trainer.py", line 1847, in hyperparameter_search
    best_run = backend_dict[backend](self, n_trials, direction, **kwargs)
  File "C:\Users\Usuario\anaconda3\envs\rigobenchmarks\lib\site-packages\transformers\integrations.py", line 160, in run_hp_search_optuna
    study.optimize(_objective, n_trials=n_trials, timeout=timeout, n_jobs=n_jobs)
  File "C:\Users\Usuario\anaconda3\envs\rigobenchmarks\lib\site-packages\optuna\study\study.py", line 409, in optimize
    show_progress_bar=show_progress_bar,
  File "C:\Users\Usuario\anaconda3\envs\rigobenchmarks\lib\site-packages\optuna\study\_optimize.py", line 76, in _optimize
    progress_bar=progress_bar,
  File "C:\Users\Usuario\anaconda3\envs\rigobenchmarks\lib\site-packages\optuna\study\_optimize.py", line 163, in _optimize_sequential
    trial = _run_trial(study, func, catch)
  File "C:\Users\Usuario\anaconda3\envs\rigobenchmarks\lib\site-packages\optuna\study\_optimize.py", line 264, in _run_trial
    raise func_err
  File "C:\Users\Usuario\anaconda3\envs\rigobenchmarks\lib\site-packages\optuna\study\_optimize.py", line 213, in _run_trial
    value_or_values = func(trial)
  File "C:\Users\Usuario\anaconda3\envs\rigobenchmarks\lib\site-packages\transformers\integrations.py", line 150, in _objective
    trainer.train(resume_from_checkpoint=checkpoint, trial=trial)
  File "C:\Users\Usuario\anaconda3\envs\rigobenchmarks\lib\site-packages\transformers\trainer.py", line 1365, in train
    tr_loss_step = self.training_step(model, inputs)
  File "C:\Users\Usuario\anaconda3\envs\rigobenchmarks\lib\site-packages\transformers\trainer.py", line 1950, in training_step
    self.scaler.scale(loss).backward()
  File "C:\Users\Usuario\anaconda3\envs\rigobenchmarks\lib\site-packages\torch\_tensor.py", line 307, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "C:\Users\Usuario\anaconda3\envs\rigobenchmarks\lib\site-packages\torch\autograd\__init__.py", line 156, in backward
    allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

    I have tried many other models in spanish and this doesn't happen, therefore it's directly related to your model, and not the model architecture (coming from transformers).

    opened by avacaondata 4
  • Model fails with index out of range in self when used on a text-classification pipeline

    Model fails with index out of range in self when used on a text-classification pipeline

    Steps to reproduce

    nlp = pipeline("text-classification", "pysentimiento/robertuito-sentiment-analysis")
nlp("long text"*300)

    IndexError                                Traceback (most recent call last)
Input In [23], in <cell line: 1>()
----> 1 nlp("long text"*300)

File /opt/conda/lib/python3.8/site-packages/transformers/pipelines/text_classification.py:125, in TextClassificationPipeline.__call__(self, *args, **kwargs)
     [92](file:///opt/conda/lib/python3.8/site-packages/transformers/pipelines/text_classification.py?line=91) def __call__(self, *args, **kwargs):
     [93](file:///opt/conda/lib/python3.8/site-packages/transformers/pipelines/text_classification.py?line=92)     """
     [94](file:///opt/conda/lib/python3.8/site-packages/transformers/pipelines/text_classification.py?line=93)     Classify the text(s) given as inputs.
     [95](file:///opt/conda/lib/python3.8/site-packages/transformers/pipelines/text_classification.py?line=94) 
   (...)
    [123](file:///opt/conda/lib/python3.8/site-packages/transformers/pipelines/text_classification.py?line=122)         If `self.return_all_scores=True`, one such dictionary is returned per label.
    [124](file:///opt/conda/lib/python3.8/site-packages/transformers/pipelines/text_classification.py?line=123)     """
--> [125](file:///opt/conda/lib/python3.8/site-packages/transformers/pipelines/text_classification.py?line=124)     result = super().__call__(*args, **kwargs)
    [126](file:///opt/conda/lib/python3.8/site-packages/transformers/pipelines/text_classification.py?line=125)     if isinstance(args[0], str):
    [127](file:///opt/conda/lib/python3.8/site-packages/transformers/pipelines/text_classification.py?line=126)         # This pipeline is odd, and return a list when single item is run
    [128](file:///opt/conda/lib/python3.8/site-packages/transformers/pipelines/text_classification.py?line=127)         return [result]

File /opt/conda/lib/python3.8/site-packages/transformers/pipelines/base.py:1027, in Pipeline.__call__(self, inputs, num_workers, batch_size, *args, **kwargs)
   [1025](file:///opt/conda/lib/python3.8/site-packages/transformers/pipelines/base.py?line=1024)     return self.iterate(inputs, preprocess_params, forward_params, postprocess_params)
   [1026](file:///opt/conda/lib/python3.8/site-packages/transformers/pipelines/base.py?line=1025) else:
-> [1027](file:///opt/conda/lib/python3.8/site-packages/transformers/pipelines/base.py?line=1026)     return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)

File /opt/conda/lib/python3.8/site-packages/transformers/pipelines/base.py:1034, in Pipeline.run_single(self, inputs, preprocess_params, forward_params, postprocess_params)
   [1032](file:///opt/conda/lib/python3.8/site-packages/transformers/pipelines/base.py?line=1031) def run_single(self, inputs, preprocess_params, forward_params, postprocess_params):
   [1033](file:///opt/conda/lib/python3.8/site-packages/transformers/pipelines/base.py?line=1032)     model_inputs = self.preprocess(inputs, **preprocess_params)
-> [1034](file:///opt/conda/lib/python3.8/site-packages/transformers/pipelines/base.py?line=1033)     model_outputs = self.forward(model_inputs, **forward_params)
   [1035](file:///opt/conda/lib/python3.8/site-packages/transformers/pipelines/base.py?line=1034)     outputs = self.postprocess(model_outputs, **postprocess_params)
   [1036](file:///opt/conda/lib/python3.8/site-packages/transformers/pipelines/base.py?line=1035)     return outputs

File /opt/conda/lib/python3.8/site-packages/transformers/pipelines/base.py:944, in Pipeline.forward(self, model_inputs, **forward_params)
    [942](file:///opt/conda/lib/python3.8/site-packages/transformers/pipelines/base.py?line=941)     with inference_context():
    [943](file:///opt/conda/lib/python3.8/site-packages/transformers/pipelines/base.py?line=942)         model_inputs = self._ensure_tensor_on_device(model_inputs, device=self.device)
--> [944](file:///opt/conda/lib/python3.8/site-packages/transformers/pipelines/base.py?line=943)         model_outputs = self._forward(model_inputs, **forward_params)
    [945](file:///opt/conda/lib/python3.8/site-packages/transformers/pipelines/base.py?line=944)         model_outputs = self._ensure_tensor_on_device(model_outputs, device=torch.device("cpu"))
    [946](file:///opt/conda/lib/python3.8/site-packages/transformers/pipelines/base.py?line=945) else:

File /opt/conda/lib/python3.8/site-packages/transformers/pipelines/text_classification.py:137, in TextClassificationPipeline._forward(self, model_inputs)
    [136](file:///opt/conda/lib/python3.8/site-packages/transformers/pipelines/text_classification.py?line=135) def _forward(self, model_inputs):
--> [137](file:///opt/conda/lib/python3.8/site-packages/transformers/pipelines/text_classification.py?line=136)     return self.model(**model_inputs)

File /opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:1102, in Module._call_impl(self, *input, **kwargs)
   [1098](file:///opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py?line=1097) # If we don't have any hooks, we want to skip the rest of the logic in
   [1099](file:///opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py?line=1098) # this function, and just call forward.
   [1100](file:///opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py?line=1099) if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   [1101](file:///opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py?line=1100)         or _global_forward_hooks or _global_forward_pre_hooks):
-> [1102](file:///opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py?line=1101)     return forward_call(*input, **kwargs)
   [1103](file:///opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py?line=1102) # Do not call functions when jit is used
   [1104](file:///opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py?line=1103) full_backward_hooks, non_full_backward_hooks = [], []

File /opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py:1204, in RobertaForSequenceClassification.forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, labels, output_attentions, output_hidden_states, return_dict)
   [1196](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=1195) r"""
   [1197](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=1196) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
   [1198](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=1197)     Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,
   [1199](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=1198)     config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
   [1200](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=1199)     `config.num_labels > 1` a classification loss is computed (Cross-Entropy).
   [1201](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=1200) """
   [1202](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=1201) return_dict = return_dict if return_dict is not None else self.config.use_return_dict
-> [1204](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=1203) outputs = self.roberta(
   [1205](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=1204)     input_ids,
   [1206](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=1205)     attention_mask=attention_mask,
   [1207](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=1206)     token_type_ids=token_type_ids,
   [1208](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=1207)     position_ids=position_ids,
   [1209](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=1208)     head_mask=head_mask,
   [1210](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=1209)     inputs_embeds=inputs_embeds,
   [1211](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=1210)     output_attentions=output_attentions,
   [1212](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=1211)     output_hidden_states=output_hidden_states,
   [1213](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=1212)     return_dict=return_dict,
   [1214](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=1213) )
   [1215](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=1214) sequence_output = outputs[0]
   [1216](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=1215) logits = self.classifier(sequence_output)

File /opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:1102, in Module._call_impl(self, *input, **kwargs)
   [1098](file:///opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py?line=1097) # If we don't have any hooks, we want to skip the rest of the logic in
   [1099](file:///opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py?line=1098) # this function, and just call forward.
   [1100](file:///opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py?line=1099) if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   [1101](file:///opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py?line=1100)         or _global_forward_hooks or _global_forward_pre_hooks):
-> [1102](file:///opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py?line=1101)     return forward_call(*input, **kwargs)
   [1103](file:///opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py?line=1102) # Do not call functions when jit is used
   [1104](file:///opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py?line=1103) full_backward_hooks, non_full_backward_hooks = [], []

File /opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py:843, in RobertaModel.forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask, past_key_values, use_cache, output_attentions, output_hidden_states, return_dict)
    [836](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=835) # Prepare head mask if needed
    [837](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=836) # 1.0 in head_mask indicate we keep the head
    [838](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=837) # attention_probs has shape bsz x n_heads x N x N
    [839](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=838) # input head_mask has shape [num_heads] or [num_hidden_layers x num_heads]
    [840](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=839) # and head_mask is converted to shape [num_hidden_layers x batch x num_heads x seq_length x seq_length]
    [841](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=840) head_mask = self.get_head_mask(head_mask, self.config.num_hidden_layers)
--> [843](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=842) embedding_output = self.embeddings(
    [844](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=843)     input_ids=input_ids,
    [845](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=844)     position_ids=position_ids,
    [846](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=845)     token_type_ids=token_type_ids,
    [847](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=846)     inputs_embeds=inputs_embeds,
    [848](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=847)     past_key_values_length=past_key_values_length,
    [849](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=848) )
    [850](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=849) encoder_outputs = self.encoder(
    [851](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=850)     embedding_output,
    [852](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=851)     attention_mask=extended_attention_mask,
   (...)
    [860](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=859)     return_dict=return_dict,
    [861](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=860) )
    [862](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=861) sequence_output = encoder_outputs[0]

File /opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:1102, in Module._call_impl(self, *input, **kwargs)
   [1098](file:///opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py?line=1097) # If we don't have any hooks, we want to skip the rest of the logic in
   [1099](file:///opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py?line=1098) # this function, and just call forward.
   [1100](file:///opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py?line=1099) if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   [1101](file:///opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py?line=1100)         or _global_forward_hooks or _global_forward_pre_hooks):
-> [1102](file:///opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py?line=1101)     return forward_call(*input, **kwargs)
   [1103](file:///opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py?line=1102) # Do not call functions when jit is used
   [1104](file:///opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py?line=1103) full_backward_hooks, non_full_backward_hooks = [], []

File /opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py:136, in RobertaEmbeddings.forward(self, input_ids, token_type_ids, position_ids, inputs_embeds, past_key_values_length)
    [134](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=133) embeddings = inputs_embeds + token_type_embeddings
    [135](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=134) if self.position_embedding_type == "absolute":
--> [136](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=135)     position_embeddings = self.position_embeddings(position_ids)
    [137](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=136)     embeddings += position_embeddings
    [138](file:///opt/conda/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py?line=137) embeddings = self.LayerNorm(embeddings)

File /opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:1102, in Module._call_impl(self, *input, **kwargs)
   [1098](file:///opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py?line=1097) # If we don't have any hooks, we want to skip the rest of the logic in
   [1099](file:///opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py?line=1098) # this function, and just call forward.
   [1100](file:///opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py?line=1099) if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   [1101](file:///opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py?line=1100)         or _global_forward_hooks or _global_forward_pre_hooks):
-> [1102](file:///opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py?line=1101)     return forward_call(*input, **kwargs)
   [1103](file:///opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py?line=1102) # Do not call functions when jit is used
   [1104](file:///opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py?line=1103) full_backward_hooks, non_full_backward_hooks = [], []

File /opt/conda/lib/python3.8/site-packages/torch/nn/modules/sparse.py:158, in Embedding.forward(self, input)
    [157](file:///opt/conda/lib/python3.8/site-packages/torch/nn/modules/sparse.py?line=156) def forward(self, input: Tensor) -> Tensor:
--> [158](file:///opt/conda/lib/python3.8/site-packages/torch/nn/modules/sparse.py?line=157)     return F.embedding(
    [159](file:///opt/conda/lib/python3.8/site-packages/torch/nn/modules/sparse.py?line=158)         input, self.weight, self.padding_idx, self.max_norm,
    [160](file:///opt/conda/lib/python3.8/site-packages/torch/nn/modules/sparse.py?line=159)         self.norm_type, self.scale_grad_by_freq, self.sparse)

File /opt/conda/lib/python3.8/site-packages/torch/nn/functional.py:2044, in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   [2038](file:///opt/conda/lib/python3.8/site-packages/torch/nn/functional.py?line=2037)     # Note [embedding_renorm set_grad_enabled]
   [2039](file:///opt/conda/lib/python3.8/site-packages/torch/nn/functional.py?line=2038)     # XXX: equivalent to
   [2040](file:///opt/conda/lib/python3.8/site-packages/torch/nn/functional.py?line=2039)     # with torch.no_grad():
   [2041](file:///opt/conda/lib/python3.8/site-packages/torch/nn/functional.py?line=2040)     #   torch.embedding_renorm_
   [2042](file:///opt/conda/lib/python3.8/site-packages/torch/nn/functional.py?line=2041)     # remove once script supports set_grad_enabled
   [2043](file:///opt/conda/lib/python3.8/site-packages/torch/nn/functional.py?line=2042)     _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> [2044](file:///opt/conda/lib/python3.8/site-packages/torch/nn/functional.py?line=2043) return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)

IndexError: index out of range in self
    opened by riolcrt 2
  • docs: demo, experiments and live inference API on Tiyaro

    docs: demo, experiments and live inference API on Tiyaro

    Hello Maintainer of Github repo pysentimiento/robertuito!

    Thank you for your work on pysentimiento/robertuito. This GitHub project is interesting, and we think that it would be a great addition to make this work instantly discoverable & available as an API for all your users, to quickly try and use it in their applications.

    The list of model card(s) covered by this PR are:

    • https://console.tiyaro.ai/explore/huggingface-pysentimiento-robertuito-emotion-analysis-1
    • https://console.tiyaro.ai/explore/huggingface-pysentimiento-robertuito-sentiment-analysis-1

    On Tiyaro, every model in pysentimiento/robertuito will get its own:

    • Dedicated model card (e.g. https://console.tiyaro.ai/explore/huggingface-pysentimiento-robertuito-emotion-analysis-1
    • Model demo (e.g. https://console.tiyaro.ai/explore/huggingface-pysentimiento-robertuito-emotion-analysis-1/demo)
    • Unique Inference API (e.g. https://api.tiyaro.ai/explore/huggingface/1//pysentimiento/robertuito-emotion-analysis)
    • Sample code snippets and swagger spec for the API

    Users will also be able to compare your model with other models of similar types on various parameters using Tiyaro Experiments (https://tiyaro.ai/blog/ocr/)

    —- I am from Tiyaro.ai (https://tiyaro.ai/). We are working on enabling developers to instantly evaluate, use and customize the world’s best AI. We are constantly working on adding new features to Tiyaro EasyTrain, EasyServe & Experiments, to make the best use of your ML model, and making AI more accessible for anyone.

    Sincerely, I-Jong Lin

    opened by ijonglin 0
Owner
null
GitHub
A collection of pre-trained StyleGAN2 models trained on different datasets at different resolution.

Awesome Pretrained StyleGAN2 A collection of pre-trained StyleGAN2 models trained on different datasets at different resolution. Note the readme is a

Justin 1.1k Dec 24, 2022
🐥A PyTorch implementation of OpenAI's finetuned transformer language model with a script to import the weights pre-trained by OpenAI

PyTorch implementation of OpenAI's Finetuned Transformer Language Model This is a PyTorch implementation of the TensorFlow code provided with OpenAI's

Hugging Face 1.4k Jan 5, 2023
DziriBERT: a Pre-trained Language Model for the Algerian Dialect

DziriBERT DziriBERT is the first Transformer-based Language Model that has been pre-trained specifically for the Algerian Dialect. It handles Algerian

null 117 Jan 7, 2023
A Comprehensive Empirical Study of Vision-Language Pre-trained Model for Supervised Cross-Modal Retrieval

CLIP4CMR A Comprehensive Empirical Study of Vision-Language Pre-trained Model for Supervised Cross-Modal Retrieval The original data and pre-calculate

null 9 Jan 12, 2022
SUPERVISED-CONTRASTIVE-LEARNING-FOR-PRE-TRAINED-LANGUAGE-MODEL-FINE-TUNING - The Facebook paper about fine tuning RoBERTa with contrastive loss

"# SUPERVISED-CONTRASTIVE-LEARNING-FOR-PRE-TRAINED-LANGUAGE-MODEL-FINE-TUNING" i

Saeed Lotfi 28 Dec 12, 2022
Annotate datasets with a semi-trained or fully trained YOLOv5 model

YOLOv5 Auto Annotator Annotate datasets with a semi-trained or fully trained YOLOv5 model Prerequisites Ubuntu >=20.04 Python >=3.7 System dependencie

Akash James 3 May 14, 2022
Tensorflow python implementation of "Learning High Fidelity Depths of Dressed Humans by Watching Social Media Dance Videos"

Learning High Fidelity Depths of Dressed Humans by Watching Social Media Dance Videos This repository is the official tensorflow python implementation

Yasamin Jafarian 287 Jan 6, 2023
This was initially the repo for the project of PSYC626@USC of Asaf Mazar, Millad Kassaie and Georgios Chochlakis named "Powered by the Will? Exploring Lay Theories of Behavior Change through Social Media"

Subreddit Analysis This repo includes tools for Subreddit analysis, originally developed for our class project of PSYC 626 in USC, titled "Powered by

Georgios Chochlakis 1 Dec 17, 2021
Machine learning and Deep learning models, deploy on telegram (the best social media)

Semi Intelligent BOT The project involves : Classifying fake news Classifying objects such as aeroplane, automobile, bird, cat, deer, dog, frog, horse

MohammadReza Norouzi 5 Mar 6, 2022
[LREC] MMChat: Multi-Modal Chat Dataset on Social Media

MMChat This repo contains the code and data for the LREC2022 paper MMChat: Multi-Modal Chat Dataset on Social Media. Dataset MMChat is a large-scale d

Silver 47 Jan 3, 2023