HuggingTweets - Train a model to generate tweets

Boris Dayma

Last update: Jan 4, 2023

Related tags

Text Data & NLP huggingtweets

Overview

HuggingTweets - Train a model to generate tweets

Create in 5 minutes a tweet generator based on your favorite Tweeter

Make my own model with the demo →

or access existing models →

Introduction

I developed HuggingTweets to try to predict Elon Musk's next breakthrough 😉

This project fine-tunes a pre-trained neural network on a user's tweets using HuggingFace Transformers, an awesome open source library for Natural Language Processing. The resulting model can then generate new tweets for you!

Training and results are automatically logged into W&B through the HuggingFace integration.

Usage

To test the demo, click on below link and share your predictions!

You can also use it locally by installing the dependencies with pipenv or pip and use huggingtweets-demo.ipynb

Results

My favorite sample is definitely on Andrej Karpathy, start of sentence "I don't like":

I don't like this :) 9:20am: Forget this little low code and preprocessor optimization. Even if it's neat, for top-level projects. 9:27am: Other useful code examples? It's not kind of best code, :) 9:37am: Python drawing bug like crazy, restarts regular web browsing ;) 9:46am: Okay, I don't mind. Maybe I should try that out! I'll investigate it :) 10:00am: I think I should try Shigemitsu's imgur page. Or the minimalist website if you're after 10/10 results :) Also maybe Google ImageNet on "Yelp" instead :) 10:05am: Looking forward to watching it talk!

I had a lot of fun running predictions on other people too!

How does it work?

To understand how the model was developed, check my W&B report.

You can also explore the development version huggingtweets-dev.ipynb or use the following link.

Required files to run W&B sweeps are in dev folder.

Future research

I still have more research to do:

evaluate how to "merge" two different personalities ;
test training top layers vs bottom layers to see how it affects learning of lexical field (subject of content) vs word predictions, memorization vs creativity ;
augment text data with adversarial approaches ;
pre-train on large Twitter dataset of many people ;
explore few-shot learning approaches as we have limited data per user though there are probably only few writing styles ;
implement a pipeline to continuously train the network on new tweets ;
cluster users and identify topics, writing style…

About

Built by Boris Dayma

My main goals with this project are:

to experiment with how to train, deploy and maintain neural networks in production ;
to make AI accessible to everyone ;
to have fun!

For more details, visit the project repository.

Disclaimer: this project is not to be used to publish any false generated information but to perform research on Natural Language Generation.

FAQ

Does this project pose a risk of being used for disinformation?

Large NLP models can be misused to publish false data. OpenAI performed a staged release of GPT-2 to study any potential misuse of their models.

I want to ensure latest AI technologies are accessible to everyone to ensure fairness and prevent social inequality.

HuggingTweets shall not be used for creating innapropriate content, nor for any illicit or unethical purposes. Any generated text from other users tweets must explicitly be referenced as such and cannot be published with the intent of hiding their origin. No generated content can be published against a person unwilling to have their data used as such.
Why is the demo in colab instead of being a real independent web app?

It actually looks much better with Voilà as the code cells are hidden and automatically executed. Also we can easily deploy it through for free on Binder.

However training such large neural networks requires GPU (not available on Binder, and not cheap) and I wanted to make HuggingTweets accessible to everybody. Google Colab generously offers free GPU so is the perfect place to host the demo.

Resources

Explore the W&B report to understand how the model works
HuggingFace and W&B integration documentation

Got questions about W&B?

If you have any questions about using W&B to track your model performance and predictions, please reach out to the slack community.

Acknowledgements

I was able to make the first version of this program in just a few days.

It would not have been possible without these people and these open-source tools:

W&B for the great tracking & visualization tools for ML experiments ;
HuggingFace for providing a great framework for Natural Language Understanding ;
Tweepy for providing a great API to interact with Twitter (used in the dev notebook) ;
Chris Van Pelt for hacking with me on the demo ;
Lavanya Shukla and Carey Phelps for their continuous feedback ;
Google Colab for letting people access free GPU!

Comments

Deleting model from HuggingFace

This library is awesome! One thing though is that creates a public model in huggingface. I don't plan to do it, but I was wondering if it is possible to delete them easily.

opened by davidgasquez 2

an error occurred

in step 3 when I try to run the trained model

Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.

An error occured...

Expecting value: line 1 column 1 (char 0)
Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.

An error occured...

Expecting value: line 1 column 1 (char 0)

opened by ngmisl 1

Train on gpt2 arabic or AraBERT

i'm trying to train huggingtweets with arabic models , i've tried everything and every arabic model out there but nothing seems to work

it always gives this error:

An error occured... num_samples should be a positive integer value, but got num_samples=0

even if tried to change the block size , same thing happens , is there anyway to make this work?

opened by antiihope 1

Spanish version

Hi Boris! Really enjoyed your google colab "app", thanks for sharing it!

Recently GPT-2 in Spanish was released, so I tried to play around but I'm getting errors:

Not a big deal, from your code

...

global trainer

from transformers import AutoTokenizer, AutoModelWithLMHead

tokenizer = AutoTokenizer.from_pretrained("mrm8488/spanish-gpt2")

model = AutoModelWithLMHead.from_pretrained("mrm8488/spanish-gpt2")

#tokenizer = AutoTokenizer.from_pretrained('gpt2')
#model = AutoModelForCausalLM.from_pretrained('gpt2', cache_dir=pathlib.Path('cache').resolve())

...

But I get:

An error occured...

num_samples should be a positive integer value, but got num_samples=0

If you can point me in the right direction, I can try more modifications :) Thanks in advance!

opened by pablo14 4

Creating a Dashboard

Hello, I was very impressed with the work that has been done on this repository. I think we can create a fast API and deploy it on server and then create a dashboard on streamlit to get the results and visualizations.

Streamlit: https://github.com/streamlit/streamlit Fast API: https://github.com/tiangolo/fastapi

opened by gagan3012 5