Google's Meena transformer chatbot implementation

Francesco Pham

Last update: Dec 25, 2022

Related tags

Text Data & NLP nlp natural-language-processing chatbot transformer opensubtitles meena-chatbot perplexity-score

Overview

Meena chatbot

Here's my attempt at recreating Meena, a state of the art chatbot developed by Google Research and described in the paper Towards a Human-like Open-Domain Chatbot.

For this implementation I used the tensor2tensor deep learning library, using an evolved transformer model as described in the paper.

The training set used is the OpenSubtitles corpus in the Italian language. Many other languages are available here.

Model

Similarly to the work done in the paper, this model consists of 1 encoder block and 12 decoder blocks for a total of 108M parameters. The optimizer used is Adafactor with the same training rate schedule as described in the paper.

Training

Here are the results after training the model on 40M sentences of the OpenSubtitles dataset in the italian language. The learning rate starts at 0.01 and remains constant for 10k steps then decay with the inverse square root of the number of steps.

Here's the plot of the evaluation loss during training.

The final perplexity score is 10.4 which is very close to the perplexity score achieved by Google's meena chatbot 10.2.

The paper shows a correlation between perplexity score and the Sensibleness and Specificity Average which is correlated with the "human likeness" of the chatbot. Our perplexity score shows that our bot is better than other chatbots such as Cleverbot and DialoGPT:

The dataset used however does not represent well normal conversations between humans. However Opensubtitles provide very large datasets in many languages.

Run pretrained model

Simply run notebook meena_chatbot_inference.ipynb.

Otherwise download the following model and extract it. Set proper MODEL_DIR and CHECKPOINT_NAME in predict.py and run main.py

Pretrained model checkpoint

Italian, 108M parameters, 200k steps, 40M sentences

Train a new model

For training simply run the ipython notebook on Google Colab, the model will be saved on Google Drive. At the end of the execution you can interact with the chatbot.

Export the model

The model can be exported by copying the following files in a folder:

hparams.json
The trained model checkpoint
The vocabulary .subwords file

and run main.py after setting the proper model directory.

Serving

server.py provides a simple HTTP API for serving the chatbot.

Comments

bot response is so slow?

Hi, first of all am so thankful for your good job creating this bot. But, as i run your bot on google colab it response to me very very slow? it's normal? or anything else? please let me know?

Best Wishes,

opened by danial1995 4
Not getting good results ?

hi @frankplus first of all thanks for your work, so I use your Jupyter notebook to train the model on the open subtitles English dataset but not getting good results. I trained it for like 3 hours on colab pro gpu, but the training goes on then i stopped it and tried the code for testing the model . but seems like the results were not that good. Can you you please suggest some better way to make mods of the meena, or you can guide me how can use your repo to use efficiently.

opened by alan-ai-learner 2
How make working under TF2?
First of all, thank you for your effort. I'd like to implement your code on my personal robot, beginning with some tests on PC Unfortunately without success, maybe because I've installed TF2 (2.10.0) on python 3.8.10 and the code seems written for TF1 . Is it true? I spent several days trying to convert code to TF2 using 'tf.compat.v1.xxxx' , but without success ( I'm an absolute newbie in Deep Learning). After loading the model 'model.ckpt-200000' from Italian_108M, in predict.py with :

with tf.compat.v1.Session() as sess: new_saver = tf.compat.v1.train.import_meta_graph(MODEL_DIR + CHECKPOINT_NAME +'.meta') new_saver.restore(sess, MODEL_DIR+ CHECKPOINT_NAME) #'model.ckpt-200000'

I get exception 'Attempting to capture an EagerTensor without building a function' on nest statement: output_candidates = [chatbot_model.infer(encoded_inputs, decode_length=1) for _ in range(NUM_SAMPLES)]

At this point I don't know anymore how to go on... So my question is if is it possible to have some help on porting the code to TF2. Thank you in advance
opened by Luke1962 0
Not getting better results after training for several days?

Hi @frankplus, as you got the better results then might be the problem is from my side so, i used this dataset , so please take a look into this dataset and let me know. Can you suggest me a dataset for the english conversation

Thanks

opened by alan-ai-learner 4
Where is the source file on the nlpl page exactly?

The notebook references http://opus.nlpl.eu/download.php?f=OpenSubtitles/v2018/mono/OpenSubtitles.it.gz as the source, when I visit the linked opus.nlpl.eu page I see this grid with a bunch of LANG.xml.gz files - I cannot seem to locate a different file than Italian - can you link me to the exact page where I can find alternatives to Italian language so that I can train the model with a different data source please?

opened by tgmerritt 1