Changing the Mind of Transformers for Topically-Controllable Language Generation
We will first introduce the how to run the IPython notebook demo by downloading our pretrained models. Then, we will introduce how to run our training and evaluation code.
Requirements and Setup
- An Unix like OS with at least one GPU
- To set up the python environment, run
pip install -r requirements.txt. I use python 3.7 and pytorch 1.3.1, but I think other python 3 or pytorch > 1.0 versions might also be fine or just require very simple revision of the code. Our codes also use IPython notebook (for running the interactive demo), Spacy (for tokenization), nltk (for running evaluation and pplm), and gensim (for running the LDA baseline). - If your python path is not
~/anaconda3/bin/python, change your PY_PATH in the all the scripts in./bin
Running IPython Notebook Demo
- Download the pretrained models and dictionary file from here or following the instructions for training code below
- Use IPython notebook to open
./src/evaluation/test_conditional_LM.ipynb - Run the 1st block after putting the models into the corresponding directory or revising the paths of TOPIC_MODEL_DIR, GENERATION_MODEL_DIR, DICT_FILE in the first block.
- Modify the input context prompt in the 2nd block and run the block to see the generated topics
- Choose some topics or specify some words and run the 3rd block to see the generated continuations that start with
conditional x:. We will also generate the continuation without the condition that start withoriginal x:as a baseline. The topical words that appear in the continuation will be highlighted. - You can append a genearted continuation to the 2nd block and repeat the process
Preprocessing Wikipedia for Training and Evaluation
- First, download only the text from Wikipedia into json format using WikiExtractor
- Check the path in
./bin/preprocessing_single_proc.shand run the script. In the preprocessing, we will run Spacy tokenizer and GPT2 tokenizer, heuristically align their resulting tokens, split the corpus into training/validation/testing sets, and store the word indices into tensors. - Note that
./bin/preprocessing_single_proc.shmight be slow because it does not parallelize the tokenization processes. If you use job scheduler like slurm in your server, you might want to see the parallized scripts for tokenization in./bin/old/tokenize_all_wiki_gpt2.shand./bin/old/tokenize_all_wiki.sh
Running Training
- Prepare a word embedding file (e.g., we download the GloVe embedding from here)
- Train our option generator using
./bin/train_option_generator.sh - Train our conditional text generator using
./bin/train_conditional_generator.sh(could train option generator and text generator at the same time) - You can start from original GPT2 model or start from our pretrained models. In our paper, we use learning rate = 1e-4. You can also try other values between 1e-4 and 1e-5.
Running Evaluation using Automatic Metrics
- To evaluate/visualize conditional text generator, update the GENERATION_MODEL_DIR and TOPIC_MODEL_DIR using the model path from the previous step to run
./bin/train_conditional_generator.sh. - To evaluate/visualize option generator, update the GENERATION_MODEL_DIR and TOPIC_MODEL_DIR and run
./bin/eval_option_generator.sh. Set VISUALIZATION='Y' to visualize the topics given some randomly selected prompt. Set AUTO_EVAL_TOPICS='Y' to compare the quality of topics from different methods as we did in Table 1 in our EACL paper. Set AUTO_EVAL_GENRATION='Y' to evaluate the topics by the quality of text that is generated given these topics as we did in Table 6 in our paper appendix. - Our scores are stored at the end of each OUT_FILE file when AUTO_EVAL*='Y'. Our text generator is called "model condition", and our option generator is called NSD_topic in our code, where NSD stands for neural set decoder.
- In our code, we also evaluate some globally clustering baselines such as LDA and kmeans. In order to test them, you can train a LDA model by following the steps here. You can also see an example code at
./src/preprocessing/tools/train_LDA_model.py. For kmeans clustering, we use./src/preprocessing/tools/word_emb_global_clustering.py. If you do not want to test them, just remove LDA_org and global_centers from METHOD_LIST
Running Evaluation using Amazon Mechanical Turk
- Download STSb dataset from here
- Preprocessing STS using
./src/evaluation/filter_STS_for_GPT2.pyand remove the duplication bysort sts-train_longer.csv | uniq > sts-train_longer_uniq.csv - Set OUTPUT_CSV_FOR_MTURK='Y' in
./bin/train_conditional_generator.shand./bin/eval_option_generator.shto generate CSV files for MTurk tasks. - Our crowdsourcing templates and responses from workers could be found in
./MTurk_eval
Citation
If you use the code in a publication, please cite our paper.
Haw-Shiuan Chang, Jiaming Yuan, Mohit Iyyer, and Andrew McCallum,
“Changing the Mind of Transformers for Topically-Controllable Language Generation.”
Conference of the European Chapter of the Association for Computational Linguistics (EACL), 2021
95 Oct 24, 2022
1.1k Jan 1, 2023
39 Sep 28, 2022
71 Nov 15, 2022
37 Nov 4, 2022
7 Apr 4, 2022
![[CVPR 2022] TransEditor: Transformer-Based Dual-Space GAN for Highly Controllable Facial Editing](https://github.com/BillyXYB/TransEditor/raw/main/resources/teaser_change_order.png)
128 Jan 3, 2023
219 Dec 28, 2022
1.3k Dec 31, 2022
628 Dec 28, 2022
153 Jan 7, 2023
261 Jan 9, 2023
4 Aug 28, 2022
10 Sep 10, 2021
6 Sep 22, 2021
22 Jan 2, 2023
44 Nov 24, 2022
114 Dec 12, 2022