Code for the paper A Theoretical Analysis of the Repetition Problem in Text Generation

Zihao Fu

Last update: Nov 21, 2022

Related tags

Deep Learning repetition-problem-nlg

Overview

A Theoretical Analysis of the Repetition Problem in Text Generation

This repository share the code for the paper "A Theoretical Analysis of the Repetition Problem in Text Generation" in AAAI 2021. The repetition problem has been observed in nearly all text generation models. We theoretically prove that this problem is, unfortunately, caused by the traits of our language itself. There exists too many words predicting the same word as the subsequent word with high probability. Consequently, it is easy to go back to that word and form repetitions. We dub this problem as the high inflow problem. Based on the theoretical analysis, we propose a novel rebalanced encoding approach to alleviate the high inflow problem.

[arXiv]

Requirements

GCC >= 4.8
Python >= 3.7

Install

git clone https://github.com/fuzihaofzh/repetition-problem-nlg.git
cd repetition-problem-nlg
./scripts/setup.sh

iwslt14

Preprocess Data

./scripts/iwslt14_preprocess.sh

Train

./scripts/iwslt14_train.sh iwslt14deen_fastbpe_10000
./scripts/iwslt14_train.sh iwslt14deen_fastbpe_10000_re0.1

Test

./scripts/iwslt14_test.sh

Results can be found in output/eval/*

wiki103

Download the preprocessed data

git clone https://github.com/fuzihaofzh/preprocessed_wiki103.git output/preprocessed/wiki103

This may take few minutes to complete.

Preprocess Data

./scripts/wiki103_preprocess.sh

Train

./scripts/wiki103_train.sh wiki103_fastbpe_10000
./scripts/wiki103_train.sh wiki103_fastbpe_10000_re0.1

Test

./scripts/wiki103_test.sh

Results can be found in output/eval/*

Cite

@inproceedings{fu2020a,
  title={A Theoretical Analysis of the Repetition Problem in Text Generation.},
  author={Fu, Zihao and Lam, Wai and So, Anthony Man-Cho and Shi, Bei },
  booktitle={Thirty-Fifth AAAI Conference on Artificial Intelligence},
  year={2021}
}

Comments

Decoding method of RE model

Hi, thank you for your paper. The mathematical analysis of the repetition phenomenon is wonderful. What kind of decoding method the model with rebalanced encoding uses in inference stage? As I know, the rebalanced encoding is an encoding method rather than a decoding method.

opened by yunhaoli1995 2

This repository contains the data and code for the paper "Diverse Text Generation via Variational Encoder-Decoder Models with Gaussian Process Priors" (SPNLP@ACL2022)

GP-VAE This repository provides datasets and code for preprocessing, training and testing models for the paper: Diverse Text Generation via Variationa

18 Dec 29, 2022

Pytorch re-implementation of Paper: SwinTextSpotter: Scene Text Spotting via Better Synergy between Text Detection and Text Recognition (CVPR 2022)

SwinTextSpotter This is the pytorch implementation of Paper: SwinTextSpotter: Scene Text Spotting via Better Synergy between Text Detection and Text R

183 Jan 3, 2023

BARTScore: Evaluating Generated Text as Text Generation

This is the Repo for the paper: BARTScore: Evaluating Generated Text as Text Generation Updates 2021.06.28 Release online evaluation Demo 2021.06.25 R

196 Dec 17, 2022

A 1.3B text-to-image generation model trained on 14 million image-text pairs

minDALL-E on Conceptual Captions minDALL-E, named after minGPT, is a 1.3B text-to-image generation model trained on 14 million image-text pairs for no

604 Dec 14, 2022

Utility tools for the "Divide and Remaster" dataset, introduced as part of the Cocktail Fork problem paper

Divide and Remaster Utility Tools Utility tools for the "Divide and Remaster" dataset, introduced as part of the Cocktail Fork problem paper The DnR d

46 Dec 11, 2022

Codes for AAAI22 paper "Learning to Solve Travelling Salesman Problem with Hardness-Adaptive Curriculum"

Paper For more details, please see our paper Learning to Solve Travelling Salesman Problem with Hardness-Adaptive Curriculum which has been accepted a

14 Sep 30, 2022

PGPortfolio: Policy Gradient Portfolio, the source code of "A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem"(https://arxiv.org/pdf/1706.10059.pdf).

This is the original implementation of our paper, A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem (arXiv:1706.1

1.5k Dec 29, 2022

Related resources for our EMNLP 2021 paper Plan-then-Generate: Controlled Data-to-Text Generation via Planning

Plan-then-Generate: Controlled Data-to-Text Generation via Planning Authors: Yixuan Su, David Vandyke, Sihui Wang, Yimai Fang, and Nigel Collier Code

61 Jan 3, 2023

Code for EMNLP 2021 main conference paper "Text AutoAugment: Learning Compositional Augmentation Policy for Text Classification"

Text-AutoAugment (TAA) This repository contains the code for our paper Text AutoAugment: Learning Compositional Augmentation Policy for Text Classific

105 Jan 3, 2023

Code for the paper A Theoretical Analysis of the Repetition Problem in Text Generation

Related tags

Overview

A Theoretical Analysis of the Repetition Problem in Text Generation

Requirements

Install

iwslt14

Preprocess Data

Train

Test

wiki103

Download the preprocessed data

Preprocess Data

Train

Test

Cite

You might also like...

This repository contains the data and code for the paper "Diverse Text Generation via Variational Encoder-Decoder Models with Gaussian Process Priors" (SPNLP@ACL2022)

Pytorch re-implementation of Paper: SwinTextSpotter: Scene Text Spotting via Better Synergy between Text Detection and Text Recognition (CVPR 2022)

BARTScore: Evaluating Generated Text as Text Generation

A 1.3B text-to-image generation model trained on 14 million image-text pairs

Utility tools for the "Divide and Remaster" dataset, introduced as part of the Cocktail Fork problem paper

Codes for AAAI22 paper "Learning to Solve Travelling Salesman Problem with Hardness-Adaptive Curriculum"

PGPortfolio: Policy Gradient Portfolio, the source code of "A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem"(https://arxiv.org/pdf/1706.10059.pdf).

Related resources for our EMNLP 2021 paper Plan-then-Generate: Controlled Data-to-Text Generation via Planning

Code for EMNLP 2021 main conference paper "Text AutoAugment: Learning Compositional Augmentation Policy for Text Classification"

Comments

Decoding method of RE model

Owner

Zihao Fu

Problem-943.-ACMP - Problem 943. ACMP

Birthday-problem - The birthday problem asks for the probability that, in a set of n randomly chosen people, at least two will share a birthday

Source code for the GPT-2 story generation models in the EMNLP 2020 paper "STORIUM: A Dataset and Evaluation Platform for Human-in-the-Loop Story Generation"

Image-generation-baseline - MUGE Text To Image Generation Baseline

Code for the paper "Relation of the Relations: A New Formalization of the Relation Extraction Problem"

Source code for the Paper: CombOptNet: Fit the Right NP-Hard Problem by Learning Integer Programming Constraints}

Data and Code for ACL 2021 Paper "Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning"

Code for the paper "Reinforcement Learning as One Big Sequence Modeling Problem"

Code for the paper "Offline Reinforcement Learning as One Big Sequence Modeling Problem"

A weakly-supervised scene graph generation codebase. The implementation of our CVPR2021 paper ``Linguistic Structures as Weak Supervision for Visual Scene Graph Generation''