Seq2Seq in PyTorch
This is a complete suite for training sequence-to-sequence models in PyTorch. It consists of several models and code to both train and infer using them.
Using this code you can train:
- Neural-machine-translation (NMT) models
- Language models
- Image to caption generation
- Skip-thought sentence representations
- And more...
Installation
git clone --recursive https://github.com/eladhoffer/seq2seq.pytorch
cd seq2seq.pytorch; python setup.py develop
Models
Models currently available:
- Simple Seq2Seq recurrent model
- Recurrent Seq2Seq with attentional decoder
- Google neural machine translation (GNMT) recurrent model
- Transformer - attention-only model from "Attention Is All You Need"
Datasets
Datasets currently available:
- WMT16
- WMT17
- OpenSubtitles 2016
- COCO image captions
- Conceptual captions
All datasets can be tokenized using 3 available segmentation methods:
- Character based segmentation
- Word based segmentation
- Byte-pair-encoding (BPE) as suggested by bpe with selectable number of tokens.
After choosing a tokenization method, a vocabulary will be generated and saved for future inference.
Training methods
The models can be trained using several methods:
- Basic Seq2Seq - given encoded sequence, generate (decode) output sequence. Training is done with teacher-forcing.
- Multi Seq2Seq - where several tasks (such as multiple languages) are trained simultaneously by using the data sequences as both input to the encoder and output for decoder.
- Image2Seq - used to train image to caption generators.
Usage
Example training scripts are available in scripts folder. Inference examples are available in examples folder.
- example for training a transformer on WMT16 according to original paper regime:
DATASET=${1:-"WMT16_de_en"}
DATASET_DIR=${2:-"./data/wmt16_de_en"}
OUTPUT_DIR=${3:-"./results"}
WARMUP="4000"
LR0="512**(-0.5)"
python main.py \
--save transformer \
--dataset ${DATASET} \
--dataset-dir ${DATASET_DIR} \
--results-dir ${OUTPUT_DIR} \
--model Transformer \
--model-config "{'num_layers': 6, 'hidden_size': 512, 'num_heads': 8, 'inner_linear': 2048}" \
--data-config "{'moses_pretok': True, 'tokenization':'bpe', 'num_symbols':32000, 'shared_vocab':True}" \
--b 128 \
--max-length 100 \
--device-ids 0 \
--label-smoothing 0.1 \
--trainer Seq2SeqTrainer \
--optimization-config "[{'step_lambda':
\"lambda t: { \
'optimizer': 'Adam', \
'lr': ${LR0} * min(t ** -0.5, t * ${WARMUP} ** -1.5), \
'betas': (0.9, 0.98), 'eps':1e-9}\"
}]"
- example for training attentional LSTM based model with 3 layers in both encoder and decoder:
python main.py \
--save de_en_wmt17 \
--dataset ${DATASET} \
--dataset-dir ${DATASET_DIR} \
--results-dir ${OUTPUT_DIR} \
--model RecurrentAttentionSeq2Seq \
--model-config "{'hidden_size': 512, 'dropout': 0.2, \
'tie_embedding': True, 'transfer_hidden': False, \
'encoder': {'num_layers': 3, 'bidirectional': True, 'num_bidirectional': 1, 'context_transform': 512}, \
'decoder': {'num_layers': 3, 'concat_attention': True,\
'attention': {'mode': 'dot_prod', 'dropout': 0, 'output_transform': True, 'output_nonlinearity': 'relu'}}}" \
--data-config "{'moses_pretok': True, 'tokenization':'bpe', 'num_symbols':32000, 'shared_vocab':True}" \
--b 128 \
--max-length 80 \
--device-ids 0 \
--trainer Seq2SeqTrainer \
--optimization-config "[{'epoch': 0, 'optimizer': 'Adam', 'lr': 1e-3},
{'epoch': 6, 'lr': 5e-4},
{'epoch': 8, 'lr':1e-4},
{'epoch': 10, 'lr': 5e-5},
{'epoch': 12, 'lr': 1e-5}]" \
43 Dec 23, 2022
20.5k Jan 8, 2023
1.1k Dec 27, 2022
39 Aug 3, 2021
490 Dec 15, 2022
1.1k Dec 17, 2022
186 Dec 24, 2022
11 Mar 28, 2022
184 Dec 18, 2022
9 Jun 27, 2022
12 Aug 19, 2021
290 Dec 26, 2022
2.5k Jan 3, 2023
460 Oct 13, 2022