Character-Aware Neural Language Models
Tensorflow implementation of Character-Aware Neural Language Models. The original code of author can be found here.
This implementation contains:
- Word-level and Character-level Convolutional Neural Network
- Highway Network
- Recurrent Neural Network Language Model
The current implementation has a performance issue. See #3.
Prerequisites
- Python 2.7 or Python 3.3+
- Tensorflow
Usage
To train a model with ptb dataset:
$ python main.py --dataset ptb
To test an existing model:
$ python main.py --dataset ptb --forward_only True
To see all training options, run:
$ python main.py --help
which will print
usage: main.py [-h] [--epoch EPOCH] [--word_embed_dim WORD_EMBED_DIM]
[--char_embed_dim CHAR_EMBED_DIM]
[--max_word_length MAX_WORD_LENGTH] [--batch_size BATCH_SIZE]
[--seq_length SEQ_LENGTH] [--learning_rate LEARNING_RATE]
[--decay DECAY] [--dropout_prob DROPOUT_PROB]
[--feature_maps FEATURE_MAPS] [--kernels KERNELS]
[--model MODEL] [--data_dir DATA_DIR] [--dataset DATASET]
[--checkpoint_dir CHECKPOINT_DIR]
[--forward_only [FORWARD_ONLY]] [--noforward_only]
[--use_char [USE_CHAR]] [--nouse_char] [--use_word [USE_WORD]]
[--nouse_word]
optional arguments:
-h, --help show this help message and exit
--epoch EPOCH Epoch to train [25]
--word_embed_dim WORD_EMBED_DIM
The dimension of word embedding matrix [650]
--char_embed_dim CHAR_EMBED_DIM
The dimension of char embedding matrix [15]
--max_word_length MAX_WORD_LENGTH
The maximum length of word [65]
--batch_size BATCH_SIZE
The size of batch images [100]
--seq_length SEQ_LENGTH
The # of timesteps to unroll for [35]
--learning_rate LEARNING_RATE
Learning rate [1.0]
--decay DECAY Decay of SGD [0.5]
--dropout_prob DROPOUT_PROB
Probability of dropout layer [0.5]
--feature_maps FEATURE_MAPS
The # of feature maps in CNN
[50,100,150,200,200,200,200]
--kernels KERNELS The width of CNN kernels [1,2,3,4,5,6,7]
--model MODEL The type of model to train and test [LSTM, LSTMTDNN]
--data_dir DATA_DIR The name of data directory [data]
--dataset DATASET The name of dataset [ptb]
--checkpoint_dir CHECKPOINT_DIR
Directory name to save the checkpoints [checkpoint]
--forward_only [FORWARD_ONLY]
True for forward only, False for training [False]
--noforward_only
--use_char [USE_CHAR]
Use character-level language model [True]
--nouse_char
--use_word [USE_WORD]
Use word-level language [False]
--nouse_word
but more options can be found in models/LSTMTDNN and models/TDNN.
Performance
Failed to reproduce the results of paper (2016.02.12). If you are looking for a code that reproduced the paper's result, see https://github.com/mkroutikov/tf-lstm-char-cnn.
The perplexity on the test sets of Penn Treebank (PTB) corpora.
| Name | Character embed | LSTM hidden units | Paper (Y Kim 2016) | This repo. |
|---|---|---|---|---|
| LSTM-Char-Small | 15 | 100 | 92.3 | in progress |
| LSTM-Char-Large | 15 | 150 | 78.9 | in progress |
Author
Taehoon Kim / @carpedm20
![ValueError: Dimensions must be equal, but are 1300 and 1750 for 'LSTMTDNN/LSTM/rnn/rnn/multi_rnn_cell/cell_0/cell_0/basic_lstm_cell/MatMul_1' (op: 'MatMul') with input shapes: [100,1300], [1750,2600].](https://avatars.githubusercontent.com/u/19690332?v=4)
2.4k Jan 5, 2023
260 Jan 5, 2023
3 Apr 7, 2022
248 Nov 21, 2022
73 Nov 6, 2022
1 Oct 29, 2021
25 Jul 20, 2022
26 Nov 18, 2022
892 Dec 28, 2022
63 May 22, 2021
3.3k Jan 6, 2023
181 Dec 27, 2022
165 Jan 3, 2023
57 Dec 29, 2022
153 Dec 14, 2022
166 Jan 1, 2023
0 Feb 6, 2022
1 Feb 26, 2022
27 Sep 24, 2022