Hey guys, looking at recent pull requests and issues, it looks like a common interest of contributors (On top of NMT obv) is Abstractive Summarization.

Any suggestions of how to train a model that will get close results to recent papers on the CNN-Daily Mail Database? Any additional preprocessing?

Thanks?

Hi, I want to reproduce the results on WMT'14 ENDE datasets of "Attention is All You Need" paper. I have read OpenNMT-FAQ and I want to know the exact details about your experiments:

1. Did you do the BPE or word spiece?
2. What's the exact BLEU score of your experiments on WMT'14 ENDE dataset?
3. Is there some other differences between README's steps and transformer experiments? If yes please provide a complete tutorial so as to help us reproduce your results.

Thank you very much! @srush

This occurs on systems with either cuda8/cudnn6 or cuda9/cudnn7, Ubuntu 16.04. Pytorch built from source.

python3 train.py -data data/demo -save_model demo-model -gpuid 1

    main()
  File "train.py", line 299, in main
    train_model(model, train, valid, fields, optim)
  File "train.py", line 159, in train_model
    train_stats = trainer.train(epoch, report_func)
  File "/home/levinth/OpenNMT-py/onmt/Trainer.py", line 133, in train
    dec_state.detach()
  File "/home/levinth/OpenNMT-py/onmt/Models.py", line 444, in detach
    h.detach_()
RuntimeError: Can't detach views in-place. Use detach() instead

Update: I made a post dedicated to this implementation: http://forum.opennmt.net/t/about-a-deep-reinforced-model-for-abstractive-summarization/1347

Working to implement Paulus et al (2017), A Deep Reinforced Model for Abstractive Summarization.

What's in the model (and what is implemented):

• [x] temporal attention on source
  • attention between decoding h and encoder outputs
  • normalizing w.r.t. previous decoding step => penalizing previous high attention scores
• [x] intra-decoder attention
  • attention between hd_t and [hd_1; hd_t-1] (for t>0)
• [x] pointer-generator
  • [x] using a soft-switch
  • [x] using tgt_embedding i.e. pointer_generator.linear = proj(tgt_embedding) (helps learning by using semantic informations)(requires partially shared embeddings)
• [x] Reducing Exposure Bias
  • [x] output predictions at each decoding steps
  • [x] using predicted token with 0.25 probability
• [x] partially sharing embeddings
  • [x] sharing tgt & src embeddings while having different vocabulary size src_vocab > tgt_vocab ; src_embedding = linear( tgt_embedding )
• [x] Reinforcement Policy
  • [x] Using ROUGE score to adjust learning

Implementations

• Most of my edits are in onmt.Reinforced
• I added the -reinforced flag to use this model.
• I tried to used as much existing functions as possible
• note that, because we need prediction at each time step the training process is quite different (run .backward() at each time steps. This is why I use a slightly modified Trainer class. It seems to have a bad impact on performance tho, GPU load < 100% most of the time.
• I currently use a different generator that is, in his current version, just using the CopyGenerator. I will be then upgraded in order to use the embedding matrix as part of its linear projection

Discussion

• I was first testing on really simple case like sequence copy i.e. using src = tgt = ptb.train.txt the model learns it really easily (more than >90% in 1 epoch on PTB). I specifically tested the copy mechanims by setting very low vocab size and looking at translation, which is correct even with a bad source (lots of unk).

• I am still finding lots of repetitions even on copy tasks. The model may continue to generate the last word for some time i.e. probably don't manage to generate <eos> for some reason.

• I am currently investigating it, I ran some not very promising training on summarization task (CNN/DM). The ppl goes as low as 30 (and ~30% acc) but translation give really poor results (lots of repetitions etc) there may be some issues about it.

What's next

• I would like to reproduce the "ML part" (without RL) result of the paper before moving to Reinforcement learning.
• Small implementation improvement like partial embeddings.

If people here are interested by this, running tests and suggesting improvement would help a lot :)

This the report for Chinese Abstractive summarization performance. Welcome to discuss.

Result

LCSTS dataset
Rouge-1 / ROUGE-2 / ROUGE-L
34.8 / 22.5 / 32.3

Gigaword Chinese dataset
Rouge-1 / ROUGE-2 / ROUGE-L
51.92 / 38.39 / 49.12

Preprocessing script

python3 preprocess.py \  
      -train_src $ORIGIN_DIR/train.source \  
      -train_tgt $ORIGIN_DIR/train.target \  
      -valid_src $ORIGIN_DIR/valid.source \
      -valid_tgt $ORIGIN_DIR/valid.target \
      -src_vocab_size 8000 \
      -tgt_vocab_size 8000 \
      -src_seq_length 400 \
      -tgt_seq_length 30 \
      -src_seq_length_trunc 400 \
      -tgt_seq_length_trunc 100 \
      -max_shard_size 20000000 \
      -save_data $DATA_DIR/processed

Training script

 python3 train.py \
      -data $DATA_DIR/processed \
      -word_vec_size 500 \
      -encoder_type brnn \
      -epochs 30 \
      -enc_layers 1 \
      -dec_layers 1 \
      -rnn_size 300 \
      -gpuid 0 \
      -save_model $MODEL_DIR/ \
      > $MODEL_DIR/log.txt

Generating script

 python3 translate.py \
      -model $MODEL_DIR/$BEST_MODEL \
      -beam_size 5 \
      -verbose \
      -batch_size 1 \
      -tgt $GOLD \
      -output $MODEL_DIR/$PRED \
      -src $TEST

Draft PR to add new vocabulary to existing checkpoint by reusing checkpoint's embeddings

This is what the new training procedure would be:

1. Run build_vocab as usual. This would generate the vocabulary files for the new corpora.
2. Then in the training procedure:

• Load checkpoint (fields would be populated with the checkpoint’s vocab)
• Build fields for the new vocabulary as usual from the vocabulary files generated in step1.
• Extend checkpoint fields vocabulary with the new vocabulary fields (only appending new words at the end, as stated in Torch docs)
• Assign the extended vocabulary to the new models fields
• In build_base_model, after loading the checkpoint’s state_dicts, replace new model’s embeddings from 0 to len(checkpoint.vocab_size) with checkpoint’s embeddings. It should be in the same order as in the extended vocabulary new words were appended at the end.
• Remove embeddings parameters from checkpoint as new embeddings have been already added to the model
• Continue as usual

When I use -gpuid option in current master of OpenNMT, regardless of what gpu I choose, the training script always chooses gpu 0. If I use multigpu, for example, -gpuid 2 3, the training goes to gpus 0 and 1.

Is this a known issue?

pytorch: 0.4 torchtext: 0.3

Hello, I've got one error when doing the first preprocessing step from here.

Traceback (most recent call last):
  File "preprocess.py", line 204, in <module>
    main()
  File "preprocess.py", line 191, in main
    fields = onmt.io.get_fields(opt.data_type, src_nfeats, tgt_nfeats)
  File "/home/lr/yukun/OpenNMT-py/onmt/io/IO.py", line 44, in get_fields
    return TextDataset.get_fields(n_src_features, n_tgt_features)
  File "/home/lr/yukun/OpenNMT-py/onmt/io/TextDataset.py", line 229, in get_fields
    postprocessing=make_src, sequential=False)
TypeError: __init__() got an unexpected keyword argument 'tensor_type'

It seems that torchtext 0.3 has changed their interface parameter in torchtext.data.Field()

But in torchtext 0.2.3, it works well except following warning:

xxx/torchtext/data/field.py:321: UserWarning: volatile was removed and now has no effect. Use with torch.no_grad(): instead.

And it seems new version of torchtext has fixed this from here. I am new to pytorch and opennmt. I don't know whether this warning matter?

1. forward() in Transformer is not compatible with Translator.py (line 153), which requires context_lengths.
2. Transformer actually doesn't converge using default options. (PPL is just ridiculous)
3. Too many parameters are reported.

Some of them are already reported in other threads. It could be hardly used in current state.

Is there a strong reason for slurping the entire source file in translate.py? On a source file of 2M single-character-per-token sentences, memory usage exceeds 256 GB. I figure that since evaluation is sentence-by-sentence (or batch-by-batch), this can be avoided.

Lab-mates and I have written an itertools.islice-based workaround. I'm happy to convert it into a PR if I have more information about the design goals and decisions, so I don't step on toes.

This pull request adds support fixed word embeddings (that is, embeddings that will not be changed as the parameter is changed) on the source or target side. In order to make this go more smoothly, I also refactored Embeddings -- the constructor now takes the parameters it needs explicitly, instead of just taking the relevant dictionaries and opt. If opt were passed, the model would somehow need to decide whether the -fix_word_vecs_enc or -fix_word_vecs_dec is relevant. This refactoring also allows loading of pretrained embeddings to be done when the object is instantiated, instead of later on in train.py. Future work could also be done to allow pretrained and/or fixed feature vectors.

In v3.0 in the TransformerDecoder class, the layer cache is no longer reset. This leads to an error when switiching between training and inference. During inference, in the layer cache the keys, etc. of the last tokens are saved, but they need to be removed when switching back to training.

Hi,

When training from a checkpoint, I consistently get an error when using multiple GPUs. The error doesn't occur if I use a single GPU.

I'm using version 2.2.0 but have verified the same behaviour in 2.1.2 as well.

See below example:

/usr/bin/python3 -m onmt.bin.train -config <path_to_train.yaml> --tensorboard --tensorboard_log_dir <path_to_log_dir> --world_size 2 --gpu_ranks 0 1 --train_from <path_to_checkpoint>
[2022-12-06 15:43:17,044 INFO] Missing transforms field for valid data, set to default: [].
[2022-12-06 15:43:17,048 INFO] Parsed 3 corpora from -data.
[2022-12-06 15:43:17,049 INFO] Loading checkpoint from en_es_step_285000.pt
[2022-12-06 15:43:19,364 INFO] Loading fields from checkpoint...
[2022-12-06 15:43:19,364 INFO]  * src vocab size = 38000
[2022-12-06 15:43:19,365 INFO]  * tgt vocab size = 38000
/usr/lib/python3.6/multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 6 leaked semaphores to clean up at shutdown
  len(cache))
Bus error (core dumped)

The training will succeed either by using only one GPU, or by removing the train_from argument altogether.

Thanks, Rob

If there is a line in the vocab file that doesn't have a token and a frequency correctly separated by whitespace skip the line instead of failing.

Example vocab file with broken last line:

В        118
ισχύουν  118
ΓΙΚΑ     118
ÁČ       117
✓        117
117

• https://github.com/OpenNMT/OpenNMT-py/issues/2242
• https://github.com/OpenNMT/OpenNMT-py/commit/baa3d73f645f10e8f3a94b334fc217afdfc4e550

This PR intends to add target features support to OpenNMT-py v3.0. All the code has been adapted for this new version.

Both source and target features support has been refactored for a more simplified handling of features. The way features are passed to the system has been changed and now features are appended to the actual textual data instead of providing a separate file. This also simplifies the way features are passed during inference and to the server. It uses the special character ￨ as a feature separator, as in the previous versions of the OpenNMT framework. For instance:

 I￨1￨3 love￨0￨1 eating￨0￨1 pizza￨0￨1

I've also added a way to provide default values for features. This can be really useful when mixing task specific data (with features) with general data which has not been annotated. Additionally, the filterfeats transform is no longer required and features are checked in the corpus loading process.

A YAML configuration file would look like this:

data:
    train:
        path_src: src_with_features.txt  #  I￨1￨3 love￨0￨1 eating￨0￨1 pizza￨0￨1
        path_tgt: tgt_with_features.txt  #  Me￨1 gusta￨0 comer￨0 pizza￨0
        transforms: [onmt_tokenize, inferfeats, filtertoolong]
    valid:
        path_src: src_with_features.txt
        path_tgt: tgt_with_features.txt
        transforms: [onmt_tokenize, inferfeats]

save_data: ./data
n_sample: -1

# # Vocab opts
src_vocab: data.vocab.src
tgt_vocab: data.vocab.tgt
n_src_feats: 2
n_tgt_feats: 1
src_feats_defaults: "0￨1"
tgt_feats_defaults: "1"
feat_merge: "sum"

For now, I've made the necessary changes in the code for vocabulary generation. That is, to make onmt_build_vocab work.

@vince62s , do you think this a good starting point?

This PR provides the ability to override the checkpoint's dropout settings (dropout, attention_dropout) when finetuning a Transformer model. A new flag override_opts is added. This overriding can thus be disabled with --override_opts False It is not possible at the moment because the checkpoint's model_optsoverride those set in the config. See the following discussion about this topic: https://forum.opennmt.net/t/tranformer-change-dropout-when-finetuning/4625

Hi,

I've recently noticed that I get slightly different translation results when translating with different batch sizes. I guess this is not expected...

For example,

Batch size 100 --> PRED SCORE: -6.0490, PRED PPL: 423.68 NB SENTENCES: 5000
Batch size 150 --> PRED SCORE: -5.2232, PRED PPL: 185.54 NB SENTENCES: 5000

I'm using the latest version of OpenNMT-py

Thanks