Not sure if anyone else has encountered this problem, but when I download the pretrained model and use it to evaluate the data, the scores that I get are abysmally low. It's something like:

---------------------------------------------
1 ROUGE-1 Average_R: 0.01291 (95%-conf.int. 0.01247 - 0.01334)
1 ROUGE-1 Average_P: 0.05262 (95%-conf.int. 0.05030 - 0.05487)
1 ROUGE-1 Average_F: 0.01769 (95%-conf.int. 0.01710 - 0.01824)
---------------------------------------------
1 ROUGE-2 Average_R: 0.00004 (95%-conf.int. 0.00003 - 0.00007)
1 ROUGE-2 Average_P: 0.00030 (95%-conf.int. 0.00015 - 0.00049)
1 ROUGE-2 Average_F: 0.00007 (95%-conf.int. 0.00004 - 0.00010)
---------------------------------------------
1 ROUGE-L Average_R: 0.01260 (95%-conf.int. 0.01219 - 0.01302)
1 ROUGE-L Average_P: 0.05109 (95%-conf.int. 0.04888 - 0.05321)
1 ROUGE-L Average_F: 0.01725 (95%-conf.int. 0.01668 - 0.01780)

Which is weird considering I used the same data and the same model. Anyone know what might be some causes? I've been trying to get this code to work properly for a while now and would appreciate any tips. Thanks.

I m trying to train a model and after 40K step I still have an accuracy of 6-6.3. When you were training your model, what was your accuracy around this step? I did make a few changes to your parameters and some of the code. So I m thinking I m going in the wrong direction.

ENV

Python: python 3.7.4 PyTorch: torch==1.1.0

Hang problem

When I running the BertAbs as the script of README, I got hanging. Specifically, using 4 V100 32G cards training the BertAbs, it hanged at nearly the end part: I run 2 times, they always hang at about 170K / 200K steps(not Exactly same). But when I use a bigger batch-size (140 x 4), or use 1 cards, it can finish correctly.

After read the code, I'm suspect the Dataloader for distributed multi-cards may cause Problem: The Dataload don't guarantee that every card will have same number of batch data, that is, in the Edge condition: the end of 1 epoch, the only left 3 batch data, so 3 cards get the data, but the 4th gpu-card haven't! so It will Running the Next Epoch and do Param-Sync with other cards which actually in previous Epoch. And according the time accumulating, the bias become bigger and bigger, and hang at some step (such as, 1 process has go over all data, while other process are stilling running, so the distributed sync will hang to wait the finished-process).

I'm just guessing... Currently I haven't test the suspecting.

Anyway, The actual reason may be other. I'm here mostly to ask Whether the Author or someone has encountered this condition?

What's more, I'm anxious because I can't reproduce the result of paper (one for this hanging condition, and under other not-hanging setting, the ROUGE1 is less than paper about 0.36) and I want to do some continuous work based on the result.

Please indicating/help me...

I am trying to run the model on the test data provided in the raw_data folder (dev branch). The summary I am getting is always the 1st sentence of every record in the source file. Is there a way to change the no of sentences in the summary, and get more sentences as a part of summary? Tried changing in trainer_ext.py

trainer_ext.py
if ((not cal_oracle) and (not self.args.recall_eval) and len(_pred) == 5):

but it does not work. The arguments I am using are: -task ext -mode test_text -test_from ../models/bertext_cnndm_transformer.pt -text_src ../raw_data/temp_ext_raw_src.txt -result_path ../results/ootb_output -alpha 0.95 -log_file ../logs/test.log -visible_gpus -1

While training model on custom data, where should we put summaries corresponding to the text articles? As in RAW_DATA we put stories that is articles.

I ran this command on my command prompt

python train.py -task ext -mode train -bert_data_path C:\Users\Ayesha\Downloads\Tasks\BERTSumm\PreSumm-dev\PreSumm-dev\bert_data\cnndm -ext_dropout 0.1 -model_path C:\Users\Ayesha\Downloads\Tasks\BERTSumm\PreSumm-dev\PreSumm-dev\models -lr 2e-3 -visible_gpus 0 -report_every 50 -save_checkpoint_steps 1000 -batch_size 3000 -train_steps 50 -accum_count 2 -log_file ../logs/ext_bert_cnndm -use_interval true -warmup_steps 10000 -max_pos 512

and I am getting this error

System Specifications Windows 10 NVIDIA GPU RTX 2060

How can I solve this?

Hi,

I used the same commands in the README to run the code. However, my reproduced results are around 0.8 lower than the reported ones for TransformerAbs and BertExtAbs.

Am I missing anything?

Thanks.

Hi, I want to use the trained model so tried to put the code train_from ../models path but it doesn't work with below key error.

I found the same issues on this github so followed the solution I revised the code from "optim = checkpoint['optim'][0]" to "optim = checkpoint['optim']" but still there is same issue. how can i fix it?

I'm having difficulties to wrap my head around the batch_size parameter.

What exactly is the batch_size parameter ?

It's not the real batch size (i.e. how many samples can be processed at once).
So what is it exactly ? And how can I choose the real batch size from this argument ?

Downloaded, then processed and trained 'cnndm_sample.train.0.json' (documented here: https://github.com/antonysama/PreSumm/blob/master/README.md). However Evaluate and Test, steps give the following errors:

###Evaluate code: python PreSumm/src/train.py -task abs -mode validate -test_all -batch_size 8 -test_batch_size 2 -bert_data_path ~/o3/PreSumm/bert_data/ -log_file ~/o3/PreSumm/logs/val_abs_bert_cnndm -model_path ~/o3/PreSumm/models -sep_optim true -use_interval true -visible_gpus -1 -max_pos 512 -max_length 10 -alpha 0.95 -min_length 5 -result_path ~/o3/PreSumm/logs/abs_bert_cnndm

##Evaluate Error: loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-pytorch_model.bin from cache at ../temp/aa1ef1aede4482d0dbcd4d52baad8ae300e60902e88fcb0bebdec09afd232066.36ca03ab34a1a5d5fa7bc3d03d55c4fa650fed07220e2eeebc06ce58d0e9a157 Traceback (most recent call last): File "PreSumm/src/train.py", line 124, in <module> validate_abs(args, device_id) File "/home/antony/o3/PreSumm/src/train_abstractive.py", line 131, in validate_abs xent = validate(args, device_id, cp, step) File "/home/antony/o3/PreSumm/src/train_abstractive.py", line 187, in validate shuffle=False, is_test=False) File "/home/antony/o3/PreSumm/src/models/data_loader.py", line 136, in __init__ self.cur_iter = self._next_dataset_iterator(datasets) File "/home/antony/o3/PreSumm/src/models/data_loader.py", line 156, in _next_dataset_iterator self.cur_dataset = next(dataset_iter) File "/home/antony/o3/PreSumm/src/models/data_loader.py", line 94, in load_dataset yield _lazy_dataset_loader(pt, corpus_type) File "/home/antony/o3/PreSumm/src/models/data_loader.py", line 78, in _lazy_dataset_loader dataset = torch.load(pt_file) File "/home/antony/.local/lib/python3.6/site-packages/torch/serialization.py", line 382, in load f = open(f, 'rb') FileNotFoundError: [Errno 2] No such file or directory: '/home/antony/o3/PreSumm/bert_data/.valid.pt'

##Test code: python PreSumm/src/train.py -task abs -mode test -test_from ~/o3/PreSumm/bert_data/model_step_148000.pt -batch_size 16 -test_batch_size 2 -bert_data_path ~/o3/PreSumm/bert_data/ -log_file ~/o3/PreSumm/logs/val_abs_bert_cnndm -sep_optim true -use_interval true -visible_gpus -1 -max_pos 512 -max_length 10 -alpha 0.95 -min_length 5 -result_path ~/o3/PreSumm/logs/abs_bert_cnndm

##Test error: Similar to 'Evaluate' error above, but ends with 'File not found...test.pt'

I am using the abstractive model for inference using following command

python3 train.py -task abs -mode validate -batch_size 3000 -test_batch_size 500 -bert_data_path ../bert_data/cnndm -log_file ../logs/val_abs_bert_cnndm -model_path ../../model/ -sep_optim true -use_interval true -visible_gpus -1 -max_pos 512 -max_length 200 -alpha 0.95 -min_length 50 -result_path ../logs/abs_bert_cnndm

I am getting below error.

File "train.py", line 124, in validate_abs(args, device_id) File "/home/exa00117/Practice/textSummerization/presumm/PreSumm-master/src/train_abstractive.py", line 154, in validate_abs validate(args, device_id, cp, step) File "/home/exa00117/Practice/textSummerization/presumm/PreSumm-master/src/train_abstractive.py", line 196, in validate stats = trainer.validate(valid_iter, step) File "/home/exa00117/Practice/textSummerization/presumm/PreSumm-master/src/models/trainer.py", line 197, in validate outputs, _ = self.model(src, tgt, segs, clss, mask_src, mask_tgt, mask_cls) File "/home/exa00117/.local/lib/python3.5/site-packages/torch/nn/modules/module.py", line 547, in call result = self.forward(*input, **kwargs) File "/home/exa00117/Practice/textSummerization/presumm/PreSumm-master/src/models/model_builder.py", line 243, in forward decoder_outputs, state = self.decoder(tgt[:, :-1], top_vec, dec_state) File "/home/exa00117/.local/lib/python3.5/site-packages/torch/nn/modules/module.py", line 547, in call result = self.forward(*input, **kwargs) File "/home/exa00117/Practice/textSummerization/presumm/PreSumm-master/src/models/decoder.py", line 202, in forward step=step) File "/home/exa00117/.local/lib/python3.5/site-packages/torch/nn/modules/module.py", line 547, in call result = self.forward(*input, **kwargs) File "/home/exa00117/Practice/textSummerization/presumm/PreSumm-master/src/models/decoder.py", line 64, in forward dec_mask = torch.gt(tgt_pad_mask + self.mask[:, :tgt_pad_mask.size(1), :tgt_pad_mask.size(1)], torch.tensor(0)) RuntimeError: expected device cpu and dtype Byte but got device cpu and dtype Bool

Hello,

Are the batch sizes and accum count for the bert large exactly the same as the base model? I have been trying to get the results but my bert large has been strictly performing worse than the base model( about 3-4 rouge points) and I have no idea why

Hi folks,

I want to use these pre-trained models for summarization with my custom input text. Let's say, I have 10 articles that I want to summarize using BertSumExt, so first I have to preprocess my raw inputs.

The README has following steps: Step 1 Download Stories Here, it will be my custom articles.

Step 2. Download Stanford CoreNLP This part has no issue.

Step 3. Sentence Splitting and Tokenization python preprocess.py -mode tokenize -raw_path RAW_PATH -save_path TOKENIZED_PATHRAW_PATH is the directory containing articles, JSON_PATH is the target directory to save the generated json files

Step 4. Format to Simpler Json Files python preprocess.py -mode format_to_lines -raw_path RAW_PATH -save_path JSON_PATH -n_cpus 1 -use_bert_basic_tokenizer false -map_path MAP_PATHRAW_PATH is the directory containing tokenized files, JSON_PATH is the target directory to save the generated json files, MAP_PATH is the directory containing the urls files (../urls)

Step 5. Format to PyTorch Files python preprocess.py -mode format_to_bert -raw_path JSON_PATH -save_path BERT_DATA_PATH -lower -n_cpus 1 -log_file ../logs/preprocess.log

So, my question is, what is the MAP_PATH i.e. mapping urls, I don't have anything related to this for my custom data. My task is very simple, given raw text input(i.e. article text), get the summary.

Can I get understandable steps for this?

e.g. input: This is the article .... output: Summary...

Hello, I was using PreSumm code to run on a custom dataset. I made the format of data compatible with model input. I trained Transformer baseline, a simple encoder and decoder, and stopped training when perplexity was low, around 2. However, in the inference, I get a very low rouge score. When I checked the actual generated candidates, I saw for all the inputs, the model generates the same candidate. I could not figure out the issue. Any help is greatly appreciated.

I am trying to load a pretrained model in my notebook. Using google colab for the experiment,

import torch

PATH = "path/to/the/model/model_step_148000.pt" model = torch.load(PATH)

it gives me this error ; ModuleNotFoundError: No module named 'models'

How can I solve this?

Bumps numpy from 1.17.2 to 1.22.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.