BARTScore: Evaluating Generated Text as Text Generation

I'm sorry to trouble you that I am interesting in paraphrase-fine tuing part.
I use the concise code you provide in train/bart.py. However when i use the command “ python bart.py" , I encounter OOM problem ( On single V100-32GB ) I reread your paper, and notice that you can even use batchsize=20 to fine-tuing one epoch within a hour ( impressive speed~).

And i check the train/args.py in this repo, the batchsize is 8 and Max-epoch is 3 which is not consistency with the setting in paper (batch = 20,epoch =1).

Can you release the setting/code you used in fine-tuing? I want to check whats the problem for OOV ( maybe the larger max_length? or larger batchsize? Currently i cut TrainBatchSize to 4 (EvalBatchSize to 2) and the model fine-tuned very very very slowly....

Or can you give me some reproducing advice ? I am not sure lower batchsize or shoter sequence length could achieve same precision with paper's result.

Thank you vey much and any suggestions are greatly appreciated. Sorry to trouble you QAQ

Thank you for your work. Could you please tell us what is corresponding description of the value range? Such as [-1, -3]: similar; [-3, -6] normal; [-6, -100] not similar or so. Thanks.

Awesome Work~~

While using the implement in Repo, I have a question to comfirm:

BARTScore use generate loss as the score to estimate the quality of sentence. In bart_score.score's implement, I notice the code under:

curr_score_list = [-x.item() for x in loss]

The returned ScoreList contains some negetive value. if SummaryA get score -1 while SummaryB get Score -100 Does that means SummaryA is better than summaryB ? (becase the loss has been Negated?, loss1 is smaller than loss100) appreciate any reply ~

Hi,

thanks for your great work.

When I use the extracted dataset of QAGS_XSUM, I found that all fact scores are 0 or 1, which differ from QAGS_CNN. Should it be like that or there is something wrong?

Thanks a lot!

Best,

Hi, I notice that the QAGSCNN/QAGSXsum ROUGE result in Table5 (Factual Test) is different from other paper (eg. the paper which release this QAGS dataset). Whether it is because this paper calculates the ROUGE between document and summary while other papers calculate the ROUGE between reference and summary? ( In your experiment setting, you set "ref_sum" same as "src") I am confused whether baselines for comparison are the same

Hello, this is a very meaningful work.

I have a question, why is the prompt added to the right of y instead of the left?

For example, if we do a summary task, X is a long document, Y is its summary, and z is the prompt "in summary". I feel "[X]. In summay, [Y]" is more natural than "[X]. [Y], in summary".

Looking forward to your reply, thank you!

Hi @yyy-Apple

I had a couple of questions regarding reproducing the scores in Table 4 in the paper, I hope you could help me out

• Are the correlations at the system or sentence level? I went through the experiment setup section and also the SUM folder but couldn't find an answer
• When calculating correlations are the BARTScore values scaled to something like 0-1 range or taken as it is?

Thanks a lot for your time and amazing work!

Hey there!

I am trying to use the BARTScore metric as follows, but my dataset contains multiple references, is there any feature that I should enable to allow multiple references? I can't seem to find any...

from bart_score import BARTScorer
bart_scorer = BARTScorer(device='cuda:0', checkpoint='facebook/bart-large-cnn')
bart_scorer.score(['This is interesting.'], [['This is fun.'],['This is amazing']]) # generation scores from the first list of texts to the second list of texts.

Thanks a lot!

In equation 2 of the paper, Bartscore is calculated as the sum of log probabilities. However, in line 63 of the bart_score.py is calculated as the average of the log probabilities.

 loss = loss.sum(dim=1) / tgt_len

could you let us know which equation is the correct one?

Hello, thanks for this useful software!

I hit the following error when trying to load the file QAGS_CNN/data.pkl. If you have a solution I would appreciate it!

>>> import pickle
>>> pickle.load(open('BARTScore/SUM/QAGS_CNN/data.pkl', 'rb'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
EOFError: Ran out of input

environment

• ubuntu:20.04
• python3.8.10

I am having trouble running the score.py script from SUM when --rouge is passed

python score.py --file "REALSumm/data.pkl" --device "cuda:0" --output "REALSumm/scores.pkl" --rouge

Leads to

Cannot open /content/BARTScore/SUM/gehrmann_rouge_opennmt/rouge_baselines/pyrouge/RELEASE-1.5.5/smart_common_words.txt

I can find smart_common_words.txt under BARTScore/SUM/gehrmann_rouge_opennmt/rouge_baselines/pyrouge/RELEASE-1.5.5/data, but even if I move it to RELEASE-1.5.5 (as the script seems to expect) this just leads to other issues

Cannot open exception db file for reading: /content/BARTScore/SUM/gehrmann_rouge_opennmt/rouge_baselines/pyrouge/RELEASE-1.5.5/WordNet-2.0.exc.db

Are there extra dependencies/instructions not currently listed that are needed to run the scripts with rouge?

Hi (:

Great work with this paper! I was wondering if you could provide me with some additional feedback on the dataset you used for MT. If I understand correctly, you used the WMT-19 DARR dataset, but would you mind confirming if this was the validation or test set? Also, were you able to get actual scores for each of the sentences?

Thanks in advance :)

In Table-4 in the paper, for summEval dataset you have measured COH, FAC, FLU, INFO. I wanted to know which variants of bart-score you used.

From my understanding of the paper, For factuality(FAC) you must have used BARTScore(s->h) i.e source -> hypothesis.

But i am not clear about FLU, COH and INFO.

If you could please elaborate that will be really helpful.

Hello

I was wondering whether you were planning to make the library pip installable. That would make further integration a lot easier. Inspiration may be taken from BERTScore's implementation.

Thanks