I also have a problem with reproducing your results. If I run your command_line.sh on WMT15 de-en I get a pearson correlation of -0.3559000480869218, but in your paper you reported 75.9. how did you run these experiments?

Hi,

thanks for your great work! I'm interested in BaryScore and want to reproduce your results. May I know with which human judgments you correlate BaryScore on WMT15? As far as I know, WMT15 officially calculates Kendall's tau with ranking-based human judgments, but you reported three types of correlations, so I guess there should be some human ratings like DA scores. Unfortunately, I was not able to find the DA scores. Could you help me with this? And the link to WMT16 DA judgments on its official website is currently invalid, do you still have this data? Besides, I'm not sure if I did something wrong, but BaryScore runs really slowly on my computer (over 200s for one language pair in WMT15 (BERTScore about 8s)). It seems the code already supports GPU computation?

Thanks a lot!

What does this PR do?

Fixes #6

This PR updates the structure of this repo and adds some necessary components to make this package easily installable from the source. Furthermore, it defines some CI jobs checking the package and also handling publishing the package distribution to PyPI once a package version is released.

@PierreColombo - Could I kindly ask you to check the author information? I'm not really sure if I set these properly. Also, could you please confirm for me that the dependency versions (in requirements.txt) make sense w.r.t. your setup?

@Borda - Could I kindly ask you to have quick a look at the changes I made so that we will be sure that everything is alright and the package will be ready soon as a reference one for torchmetrics testing? :] (Not sure if there's anything missing)

Hello,

I am having trouble reproducing the scores from the toy example Jupyter Notebook. I want to make sure I have faithfully reproduced the results from the metrics when I incorporate them into Repro.

Here are some issues I'm running into:

• Could you provide a requirements.txt with the package versions you are using (e.g., what transformers/pytorch versions)?
• The Jupyter Notebook passes in the references first (metric_call.evaluate_batch(refs, hypothesis)) but the evaluate_batch methods seem to expect the arguments in the opposite order
• When I run InfoLM on the GPU, there is a tensor which is still on the CPU. It occurs here:

File "/app/nlg_eval_via_simi_measures/infolm.py", line 280, in evaluate_batch
    dict_final_distribution_batch_refs, idfs_ref = self.get_distribution(batch_refs, idf_ref)
File "/app/nlg_eval_via_simi_measures/infolm.py", line 245, in get_distribution
    outputs = self.model(**unmasked_data, labels=labels)
...
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument index in method wrapper__index_select)

• It would also be very helpful to have the expected output for the data in the "samples" directory

Thanks!

We found the word frequency of candidate sentences might affect the score. We used the following codes to test the performance of InfoLM, but got different results for the same pair of candidate and reference sentences in different subsets.

Firstly, we made the candidate and reference sentences exactly the same; as expected, the distances between sentences are 0.

metric = InfoLM(measure_to_use='fisher_rao', alpha=0.5, beta=2, temperature=1.5, )
ref = [
       "germany produces what kind of beer?", ]
hypothesis = [
              "germany produces what kind of beer?", ]
metric.prepare_idfs(ref, hypothesis)
final_preds = metric.evaluate_batch(hypothesis, ref)
print(final_preds)

result: {'fisher_rao': [0.0], 'r_fisher_rao': [0.0], 'sim_fisher_rao': [0.0]}

However, when we make the set of candidate and reference sentences different, but including one exactly the same sentence inside each, the distance between these two sentences is not 0,

metric = InfoLM(measure_to_use='fisher_rao', alpha=0.5, beta=2, temperature=1.5, )
ref = [
       "name all of the rivers located in china.",
       "germany produces what kind of beer?",
       "lisp was designed by who?",
       "which of the bassists has a height of at least 1.94?",
       "name the aircraft that has the least amount of cargo space.",
       "what building has the earliest construction starting date?",
       "what lighthouses were constructed with sandstones?",
       "germany produces what kind of beer?",
       "belgium produces what kind of beer?",
       "belgium produces what kind of wine?",
       "belgium produces what type of wine?",
       "belgium produces which type of wine?",
       "belgium has which type of wine?",
       ]
hypothesis = [
              "name all of the rivers located in korea.",
              "france produces what kind of beer?",
              "ocaml was designed by who?",
              "which of the bassists has a height of at most 1.94?",
              "name the aircraft that has the most amount of cargo space.",
              "what building has the oldest construction starting date?",
              "what lighthouses were constructed with cements?",
              "germany produces what kind of beer?",
              "germany produces what kind of beer?",
              "germany produces what kind of beer?",
              "germany produces what kind of beer?",
              "germany produces what kind of beer?",
              "germany produces what kind of beer?"
]
metric.prepare_idfs(ref, hypothesis)
final_preds = metric.evaluate_batch(hypothesis, ref)
print(final_preds)

result: {'fisher_rao': [0.13154096901416779, 0.3968910872936249, 0.8951906561851501, 0.17877444624900818, 0.3418828845024109, 0.28145506978034973, 0.42686304450035095, 0.23330892622470856, 0.4328637421131134, 0.5381670594215393, 0.6704646348953247, 0.8761745095252991, 0.7861247658729553], 'r_fisher_rao': [0.13154096901416779, 0.3968910872936249, 0.8951906561851501, 0.17877444624900818, 0.3418828845024109, 0.28145506978034973, 0.42686304450035095, 0.23330892622470856, 0.4328637421131134, 0.5381670594215393, 0.6704646348953247, 0.8761745095252991, 0.7861247658729553], ' sim_fisher_rao': [0.13154096901416779, 0.3968910872936249, 0.8951906561851501, 0.17877444624900818, 0.3418828845024109, 0.28145506978034973, 0.42686304450035095, 0.23330892622470856, 0.4328637421131134, 0.5381670594215393, 0.6704646348953247, 0.8761745095252991, 0.7861247658729553]}

Here we found that exactly the same sentence, "germany produces what kind of beer?" in reference and candidate sets don’t lead to a 0 distance, but their distance is 0.23330892622470856 according to the code you provided.

After reading the paper carefully, we found that the word frequency in candidate sentences might affect the InfoLM score. On page 3 of your paper, you wrote “where $\widetilde{γ_k}$ and $γ_k$ are measures of the importance of the k-th token in the candidate and reference text, respectively”. From our understanding, since the set of candidate sentences each time might be different, then if each time you compute the IDF of words according to the candidate sentences, the distance obtained for a given (fixed) pair of (candidate sentence, reference sentence) could be different at each time of testing. Would you confirm our understanding here - is this a feature from the design your metric?

Thank you for spending some time reading our mail. I’d appreciate it if you could help me with my questions.

Hi @PierreColombo,

First of all, I'd like to thank you for the great work of you and your team in developing NLG metrics. I'm just curious if you'd be interested in my contribution to creating setup.py file and preparing other related stuff so that the repo can be installed from the source with pip and also make it potentially published via pypi. I believe this could help further adopt your evaluation metrics and make them easier to use for various use cases. Also, we would like to implement these metrics within torchmetrics package (see InfoLM, BaryScore and DepthScore issues), therefore, the possibility of installing the package from pypi would facilitate us to test our implementation against yours so that we can be sure our implementation is correct.

Thanks once again and please let me know what you do think about my proposal :]

cc: @Borda

Thanks a lot for publishing your code :)

I have a problem with bary_score.py. There is an import ot, which I couldn't install with pip. Does it refer to some file that I am missing?

Hello,

when you call

info_dic = self.compute_infolm(sum_distribution_hypothesis, sum_distribution_refs)

in evaluate_batch method, you pass the arguments in the incorrect order as the signature of compute_infolm is

def compute_infolm(self, ref_distribution, hyp_distribution):