I am trying to reproduce the end-to-end entity linking results from the paper (Table 2). I have read appendix A.2 and issues #30 and #37 and try to follow it as closely as possible, but I fail to get the same numbers as in the paper, or even numbers close to it. In particular, I get very bad results from the model without fine-tuning.
I would appreciate very much if you could tell me what I am doing wrong. If you could explain the exact setup with which you got the results in Table 2, this would not only help me in reproducing the results, but also allow other users of your system to get the best performance.
Candidate sets
I got from issues #30 and #37 that you use candidate sets and a mention trie. I first create the candidate sets and then create a mention trie with all mentions which have at least 1 candidate.
In appendix A.2 you write that you used the candidate sets from Kolitsas et al. with additions of the table computed by Hoffart et al.
I downloaded the data by Kolitsas et al. from the link provided in their GitHub repository (https://github.com/dalab/end2end_neural_el, file prob_yago_crosswikis_wikipedia_p_e_m.txt), and the data by Hoffart et al. (https://www.mpi-inf.mpg.de/departments/databases-and-information-systems/research/ambiverse-nlu/aida/downloads, file aida_means.tsv).
Since you mentioned in isse #30 that you removed all mentions that do not start with an uppercase letter, and in issue #37 that you probably removed mentions that start with punctuation, I do that too.
The following code produces my mention_to_candidates_dict:
import re
import string
import itertools
import pickle
def read_dalab_candidates():
for line in open("data/dalab/prob_yago_crosswikis_wikipedia_p_e_m.txt"):
line = line[:-1]
columns = line.split("\t")
mention = columns[0]
for column in columns[2:]:
if len(column.strip()) == 0:
continue
values = column.split(",")
candidate = ",".join(values[2:])
candidate = candidate.replace("_", " ")
yield mention, candidate
def hex2int(hexa: str) -> int:
return int(hexa, 16)
def replace_unicode(u_str):
matches = set(re.findall("\\\\u....", u_str))
for match in matches:
u_str = u_str.replace(match, chr(hex2int(match[2:])))
return u_str
PUNCTUATION_CHARS = set(string.punctuation)
def filter_mention(mention):
if mention[0].islower():
return True
if mention[0] in PUNCTUATION_CHARS:
return True
return False
def read_aida_candidates():
for line in open("data/aida/aida_means.tsv"):
line = line[:-1]
values = line.split("\t")
mention = replace_unicode(values[0][1:-1])
candidate = replace_unicode(values[1]).replace("_", " ")
yield mention, candidate
if __name__ == "__main__":
mention_candidates_dict = {}
for mention, candidate in itertools.chain(read_dalab_candidates(), read_aida_candidates()):
if filter_mention(mention):
continue
if mention not in mention_candidates_dict:
mention_candidates_dict[mention] = set()
mention_candidates_dict[mention].add(candidate)
for mention in mention_candidates_dict:
mention_candidates_dict[mention] = sorted(mention_candidates_dict[mention])
with open("data/mention_candidates_dict.pkl", "wb") as f:
pickle.dump(mention_candidates_dict, f)
Mention trie
The following code creates a mention trie with all mentions from the mention_to_candidates_dict.
import sys
import pickle
from tqdm import tqdm
from genre.fairseq_model import GENRE
from genre.trie import Trie
if __name__ == "__main__":
sys.setrecursionlimit(10000)
model_path = "models/fairseq_e2e_entity_linking_wiki_abs"
model = GENRE.from_pretrained(model_path).eval()
with open("data/mention_to_candidates_dict.pkl", "rb") as f:
mention_to_candidates_dict = pickle.load(f)
mention_trie = Trie()
for mention in tqdm(mention_to_candidates_dict):
encoded = model.encode(" {}".format(mention))[1:].tolist()
mention_trie.add(encoded)
out_file = "data/mention_trie.pkl"
with open(out_file, "wb") as f:
pickle.dump(mention_trie, f)
Results
I use the mention_trie and mention_to_candidates_dict to restrict the beam search, as in the example.
When an article is too long, I split it iteratively into 2, 3, 4, ... parts, until the parts are short enough to be processed by the model. I split the article into sentences first (using spaCy), and then concatenate n/k sentences to create the k parts.
The results are much worse than the results reported in the paper. With the Wikipedia model (fairseq_e2e_entity_linking_wiki_abs) I get an F1-score of ~33% (28% precision, 40% recall) on the MSNBC benchmark. With the model fine-tuned on AidaYago (fairseq_e2e_entity_linking_aidayago) the result is better, with ~68% F1-score (72% precision, 64% recall).
Since you write in the paper that you "[...] considered only mentions that have entities in the KB" and in issue #30 that you "... used the entity universe from https://github.com/dalab/end2end_neural_el", I also tried to restrict candidates to entities from the file entities_universe.txt from that source. However, the results are worse, with ~26% F1-score for the Wikipedia model and ~41% F1-score for the fine-tuned model.
Example
Let's look at an example. The text is the first half of the first article in the MSNBC benchmark.
import pickle
import sys
from genre.utils import get_entity_spans_fairseq as get_entity_spans
from genre.fairseq_model import GENRE
from genre.utils import get_markdown
if __name__ == "__main__":
model_path = "models/fairseq_e2e_entity_linking_wiki_abs"
dict_path = "data/mention_to_candidates_dict.pkl"
trie_path = "data/mention_trie.pkl"
model = GENRE.from_pretrained(model_path).eval()
with open(trie_path, "rb") as f:
mention_trie = pickle.load(f)
with open(dict_path, "rb") as f:
mention_to_candidates_dict = pickle.load(f)
text = """Home Depot CEO Nardelli quits Home-improvement retailer's chief executive had been criticized over pay ATLANTA - Bob Nardelli abruptly resigned Wednesday as chairman and chief executive of The Home Depot Inc. after a six-year tenure that saw the world’s largest home improvement store chain post big profits but left investors disheartened by poor stock performance. Nardelli has also been under fire by investors for his hefty pay and is leaving with a severance package valued at about $210 million. He became CEO in December 2000 after being passed over for the top job at General Electric Co., where Nardelli had been a senior executive. Home Depot said Nardelli was being replaced by Frank Blake, its vice chairman, effective immediately. Blake’s appointment is permanent, Home Depot spokesman Jerry Shields said. What he will be paid was not immediately disclosed, Shields said. The company declined to make Blake available for comment, and a message left for Nardelli with his secretary was not immediately returned. Before Wednesday’s news, Home Depot’s stock had been down more than 3 percent on a split-adjusted basis since Nardelli took over. Nardelli’s sudden departure was stunning in that he told The Associated Press as recently as Sept. 1 that he had no intention of leaving, and a key director also said that the board was pleased with Nardelli despite the uproar by some investors. Asked in that interview if he had thought of hanging up his orange apron and leaving Home Depot, Nardelli said unequivocally that he hadn’t. Asked what he thought he would be doing 10 years from now, Nardelli said, “Selling hammers.” For The Home Depot? “Absolutely,” he said at the time. Home Depot said Nardelli’s decision to resign was by mutual agreement with the Atlanta-based company. “We are very grateful to Bob for his strong leadership of The Home Depot over the past six years. Under Bob’s tenure, the company made significant and necessary investments that greatly improved the company’s infrastructure and operations, expanded our markets to include wholesale distribution and new geographies, and undertook key strategic initiatives to strengthen the company’s foundation for the future,” Home Depot’s board said in a statement. Nardelli was a nuts-and-bolts leader, a former college football player and friend of President Bush. He helped increase revenue and profits at Home Depot and increase the number of stores the company operates to more than 2,000. Home Depot’s earnings per share have increased by approximately 150 percent over the last five years."""
sentences = [text]
entity_spans = get_entity_spans(
model,
sentences,
mention_trie=mention_trie,
mention_to_candidates_dict=mention_to_candidates_dict
)
markdown = get_markdown(sentences, entity_spans)[0]
print(markdown)
Output: Home Depot CEO Nardelli quits Home-improvement retailer's chief executive had been criticized over pay ATLANTA - Bob Nardelli abruptly resigned Wednesday as chairman and chief executive of The Home Depot Inc. after a six-year tenure that saw the world’s largest home improvement store chain post big profits but left investors disheartened by poor stock performance. Nardelli has also been under fire by investors for his hefty pay and is leaving with a severance package valued at about $210 million. He became CEO in December 2000 after being passed over for the top job at General Electric Co., where Nardelli had been a senior executive. Home Depot said Nardelli was being replaced by Frank Blake, its vice chairman, effective immediately. Blake’s appointment is permanent, Home Depot spokesman Jerry Shields said. What he will be paid was not immediately disclosed, Shields said. The company declined to make Blake available for comment, and a message left for Nardelli with his secretary was not immediately returned. Before Wednesday’s news, Home Depot’s stock had been down more than 3 percent on a split-adjusted basis since Nardelli took over. Nardelli’s sudden departure was stunning in that he told The Associated Press as recently as Sept. 1 that he had no intention of leaving, and a key director also said that the board was pleased with Nardelli despite the uproar by some investors. Asked in that interview if he had thought of hanging up his orange apron and leaving Home Depot, Nardelli said unequivocally that he hadn’t. Asked what he thought he would be doing 10 years from now, Nardelli said, “Selling hammers.” For The Home Depot? “Absolutely,” he said at the time. Home Depot said Nardelli’s decision to resign was by mutual agreement with the Atlanta-based company. “We are very grateful to Bob for his strong leadership of The Home Depot over the past six years. Under Bob’s tenure, the company made significant and necessary investments that greatly improved the company’s infrastructure and operations, expanded our markets to include wholesale distribution and new geographies, and undertook key strategic initiatives to strengthen the company’s foundation for the future,” Home Depot’s board said in a statement. Nardelli was a nuts-and-bolts leader, a former college football player and friend of President Bush. He helped increase revenue and profits at Home Depot and increase the number of stores the company operates to more than 2,000. Home Depot’s earnings per share have increased by approximately 150 percent over the last five years.
There are many wrong predictions:
- Mismatching boundaries ("The Home Depot" vs. "The Home Depot Inc.", "General Electric" vs. "General Electric Co.")
- Stopwords (The company, The Associated Press, He)
- Numbers (2,000, 150)
- Disambiguation errors (Bob)
- Other weird stuff (President Bush)
Clearly we can't expect the model to predict everything correctly, but the amount of mistakes is unexpected and does not fit with the good results reported in the paper.
The result of the fine-tuned model looks better:
Output: Home Depot CEO Nardelli quits Home-improvement retailer's chief executive had been criticized over pay ATLANTA - Bob Nardelli abruptly resigned Wednesday as chairman and chief executive of The Home Depot Inc. after a six-year tenure that saw the world’s largest home improvement store chain post big profits but left investors disheartened by poor stock performance. Nardelli has also been under fire by investors for his hefty pay and is leaving with a severance package valued at about $210 million. He became CEO in December 2000 after being passed over for the top job at General Electric Co., where Nardelli had been a senior executive. Home Depot said Nardelli was being replaced by Frank Blake, its vice chairman, effective immediately. Blake’s appointment is permanent, Home Depot spokesman Jerry Shields said. What he will be paid was not immediately disclosed, Shields said. The company declined to make Blake available for comment, and a message left for Nardelli with his secretary was not immediately returned. Before Wednesday’s news, Home Depot’s stock had been down more than 3 percent on a split-adjusted basis since Nardelli took over. Nardelli’s sudden departure was stunning in that he told The Associated Press as recently as Sept. 1 that he had no intention of leaving, and a key director also said that the board was pleased with Nardelli despite the uproar by some investors. Asked in that interview if he had thought of hanging up his orange apron and leaving Home Depot, Nardelli said unequivocally that he hadn’t. Asked what he thought he would be doing 10 years from now, Nardelli said, “Selling hammers.” For The Home Depot? “Absolutely,” he said at the time. Home Depot said Nardelli’s decision to resign was by mutual agreement with the Atlanta-based company. “We are very grateful to Bob for his strong leadership of The Home Depot over the past six years. Under Bob’s tenure, the company made significant and necessary investments that greatly improved the company’s infrastructure and operations, expanded our markets to include wholesale distribution and new geographies, and undertook key strategic initiatives to strengthen the company’s foundation for the future,” Home Depot’s board said in a statement. Nardelli was a nuts-and-bolts leader, a former college football player and friend of President Bush. He helped increase revenue and profits at Home Depot and increase the number of stores the company operates to more than 2,000. Home Depot’s earnings per share have increased by approximately 150 percent over the last five years.
Questions
Could you please help me (and other users of your system) in answering the following questions to clarify the best-performing setup:
- I understand the paper that in Table 2, the results on all benchmarks except for AIDA are from the Wikipedia model (out-of-domain scenario), and only the results on the AIDA benchmark are from the fine-tuned model (in-domain scenario). Is that correct?
- Do you have an idea why in my case the Wikipedia model is so much worse than the fine-tuned model? I get this result consistently on MSNBC, AIDA (here it is expected), KORE50, Spotlight, another benchmark with news articles and even on a benchmark with (excerpts of) Wikipedia articles (where the Wikipedia model should clearly be better).
- What exactly do you use as a knowledge base (that is, the set of entities)? The entities_universe.txt, all Wikipedia article titles, or something else? Is it the same for the in-domain and out-of-domain experiments?
- What exactly do you use as candidate sets? In the paper you write that you combine the sets from Kolitsas et al. and Hoffart et al., but in issue #37 it seems you only use the sets by Kolitsas et al. Do the candidate sets in the in-domain and out-of-domain experiments differ?
- I experience that the Wikipedia model links numbers (e.g., 2000, 12,580.35) and stop words (e.g., The, It), all resulting in false positives. Do you filter more mentions than lowercase mentions, mentions starting with punctuation and mentions with non-English characters (as you say in issue #37)?
- How do you split longer texts? Here I mean texts that are too long to be processed by the model, resulting in
None, [] (empty list), or truncated beams which are shorter than the input texts. I experience that the predictions of the model can differ a lot with different contexts (single sentences, paragraphs, texts spanning multiple paragraphs, or full articles).
Best regards,
Matthias
43 Nov 11, 2022
475 Jan 4, 2023
147 Dec 5, 2022
138 Oct 28, 2022
67 Nov 14, 2022
261 Nov 12, 2022
32 Oct 14, 2022
20 Dec 25, 2022
760 Jan 3, 2023
3.6k Jan 2, 2023
1.6k Dec 27, 2022
141 Dec 30, 2022
89 Nov 10, 2022
109 Dec 2, 2022
461 Dec 28, 2022
1.5k Dec 5, 2022