This repo provides the code of the following papers:
(GAR) "Generation-Augmented Retrieval for Open-domain Question Answering", ACL 2021
(RIDER) "Reader-Guided Passage Reranking for Open-Domain Question Answering", Findings of ACL 2021.
GAR augments a question with relevant contexts generated by seq2seq learning, with the question as input and target outputs such as the answer, the sentence where the answer belongs to, and the title of a passage that contains the answer. With the generated contexts appended to the original questions, GAR achieves state-of-the-art OpenQA performance with a simple BM25 retriever.
RIDER is a simple and effective passage reranker, which reranks retrieved passages by reader predictions without any training. RIDER achieves 10~20 gains in top-1 retrieval accuracy, 1~4 gains in Exact Match (EM), and even outperforms supervised transformer-based rerankers.
Code
Generation
The codebase of seq2seq models is based on (old) huggingface/transformers (version==2.11.0) examples.
See train_gen.yml for the package requirements and example commands to run the models.
train_generator.py: training of seq2seq models.
conf.py: configurations for train_generator.py. There are some default parameters but it might be easier to set e.g., --data_dir and --output_dir directly.
test_generator.py: test of seq2seq models (if not already done in train_generator.py).
Retrieval
We use pyserini for BM25 retrieval. Please refer to its document for indexing and searching wiki passages (wiki passages can be downloaded here). Alternatively, you may take a look at its effort to reproduce DPR results, which gives more detailed instructions and incorporates the passage-level span voting in GAR.
Reranking
Please see the instructions in rider/rider.py.
Reading
We experiment with one extractive reader and one generative reader.
For the extractive reader, we take the one used by dense passage retrieval. Please refer to DPR for more details.
For the generative reader, we reuse the codebase in the generation stage above, with [question; top-retrieved passages] as the source input and one ground-truth answer as the target output. Example script is provided in train_gen.yml.
Data
Please refer to DPR for dataset downloading.
For seq2seq learning, use {train/val/test}.source as the input and {train/val/test}.target as the output, where each line is one example.
In the same folder, save the list of ground-truth answers with name {val/test}.target.json if you want to evaluate EM during training.
Cite
Please use the following bibtex to cite our papers.
@article{mao2020generation,
title={Generation-augmented retrieval for open-domain question answering},
author={Mao, Yuning and He, Pengcheng and Liu, Xiaodong and Shen, Yelong and Gao, Jianfeng and Han, Jiawei and Chen, Weizhu},
journal={arXiv preprint arXiv:2009.08553},
year={2020}
}
@article{mao2021reader,
title={Reader-Guided Passage Reranking for Open-Domain Question Answering},
author={Mao, Yuning and He, Pengcheng and Liu, Xiaodong and Shen, Yelong and Gao, Jianfeng and Han, Jiawei and Chen, Weizhu},
journal={arXiv preprint arXiv:2101.00294}
}
and the iteration only runs at 73it. How should I fix it and is the 'passage_idx = idxs[q, p].item()' takes a long time that results in timed out?
2 Feb 3, 2022
189 Jan 2, 2023
90 Dec 27, 2022
46 Dec 15, 2022
478 Dec 25, 2022
5 Sep 13, 2022
16 Feb 24, 2022
43 Dec 28, 2022
14 Aug 24, 2022
540 Dec 30, 2022
10 Jul 1, 2022
14 Jan 3, 2023
45 Nov 29, 2022
78 Dec 12, 2022
20 Dec 12, 2022
6 Apr 29, 2022