ReConsider is a re-ranking model that re-ranks the top-K (passage, answer-span) predictions of an Open-Domain QA Model like DPR (Karpukhin et al., 2020).

Overview

ReConsider

ReConsider is a re-ranking model that re-ranks the top-K (passage, answer-span) predictions of an Open-Domain QA Model like DPR (Karpukhin et al., 2020).

The technical details are described in:

@inproceedings{iyer2020reconsider,
 title={RECONSIDER: Re-Ranking using Span-Focused Cross-Attention for Open Domain Question Answering},
 author={Iyer, Srinivasan and Min, Sewon and Mehdad, Yashar and Yih, Wen-tau},
 booktitle={NAACL},
 year={2021}
}

https://arxiv.org/abs/2010.10757

LICENSE

The majority of ReConsider is licensed under CC-BY-NC, however portions of the project are available under separate license terms: huggingface transformers and HotpotQA Utils are licensed under the Apache 2.0 license.

Re-producing results from the paper

The ReConsider models in the paper are trained on the top-100 predictions from the DPR Retriever + Reader model (Karpukhin et al., 2020) on four datasets: NaturalQuestions, TriviaQA, Trec, and WebQ.

We outline all the steps here for NaturalQuestions, but the same steps can be followed for the other datasets.

  1. Environment Setup
pip install -r requirements.txt
  1. [optional] Get the top-100 retrieved passages for each question using the best DPR retriever model for the NQ train, dev, and test sets. We provide these in our repo, but alternatively, you can obtain them by training the DPR retriever from scratch (from here). You can skip this entire step if you are only running ReConsider.
wget http://dl.fbaipublicfiles.com/reconsider/dpr_retriever_outputs/{nq|webq|trec|tqa}-{train|dev|test}-multi.json
  1. [optional] Get the top-100 predictions from the DPR reader (Karpukhin et al., 2020) executed on the output of the DPR retriever, on the NQ train, dev, and test sets. We provide these in our repo, but alternatively, you can obtain them by training the DPR reader from scratch (from here). You can skip this entire step if you are only running ReConsider.
wget http://dl.fbaipublicfiles.com/reconsider/dpr_reader_outputs/ttttt_{train|dev|test}.{nq|tqa|trec|webq}.{bbase|blarge}.output.nopp.title.json
  1. [optional] Convert DPR reader predictions to the marked-passage format required by ReConsider.
python prepare_marked_dataset.py --answer_json ttttt__train.{nq|tqa|trec|webq}.{bbase|blarge}.output.nopp.title.json --orig_json {nq|webq|trec|tqa}-train-multi.json --out_json paraphrase_selection_train.{nq|tqa|trec|webq}.{bbase|blarge}.100.qp_mp.nopp.title.json --train_M 100

python prepare_marked_dataset.py --answer_json ttttt_dev.{nq|tqa|trec|webq}.{bbase|blarge}.output.nopp.title.json --orig_json {nq|webq|trec|tqa}-dev-multi.json --out_json paraphrase_selection_dev.{nq|tqa|trec|webq}.{bbase|blarge}.5.qp_mp.nopp.title.json --dev --test_M 5

python prepare_marked_dataset.py --answer_json ttttt_test.{nq|tqa|trec|webq}.{bbase|blarge}.output.nopp.title.json --orig_json {nq|webq|trec|tqa}-test-multi.json --out_json paraphrase_selection_test.{nq|tqa|trec|webq}.{bbase|blarge}.5.qp_mp.nopp.title.json --dev --test_M 5

We also provide these files, so that you don't need to execute this command. You can directly download the output files using:

wget http://dl.fbaipublicfiles.com/reconsider/reconsider_inputs/paraphrase_selection_{train|dev|test}.{nq|tqa|trec|webq}.{bbase|blarge}.qp_mp.nopp.title.json
  1. Train ReConsider Models For Base models:
dset={nq|tqa|trec|webq}
python main.py --do_train --output_dir ps.$dset.bbase --train_file paraphrase_selection_train.$dset.bbase.qp_mp.nopp.title.json --predict_file paraphrase_selection_dev.$dset.bbase.qp_mp.nopp.title.json --train_batch_size 16 --predict_batch_size 144 --eval_period 500 --threads 80 --pad_question --max_question_length 0 --max_passage_length 240 --train_M 30 --test_M 5

For Large models:

dset={nq|tqa|trec|webq}
python main.py --do_train --output_dir ps.$dset.bbase --train_file paraphrase_selection_train.$dset.bbase.qp_mp.nopp.title.json --predict_file paraphrase_selection_dev.$dset.bbase.qp_mp.nopp.title.json --train_batch_size 16 --predict_batch_size 144 --eval_period 500 --threads 80 --pad_question --max_question_length 0 --max_passage_length 240 --train_M 10 --test_M 5 --bert_name bert-large-uncased

Note: If training on Trec or Webq, initialize the model with the model trained on NQ of the corresponding size by adding this parameter: --checkpoint $model_nq_{bbase|blarge}. You can either train this NQ model using the commands above, or directly download it as described below:

We also provide our pre-trained models for download, using this script:

python download_reconsider_models.py --model {nq|trec|tqa|webq}_{bbase|blarse}
  1. Predict on the test set using ReConsider Models
python main.py --do_predict --output_dir /tmp/ --predict_file paraphrase_selection_test.{nq|trec|webq|tqa}.{bbase|blarge}.qp_mp.nopp.title.json  --checkpoint {path_to_model} --predict_batch_size 72 --threads 80 --n_paragraphs 100  --verbose --prefix test_  --pad_question --max_question_length 0 --max_passage_length 240 --predict_batch_size 72 --test_M 5 --bert_name {bert-base-uncased|bert-large-uncased}
You might also like...
The code for two papers: Feedback Transformer and Expire-Span.

transformer-sequential This repo contains the code for two papers: Feedback Transformer Expire-Span The training code is structured for long sequentia

SpanNER: Named EntityRe-/Recognition as Span Prediction
SpanNER: Named EntityRe-/Recognition as Span Prediction

SpanNER: Named EntityRe-/Recognition as Span Prediction Overview | Demo | Installation | Preprocessing | Prepare Models | Running | System Combination

A sample pytorch Implementation of ACL 2021 research paper "Learning Span-Level Interactions for Aspect Sentiment Triplet Extraction".

Span-ASTE-Pytorch This repository is a pytorch version that implements Ali's ACL 2021 research paper Learning Span-Level Interactions for Aspect Senti

naked is a Python tool which allows you to strip a model and only keep what matters for making predictions.

naked is a Python tool which allows you to strip a model and only keep what matters for making predictions. The result is a pure Python function with no third-party dependencies that you can simply copy/paste wherever you wish.

🥈78th place in Riiid Answer Correctness Prediction competition

Riiid Answer Correctness Prediction Introduction This repository is the code that placed 78th in Riiid Answer Correctness Prediction competition. Requ

Transformer part of 12th place solution in Riiid! Answer Correctness Prediction

kaggle_riiid Transformer part of 12th place solution in Riiid! Answer Correctness Prediction. Please see here for more information. Execution You need

Predicts an answer in yes or no.

Oui-ou-non-prediction Predicts an answer in 'yes' or 'no'. It is based on the game 'effeuiller la marguerite' in which the person plucks flower petals

Wordle-solver - Wordle answer generation program in python
Wordle-solver - Wordle answer generation program in python

🟨 Wordle Solver 🟩 Wordle answer generation program in python ✔️ Requirements U

nfelo: a power ranking, prediction, and betting model for the NFL

nfelo nfelo is a power ranking, prediction, and betting model for the NFL. Nfelo take's 538's Elo framework and further adapts it for the NFL, hence t

Comments
  • About the prediction file

    About the prediction file

    Hello, I use the checkpoint model and prediction file you provided. The following instructions are executed: --do_predict --output_dir ./output --predict_file "./data/paraphrase_selection_dev.nq.bbase.qp_mp.nopp.title.json" --checkpoint "./checkpoint/nq-base.PT" --predict_batch_size 72 --threads 80 --n_paragraphs 100 --verbose --prefix test_ --pad_question --max_question_length 0 --max_passage_length 240 --predict_batch_size 72 --test_M 5 --bert_name bert-base-uncased

    But I get the following error. image

    This is because in the feature's file the 'positive_input_ids' is 'None'. image

    Could you tell me how I can solve this mistake? Thanks!

    opened by WildLight 0
Owner
Facebook Research
Facebook Research
A multilingual version of MS MARCO passage ranking dataset

mMARCO A multilingual version of MS MARCO passage ranking dataset This repository presents a neural machine translation-based method for translating t

null 75 Dec 27, 2022
Wandb-predictions - WANDB Predictions With Python

WANDB API CI/CD Below we capture the CI/CD scenarios that we would expect with o

Anish Shah 6 Oct 7, 2022
An efficient and effective learning to rank algorithm by mining information across ranking candidates. This repository contains the tensorflow implementation of SERank model. The code is developed based on TF-Ranking.

SERank An efficient and effective learning to rank algorithm by mining information across ranking candidates. This repository contains the tensorflow

Zhihu 44 Oct 20, 2022
A pytorch implementation of Reading Wikipedia to Answer Open-Domain Questions.

DrQA A pytorch implementation of the ACL 2017 paper Reading Wikipedia to Answer Open-Domain Questions (DrQA). Reading comprehension is a task to produ

Runqi Yang 394 Nov 8, 2022
This is the code for our KILT leaderboard submission to the T-REx and zsRE tasks. It includes code for training a DPR model then continuing training with RAG.

KGI (Knowledge Graph Induction) for slot filling This is the code for our KILT leaderboard submission to the T-REx and zsRE tasks. It includes code fo

International Business Machines 72 Jan 6, 2023
This is the official implementation of "One Question Answering Model for Many Languages with Cross-lingual Dense Passage Retrieval".

CORA This is the official implementation of the following paper: Akari Asai, Xinyan Yu, Jungo Kasai and Hannaneh Hajishirzi. One Question Answering Mo

Akari Asai 59 Dec 28, 2022
SymPy-powered, Wolfram|Alpha-like answer engine totally in your browser, without backend computation

SymPy Beta SymPy Beta is a fork of SymPy Gamma. The purpose of this project is to run a SymPy-powered, Wolfram|Alpha-like answer engine totally in you

Liumeo 25 Dec 21, 2022
Pomodoro timer that acknowledges the inexorable, infinite passage of time

Pomodouroboros Most pomodoro trackers assume you're going to start them. But time and tide wait for no one - the great pomodoro of the cosmos is cold

Glyph 66 Dec 13, 2022
Stroke-predictions-ml-model - Machine learning model to predict individuals chances of having a stroke

stroke-predictions-ml-model machine learning model to predict individuals chance

Alex Volchek 1 Jan 3, 2022
Adaptive Attention Span for Reinforcement Learning

Adaptive Transformers in RL Official implementation of Adaptive Transformers in RL In this work we replicate several results from Stabilizing Transfor

null 100 Nov 15, 2022