PyTorch reimplementation of REALM and ORQA

Related tags

Deep Learning pytorch-realm-orqa
Overview

PyTorch Reimplementation of REALM and ORQA

This is PyTorch reimplementation of REALM (paper, codebase) and ORQA (paper, codebase).

Some features have not been implemented yet, currently the predictor and finetuning script are available.

The term retriever and searcher in the code are basically interchangeable, their difference is that retriever is for REALM pretraining, and searcher is for ORQA finetuning.

Prerequisite

cd transformers && pip install -U -e ".[dev]"
pip install -U scann, apache_beam

Data

To download pretrained checkpoints and preprocessed data, please follow the instructions below:

cd data
pip install -U -r requirements.txt
sh download.sh

Finetune (Experimental)

The default finetuning dataset is Natural Question(NQ). To laod your custom dataset, please change the loading function in data.py.

Training:

python run_finetune.py --is_train \
    --model_dir "./" \
    --num_epochs 2 \
    --device cuda

Evaluation:

python run_finetune.py \
    --retriever_pretrained_name "retriever" \
    --checkpoint_pretrained_name "reader" \
    --model_dir "./" \
    --device cuda

Predict

The default checkpoints of retriever and reader are orqa_nq_model_from_realm. To change them, kindly specify --retriever_path and --checkpoint_path.

python predictor.py --question "Who is the pioneer in modern computer science?"

Output: alan mathison turing

License

Apache License 2.0

Comments
  • Parameters of the retriever in fine-tuning

    Parameters of the retriever in fine-tuning

    Hi! I am wondering why the retriever is frozen during fine-tuning time. I think the retriever will learn more in fine-tuning. I am not very familiar with tensorflow. Is it possible to update the parameters of the retriever during fine-tuning time with this repository? How?

    opened by catalwaysright 17
  • run_finetune takes too much time.

    run_finetune takes too much time.

    Under the default settings, one epoch cost 700+h on one Tesla V100, which makes training impossible. I found the retriever will select top 5000 docs and tokenize them, which takes up most of the time. How to fix it? Can I make beam_size smaller?

    opened by Boltzmachine 11
  • Not Found for url: https://huggingface.co/google/realm-cc-news-pretrained-openqa/resolve/main/block_records.npy

    Not Found for url: https://huggingface.co/google/realm-cc-news-pretrained-openqa/resolve/main/block_records.npy

    Traceback (most recent call last): File "run_finetune.py", line 305, in main(args) File "run_finetune.py", line 127, in main openqa = get_openqa(args, config) File "/home/chuanqi.tcq/qwk/pytorch-realm-orqa/model.py", line 99, in get_openqa retriever = RealmRetriever.from_pretrained(args.checkpoint_pretrained_name) File "/home/chuanqi.tcq/anaconda3/envs/MRC-KI/lib/python3.7/site-packages/transformers/models/realm/retrieval_realm.py", line 114, in from_pretrained repo_id=pretrained_model_name_or_path, filename=_REALM_BLOCK_RECORDS_FILENAME, **kwargs File "/home/chuanqi.tcq/anaconda3/envs/MRC-KI/lib/python3.7/site-packages/huggingface_hub/file_download.py", line 545, in hf_hub_download local_files_only=local_files_only, File "/home/chuanqi.tcq/anaconda3/envs/MRC-KI/lib/python3.7/site-packages/huggingface_hub/file_download.py", line 352, in cached_download r.raise_for_status() File "/home/chuanqi.tcq/anaconda3/envs/MRC-KI/lib/python3.7/site-packages/requests/models.py", line 960, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/google/realm-cc-news-pretrained-openqa/resolve/main/block_records.npy

    opened by Boltzmachine 1
  • Official EM and Finetuning program

    Official EM and Finetuning program

    Hi, Thanks for the great work! It really helps me a lot.

    I am curious about the difference between EM and official EM. And what is to be improved in run_finetune.py considering its experimental flag. Thanks so much

    opened by Hannibal046 6
  • Does the code supports for the entire end-to-end fine-tuning including the retriever ?

    Does the code supports for the entire end-to-end fine-tuning including the retriever ?

    The REALM paper highlights that for downstream tasks they kept the retriever frozen. What about a task like domain-specific open domain question answering? In that kind of a scenario can we train the entire REALM with this code.

    if yes: we might able to compare results with RAG-end2end

    https://github.com/huggingface/transformers/tree/master/examples/research_projects/rag-end2end-retriever

    opened by shamanez 7
Owner
Li-Huai (Allan) Lin
DL & NLP Enthusiast
Li-Huai (Allan) Lin
GitHub
PyTorch reimplementation of the Smooth ReLU activation function proposed in the paper "Real World Large Scale Recommendation Systems Reproducibility and Smooth Activations" [arXiv 2022].

Smooth ReLU in PyTorch Unofficial PyTorch reimplementation of the Smooth ReLU (SmeLU) activation function proposed in the paper Real World Large Scale

Christoph Reich 10 Jan 2, 2023
text_recognition_toolbox: The reimplementation of a series of classical scene text recognition papers with Pytorch in a uniform way.

text recognition toolbox 1. 项目介绍 该项目是基于pytorch深度学习框架，以统一的改写方式实现了以下6篇经典的文字识别论文，论文的详情如下。该项目会持续进行更新，欢迎大家提出问题以及对代码进行贡献。 模型 论文标题 发表年份 模型方法划分 CRNN 《An End-t

null 168 Dec 24, 2022
PyTorch reimplementation of minimal-hand (CVPR2020)

Minimal Hand Pytorch Unofficial PyTorch reimplementation of minimal-hand (CVPR2020). you can also find in youtube or bilibili bare hand youtube or bil

Hao Meng 228 Dec 29, 2022
PyTorch reimplementation of the paper Involution: Inverting the Inherence of Convolution for Visual Recognition [CVPR 2021].

Involution: Inverting the Inherence of Convolution for Visual Recognition Unofficial PyTorch reimplementation of the paper Involution: Inverting the I

Christoph Reich 100 Dec 1, 2022
PyTorch reimplementation of hand-biomechanical-constraints (ECCV2020)

Hand Biomechanical Constraints Pytorch Unofficial PyTorch reimplementation of Hand-Biomechanical-Constraints (ECCV2020). This project reimplement foll

Hao Meng 59 Dec 20, 2022
A PyTorch Reimplementation of TecoGAN: Temporally Coherent GAN for Video Super-Resolution

TecoGAN-PyTorch Introduction This is a PyTorch reimplementation of TecoGAN: Temporally Coherent GAN for Video Super-Resolution (VSR). Please refer to

null 165 Dec 17, 2022
a reimplementation of Optical Flow Estimation using a Spatial Pyramid Network in PyTorch

pytorch-spynet This is a personal reimplementation of SPyNet [1] using PyTorch. Should you be making use of this work, please cite the paper according

Simon Niklaus 269 Jan 2, 2023
a reimplementation of UnFlow in PyTorch that matches the official TensorFlow version

pytorch-unflow This is a personal reimplementation of UnFlow [1] using PyTorch. Should you be making use of this work, please cite the paper according

Simon Niklaus 134 Nov 20, 2022
a reimplementation of LiteFlowNet in PyTorch that matches the official Caffe version

pytorch-liteflownet This is a personal reimplementation of LiteFlowNet [1] using PyTorch. Should you be making use of this work, please cite the paper

Simon Niklaus 365 Dec 31, 2022
a reimplementation of Holistically-Nested Edge Detection in PyTorch

pytorch-hed This is a personal reimplementation of Holistically-Nested Edge Detection [1] using PyTorch. Should you be making use of this work, please

Simon Niklaus 375 Dec 6, 2022
Pytorch reimplementation of PSM-Net: "Pyramid Stereo Matching Network"

This is a Pytorch Lightning version PSMNet which is based on JiaRenChang/PSMNet. use python main.py to start training. PSM-Net Pytorch reimplementatio

XIAOTIAN LIU 1 Nov 25, 2021
[ECCV 2020] Reimplementation of 3DDFAv2, including face mesh, head pose, landmarks, and more.

Stable Head Pose Estimation and Landmark Regression via 3D Dense Face Reconstruction Reimplementation of (ECCV 2020) Towards Fast, Accurate and Stable

Remilia Scarlet 221 Dec 30, 2022
Reimplementation of the paper `Human Attention Maps for Text Classification: Do Humans and Neural Networks Focus on the Same Words? (ACL2020)`

Human Attention for Text Classification Re-implementation of the paper Human Attention Maps for Text Classification: Do Humans and Neural Networks Foc

Shunsuke KITADA 15 Dec 13, 2021
An official reimplementation of the method described in the INTERSPEECH 2021 paper - Speech Resynthesis from Discrete Disentangled Self-Supervised Representations.

Speech Resynthesis from Discrete Disentangled Self-Supervised Representations Implementation of the method described in the Speech Resynthesis from Di

Facebook Research 253 Jan 6, 2023
Reimplementation of the paper "Attention, Learn to Solve Routing Problems!" in jax/flax.

JAX + Attention Learn To Solve Routing Problems Reinplementation of the paper Attention, Learn to Solve Routing Problems! using Jax and Flax. Fully su

Gabriela Surita 7 Dec 1, 2022
Reimplementation of Dynamic Multi-scale filters for Semantic Segmentation.

Paddle implementation of Dynamic Multi-scale filters for Semantic Segmentation.

Hongqiang.Wang 2 Nov 1, 2021
Unofficial reimplementation of ECAPA-TDNN for speaker recognition (EER=0.86 for Vox1_O when train only in Vox2)

Introduction This repository contains my unofficial reimplementation of the standard ECAPA-TDNN, which is the speaker recognition in VoxCeleb2 dataset

Tao Ruijie 277 Dec 31, 2022
Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

Pytorch Lightning 1.4k Jan 1, 2023
Generic template to bootstrap your PyTorch project with PyTorch Lightning, Hydra, W&B, and DVC.

NN Template Generic template to bootstrap your PyTorch project. Click on Use this Template and avoid writing boilerplate code for: PyTorch Lightning,

Luca Moschella 520 Dec 30, 2022