Tracking Progress in Question Answering over Knowledge Graphs

Related tags

Deep Learning benchmark dataset question-answering kgqa kbqa
Overview

Tracking Progress in Question Answering over Knowledge Graphs

Table of contents

Question Answering Systems with Descriptions

The QA Systems Table contains links to publications, demo/APIs (if available) and short descriptions of ca. 100 QA systems.

DBpedia

Wikidata

Freebase

Other KGs

This leaderboard aims to provide a central place to compare the capabilities of different Knowledge Graph Question Answering (KGQA) approaches. It gives a global view of the state-of-the-art (SOTA) across many KGQA benchmarks.

Using a global and open resource, trusting evaluation results will be easier. In particular, we want to close gaps in evaluation campaigns to avoid incomplete or missing comparisons. The ultimate goal is to prevent a replication crisis before it even starts.

Contributing

Adding a new result

If you would like to add a new result, you can just click on the small edit button in the top-right corner of the file for the respective dataset. This allows you to edit the file in Markdown. Simply add a row to the corresponding table in the same format. Make sure that the table stays sorted (with the best result on top). After you've made your change, make sure that the table still looks ok by clicking on the "Preview changes" tab at the top of the page. If everything looks good, go to the bottom of the page, where you see the below form.

Add a name for your proposed change, an optional description, indicate that you would like to "Create a new branch for this commit and start a pull request", and click on "Propose file change".

Adding a new dataset or task

For adding a new dataset or task, you can also follow the steps above. Alternatively, you can fork the repository. In both cases, follow the steps below:

  1. If your dataset is completely new, create a new file and link to it in the table of contents above.
  2. Briefly describe the dataset and include relevant references.
  3. Describe the evaluation setting and evaluation metric.
  4. Show how an annotated example of the dataset looks like.
  5. Add a download link if available.
  6. Copy the below table and fill in at least two results (including the state-of-the-art) for your dataset (change Metric1/Metric2/Metric3 to the metric of your dataset).
  7. Submit your change as a pull request.
Model / System Year Metric1 Metric2 Metric3 Reported by

Instructions for building the site locally

Instructions for building the website locally using Jekyll can be found here.

Citation

Please cite the following:

Perevalov, A., Yan, X., Kovriguina, L., Jiang, L., Both, A., & Usbeck, R. (2022). Knowledge Graph Question Answering Leaderboard: A Community Resource to Prevent a Replication Crisis. arXiv preprint arXiv:2201.08174.

Acknowledgement

This site is based on https://nlpprogress.com/ and thus, a great thanks goes to Sebastian Ruder.

Comments
  • Add KSEM papers

    Add KSEM papers

    • https://link.springer.com/chapter/10.1007/978-3-031-10983-6_15
    • https://link.springer.com/chapter/10.1007/978-3-031-10983-6_11
    • https://link.springer.com/chapter/10.1007/978-3-031-10983-6_12
    • https://link.springer.com/chapter/10.1007/978-3-031-10983-6_39
    opened by RicardoUsbeck 9
  • Add qald-9-plus dataset (dbpdia ver) and eval

    Add qald-9-plus dataset (dbpdia ver) and eval

    Add qald-9-plus dataset (dbpdia ver) and eval based on paper https://arxiv.org/pdf/2202.00120v2.pdf It turns out this paper: https://dl.acm.org/doi/pdf/10.1145/3485447.3511940 is only tested on the qald-9-plus wikidata version.

    opened by xixi019 8
  • Fixing overflowing table

    Fixing overflowing table

    Fixing style bug where table would overflow the page. Added break-word wrapping to make the URLs go into new line and not overflow the table as a result of the adjustment.

    opened by rudygarrido 3
  • add https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9893129

    add https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9893129

    Efficient SPARQL Queries Generator for Question Answering Systems YI-HUI CHEN1, 2, (Member, IEEE), ERIC JUI-LIN LU3 , (Member, IEEE), and YING-YEN LIN4

    opened by RicardoUsbeck 2
  • Add papers to leaderboard April

    Add papers to leaderboard April

    • https://assets.researchsquare.com/files/rs-1563040/v1_covered.pdf?c=1650473022
    • https://arxiv.org/pdf/2204.10194.pdf
    • https://arxiv.org/pdf/2204.10629.pdf
    opened by RicardoUsbeck 2
  • Download link not working for MLPQ dataset

    Download link not working for MLPQ dataset

    opened by acxcv 1
Owner
Knowledge Graph Question Answering
Knowledge Graph Question Answering
GitHub https://kgqa.github.io/leaderboard/
Codes for NAACL 2021 Paper "Unsupervised Multi-hop Question Answering by Question Generation"

Unsupervised-Multi-hop-QA This repository contains code and models for the paper: Unsupervised Multi-hop Question Answering by Question Generation (NA

Liangming Pan 70 Nov 27, 2022
improvement of CLIP features over the traditional resnet features on the visual question answering, image captioning, navigation and visual entailment tasks.

CLIP-ViL In our paper "How Much Can CLIP Benefit Vision-and-Language Tasks?", we show the improvement of CLIP features over the traditional resnet fea

null 310 Dec 28, 2022
MOT-Tracking-by-Detection-Pipeline - For Tracking-by-Detection format MOT (Multi Object Tracking), is it a framework that separates Detection and Tracking processes?

MOT-Tracking-by-Detection-Pipeline Tracking-by-Detection形式のMOT(Multi Object Trac

KazuhitoTakahashi 41 Nov 23, 2022
RNG-KBQA: Generation Augmented Iterative Ranking for Knowledge Base Question Answering

RNG-KBQA: Generation Augmented Iterative Ranking for Knowledge Base Question Answering Authors: Xi Ye, Semih Yavuz, Kazuma Hashimoto, Yingbo Zhou and

Salesforce 72 Dec 5, 2022
Continuous Query Decomposition for Complex Query Answering in Incomplete Knowledge Graphs

Continuous Query Decomposition This repository contains the official implementation for our ICLR 2021 (Oral) paper, Complex Query Answering with Neura

UCL Natural Language Processing 71 Dec 29, 2022
ConE: Cone Embeddings for Multi-Hop Reasoning over Knowledge Graphs

ConE: Cone Embeddings for Multi-Hop Reasoning over Knowledge Graphs This is the code of paper ConE: Cone Embeddings for Multi-Hop Reasoning over Knowl

MIRA Lab 33 Dec 7, 2022
Designing a Minimal Retrieve-and-Read System for Open-Domain Question Answering (NAACL 2021)

Designing a Minimal Retrieve-and-Read System for Open-Domain Question Answering Abstract In open-domain question answering (QA), retrieve-and-read mec

Clova AI Research 34 Apr 13, 2022
GrailQA: Strongly Generalizable Question Answering

GrailQA is a new large-scale, high-quality KBQA dataset with 64,331 questions annotated with both answers and corresponding logical forms in different syntax (i.e., SPARQL, S-expression, etc.). It can be used to test three levels of generalization in KBQA: i.i.d., compositional, and zero-shot.

OSU DKI Lab 76 Dec 21, 2022
Binary Passage Retriever (BPR) - an efficient passage retriever for open-domain question answering

BPR Binary Passage Retriever (BPR) is an efficient neural retrieval model for open-domain question answering. BPR integrates a learning-to-hash techni

Studio Ousia 147 Dec 7, 2022
covid question answering datasets and fine tuned models

Covid-QA Fine tuned models for question answering on Covid-19 data. Hosted Inference This model has been contributed to huggingface.Click here to see

Abhijith Neil Abraham 19 Sep 9, 2021
NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions (CVPR2021)

NExT-QA We reproduce some SOTA VideoQA methods to provide benchmark results for our NExT-QA dataset accepted to CVPR2021 (with 1 'Strong Accept' and 2

Junbin Xiao 50 Nov 24, 2022
FeTaQA: Free-form Table Question Answering

FeTaQA: Free-form Table Question Answering FeTaQA is a Free-form Table Question Answering dataset with 10K Wikipedia-based {table, question, free-form

Language, Information, and Learning at Yale 40 Dec 13, 2022
Pytorch implementation for the EMNLP 2020 (Findings) paper: Connecting the Dots: A Knowledgeable Path Generator for Commonsense Question Answering

Path-Generator-QA This is a Pytorch implementation for the EMNLP 2020 (Findings) paper: Connecting the Dots: A Knowledgeable Path Generator for Common

Peifeng Wang 33 Dec 5, 2022
This is the official implementation of "One Question Answering Model for Many Languages with Cross-lingual Dense Passage Retrieval".

CORA This is the official implementation of the following paper: Akari Asai, Xinyan Yu, Jungo Kasai and Hannaneh Hajishirzi. One Question Answering Mo

Akari Asai 59 Dec 28, 2022
Bilinear attention networks for visual question answering

Bilinear Attention Networks This repository is the implementation of Bilinear Attention Networks for the visual question answering and Flickr30k Entit

Jin-Hwa Kim 506 Nov 29, 2022
Visual Question Answering in Pytorch

Visual Question Answering in pytorch /!\ New version of pytorch for VQA available here: https://github.com/Cadene/block.bootstrap.pytorch This repo wa

Remi 672 Jan 1, 2023
This reporistory contains the test-dev data of the paper "xGQA: Cross-lingual Visual Question Answering".

This reporistory contains the test-dev data of the paper "xGQA: Cross-lingual Visual Question Answering".

AdapterHub 18 Dec 9, 2022
EMNLP 2021: Single-dataset Experts for Multi-dataset Question-Answering

MADE (Multi-Adapter Dataset Experts) This repository contains the implementation of MADE (Multi-adapter dataset experts), which is described in the pa

Princeton Natural Language Processing 68 Jul 18, 2022
EMNLP 2021: Single-dataset Experts for Multi-dataset Question-Answering

MADE (Multi-Adapter Dataset Experts) This repository contains the implementation of MADE (Multi-adapter dataset experts), which is described in the pa

Princeton Natural Language Processing 39 Oct 5, 2021