Explainable Fact Checking: A Survey

This repository and the accompanying webpage contain resources for the paper "Explainable Fact Checking: A Survey". In the paper, we offer a critical review of the state-of-the-art in automated fact-checking with a particular focus on interpretability.

Reference

If you find our work useful, please cite the paper as formatted below.

  @inproceedings{kotonya-toni-2020-explainable-automated,
      title = "Explainable Automated Fact-Checking: A Survey",
      author = "Kotonya, Neema  and
        Toni, Francesca",
      booktitle = "Proceedings of the 28th International Conference on Computational Linguistics",
      month = dec,
      year = "2020",
      address = "Barcelona, Spain (Online)",
      publisher = "International Committee on Computational Linguistics",
      url = "https://www.aclweb.org/anthology/2020.coling-main.474",
      pages = "5430--5443"
  }

Contents

Here is an overview of papers mentioned in this work, and more recent papers which have been added.

• Introduction
• Task Formulations
• Datasets
  • Naturally occurring claims
    • From social media
    • From fact-checking and news websites
  • Hand crafted claims
    • From Wikipedia
    • From scientific journals
• Fact Checking Systems
  • By dataset
    • LIAR dataset
    • FEVER dataset
    • MultiFC dataset
  • By method
    • Support Vector Machines (SVMs)
    • Convolutional Neural Networks (CNNs)
    • Recurrent Neural Networks (RNNs)
    • Attention Networks
    • Hybrid
• Shared Tasks
• Explainable Fact Checking
  • Systems
    • Saliency maps
    • Rule discovery and graph based methods
    • Summarization
  • Adversarial Training for Fact Checking
  • Error Correction of Claims
  • Multi-hop Fact Checking
  • Fact Checking with Intrepretable Features
• Surveys
• Tutorials

Introduction

Fact checking is the process of establishing the veracity of claims i.e., to distinguish between false stories (e.g., misattributions, rumours, hoaxes) and facts.

Over the past few years the use of deep learning methods for fact checking and fake news detection have become a popular. Indeed, several exciting breakthroughs have occurred in automated fact checking thanks in large part due to new datasets (e.g., FEVER) and advances in machine learning for NLP. However there are still some limitations in this research area, the one we focus on in this work in our work is explanations for automated fact checking.

The pipeline commonly employed for automated fact-checking consists of four parts (subtasks). We propose that post-hoc explanations are an important and necessary extension of this pipeline.

For an overview of the data and results mentioned in our survey, please visit this webpage.

Task Formulations

Here we list papers which address varied tasks related to fact checking and fake news detection.

• Check-worthy Claim Detection

  • Toward Automated Fact-Checking: Detecting Check-worthy Factual Claims by ClaimBuster (Hassan et al., 2017). [paper] [bib]

• Fauxtography and Multimodal Fake News Detection

  • FauxBuster: A Content-free Fauxtography Detector Using Social Media Comments (Zhang et al., 2018). [paper] [bib] [slides]
  • Fact-Checking Meets Fauxtography: Verifying Claims About Images (Zlatkova et al., 2019). [paper] [bib]
  • Eann: Event adversarial neural networks for multi-modal fake news detection (Wang et al., 2018). [paper] [bib]

• Identifying Previously Fact-Checked Claims

  • That is a Known Lie: Detecting Previously Fact-Checked Claims (Shaar et al., 2020). [paper] [bib]
  • Where Are the Facts? Searching for Fact-checked Information to Alleviate the Spread of Fake News (Vo and Lee, 2020). [paper] [bib]

• Neural Fake News Detection

  • Defending against neural fake news (Zellers et al., 2019). [paper] [bib]
  • Detecting Cross-Modal Inconsistency to Defend Against Neural Fake News (Tan et al., 2020). [paper] [bib]

• Rumour Verification and Resolution

  • SemEval-2019 Task 7: RumourEval, Determining Rumour Veracity and Support for Rumours (Gorrell et al., 2019). [paper] [bib]
  • Can Rumour Stance Alone Predict Veracity? (Dungs et al., 2018). [paper] [bib]
  • SemEval-2017 Task 8: RumourEval: Determining rumour veracity and support for rumours (Derczynski et al., 2017). [paper] [bib]

• Stylometric Analysis of News Articles

  • A stylometric inquiry into hyperpartisan and fake news (Potthast et al., 2017) [paper] [bib] [video]
  • Truth of Varying Shades: Analyzing Language in Fake News and Political Fact-Checking (Rashkin et al., 2017). [paper] [bib]

• Table-based Fact Verification

  • TabFact: A Large-scale Dataset for Table-based Fact Verification (Chen et al., 2020). [paper] [bib]

• Multi-hop Fact Checking

• Error Correction of Claims

Fact Checking Datasets

List of fact checking, rumour verification and fake news detection datasets:

Datasets of naturally occurring claims

Social media

Claims from social media platforms sources e.g., Twitter, Facebook.

• r/Fakeddit: A New Multimodal Benchmark Dataset for Fine-grained Fake News Detection (Nakamura et al., 2020). [paper] [data] [bib]
• SemEval-2019 Task 7: RumourEval, Determining Rumour Veracity and Support for Rumours (Gorrell et al., 2019). [paper] [data] [bib]
• All-in-one: Multi-task Learning for Rumour Verification (Kochkina et al., 2018). [paper] [data]† [bib]
• SemEval-2017 Task 8: RumourEval: Determining rumour veracity and support for rumours (Derczynski et al., 2017). [paper] [data] [bib]
• Hyperpartisan Facebook Pages Are Publishing False And Misleading Information At An Alarming Rate (Silverman et al., 2017). [article] [data]
• Detect Rumors in Microblog Posts Using Propagation Structure via Kernel Learning (Ma et al., 2017). [paper] [data] [bib]
• Analysing How People Orient to and Spread Rumours in Social Media by Looking at Conversational Threads (Zubiaga et al., 2016). [paper] [data] [bib]
• CREDBANK: A Large-Scale Social Media Corpus with Associated Credibility Annotations (Mitra and Gilbert, 2015). [paper] [data] [bib]

† This dataset is an extention of the PHEME rumour dataset.

Fact checking and news websites

Claims for news and fact-checking platforms e.g., Snopes, Politifact.

• Explainable Automated Fact-Checking for Public Health Claims (Kotonya and Toni, 2020). [paper] [data] [bib]
• STANDER: An Expert-Annotated Dataset for News Stance Detection and Evidence Retrieval (Conforti et al., 2020). [paper] [data] [bib]
• FakeCovid-- A Multilingual Cross-domain Fact Check News Dataset for COVID-19 (Shahi and Nandini, 2020). [paper] [data] [bib]
• MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims (Augenstein et al., 2019). [paper] [bib] [data]
• A Richly Annotated Corpus for Different Tasks in Automated Fact-Checking (Hanselowski et al., 2019). [paper] [code] [data] [bib]
• Integrating Stance Detection and Fact Checking in a Unified Corpus (Baly et al., 2018). [paper] [data] [bib]
• FakeNewsNet: A Data Repository with News Content, Social Context and Spatialtemporal Information for Studying Fake News on Social Media (Shu et al., 2018). [paper] [data]
• “Liar, Liar Pants on Fire”: A New Benchmark Dataset for Fake News Detection (Wang, 2017). [paper] [data] [bib]
• Truth of Varying Shades: Analyzing Language in Fake News and Political Fact-Checking (Rashkin et al., 2017). [paper] [data] [bib]
• The Fake News Challenge (Pomerleau and Rao, 2017) [data]

Hand crafted

This covers claims which are generated manually e.g. through re-writing statements.

Wikipedia

• TabFact: A Large-scale Dataset for Table-based Fact Verification (Chen et al., 2020). [paper] [data] [bib]
• FEVER: a Large-scale Dataset for Fact Extraction and VERification (Thorne et al., 2018). [paper] [data] [bib]
• Automated Fact-Checking of Claims from Wikipedia (Sathe et al., 2020). [paper] [data] [bib]
• Generating Fact Checking Briefs (Fan et al., 2020). [paper] [bib]

Scientific journals

• Fact or Fiction: Verifying Scientific Claims (Wadden et al., EMNLP 2020). [paper] [data] [bib]

Fact Checking Systems

A list of fact-checking and fake news detection systems.

Systems by Dataset

LIAR

• Where is your Evidence: Improving Fact-checking by Justification Modeling (Alhindi et al., 2018). [paper] [bib] [code]
• Generating Fact Checking Explanations (Atanasova et al., 2020). [paper] [bib]

FEVER

• FEVER 1.0 Baseline
• Combining Fact Extraction and Verification with Neural Semantic Matching Networks (Nie et al., 2019). [paper] [bib] [code]
• UCL Machine Reading Group: Four Factor Framework For Fact Finding (HexaF) (Yoneda et al., 2018). [paper] [bib] [code]
• UKP-Athene: Multi-Sentence Textual Entailment for Claim Verification (Hanselowski et al., 2018). [paper] [bib] [code]
• Team Papelo: Transformer Networks at FEVER (Malon, 2018). [paper] [bib] [code]
• Team DOMLIN: Exploiting Evidence Enhancement for the FEVER Shared Task (Stammbach and Neumann, 2019). [paper] [bib] [code]
• GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification [paper] [bib] [code]
• Fine-grained Fact Verification with Kernel Graph Attention Network [paper] [bib] [code]

MultiFC

• Time-Aware Evidence Ranking for Fact-Checking (Allein et al., 2020). [paper] [bib]

Systems by Method

Support Vector Machines

• Fake News or Truth? Using Satirical Cues to Detect Potentially Misleading News. [paper] [bib]

Convolutional Neural Networks

• “Liar, Liar Pants on Fire”: A New Benchmark Dataset for Fake News Detection (Wang, 2017). [paper] [bib]
• FAKTA: An Automatic End-to-End Fact Checking System (Nadeeem et al., 2019). [paper] [bib]

Recurrent Neural Networks

• CSI: A Hybrid Deep Model for Fake News Detection (Ruchansky et al., 2017). [paper]
• DeClarE: Debunking Fake News and False Claims using Evidence-Aware Deep Learning (Popat et al., 2018). [paper] [bib]
• Truth of Varying Shades: Analyzing Language in Fake News and Political Fact-Checking (Rashkin et al., 2017). [paper] [bib]
• Where is your Evidence: Improving Fact-checking by Justification Modeling (Alhindi et al., 2018). [paper] [bib]

Transformers and Attention Networks

• Two Stage Transformer Model for COVID-19 Fake News Detection and Fact Checking (Vijjali et al., 2020). [paper] [bib]

Hybrid

• GCAN: Graph-aware Co-Attention Networks for Explainable Fake News Detection on Social Media (Lu and Li, 2020). [paper] [bib]
• DTCA: Decision Tree-based Co-Attention Networks for Explainable Claim Verification (Wu et al., 2020). [paper] [bib]
• XFake: Explainable Fake News Detector with Visualizations (Yang et al., 2019). [paper] [bib]

Shared Tasks

📣
indicates the shared task is ongoing!

• Statement Verification and Evidence Finding with Tables (SEM-TAB-FACT) [Wang et al., 2021] 📣 (Ends on Jan 29 2021)
• SciFact Claim Verifiation [Wadden et al., 2020] 📣
• Fakeddit Multimodal Fake News Detection Challenge [Nakamura et al., 2020] 📣 (Ends on Feb 16 2021)
• SemEval-2019 Task 7: RumourEval, Determining Rumour Veracity and Support for Rumours [Gorrell et al., 2019]
• SemEval-2019 Task 8: Fact Checking in Community Question Answering Forums [Mihaylova et al., 2019]
• The Fake News Challenge (FNC-1) [Pomerleau and Rao, 2017]
  • A Retrospective Analysis of the Fake News Challenge Stance-Detection Task [Hanselowski et al., 2018]
• The Fact Extraction and VERification (FEVER) Shared Task [Thorne et al., 2018]
• SemEval-2017 Task 8: RumourEval: Determining rumour veracity and support for rumours [Derczynski et al., 2017]

Explainable Fact Checking

Systems

A list of works on explainability in fact-checking and fake news detection:

Saliency maps

• dEFEND: Explainable Fake News Detection (Shu et al., 2019). [paper] [bib] [video]
• GCAN: Graph-aware Co-Attention Networks for Explainable Fake News Detection on Social Media (Lu and Li, 2020). [paper] [bib]
• DTCA: Decision Tree-based Co-Attention Networks for Explainable Claim Verification (Wu et al., 2020). [paper] [bib]
• DeClarE: Debunking Fake News and False Claims using Evidence-Aware Deep Learning (Popat et al., 2018). [paper] [bib]
• CredEye: A Credibility Lens for Analyzing and Explaining Misinformation (Popat et al., 2018). [paper] [bib] [demo]
• XFake: Explainable Fake News Detector with Visualizations (Yang et al., 2019). [paper] [bib]
• Credibility Assessment of Textual Claims on the Web (Popat et al., 2017). [paper] [bib]

Rule discovery and graph based methods

• ExFaKT: A Framework for Explaining Facts over Knowledge Graphs and Text (Gad-Elrab et al., 2019). [paper] [bib]
• Explainable Fact Checking with Probabilistic Answer Set Programming (Ahmadi et al., 2019). [paper] [bib]
• Linked Credibility Reviews for Explainable Misinformation Detection (Denaux et al., 2020). [paper [bib]

Summarization

• e-FEVER: Explanations and Summaries for Automated Fact Checking (Stammbach and Ash, 2020). [paper] [bib]
• Generating Fact Checking Explanations (Atanasova et al., 2020). [paper] [bib]
• Explainable Automated Fact Checking for Public Health Claims (Kotonya and Toni, 2020). [paper] [bib]

Adversarial Training for Automated Fact Checking

List of papers which look at adversarial training for robust fact-checking:

• Generating Label Cohesive and Well-Formed Adversarial Claims (Atanasova et al.,2020). [paper] [bib] [code]
• Evaluating adversarial attacks against multiple fact verification systems (Thorne et al., 2019). [paper] [bib] [video]
• DeSePtion: Dual Sequence Prediction and Adversarial Examples for Improved Fact-Checking (Hidey et al., 2020). [paper] [bib]
• Fake news detection via NLP is vulnerable to adversarial attacks (Zhou et al., 2019). [paper] [bib]
• GEM: Generative enhanced model for adversarial attacks (Niewinski et al., 2019). [paper] [bib]
• FEVER Breaker’s Run of Team NbAuzDrLqg (Kim and Allan, 2019). [paper] [bib]

Error Correction of Claims

• Factual Error Correction of Claims (Thorne and Vlachos, 2020). [paper] [bib]

Fact Checking with Intrepretable Features

• A Language-Based Approach to Fake News Detection Through Interpretable Features and BRNN (Qiao et al., 2020). [paper] [bib]

Multihop Reasoning for Facting Checking

Multi-hop reasoning is closely important for explainability in fact-checking, recently there have been a number of papers which look to address this task:

• HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verification (Jiang et al., 2020). [paper] [bib] [project]
• Multi-Hop Fact Checking of Political Claims (Ostrowski et al., 2020). [paper] [bib] [code]
• Chains-of-Reasoning at TextGraphs 2019 Shared Task: Reasoning over Chains of Facts for Explainable Multi-hop Inference (Das et al., 2019). [paper] [bib]

Surveys

Automated fact checking and fake news detection surveys:

• A Survey of Fake News: Fundamental Theories, Detection Methods, and Opportunities (Zhou and Zafarani, 2020). [paper] [bib]
• A Review on Fact Extraction and VERification: The FEVER case (Bekoulis et al., 2020). [paper] [bib]
• A Survey on Fake News and Rumour Detection Techniques (Bondielli and Marcelloni, 2020). [paper] [bib]
• A Survey on Natural Language Processing for Fake News Detection (Oshikawa et al., 2020). [paper] [bib]
• Fake News Detection using Stance Classification: A Survey (Lillie and Middelboe, 2019). [paper] [bib]
• Detection and Resolution of Rumours in Social Media: A Survey (Zubiaga et al., 2018). [paper] [bib]
• Automated Fact Checking: Task Formulations, Methods and Future Directions (Thorne and Vlachos, 2018). [paper] [bib]
• Media-Rich Fake News Detection: A Survey (Parikh and Atrey, 2018). [paper] [bib]
• A Content Management Perspective on Fact-Checking (Cazalens et al., 2018). [paper] [bib]
• Fake News Detection on Social Media: A Data Mining Perspective (Shu et al., 2017). [paper] [bib]

Tutorials

• Fact Checking: Theory and Practice [Dong et al., KDD 2018].
• Fact-Checking, Fake News, Propaganda, and Media Bias: Truth Seeking in the Post-Truth Era [Nakov and Da San Martino, EMNLP 2020].
• Detection and Resolution of Rumors and Misinformation with NLP [Derczynski and Zubiaga, COLING 2020] [slides].

Contact

Please feel free to contact Neema Kotonya if you have any questions or comments.