Deep Learning for Natural Language Processing SS 2021 (TU Darmstadt)

Overview

Deep Learning for Natural Language Processing SS 2021 (TU Darmstadt)

Task

Training huge unsupervised deep neural networks yields to strong progress in the field of Natural Language Processing (NLP). Using these extensively pre-trained networks for particular NLP applications is the current state-of-the-art approach. In this project, we approach the task of ranking possible clarifying questions for a given query. We fine-tuned a pre-trained BERT model to rank the possible clarifying questions in a classification manner. The achieved model scores a top-5 accuracy of 0.4565 on the provided benchmark dataset.

Installation

This project was originally developed with Python 3.8, PyTorch 1.7, and CUDA 11.0. The training requires one NVIDIA GeForce RTX 1080 (11GB memory).

  • Create conda environment:
conda create --name dl4nlp
source activate dl4nlp
  • Install the dependencies:
pip install -r requirements.txt

Run

We use a pretrained BERT-Base by Hugging Face and fine-tune it on the given training dataset. To run training, please use the following command:

python main.py --train

For evaluation on the test set, please use the following command:

python main.py --test

Arguments for training and/or testing:

  • --train: Run training on training dataset. Default: True
  • --val: Run evaluation during training on validation dataset. Default: True
  • --test: Run evaluation on test dataset. Default: True
  • --cuda-devices: Set GPU index Default: 0
  • --cpu: Run everything on CPU. Default: False
  • --data-parallel: Use DataParallel. Default: False
  • --data-root: Path to dataset folder. Default: data
  • --train-file-name: Name of training file name in data-root. Default: training.tsv
  • --test-file-name: Name of test file name in data-root. Default: test_set.tsv
  • --question-bank-name: Name of question bank file name in data-root. Default: question_bank.tsv
  • --checkpoints-root: Path to checkpoints folder. Default: checkpoints
  • --checkpoint-name: File name of checkpoint in checkpoints-root to start training or use for testing. Default: None
  • --runs-root: Path to output runs folder for tensorboard. Default: runs
  • --txt-root: Path to output txt folder for evaluation results. Default: txt
  • --lr: Learning rate. Default: 1e-5
  • --betas: Betas for optimization. Default: (0.9, 0.999)
  • --weight-decay: Weight decay. Default: 1e-2
  • --val-start: Set at which epoch to start validation. Default: 0
  • --val-step: Set at which epoch rate to valide. Default: 1
  • --val-split: Use subset of training dataset for validation. Default: 0.005
  • --num-epochs: Number of epochs for training. Default: 10
  • --batch-size: Samples per batch. Default: 32
  • --num-workers: Number of workers. Default: 4
  • --top-k-accuracy: Evaluation metric with flexible top-k-accuracy. Default: 50
  • --true-label: True label in dataset. Default: 1
  • --false-label: False label in dataset. Default: 0

Example output

User query:

Tell me about Computers

Propagated clarifying questions:

  1. do you like using computers
  2. do you want to know how to do computer programming
  3. do you want to see some closeup of a turbine
  4. are you looking for information on different computer programming languages
  5. are you referring to a software
You might also like...
Release of SPLASH: Dataset for semantic parse correction with natural language feedback in the context of text-to-SQL parsing
Release of SPLASH: Dataset for semantic parse correction with natural language feedback in the context of text-to-SQL parsing

SPLASH: Semantic Parsing with Language Assistance from Humans SPLASH is dataset for the task of semantic parse correction with natural language feedba

The source code for the Cutoff data augmentation approach proposed in this paper: "A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation".

Cutoff: A Simple Data Augmentation Approach for Natural Language This repository contains source code necessary to reproduce the results presented in

Train an RL agent to execute natural language instructions in a 3D Environment (PyTorch)
Train an RL agent to execute natural language instructions in a 3D Environment (PyTorch)

Gated-Attention Architectures for Task-Oriented Language Grounding This is a PyTorch implementation of the AAAI-18 paper: Gated-Attention Architecture

Implementation of EMNLP 2017 Paper
Implementation of EMNLP 2017 Paper "Natural Language Does Not Emerge 'Naturally' in Multi-Agent Dialog" using PyTorch and ParlAI

Language Emergence in Multi Agent Dialog Code for the Paper Natural Language Does Not Emerge 'Naturally' in Multi-Agent Dialog Satwik Kottur, José M.

Implementation of EMNLP 2017 Paper
Implementation of EMNLP 2017 Paper "Natural Language Does Not Emerge 'Naturally' in Multi-Agent Dialog" using PyTorch and ParlAI

Language Emergence in Multi Agent Dialog Code for the Paper Natural Language Does Not Emerge 'Naturally' in Multi-Agent Dialog Satwik Kottur, José M.

Flybirds - BDD-driven natural language automated testing framework, present by Trip Flight

Flybird | English Version 行为驱动开发(Behavior-driven development,缩写BDD),是一种软件过程的思想或者

It is a system used to detect bone fractures. using techniques deep learning and image processing

MohammedHussiengadalla-Intelligent-Classification-System-for-Bone-Fractures It is a system used to detect bone fractures. using techniques deep learni

[IJCAI'21] Deep Automatic Natural Image Matting
[IJCAI'21] Deep Automatic Natural Image Matting

Deep Automatic Natural Image Matting [IJCAI-21] This is the official repository of the paper Deep Automatic Natural Image Matting. Introduction | Netw

Official implementation of the paper
Official implementation of the paper "Lightweight Deep CNN for Natural Image Matting via Similarity Preserving Knowledge Distillation"

Lightweight-Deep-CNN-for-Natural-Image-Matting-via-Similarity-Preserving-Knowledge-Distillation Introduction Accepted at IEEE Signal Processing Letter

Owner
null
Prometheus Exporter for data scraped from datenplattform.darmstadt.de

darmstadt-opendata-exporter Scrapes data from https://datenplattform.darmstadt.de and presents it in the Prometheus Exposition format. Pull requests w

Martin Weinelt 2 Apr 12, 2022
Web mining module for Python, with tools for scraping, natural language processing, machine learning, network analysis and visualization.

Pattern Pattern is a web mining module for Python. It has tools for: Data Mining: web services (Google, Twitter, Wikipedia), web crawler, HTML DOM par

Computational Linguistics Research Group 8.4k Jan 3, 2023
🤗 Transformers: State-of-the-art Natural Language Processing for Pytorch, TensorFlow, and JAX.

English | 简体中文 | 繁體中文 State-of-the-art Natural Language Processing for Jax, PyTorch and TensorFlow ?? Transformers provides thousands of pretrained mo

Hugging Face 77.2k Jan 2, 2023
deep learning model that learns to code with drawing in the Processing language

sketchnet sketchnet - processing code generator can we teach a computer to draw pictures with code. We use Processing and java/jruby code paired with

null 41 Dec 12, 2022
🏆 The 1st Place Submission to AICity Challenge 2021 Natural Language-Based Vehicle Retrieval Track (Alibaba-UTS submission)

AI City 2021: Connecting Language and Vision for Natural Language-Based Vehicle Retrieval ?? The 1st Place Submission to AICity Challenge 2021 Natural

null 82 Dec 29, 2022
CLIP: Connecting Text and Image (Learning Transferable Visual Models From Natural Language Supervision)

CLIP (Contrastive Language–Image Pre-training) Experiments (Evaluation) Model Dataset Acc (%) ViT-B/32 (Paper) CIFAR100 65.1 ViT-B/32 (Our) CIFAR100 6

Myeongjun Kim 52 Jan 7, 2023
Third party Pytorch implement of Image Processing Transformer (Pre-Trained Image Processing Transformer arXiv:2012.00364v2)

ImageProcessingTransformer Third party Pytorch implement of Image Processing Transformer (Pre-Trained Image Processing Transformer arXiv:2012.00364v2)

null 61 Jan 1, 2023
Harmonious Textual Layout Generation over Natural Images via Deep Aesthetics Learning

Harmonious Textual Layout Generation over Natural Images via Deep Aesthetics Learning Code for the paper Harmonious Textual Layout Generation over Nat

null 7 Aug 9, 2022
Uncertain natural language inference

Uncertain Natural Language Inference This repository hosts the code for the following paper: Tongfei Chen*, Zhengping Jiang*, Adam Poliak, Keisuke Sak

Tongfei Chen 14 Sep 1, 2022
NaturalProofs: Mathematical Theorem Proving in Natural Language

NaturalProofs: Mathematical Theorem Proving in Natural Language NaturalProofs: Mathematical Theorem Proving in Natural Language Sean Welleck, Jiacheng

Sean Welleck 83 Jan 5, 2023