TensorFlow implementation of the paper "Hierarchical Attention Networks for Document Classification"

Quoc-Tuan Truong

Last update: Dec 5, 2022

Related tags

Deep Learning sentiment-analysis text-classification tensorflow document-classification attention-mechanism hierarchical-attention-networks

Overview

Hierarchical Attention Networks for Document Classification

This is an implementation of the paper Hierarchical Attention Networks for Document Classification, NAACL 2016.

Requirements

Python 3
Tensorflow > 1.0
Pandas
Nltk
Tqdm
Glove pre-trained word embeddings

Data

We use the data provided by Tang et al. 2015, including 4 datasets:

IMDB
Yelp 2013
Yelp 2014
Yelp 2015

Note: The original data seems to have an issue with unzipping. I re-uploaded the data to GG Drive for better downloading speed. Please request for access permission.

Usage

First, download the datasets and unzip into data folder.
Then, run script to prepare the data (default is using Yelp-2015 dataset):

python data_prepare.py

Train and evaluate the model:
(make sure Glove embeddings are ready before training)

wget http://nlp.stanford.edu/data/glove.6B.zip
unzip glove.6B.zip

python train.py

Print training arguments:

python train.py --help

optional arguments:
  -h, --help            show this help message and exit
  --cell_dim            CELL_DIM
                        Hidden dimensions of GRU cells (default: 50)
  --att_dim             ATTENTION_DIM
                        Dimensionality of attention spaces (default: 100)
  --emb_dim             EMBEDDING_DIM
                        Dimensionality of word embedding (default: 200)
  --learning_rate       LEARNING_RATE
                        Learning rate (default: 0.0005)
  --max_grad_norm       MAX_GRAD_NORM
                        Maximum value of the global norm of the gradients for clipping (default: 5.0)
  --dropout_rate        DROPOUT_RATE
                        Probability of dropping neurons (default: 0.5)
  --num_classes         NUM_CLASSES
                        Number of classes (default: 5)
  --num_checkpoints     NUM_CHECKPOINTS
                        Number of checkpoints to store (default: 1)
  --num_epochs          NUM_EPOCHS
                        Number of training epochs (default: 20)
  --batch_size          BATCH_SIZE
                        Batch size (default: 64)
  --display_step        DISPLAY_STEP
                        Number of steps to display log into TensorBoard (default: 20)
  --allow_soft_placement ALLOW_SOFT_PLACEMENT
                        Allow device soft device placement

Results

With the Yelp-2015 dataset, after 5 epochs, we achieved:

69.79% accuracy on the dev set
69.62% accuracy on the test set

No systematic hyper-parameter tunning was performed. The result reported in the paper is 71.0% for the Yelp-2015.

Comments

What hardware did you use?

Hi. I'm trying to execute and tensorflow struggles with memory exhausted error. I think it may be my video board, it's a Nvidia 1660 6GB, but I tried to execute in gcloud with NVIDIA Tesla T4 16GB and the same error appeared. (I made some modifications to original code, so the problem could be hardware limitations or my modifications)

What hardware did you use?

Did you have some memory exhausted error?

opened by VictorOF 2

Official TensorFlow code for the forthcoming paper

~ Efficient-CapsNet ~ Are you tired of over inflated and overused convolutional neural networks? You're right! It's time for CAPSULES :)

203 Jan 8, 2023

TensorFlow code for the neural network presented in the paper: "Structural Language Models of Code" (ICML'2020)

SLM: Structural Language Models of Code This is an official implementation of the model described in: "Structural Language Models of Code" [PDF] To ap

73 Nov 6, 2022

Annotated notes and summaries of the TensorFlow white paper, along with SVG figures and links to documentation

TensorFlow White Paper Notes Features Notes broken down section by section, as well as subsection by subsection Relevant links to documentation, resou

437 Oct 9, 2022

The LaTeX and Python code for generating the paper, experiments' results and visualizations reported in each paper is available (whenever possible) in the paper's directory

This repository contains the software implementation of most algorithms used or developed in my research. The LaTeX and Python code for generating the

3 Jan 3, 2023

Implementation of Restricted Boltzmann Machine (RBM) and its variants in Tensorflow

xRBM Library Implementation of Restricted Boltzmann Machine (RBM) and its variants in Tensorflow Installation Using pip: pip install xrbm Examples Tut

55 Dec 29, 2022

StyleGAN2 - Official TensorFlow Implementation

10.1k Dec 28, 2022

An efficient and effective learning to rank algorithm by mining information across ranking candidates. This repository contains the tensorflow implementation of SERank model. The code is developed based on TF-Ranking.

SERank An efficient and effective learning to rank algorithm by mining information across ranking candidates. This repository contains the tensorflow

44 Oct 20, 2022

Implementation of Perceiver, General Perception with Iterative Attention in TensorFlow

Perceiver This Python package implements Perceiver: General Perception with Iterative Attention by Andrew Jaegle in TensorFlow. This model builds on t

84 Oct 15, 2022

Minimal implementation of Denoised Smoothing: A Provable Defense for Pretrained Classifiers in TensorFlow.

Denoised-Smoothing-TF Minimal implementation of Denoised Smoothing: A Provable Defense for Pretrained Classifiers in TensorFlow. Denoised Smoothing is

19 Dec 11, 2022

TensorFlow implementation of the paper "Hierarchical Attention Networks for Document Classification"

Related tags

Overview

Hierarchical Attention Networks for Document Classification

Requirements

Data

Usage

Results

You might also like...

Official TensorFlow code for the forthcoming paper

TensorFlow code for the neural network presented in the paper: "Structural Language Models of Code" (ICML'2020)

Annotated notes and summaries of the TensorFlow white paper, along with SVG figures and links to documentation

The LaTeX and Python code for generating the paper, experiments' results and visualizations reported in each paper is available (whenever possible) in the paper's directory

Implementation of Restricted Boltzmann Machine (RBM) and its variants in Tensorflow

StyleGAN2 - Official TensorFlow Implementation

An efficient and effective learning to rank algorithm by mining information across ranking candidates. This repository contains the tensorflow implementation of SERank model. The code is developed based on TF-Ranking.

Implementation of Perceiver, General Perception with Iterative Attention in TensorFlow

Minimal implementation of Denoised Smoothing: A Provable Defense for Pretrained Classifiers in TensorFlow.

Comments

What hardware did you use?

Owner

Quoc-Tuan Truong

TensorFlow Ranking is a library for Learning-to-Rank (LTR) techniques on the TensorFlow platform

Robust Video Matting in PyTorch, TensorFlow, TensorFlow.js, ONNX, CoreML!

Robust Video Matting in PyTorch, TensorFlow, TensorFlow.js, ONNX, CoreML!

Functional TensorFlow Implementation of Singular Value Decomposition for paper Fast Graph Learning

Tensorflow implementation of the paper "HumanGPS: Geodesic PreServing Feature for Dense Human Correspondences", CVPR 2021.

Unofficial Tensorflow-Keras implementation of Fastformer based on paper [Fastformer: Additive Attention Can Be All You Need](https://arxiv.org/abs/2108.09084).

Tensorflow 2 implementation of the paper: Learning and Evaluating Representations for Deep One-class Classification published at ICLR 2021

Unofficial implementation of the paper: PonderNet: Learning to Ponder in TensorFlow

The official TensorFlow implementation of the paper Action Transformer: A Self-Attention Model for Short-Time Pose-Based Human Action Recognition

Unofficial Tensorflow 2 implementation of the paper Implicit Neural Representations with Periodic Activation Functions